Deep subspace clustering based on multiscale self-representation learning with consistency and diversity

doi:10.11772/j.issn.1001-9081.2023030275

Abstract

Abstract:

Deep Subspace Clustering （DSC） is based on the assumption that the original data lies in a collection of low-dimensional nonlinear subspaces. In the multi-scale representation learning methods for deep subspace clustering， based on deep auto-encoder， fully connected layers are added between the encoder and the corresponding decoder for each layer to capture multi-scale features， without deeply analyzing the nature of multi-scale features and considering the multi-scale reconstruction loss between input data and output data. In order to solve the above problems， firstly， the reconstruction loss function of each network layer was established to supervise the learning of encoder parameters at different levels； then， a more effective multi-scale self-representation module was proposed based on the block diagonality of the sum of the common self-representation matrix and the unique self-representation matrices for multi-scale features； finally， the diversity of unique self-representation matrices for different scale features was analyzed in depth and the multi-scale feature matrices were used effectively. On this basis， an MSCD-DSC （Multiscale Self-representation learning with Consistency and Diversity for Deep Subspace Clustering） method was proposed. Experimental results on the datasets Extended Yale B， ORL， COIL20 and Umist show that， compared to the suboptimal method MLRDSC （Multi-Level Representation learning for Deep Subspace Clustering）， the clustering error rate of MSCD-DSC is reduced by 15.44%， 2.22%， 3.37%， and 13.17%， respectively， indicating that the clustering effect of MSCD-DSC is better than those of the existing methods.

Key words: deep subspace clustering, auto-encoder, multiscale, self-representation matrix, consistency, diversity

摘要：

深度子空间聚类（DSC）基于原始数据位于低维非线性子空间的集合中的假设。其中深度子空间聚类多尺度表示学习方法在深度自编码器的基础上，将每一层的编码器与对应的解码器之间都添加全连接层，并以此捕获多尺度的特征，但它没有深度分析多尺度特征的性质，也没有考虑输入数据和输出数据之间多尺度的重构损失。为了解决上述问题，首先建立每个网络层的重构损失函数，监督不同级别编码器参数的学习；然后利用多尺度特征共有的自表示矩阵和特有的自表示矩阵的和具有块对角性，提出更有效的多尺度自表示模块；最后分析不同尺度特征特有的自表示矩阵之间的多样性，有效地利用了多尺度的特征矩阵。在此基础上，提出一种基于一致性和多样性的多尺度自表示学习的深度子空间聚类（MSCD-DSC）方法。在数据集Extended Yale B 、ORL、COIL20和Umist上的实验结果表明，相较于次优的MLRDSC（Multi-Level Representation learning for Deep Subspace Clustering），MSCD-DSC的聚类错误率分别降低了15.44%、2.22%、3.37%和13.17%，表明MSCD-DSC的聚类效果优于已有的方法。

关键词: 深度子空间聚类, 自编码器, 多尺度, 自表示矩阵, 一致性, 多样性

CLC Number:

TP751

Zhuo ZHANG, Huazhu CHEN. Deep subspace clustering based on multiscale self-representation learning with consistency and diversity[J]. Journal of Computer Applications, 2024, 44(2): 353-359.

张卓, 陈花竹. 基于一致性和多样性的多尺度自表示学习的深度子空间聚类[J]. 《计算机应用》唯一官方网站, 2024, 44(2): 353-359.

Figures/Tables 9

Tab. 1 Symbol description

符号	含义
$θ e$	编码器、解码器及全连接层的参数
$L$	网络的深度
$l$	所在网络的层级
$X ∈ R N × C × H × W$	原始输入数据
$X θ e l ∈ R N × C l × H l × W l$	第 $l$ 层编码器的输入数据
$X ̂ θ e l ∈ R N × C l × H l × W l$	第 $l$ 层解码器的输出数据
$C ∈ R N × N$	网络层嵌入特征共有的自表示系数矩阵
$C ∈ R N × N$	将自表示系数矩阵 $C$ 中的元素求绝对值
$Q ∈ R N × K$	伪标签矩阵
$D l ∈ R N × N$	第 $l$ 层的嵌入特征特有的自表示系数矩阵
$Z θ e l ∈ R M l × N$	数据经过编码器后在子空间中的数据表示
$\| \| ⋅ \| \| F$	矩阵的 $F r o b e n i u s$ 范数
$\| \| ⋅ \| \| 1$	矩阵的 $L 1$ 范数
$C T$	矩阵 C 的转置矩阵
$⊙$	矩阵的Hadamard积

Tab. 1 Symbol description

符号	含义
$θ e$	编码器、解码器及全连接层的参数
$L$	网络的深度
$l$	所在网络的层级
$X ∈ R N × C × H × W$	原始输入数据
$X θ e l ∈ R N × C l × H l × W l$	第 $l$ 层编码器的输入数据
$X ̂ θ e l ∈ R N × C l × H l × W l$	第 $l$ 层解码器的输出数据
$C ∈ R N × N$	网络层嵌入特征共有的自表示系数矩阵
$C ∈ R N × N$	将自表示系数矩阵 $C$ 中的元素求绝对值
$Q ∈ R N × K$	伪标签矩阵
$D l ∈ R N × N$	第 $l$ 层的嵌入特征特有的自表示系数矩阵
$Z θ e l ∈ R M l × N$	数据经过编码器后在子空间中的数据表示
$\| \| ⋅ \| \| F$	矩阵的 $F r o b e n i u s$ 范数
$\| \| ⋅ \| \| 1$	矩阵的 $L 1$ 范数
$C T$	矩阵 C 的转置矩阵
$⊙$	矩阵的Hadamard积

Fig.1 Overall architecture of proposed method

Fig.2 Unique self-representation matrix of lth layer

Tab. 2 Network structures of different datasets

自编码层	Extended Yale B		ORL		COIL20		Umist
自编码层	卷积核	通道数	卷积核	通道数	卷积核	通道数	卷积核	通道数
编码层-1	5×5	10	5×5	5	3×3	5	3×3	20
编码层-2	3×3	20	3×3	3	3×3	10	3×3	10
编码层-3	3×3	30	3×3	3	—	—	3×3	5
解码层-3	3×3	30	3×3	3	—	—	3×3	5
解码层-2	3×3	20	3×3	3	3×3	10	3×3	10
解码层-1	5×5	10	5×5	5	3×3	5	3×3	20

Fig. 3 Sampled images from different datasets

Tab. 3 Clustering error rates of different algorithms on Extended Yale B dataset

n	LRR	LRSC	SSC	AE+SSC	KSSC	SSC-OMP	EDSC	AE+EDSC	DSC	MLRDSC	MSCD-DSC
10	22.22	30.95	10.22	17.06	14.49	12.08	5.64	5.46	1.59	1.10	1.41
15	23.00	31.47	13.13	18.65	16.22	14.05	7.63	6.70	1.69	0.91	1.56
20	30.23	28.76	19.75	18.23	16.55	15.16	9.30	7.67	1.73	0.99	1.17
25	27.92	27.81	26.22	18.72	18.56	18.89	10.67	10.27	1.75	1.13	0.88
30	37.98	30.64	28.76	19.99	20.49	20.75	11.24	11.56	2.07	1.78	1.09
35	41.82	31.35	28.55	22.13	26.07	20.29	13.10	13.28	2.65	1.44	1.34
38	34.87	29.89	27.51	25.33	27.75	24.71	11.64	12.66	2.67	1.36	1.15

Tab. 4 Ablation experiment results on Extended Yale B

n	不同算法的聚类错误率/%
n	MLRDSC	算法1	算法2	算法3	算法4
10	1.10	1.78	2.17	2.03	1.41
15	0.91	1.35	1.35	1.56	1.56
20	0.99	1.25	1.17	1.25	1.17
25	1.13	0.94	0.88	0.94	0.88
30	1.78	1.20	1.15	1.15	1.09
35	1.44	1.43	1.43	1.43	1.34
38	1.36	1.27	1.32	1.23	1.15

Tab. 5 Parameter ablation experiment results on ORL

实验序号	$λ 1$	$λ 2$	$λ 3$	错误率/%
1	0	75	1	22.50
2	10	0	1	14.75
3	10	75	0	14.50
4	10	75	1	11.00

Tab. 5 Parameter ablation experiment results on ORL

实验序号	$λ 1$	$λ 2$	$λ 3$	错误率/%
1	0	75	1	22.50
2	10	0	1	14.75
3	10	75	0	14.50
4	10	75	1	11.00

Tab. 6 Clustering error rates for datasets ORL， COIL20 and Umist

数据集	LRR	LRSC	SSC	AE+SSC	KSSC	SSC-OMP	EDSC	AE+EDSC	DSC	DASC	MLRDSC	MSCD-DSC
ORL	33.50	32.50	29.50	26.75	34.25	37.05	27.25	26.25	14.00	11.75	11.25	11.00
COIL20	30.21	31.25	14.83	22.08	24.65	29.86	14.86	14.79	5.42	3.61	2.08	2.01
Umist	30.21	—	34.69	—	34.69	—	30.69	—	26.88	—	26.87	23.33

References 25

1	LAUER F， SCHNÖRR C. Spectral clustering of linear subspaces for motion segmentation［C］// Proceedings of the 2009 IEEE 12th International Conference on Computer Vision. Piscataway： IEEE， 2009： 678-685. 10.1109/iccv.2009.5459173
2	RAO S， TRON R， VIDAL R， et al. Motion segmentation in the presence of outlying， incomplete， or corrupted trajectories［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2009， 32（10）： 1832-1845. 10.1109/tpami.2009.191
3	CAO X， ZHANG C， FU H， et al. Diversity-induced multi-view subspace clustering［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 586-594. 10.1109/cvpr.2015.7298657
4	ZHANG C， FU H， LIU S， et al. Low-rank tensor constrained multiview subspace clustering［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 1582-1590. 10.1109/iccv.2015.185
5	YANG A Y， WRIGHT J， MA Y， et al. Unsupervised segmentation of natural images via lossy data compression［J］. Computer Vision and Image Understanding， 2008， 110（2）： 212-225. 10.1016/j.cviu.2007.07.005
6	JI P， ZHANG T， LI H， et al. Deep subspace clustering networks［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc. 2017： 23-32.
7	JI P， SALZMANN M， LI H. Efficient dense subspace clustering［C］// Proceedings of the 2014 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2014： 461-468. 10.1109/wacv.2014.6836065
8	ZHOU P， HOU Y， FENG J. Deep adversarial subspace clustering［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 1596-1604. 10.1109/cvpr.2018.00172
9	LV J， KANG Z， LU X， et al. Pseudo-supervised deep subspace clustering［J］. IEEE Transactions on Image Processing， 2021， 30： 5252-5263. 10.1109/tip.2021.3079800
10	VALANARASU J M J， PATEL V M. Overcomplete deep subspace clustering networks［C］// Proceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2021： 746-755. 10.1109/wacv48630.2021.00079
11	WU F， YUAN P， SHI G， et al. Robust subspace clustering network with dual-domain regularization［J］. Pattern Recognition Letters， 2021， 149： 44-50. 10.1016/j.patrec.2021.06.009
12	PAN J， QIAN Y， LI F， et al. Image deep clustering based on local-topology embedding［J］. Pattern Recognition Letters， 2021， 151： 88-94. 10.1016/j.patrec.2021.08.004
13	KHEIRANDISHFARD M， ZOHRIZADEH F， KAMANGAR F. Multi-level representation learning for deep subspace clustering［C］// Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2020： 2028-2037. 10.1109/wacv45572.2020.9093277
14	LIU G， LIN Z， YAN S， et al. Robust recovery of subspace structures by low-rank representation［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2013， 35（1）： 171-184. 10.1109/tpami.2012.88
15	GUO X， WANG X， LING H. Exclusivity regularized machine： a new ensemble SVM classifier［C］// Proceedings of the 26th International Joint Conference on Artificial Intelligence. Palo Alto： AAAI Press， 2017： 1739-1745. 10.24963/ijcai.2017/241
16	NG A Y， JORDAN M I， WEISS Y. On spectral clustering： Analysis and an algorithm［C］// Proceedings of the 14th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2001： 849-856.
17	GEORGHIADES A S， BELHUMEUR P N， KRIEGMAN D J. From few to many： Illumination cone models for face recognition under variable lighting and pose［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2001， 23（6）： 643-660. 10.1109/34.927464
18	SAMARIA F S， HARTER A C. Parameterisation of a stochastic model for human face identification［C］// Proceedings of 1994 IEEE Workshop on Applications of Computer Vision. Piscataway： IEEE， 1994： 138-142. 10.1109/acv.1994.341276
19	McELROY D， WALSH C， MARKWICK A J， et al. The UMIST database for astrochemistry 2012［J］. Astronomy and Astrophysics， 2013， 550： A36. 10.1051/0004-6361/201220465
20	NENE S， NAYAR S， MURASE H. Columbia Object Image Library （COIL-20）［DB/OL］. ［2022-10-18］. .
21	VIDAL R， FAVARO P. Low rank subspace clustering （LRSC）［J］. Pattern Recognition Letters， 2014， 43： 47-61. 10.1016/j.patrec.2013.08.006
22	ELHAMIFAR E， VIDAL R. Sparse subspace clustering： Algorithm， theory， and applications［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2013， 35（11）： 2765-2781. 10.1109/tpami.2013.57
23	PATEL V M， VIDAL R. Kernel sparse subspace clustering［C］// Proceedings of the 2014 IEEE International Conference on Image Processing. Piscataway： IEEE， 2014： 2849-2853. 10.1109/icip.2014.7025576
24	YOU C， ROBINSON D， VIDAL R. Scalable sparse subspace clustering by orthogonal matching pursuit［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 3918-3927. 10.1109/cvpr.2016.425
25	LU C-Y， MIN H， ZHAO Z-Q， et al. Robust and efficient subspace segmentation via least squares regression［C］// Proceedings of the 2012 European Conference on Computer Vision. Berlin： Springer， 2012： 347-360. 10.1007/978-3-642-33786-4_26

[1]	Guanglei YAO, Juxia XIONG, Guowu YANG. Flower pollination algorithm based on neural network optimization [J]. Journal of Computer Applications, 2024, 44(9): 2829-2837.
[2]	Wentao JIANG, Wanxuan LI, Shengchong ZHANG. Correlation filtering based target tracking with nonlinear temporal consistency [J]. Journal of Computer Applications, 2024, 44(8): 2558-2570.
[3]	Qing WANG, Jieyu ZHAO, Xulun YE, Nongxiao WANG. Enhanced deep subspace clustering method with unified framework [J]. Journal of Computer Applications, 2024, 44(7): 1995-2003.
[4]	Junna ZHANG, Xinxin WANG, Tianze LI, Xiaoyan ZHAO, Peiyan YUAN. Task offloading method based on dynamic service cache assistance [J]. Journal of Computer Applications, 2024, 44(5): 1493-1500.
[5]	Shunwang FU, Qian CHEN, Zhi LI, Guomei WANG, Yu LU. Two-channel progressive feature filtering network for tampered image detection and localization [J]. Journal of Computer Applications, 2024, 44(4): 1303-1309.
[6]	Yawei HUANG, Xuezhong QIAN, Wei SONG. Improved differential evolution algorithm based on dual-archive population size adaptive method [J]. Journal of Computer Applications, 2024, 44(12): 3844-3853.
[7]	Tingting GAO, Zhongyuan YAO, Miao JIA, Xueming SI. Overview of on-chain and off-chain consistency protection technologies [J]. Journal of Computer Applications, 2024, 44(12): 3658-3668.
[8]	Ziyi LI, Tingting QU, Qianpeng CHONG, Jindong XU. Remote sensing image segmentation network based on fuzzy multiscale features [J]. Journal of Computer Applications, 2024, 44(11): 3581-3586.
[9]	Shuaihua ZHANG, Shufen ZHANG, Mingchuan ZHOU, Chao XU, Xuebin CHEN. Malicious traffic detection model based on semi-supervised federated learning [J]. Journal of Computer Applications, 2024, 44(11): 3487-3494.
[10]	Lishuo YE, Zhixue HE. Multiscale time series anomaly detection incorporating wavelet decomposition [J]. Journal of Computer Applications, 2024, 44(10): 3300-3306.
[11]	Qiujie LIU, Yuan WAN, Jie WU. Deep bi-modal source domain symmetrical transfer learning for cross-modal retrieval [J]. Journal of Computer Applications, 2024, 44(1): 24-31.
[12]	Yongjian MA, Xuhua SHI, Peiyao WANG. Constrained multi-objective evolutionary algorithm based on two-stage search and dynamic resource allocation [J]. Journal of Computer Applications, 2024, 44(1): 269-277.
[13]	Jianwen GAN, Yan CHEN, Peng ZHOU, Liang DU. Clustering ensemble algorithm with high-order consistency learning [J]. Journal of Computer Applications, 2023, 43(9): 2665-2672.
[14]	Ruiqi WANG, Shujuan JI, Ning CAO, Yajie GUO. Semi-supervised fake job advertisement detection model based on consistency training [J]. Journal of Computer Applications, 2023, 43(9): 2932-2939.
[15]	Jinghong WANG, Zhixia ZHOU, Hui WANG, Haokang LI. Attribute network representation learning with dual auto-encoder [J]. Journal of Computer Applications, 2023, 43(8): 2338-2344.