Enhanced deep subspace clustering method with unified framework

doi:10.11772/j.issn.1001-9081.2023101395

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (7): 1995-2003.DOI: 10.11772/j.issn.1001-9081.2023101395

• Artificial intelligence • Previous Articles Next Articles

Enhanced deep subspace clustering method with unified framework

Qing WANG, Jieyu ZHAO(), Xulun YE, Nongxiao WANG

College of Information Science and Engineering，Ningbo University，Ningbo Zhejiang 315211，China

Received:2023-10-17 Revised:2024-01-18 Accepted:2024-01-24 Online:2024-03-08 Published:2024-07-10
Contact: Jieyu ZHAO
About author:WANG Qing， born in 1999， M. S. candidate. Her research interests include subspace clustering， few-shot learning.
YE Xulun， born in 1991， Ph. D.， associate professor. His research interests include Bayesian learning， convex optimization.
WANG Nongxiao， born in 1998， M. S. candidate. Her research interests include contrastive learning， spectral clustering， few-shot learning.
First author contact:ZHAO Jieyu， born in 1965， Ph. D.， professor. His research interests include computational intelligence， pattern recognition.
Supported by:
National Natural Science Foundation of China(62006131);Natural Science Foundation of Zhejiang Province(LQ21F020009)

统一框架的增强深度子空间聚类方法

王清, 赵杰煜(), 叶绪伦, 王弄潇

宁波大学信息科学与工程学院，浙江宁波 315211

通讯作者: 赵杰煜
作者简介:王清（1999—），女，江西上饶人，硕士研究生，CCF会员，主要研究方向：子空间聚类、小样本学习；
叶绪伦（1991—），男，安徽安庆人，副教授，博士，CCF会员，主要研究方向：贝叶斯学习、凸优化；
王弄潇（1998—），女，浙江台州人，硕士研究生，CCF会员，主要研究方向：对比学习、谱聚类、小样本学习。
第一联系人：赵杰煜（1965—），男，浙江宁波人，教授，博士，CCF会员，主要研究方向：计算智能、模式识别；
基金资助:
国家自然科学基金资助项目(62006131);浙江省自然科学基金资助项目(LQ21F020009)

Abstract

Abstract:

Deep subspace clustering is a method that performs well in processing high-dimensional data clustering tasks. However， when dealing with challenging data， current deep subspace clustering methods with fixed self-expressive matrix usually exhibit suboptimal clustering results due to the conventional practice of treating self-expressive learning and indicator learning as two separate and independent processes， and the quality of self-expressive matrix has a crucial impact on the accuracy of clustering results. To solve the above problems， an enhanced deep subspace clustering method with unified framework was proposed. Firstly， by integrating feature learning， self-expressive learning， and indicator learning together to optimize all parameters， the self-expressive matrix was dynamically learned based on the characteristics of the data， ensuring accurate capture of data features. Secondly， to improve the effects of self-representative learning， class prototype pseudo-label learning was proposed to provide self-supervised information for feature learning and indicator learning， thereby promoting self-expressive learning. Finally， to enhance the discriminative ability of embedded representations， orthogonality constraints were introduced to help achieve self-expressive attribute. The experimental results show that compared with AASSC （Adaptive Attribute and Structure Subspace Clustering network）， the proposed method improves clustering accuracy by 1.84， 0.49 and 0.34 percentage points on the MNIST， UMIST and COIL20 datasets. It can be seen that the proposed method improves the accuracy of self-representative matrix learning， thereby achieving better clustering effects.

Key words: deep subspace clustering, self-expressive learning, indicator learning, affinity matrix, orthogonality constraint

摘要：

深度子空间聚类是一种处理高维数据聚类任务的有效方法。然而，现有的深度子空间聚类方法通常将自表示学习和指标学习作为两个独立的过程，导致在处理具有挑战性的数据时，固定的自表示矩阵会导致次优的聚类结果；另外，自表示矩阵的质量对聚类结果的准确性至关重要。针对上述问题，提出一种统一框架的增强深度子空间聚类方法。首先，通过将特征学习、自表示学习和指标学习集成在一起同时优化所有参数，根据数据的特征动态地学习自表示矩阵，确保准确地捕捉数据特征；其次，为了提高自表示学习的效果，提出类原型伪标签学习，为特征学习和指标学习提供自监督信息，进而促进自表示学习；最后，为了增强嵌入表示的判别能力，引入正交性约束帮助实现自表示属性。实验结果表明，与AASSC （Adaptive Attribute and Structure Subspace Clustering network）相比，所提方法在MNIST、UMIST、COIL20数据集上的聚类准确率分别提升了1.84、0.49、0.34个百分点。可见，所提方法提高了自表示矩阵学习的准确性，聚类效果更好。

关键词: 深度子空间聚类, 自表示学习, 指标学习, 亲和矩阵, 正交约束

CLC Number:

TP181

Qing WANG, Jieyu ZHAO, Xulun YE, Nongxiao WANG. Enhanced deep subspace clustering method with unified framework[J]. Journal of Computer Applications, 2024, 44(7): 1995-2003.

王清, 赵杰煜, 叶绪伦, 王弄潇. 统一框架的增强深度子空间聚类方法[J]. 《计算机应用》唯一官方网站, 2024, 44(7): 1995-2003.

Figures/Tables 19

Fig. 1 Feature redundancy example

Fig. 2 Network framework of proposed method

Tab. 1 Notations and descriptions

符号	描述	符号	描述
X	输入数据	G，H	自表示学习模块和指标学习模块
Q	伪标签矩阵	n，k	输入数据的个数和类别数
N	近邻矩阵	C， W	自表示矩阵和亲和矩阵
$Θ e$ ， $Θ d$	E和D的参数	$C $ ， $W $	更新后的自表示矩阵和亲和矩阵
$Θ s$ ， $Θ y$	G和H的参数	$λ 1$ ， $λ 2$	控制自表示矩阵的超参数
Z	隐空间表示	B	高置信度样本集合
$X ̂$	重建输入	$ϱ$	每个类的类原型
S	指标嵌入矩阵	E，D	编码器和解码器
I	单位矩阵	$α$ ， $β$	平衡参数

Tab. 1 Notations and descriptions

符号	描述	符号	描述
X	输入数据	G，H	自表示学习模块和指标学习模块
Q	伪标签矩阵	n，k	输入数据的个数和类别数
N	近邻矩阵	C， W	自表示矩阵和亲和矩阵
$Θ e$ ， $Θ d$	E和D的参数	$C $ ， $W $	更新后的自表示矩阵和亲和矩阵
$Θ s$ ， $Θ y$	G和H的参数	$λ 1$ ， $λ 2$	控制自表示矩阵的超参数
Z	隐空间表示	B	高置信度样本集合
$X ̂$	重建输入	$ϱ$	每个类的类原型
S	指标嵌入矩阵	E，D	编码器和解码器
I	单位矩阵	$α$ ， $β$	平衡参数

Fig. 3 Illustration of orthogonality constraint

Fig. 4 Examples of two face datasets

Fig. 5 Examples of other datasets

Tab. 2 Statistics of benchmark datasets

数据集	类别数	样本数	维度
MNIST	10	1 000	28×28
ORL	40	400	32×32
UMIST	20	480	32×32
COIL20	20	1 440	32×32
COIL40	40	2 880	32×32

Tab. 3 Network architectures for COIL20 and COIL40

数据集	Encoder	C	Decoder
COIL20	$3 × 3 (15)$	$1 440 × 1 440$	$3 × 3 (15)$
COIL40	$3 × 3 (20)$	$2 880 × 2 880$	$3 × 3 (20)$

Tab. 3 Network architectures for COIL20 and COIL40

数据集	Encoder	C	Decoder
COIL20	$3 × 3 (15)$	$1 440 × 1 440$	$3 × 3 (15)$
COIL40	$3 × 3 (20)$	$2 880 × 2 880$	$3 × 3 (20)$

Tab. 4 Network architectures for MNIST， ORL and UMIST

数据集	Encoder			C	Decoder
MNIST	$5 × 5 (15)$	$3 × 3 (10)$	$3 × 3 (5)$	$1 000 × 1 000$	$3 × 3 (5)$	$3 × 3 (10)$	$5 × 5 (15)$
ORL	$5 × 5 (5)$	$3 × 3 (3)$	$3 × 3 (3)$	$400 × 400$	$3 × 3 (3)$	$3 × 3 (3)$	$5 × 5 (5)$
UMIST	$5 × 5 (20)$	$3 × 3 (10)$	$3 × 3 (5)$	$480 × 480$	$3 × 3 (5)$	$3 × 3 (10)$	$5 × 5 (20)$

Tab. 4 Network architectures for MNIST， ORL and UMIST

数据集	Encoder			C	Decoder
MNIST	$5 × 5 (15)$	$3 × 3 (10)$	$3 × 3 (5)$	$1 000 × 1 000$	$3 × 3 (5)$	$3 × 3 (10)$	$5 × 5 (15)$
ORL	$5 × 5 (5)$	$3 × 3 (3)$	$3 × 3 (3)$	$400 × 400$	$3 × 3 (3)$	$3 × 3 (3)$	$5 × 5 (5)$
UMIST	$5 × 5 (20)$	$3 × 3 (10)$	$3 × 3 (5)$	$480 × 480$	$3 × 3 (5)$	$3 × 3 (10)$	$5 × 5 (20)$

Fig. 6 Network dataflow

Tab. 5 ACC and NMI comparison for different methods on MNIST

分类	方法	ACC	NMI
传统浅层方法	LRR	53.86	56.32
	EDSC	56.50	57.52
	KSSC	52.20	56.23
	FLSR	62.10	52.31
	AGCSC	72.80	67.54
基于深度学习的方法	DEC	61.20	57.43
	DSC-Net_L1	72.80	72.17
	DSC-Net_L2	75.00	73.19
	DASC	80.40	78.00
	DKM	53.32	50.02
	DSCDL	81.20	76.10
	DSRSC	84.80	81.94
	PSSC	84.30	76.76
	AASSC	84.60	76.09
	本文方法	86.44	76.51

Tab. 6 ACC and NMI comparison for different methods on ORL and UMIST

分类	方法	ORL		UMIST
分类	方法	ACC	NMI	ACC	NMI
传统浅层方法	SSC	74.25	84.95	69.04	74.89
	EDSC	70.38	77.99	69.37	75.22
	KSSC	71.43	80.70	65.31	73.77
	LRSC	72.00	81.56	67.29	74.39
	SSC-OMP	71.00	79.52	64.34	70.68
	FLSR	77.75	86.61	—	—
	CLR2SC	69.25	84.25	—	—
	PBDR	86.40	93.20	—	—
基于深度学习的方法	AE+SSC	75.63	85.55	70.42	75.15
	DSC-Net_L1	85.00	90.23	72.42	75.56
	DSC-Net_L2	86.00	90.34	73.12	76.62
	DASC	88.25	93.15	76.88	80.42
	DSCDL	—	—	81.90	91.70
	DSRSC	88.50	93.52	—	—
	PSSC	86.75	93.49	79.17	86.70
	AASSC	90.75	94.31	83.54	89.02
	ARSSC	75.50	—	—	—
	本文方法	90.75	94.99	84.03	89.54

Tab. 7 ACC and NMI comparison for different methods on COIL20 and COIL40

分类	方法	COIL20		COIL40
分类	方法	ACC	NMI	ACC	NMI
传统浅层方法	SSC	86.31	88.92	71.91	82.12
	LRR	81.18	87.47	64.93	78.28
	EDSC	83.71	88.28	68.70	81.39
	LRSC	74.16	84.52	63.27	77.37
基于深度学习的方法	AE+SSC	87.11	89.90	73.91	83.18
	DEC	72.15	80.07	48.72	74.17
	DEPICT	86.18	92.66	80.73	92.91
	DSC-Net_L1	93.14	93.53	80.03	88.52
	DSC-Net_L2	93.68	94.08	80.75	89.41
	DASC	96.39	96.86	83.54	91.96
	DSRSC	94.95	95.32	—	—
	PSSC	97.22	97.79	83.58	92.58
	AASSC	98.40	98.29	87.19	93.43
	本文方法	98.74	98.97	85.13	92.67

Fig. 7 Visualization of learned affinity matrix W on MNIST and ORL

Fig. 8 t-SNE visualization of embedding representation during training on COIL20 dataset

Tab. 8 Ablation study results with different components of loss function on datasets COIL20 and COIL40

方法	COIL20		COIL40
方法	ACC	NMI	ACC	NMI
w/o $L 2 + L 4 + L 5$	93.68	94.08	80.75	89.41
w/o $L 4 + L 5$	94.26	96.47	81.59	90.86
w/o $L 2$	96.65	95.06	83.04	91.87
本文方法	98.74	98.97	85.13	92.67

Tab. 8 Ablation study results with different components of loss function on datasets COIL20 and COIL40

方法	COIL20		COIL40
方法	ACC	NMI	ACC	NMI
w/o $L 2 + L 4 + L 5$	93.68	94.08	80.75	89.41
w/o $L 4 + L 5$	94.26	96.47	81.59	90.86
w/o $L 2$	96.65	95.06	83.04	91.87
本文方法	98.74	98.97	85.13	92.67

Fig. 9 Average ACC and NMI for Model a and Model b on five datasets

Fig. 10 ACC results with different parameters α and β on five datasets

Fig. 11 NMI results with different parameters α and β on five datasets

References 34

1	RAO S， TRON R， VIDAL R， et al. Motion segmentation in the presence of outlying， incomplete， or corrupted trajectories ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2010， 32（10）： 1832-1845.
2	VIDAL R， MA Y， SASTRY S. Generalized Principal Component Analysis （GPCA）［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2005， 27（12）： 1945-1959.
3	ELHAMIFAR E， VIDAL R. Sparse subspace clustering： algorithm， theory， and applications ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2013， 35（11）： 2765-2781.
4	PARK S， HAN S， KIM S， et al. Improving unsupervised image clustering with robust learning ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 12273-12282.
5	CAI J， FAN J， GUO W， et al. Efficient deep embedded subspace clustering ［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2022： 21-30.
6	LI Z， TANG C， ZHENG X， et al. High-order correlation preserved incomplete multi-view subspace clustering ［J］. IEEE Transactions on Image Processing， 2022， 31： 2067-2080.
7	LIU S， WANG S， ZHANG P， et al. Efficient one-pass multi-view subspace clustering with consensus anchors ［J］. Proceedings of the AAAI Conference on Artificial Intelligence， 2022， 36（7）： 7576-7584.
8	LIU G， LIN Z， YAN S， et al. Robust recovery of subspace structures by low-rank representation ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2013， 35（1）： 171-184.
9	YOU C， LI C-G， ROBINSON D P， et al. Oracle based active set algorithm for scalable elastic net subspace clustering ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 3928-3937.
10	YOU C， ROBINSON D， VIDAL R. Scalable sparse subspace clustering by orthogonal matching pursuit ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 3918-3927.
11	MATSUSHIMA S， BRBIC M. Selective sampling-based scalable sparse subspace clustering ［C］// Proceedings of the 33 rd International Conference on Neural Information Processing Systems. Red Hook： Curran Associates， Inc.， 2019： 12425-12434.
12	HINTON G E， SALAKHUTDINOV R R. Reducing the dimensionality of data with neural networks ［J］. Science， 2006， 313（5786）： 504-507.
13	ZHOU P， HOU Y， FENG J. Deep adversarial subspace clustering ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 1596-1604.
14	JI P， ZHANG T， LI H， et al. Deep subspace clustering networks ［C］// Proceedings of the 31 st International Conference on Neural Information Processing Systems. Red Hook： Curran Associates， Inc.， 2017： 23-32.
15	尹明，吴浩杨，谢胜利，等.基于自注意力对抗的深度子空间聚类［J］.自动化学报， 2022， 48（1）： 271-281.
	YIN M， WU H Y， XIE S L， et al. Self-attention adversarial based deep subspace clustering ［J］. Acta Automatica Sinica， 2022， 48（1）： 271-281.
16	李凯，张可心.结构α-熵的加权高斯混合模型的子空间聚类［J］.电子学报， 2022， 50（3）： 718-725.
	LI K， ZHANG K X. Structural α-entropy weighting Gaussian mixture model for subspace clustering ［J］. Acta Electronica Sinica， 2022， 50（3）： 718-725.
17	LeCUN Y， BOTTOU L， BENGIO Y， et al. Gradient-based learning applied to document recognition ［J］. Proceedings of the IEEE， 1998， 86（11）： 2278-2324.
18	SAMARIA F S， HARTER A C. Parameterisation of a stochastic model for human face identification ［C］// Proceedings of the 1994 IEEE Workshop on Applications of Computer Vision. Piscataway： IEEE， 1994： 138-142.
19	GRAHAM D B， ALLINSON N M. Characterising virtual eigensignatures for general purpose face recognition ［M］// WECHSLER H， PHILLIPS J P， BRUCE V， et al. Face Recognition： from Theory to Applications， NATO Series 163. Berlin： Springer， 1998： 446-456.
20	JI P， SALZMANN M， LI H. Efficient dense subspace clustering ［C］// Proceedings of the 2014 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2014： 461-468.
21	PATEL V M， VIDAL R. Kernel sparse subspace clustering ［C］// Proceedings of the 2014 IEEE International Conference on Image Processing. Piscataway： IEEE， 2014： 2849-2853.
22	MA Z， KANG Z， LUO G， et al. Towards clustering-friendly representations： subspace clustering via graph filtering ［C］// Proceedings of the 28th ACM International Conference on Multimedia. New York： ACM， 2020： 3081-3089.
23	WEI L， CHEN Z， YIN J， et al. Adaptive graph convolutional subspace clustering ［C］// Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2023： 6262-6271.
24	XIE J， GIRSHICK R， FARHADI A. Unsupervised deep embedding for clustering analysis ［C］// Proceedings of the 33 rd International Conference on Machine Learning.New York： JMLR.org， 2016： 478-487.
25	MORADI FARD M， THONET T， GAUSSIER E. Deep k-means： jointly clustering with k-means and learning representations ［J］. Pattern Recognition Letters， 2020， 138： 185-192.
26	HUANG Q， ZHANG Y， PENG H， et al. Deep subspace clustering to achieve jointly latent feature extraction and discriminative learning ［J］. Neurocomputing， 2020， 404： 340-350.
27	BAEK S， YOON G， SONG J， et al. Deep self-representative subspace clustering network ［J］. Pattern Recognition， 2021， 118： 108041.
28	LV J， KANG Z， LU X， et al. Pseudo-supervised deep subspace clustering ［J］. IEEE Transactions on Image Processing， 2021， 30： 5252-5263.
29	PENG Z， LIU H， JIA Y， et al. Adaptive attribute and structure subspace clustering network ［J］. IEEE Transactions on Image Processing， 2022， 31： 3430-3439.
30	VIDAL R， FAVARO P. Low rank subspace clustering （LRSC）［J］. Pattern Recognition Letters， 2014， 43： 47-61.
31	ABHADIOMHEN S E， WANG Z Y， SHEN X J. Coupled low rank representation and subspace clustering ［J］. Applied Intelligence， 2022， 52（1）： 530-546.
32	XU Y， CHEN S， LI J， et al. Fast subspace clustering by learning projective block diagonal representation ［J］. Pattern Recognition， 2023， 135： 109152.
33	WANG L， WANG Y， DENG H， et al. Attention reweighted sparse subspace clustering ［J］. Pattern Recognition， 2023， 139： 109438.
34	DIZAJI K G， HERANDI A， DENG C， et al. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 5747-5756.

[1]	Zhuo ZHANG, Huazhu CHEN. Deep subspace clustering based on multiscale self-representation learning with consistency and diversity [J]. Journal of Computer Applications, 2024, 44(2): 353-359.
[2]	Ran GAO, Huazhu CHEN. Improved subspace clustering model based on spectral clustering [J]. Journal of Computer Applications, 2021, 41(12): 3645-3651.

Enhanced deep subspace clustering method with unified framework

统一框架的增强深度子空间聚类方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 19

References 34

Related Articles 2

Recommended Articles

Metrics