Skip to main content
SpringerLink
Log in

Entropy regularized fuzzy nonnegative matrix factorization for data clustering

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Clustering high-dimensional data is very challenging due to the curse of dimensionality. To address this problem, low-rank matrix approximations are widely used to identify the underlying low-dimensional structure of a dataset. Among these, nonnegative matrix factorization (NMF) is the most popular because its decomposed factors are nonnegative and meaningful. However, the NMF problem has been proved to be nonconvex and NP-hard, thus resulting in many local minima. To obtain high-quality local minima, we propose an entropy regularized fuzzy nonnegative matrix factorization (ERF-NMF) model for high-dimensional data fuzzy clustering. First, probability simplex constraints on the decomposed weight components are added to achieve dimension reduction and fuzzy clustering of a dataset simultaneously. Based on the constraints, we also introduce entropy regularization to further reduce the search space for optimal solutions. Finally, we present multiplicative update rules for solving the ERF-NMF model and provide a complexity and convergence analysis. Comprehensive experiments show that the proposed ERF-NMF performs remarkably well with promising results, and its decomposition will be sparser because of entropy regularization and have a clearer physical meaning because of probability simplex constraints.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

No applicable.

References

  1. Marx V (2013) The big challenges of big data. Nature 498(7453):255–260

    Google Scholar 

  2. Jain AK, Murty MN, Flynn PJ (1999) Data clustering: a review. ACM Comp Surv 31(3):264–323

    Google Scholar 

  3. Jain AK (2010) Data clustering: 50 years beyond k-means. Patt Recognit Lett 31(8):651–666

    Google Scholar 

  4. Xie XL, Beni G (1991) A validity measure for fuzzy clustering. IEEE Trans Patt Anal Mach Intell 13(8):841–847

    Google Scholar 

  5. Comaniciu D, Meer P (2002) Mean shift: A robust approach toward feature space analysis. IEEE Trans Patt Anal Mach Intell 24(5):603–619

    Google Scholar 

  6. Ester M, Kriegel H-P, Sander J, Xu X (1996) A density-based algorithm for discovering clusters in large spatial datasets with noise. In Kdd 96:226–231

    Google Scholar 

  7. Ankerst M, Breunig MM, Kriegel H-P, Sander J (1999) Optics: ordering points to identify the clustering structure. ACM Sigmod Rec 28(2):49–60

    Google Scholar 

  8. Rodriguez A, Laio A (2014) Clustering by fast search and find of density peak. Science 344(6191):1492–1496

    Google Scholar 

  9. Kyriakopoulou A, Kalamboukis T (2006) Text classification using clustering. In: Proceedings of The Discovery Challenge Workshop, 28-38

  10. Köppen M (2000) The curse of dimensionality 1:4–8

  11. Tenenbaum JB, Silva Vd, Langford JC (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290(5500):2319–2323

    Google Scholar 

  12. Roweis S, Saul L (2020) Nonlinear dimensionality analysis by locally linear embedding. Science 290(12):2323–2326

    Google Scholar 

  13. Gillis N (2011) Nonnegative matrix factorization Complexity algorithms and applications. PhD Thesis Universite catholique de Louvain 290(12), 2323–2326

  14. Alkandari A, Aljaber SJ (2015) Principle component analysis algorithm (pca) for image recognition. In: International Conference on Computing Technology and Information Management, pp. 76–80

  15. Hyvärinen A, Oja E (2000) Independent component analysis: algorithms and applications. Neural Net 13(4):411–430

    Google Scholar 

  16. He G, Gan J (2004) A method for singular value feature extraction of face image. In: International Symposium on Intelligent Multimedia, Video and Speech Processing, pp. 37-40

  17. Paatero P, Tapper U (1994) Positive matrix factorization: a nonnegative factor model with optimal utilization of error estimates of data values. Environmetrics 5(2):111–126

    Google Scholar 

  18. Lee DD, Seung HS (1999) Learning the parts of objects by nonnegative matrix factorization. Nature 401(6755):788–791

    Google Scholar 

  19. Drakakis K, Rickard S, de Fréin R, Cichocki A (2008) Analysis of financial data using non-negative matrix factorization. Int Math Forum 3:1853–1870

    MathSciNet  Google Scholar 

  20. Ulbrich I, Canagaratna M, Zhang Q, Worsnop D, Jimenez J (2009) Interpretation of organic components from positive matrix factorization of aerosol mass spectrometric data. Atmos Chem Phys 9:2891–2918

    Google Scholar 

  21. Cichocki A, Zdunek R, Phan AH, Amari Si (2009) Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-Way Data Analysis and Blind Source Separation,

  22. Gan J, Liu T, Li L, Zhang J (2021) Nonnegative matrix factorizations: A survey. Comp J 64(7):1080–1092

    Google Scholar 

  23. Vavasis SA (2009) On the complexity of nonnegative matrix factorization. SIAM J. Optim 20(3):1364–1377

    MathSciNet  Google Scholar 

  24. Donoho D, Stodden V (2003) When does non-negative matrix factorization give a correct decomposition into parts? Advances in Neural Information Processing Systems 16

  25. Berry MW, Browne M, Langville AN, Pauca VP, Plemmons RJ (2007) Algorithms and applications for approximate nonnegative matrix factorization. Comp Stat Data Anal 52(1):155–173

    MathSciNet  Google Scholar 

  26. Wang Y-X, Zhang Y-J (2012) Nonnegative matrix factorization: A comprehensive review. IEEE Trans Know Data Eng 25(6):1336–1353

    Google Scholar 

  27. Olshausen BA, Field DJ (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381(6583):607–609

    Google Scholar 

  28. Hoyer PO (2004) Non-negative matrix factorization with sparseness constraints. J Mach Learn Res 5:9

    MathSciNet  Google Scholar 

  29. Pauca VP, Piper J, Plemmons RJ (2006) Nonnegative matrix factorization for spectral data analysis. Lin Algeb Appl 416(1):29–47

    MathSciNet  Google Scholar 

  30. Peharz R, Pernkopf F (2012) Sparse nonnegative matrix factorization with \(\ell ^0\)-constraints. Neurocomputing 80:38–46

    Google Scholar 

  31. Nakamura T, Kameoka H (2015) \(l_p\)-norm non-negative matrix factorization and its application to singing voice enhancement. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2115–2119

  32. He W, Zhang H, Zhang L (2017) Total variation regularized reweighted sparse nonnegative matrix factorization for hyperspectral unmixing. IEEE Trans Geosci Remote Sens 55(7):3909–3921

    Google Scholar 

  33. Qian Y, Jia S, Zhou J, Robles-Kelly A (2010) L1/2 sparsity constrained nonnegative matrix factorization for hyperspectral unmixing. In: International Conference on Digital Image Computing: Techniques and Applications, pp. 447–453

  34. Peng S, Yang Z, Ling BW-K, Chen B, Lin Z (2022) Dual semi-supervised convex nonnegative matrix factorization for data representation. Infor Sci 585:571–593

    Google Scholar 

  35. Belkin M, Niyogi P, Sindhwani V (2006) Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. J Mach Learn Res 7:11

    MathSciNet  Google Scholar 

  36. Cai D, He X, Han J, Huang TS (2010) Graph regularized nonnegative matrix factorization for data representation. IEEE Trans Patt Anal Mach Intell 33(8):1548–1560

    Google Scholar 

  37. Hu J, Li Y, Xu G, Gao W (2022) Dynamic subspace dual-graph regularized multi-label feature selection. Neurocomputing 467:184–196

    Google Scholar 

  38. Huang Q, Yin X, Chen S, Wang Y, Chen B (2020) Robust nonnegative matrix factorization with structure regularization. Neurocomputing 412:72–90

    Google Scholar 

  39. Cichocki A, Zdunek R (2006) Multilayer nonnegative matrix factorisation. Elect Lett 42(16):947

    Google Scholar 

  40. Junmin L, Shuai Y, Xuehu Z, Yifan H, Qian Z (2021) Nonnegative matrix factorization with entropy regularization for hyperspectral unmixing. Int J Rem Sens 42(16):6359–6390

    Google Scholar 

  41. Hartigan JA, Wong MA (1979) Algorithm as 136: a k-means clustering algorithm. J Royal Stat Soci 28(1):100–108

    Google Scholar 

  42. Dunn JC (1973) A fuzzy relative of the isodata process and its use in detecting compact well-separated clusters

  43. Bezdek JC, Ehrlich R, Full W (1984) Fcm: The fuzzy c-means clustering algorithm. Comp Geosci 10(2–3):191–203

    Google Scholar 

  44. Baker J, Fearnhead P, Fox E, Nemeth C (2018) Large-scale stochastic sampling from the probability simplex. Advances in Neural Information Processing Systems 31

  45. MacKay DJ, Mac Kay DJ (2003) Information Theory, Inference and Learning Algorithms,

  46. Seung D, Lee L (2001) Algorithms for non-negative matrix factorization. Adv Neural Infor Proc Syst 11:556–562

    Google Scholar 

  47. Li H, Gao Y, Liu J, Zhang J, Li C (2022) Semi-supervised graph regularized nonnegative matrix factorization with local coordinate for image representation. Sig Proc Image Commun 102:116589

    Google Scholar 

  48. Chen Y, Zhang J, Cai D, Liu W, He X (2012) Nonnegative local coordinate factorization for image representation. IEEE Trans Image Proc 22(3):969–979

    MathSciNet  Google Scholar 

  49. Nene S, Nayar S, Murase H (1996) Columbia object image library (coil-20). Comput. Sci., Columbia Univ., New York.[Online] http://www. cs. columbia. edu/CAVE/coil-20. html 62

  50. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Google Scholar 

  51. Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs fisherfaces: Recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Google Scholar 

  52. Hond D, Spacek, L (1997) Distinctive descriptions for face processing. In: BMVC, pp. 0–4

  53. Lyons, M.J., Akamatsu, S., Kamachi, M., Gyoba, J., Budynek, J (1998) The japanese female facial expression (jaffe) database. In: International Conference on Automatic Face and Gesture Recognition, pp. 14–16

  54. Liu H, Wu Z, Li X, Cai D, Huang TS (2011) Constrained nonnegative matrix factorization for image representation. IEEE Trans Patt Anal Mach Intell 34(7):1299–1311

    Google Scholar 

  55. Tsinos CG, Rontogiannis AA, Berberidis K (2017) Distributed blind hyperspectral unmixing via joint sparsity and low-rank constrained non-negative matrix factorization. IEEE Trans Comp Imag 3(2):160–174

    MathSciNet  Google Scholar 

  56. He W, Zhang H, Zhang L (2017) Total variation regularized reweighted sparse nonnegative matrix factorization for hyperspectral unmixing. IEEE Trans Geosci Remote Sens 55(7):3909–3921

    Google Scholar 

  57. Peng C, Zhang Z, Kang Z, Chen C, Cheng Q (2021) Nonnegative matrix factorization with local similarity learning. Inform Sci 562:325–346

    MathSciNet  Google Scholar 

  58. Nascimento JM, Dias JM (2005) Vertex component analysis: A fast algorithm to unmix hyperspectral data. IEEE Trans Geosci Remote Sens 43(4):898–910

    Google Scholar 

  59. Bioucas-Dias, J.M (2009)A variable splitting augmented lagrangian approach to linear spectral unmixing. In: Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing, pp. 1–4

Download references

Funding

This document is a result of the research project funded by the National Natural Science Foundation of China with Grant Number 62276208.

Author information

Authors and Affiliations

  1. School of Mathematics and Statistics, Xi’an Jiaotong University, West Xianning Road, Xi’an, 710049, Shaanxi, China

    Kun Chen, Junchen Liang, Junmin Liu, Weilin Shen, Zongben Xu & Zhengjian Yao

Authors
  1. Kun Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junchen Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junmin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weilin Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zongben Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhengjian Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Junmin Liu or Zongben Xu.

Ethics declarations

Conflicts of interest

The authors declare that there is no conflict of interest

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, K., Liang, J., Liu, J. et al. Entropy regularized fuzzy nonnegative matrix factorization for data clustering. Int. J. Mach. Learn. & Cyber. 15, 459–476 (2024). https://doi.org/10.1007/s13042-023-01919-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-023-01919-1

Keywords

Search

Navigation