Document Type : Original/Review Paper
Authors
Faculty of Electrical Engineering, Shahrood University of Technology, Shahrood, Iran.
10.22044/jadm.2025.16416.2767
Abstract
The Convolutional Restricted Boltzmann Machine (CRBM) is a generative model that extracts representations from unlabeled data, achieving success in various applications. However, its unsupervised nature may yield suboptimal representations for specific classification tasks. This paper proposes adapting k-means clustering to enhance CRBM parameters, aligning features with informative cluster centers. A novel criterion combining generative and soft-K-Means objectives optimizes both cluster centers and CRBM parameters, allowing for continued unsupervised feature learning.
Experiments on MNIST, CIFAR10, and three facial expression datasets (JAFFE, KANADE, BU) show that the proposed method enhances the learning process and offers a more informative representation compared to standard and classification CRBM.
Keywords
- Convolutional Restricted Boltzmann Machine
- feature learning
- representation learning
- K-means clustering
Main Subjects
[1] T. S. Lee, "Image representation using 2D Gabor wavelets," IEEE Transactions on pattern analysis and machine intelligence, vol. 18, no. 10, pp. 959-971, 1996.
[2] Y. Bengio, A. Courville, and P. Vincent, "Representation learning: A review and new perspectives," IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 8, pp. 1798-1828, 2013.
[3] G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," science, vol. 313, no. 5786, pp. 504––507, 2006.
[4] G. E. Hinton, S. Osindero, and Y.-W. Teh, "A fast learning algorithm for deep belief nets," Neural computation, vol. 18, no. 7, pp. 1527––1554, 2006.
[5] M. Joudaki, 2025, doi: 10.20944/preprints202502.1119.v2.
[6] N. Zhang, S. Ding, J. Zhang, and Y. Xue, "An overview on restricted Boltzmann machines," Neurocomputing, vol. 275, pp. 1186––1199, 2018.
[7] A. Fischer and C. Igel, "Training restricted Boltzmann machines: An introduction," Pattern Recognition, vol. 47, no. 1, pp. 25––39, 2014.
[8] V. V. Narayana and P. Kodali, "Advanced Seizure Detection Framework Using Stacked Convolutional Restricted Boltzmann Machine (SCRBM)," IEEE Sensors Letters, vol. 9, no. 3, pp. 1–4, 2025, doi: 10.1109/lsens.2025.3543141.
[9] K. Visalini, S. Alagarsamy, and D. Nagarajan, "Event-Based Epileptic Seizure Detection with Stacked Convolutional Restricted Boltzmann Machine," IETE Journal of Research, vol. 70, no. 5, pp. 4880–4889, 2023, doi: 10.1080/03772063.2023.2234854.
[10] S. Sobczak and R. Kapela, "Hybrid Restricted Boltzmann Machine– Convolutional Neural Network Model for Image Recognition," IEEE Access, vol. 10, pp. 24985–24994, 2022, doi: 10.1109/access.2022.3155873.
[11] X. Lü, L. Long, R. Deng, and R. Meng, "Image feature extraction based on fuzzy restricted Boltzmann machine," Measurement, vol. 204, 2022, doi: 10.1016/j.measurement.2022.112063.
[12] R. Kharghanian, A. Peiravi, F. Moradi, and A. Iosifidis, "Pain detection using batch normalized discriminant restricted Boltzmann machine layers," Journal of Visual Communication and Image Representation, vol. 76, 2021, doi: 10.1016/j.jvcir.2021.103062.
[13] R. Kirubahari and S. M. J. Amali, "An improved restricted Boltzmann Machine using Bayesian Optimization for Recommender Systems," Evolving Systems, vol. 15, no. 3, pp. 1099–1111, 2023, doi: 10.1007/s12530-023-09520-1.
[14] E. Karthik and T. Sethukarasi, "A Centered Convolutional Restricted Boltzmann Machine Optimized by Hybrid Atom Search Arithmetic Optimization Algorithm for Sentimental Analysis," Neural Processing Letters, vol. 54, no. 5, pp. 4123–4151, 2022, doi: 10.1007/s11063-022-10797-7.
[15] M. Aamir, N. M. Nawi, F. Wahid, M. S. H. Zada, M. Z. Rehman, and M. Zulqarnain, "Hybrid Contractive Auto-encoder with Restricted Boltzmann Machine For Multiclass Classification," Arabian Journal for Science and Engineering, vol. 46, no. 9, pp. 9237–9251, 2021, doi: 10.1007/s13369-021-05674-9.
[16] J. Garí, E. Romero, and F. Mazzanti, "Learning restricted Boltzmann machines with pattern induced weights," Neurocomputing, vol. 610, 2024, doi: 10.1016/j.neucom.2024.128469.
[17] H. Larochelle, M. Mandel, R. Pascanu, and Y. Bengio, "Learning algorithms for the classification restricted boltzmann machine," Journal of Machine Learning Research, vol. 13, no. Mar, pp. 643––669, 2012.
[18] G. van Tulder and M. de Bruijne, "Combining generative and discriminative representation learning for lung CT analysis with convolutional restricted boltzmann machines," IEEE transactions on medical imaging, vol. 35, no. 5, pp. 1262––1272, 2016.
[19] J. Yin, J. Lv, Y. Sang, and J. Guo, "Classification model of restricted Boltzmann machine based on reconstruction error," Neural Computing and Applications, vol. 29, no. 11, pp. 1171––1186, 2018.
[20] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations," 2009.
[21] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, "Unsupervised learning of hierarchical representations with convolutional deep belief networks," Communications of the ACM, vol. 54, no. 10, pp. 95––103, 2011.
[22] M. A. Carreira-Perpinan and G. E. Hinton, "On contrastive divergence learning.," 2005.
[23] P. Nousi and A. Tefas, "Deep learning algorithms for discriminant autoencoding," Neurocomputing, vol. 266, pp. 325––335, 2017.
[24] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, and et al., "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278––2324, 1998.
[25] A. Krizhevsky, G. Hinton, and et al., "Learning multiple layers of features from tiny images," 2009.
[26] M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, "Coding facial expressions with gabor wavelets," 1998.
[27] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, "The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression," 2010.
[28] M. La Cascia, S. Sclaroff, and V. Athitsos, "Fast, reliable head tracking under varying illumination: An approach based on registration of texture-mapped 3D models," IEEE Transactions on pattern analysis and machine intelligence, vol. 22, no. 4, pp. 322––336, 2000.
[29] M. A. Ranzato, A. Krizhevsky, and G. Hinton, "Factored 3-way restricted boltzmann machines for modeling natural images," 2010.
[30] W. Luo, J. Li, J. Yang, W. Xu, and J. Zhang, "Convolutional sparse autoencoders for image classification," IEEE transactions on neural networks and learning systems, vol. 29, no. 7, pp. 3289––3294, 2017.
[31] G. E. Hinton, "A practical guide to training restricted Boltzmann machines," Springer, 2012, pp. 599––619.
[32] S. Dieleman and B. Schrauwen, "Accelerating sparse restricted Boltzmann machine training using non-Gaussianity measures," 2012.
)