Facial Expression Recognition based on Image Gradient and Deep Convolutional Neural Network

Fallahzadeh, M. R.; Farokhi, F.; Harimi, A.; Sabbaghi-Nadooshan, R.

doi:10.22044/jadm.2021.9898.2121

Document Type : Original/Review Paper

Authors

¹ Department of Technical and engineering, Central Tehran Branch, Islamic Azad University, Iran.

² Department of Technical and engineering, Shahrood Branch, Islamic Azad University, Iran.

https://doi.org/10.22044/jadm.2021.9898.2121

Abstract

Facial Expression Recognition (FER) is one of the basic ways of interacting with machines and has been getting more attention in recent years. In this paper, a novel FER system based on a deep convolutional neural network (DCNN) is presented. Motivated by the powerful ability of DCNN to learn features and image classification, the goal of this research is to design a compatible and discriminative input for pre-trained AlexNet-DCNN. The proposed method consists of 4 steps: first, extracting three channels of the image including the original gray-level image, in addition to horizontal and vertical gradients of the image similar to the red, green, and blue color channels of an RGB image as the DCNN input. Second, data augmentation including scale, rotation, width shift, height shift, zoom, horizontal flip, and vertical flip of the images are prepared in addition to the original images for training the DCNN. Then, the AlexNet-DCNN model is applied to learn high-level features corresponding to different emotion classes. Finally, transfer learning is implemented on the proposed model and the presented model is fine-tuned on target datasets. The average recognition accuracy of 92.41% and 93.66% were achieved for JAFEE and CK+ datasets, respectively. Experimental results on two benchmark emotional datasets show promising performance of the proposed model that can improve the performance of current FER systems.

Keywords

20.1001.1.23225211.2021.9.2.12.5

References

[1] A. Majumder, L. Behera, and V.K. Subramanian, "Automatic facial expression recognition system using deep network-based data fusion," IEEE transactions on cybernetics., vol. 48, pp. 103-114, Jan 2016.

[2] Y. Din, Q. Zhao, B. Li, and X. Yuan, "Facial expression recognition from image sequence based on LBP and Taylor expansion," IEEE Access., vol. 5, pp. 19409-19419, August 2017.

[3] JY. Jung, SW. Kim, CH. Yoo, WJ. Park, and S.J. Ko, "LBP-ferns-based feature extraction for robust facial recognition," IEEE Transactions on Consumer Electronics., vol. 62, pp. 446-453, November 2016.

[4] J. Deng, G. Pang, Z. Zhang, Z. Pang, H. Yang, and G. Yang, "cGAN Based Facial Expression Recognition for Human-Robot Interaction," IEEE Access., vol. 7, pp. 9848-9859, January 2019.

[5] M. Z. Uddin, M.M. Hassan, A. Almogren, A. Alamri, M. Alrubaian, and G. Fortino, "Facial expression recognition utilizing local direction-based robust features and deep belief network," IEEE Access., vol. 5, pp. 4525-4536, March 2017.

[6] Y. Zhang and Q. Ji, "Active and dynamic information fusion for facial expression understanding from image sequences," IEEE Transactions on pattern analysis and machine intelligence., vol. 27, pp. 699-714, March 2005.

[7] A. Panning, A.K. Al-Hamadi, R. Niese, and B. Michaelis, "Facial expression recognition based on haar-like feature detection," Pattern Recognition and Image Analysis., vol. 18, pp. 447-4452, Sep 2008.

[8] C. Shan, S. Gong and P.W. McOwan. "Facial expression recognition based on local binary patterns: A comprehensive study," Image and vision Computing"., vol. 27, pp. 803-816, Dec 2009.

[9] W. Liu, Y. Wang, and S. Li, "LBP feature extraction for facial expression recognition," Journal of information & computional science., vol. 8, pp.412-421, March 2011.

[10] L. Wang and k. Wang, R. Li, "Unsupervised feature selection based on spectral regression from manifold learning for facial expression recognition," IET Computer Vision., vol. 9, pp. 655-652, Oct 2015.

[11] A. Sedaghat, M. Mokhtarzade, and H. Ebadi, "Uniform robust scale-invariant feature matching for optical remote sensing images," IEEE Transactions on Geoscience and Remote Sensing., vol. 49, pp. 4516-4527, May 2011.

[12] B. Yang, J. Cao, R. Ni, and Y. Zhang. "Facial expression recognition using weighted mixture deep neural network based on double-channel facial images," IEEE Access., vol. 6, pp. 4630-40, Dec 2017.

[13] B.F. Wu and C.H. Lin, "Adaptive feature mapping for customizing deep learning based facial expression recognition model," IEEE Access., vol. 6, Feb 2018.

[14] J.H. Kim, B.G. Kim, P.P. Roy, and D.M. Jeong, "Efficient Facial Expression Recognition Algorithm Based on Hierarchical Deep Neural Network Structure," IEEE Access., vol. 7 Jan 2019.

[15] S. Xie and H. Hu, "Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks," IEEE Transactions on Multimedia., vol. 21, pp. 211-220, June 2018.

[16] Z Yu, G. Liu, Q. Liu, and J. Deng, "Spatio-temporal convolutional features with nested LSTM for facial expression recognition," Neurocomputing., vol. 317, pp. 50-57, November 2018.

[17] M. Garcia and S. Ramirez, "Deep Neural Network Architecture: Application for Facial Expression Recognition," IEEE Latin America Transactions., vol. 18, pp. 1311-1319, May 2020.

[18] Y. Yan, Y. Huang, S. Chen, C. Shen, , "Joint Deep Learning of Facial Expression Synthesis and Recognition," Computer Vision., Feb 2020

[19] A. Krizhevsky, I. Sutskever, and G.E. Hinton, "Imagenet classification with deep convolutional neural networks," Advances in neural information processing systems., pp. 1097-105, Sep 2012.

[20] M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, "Coding facial expressions with gabor wavelets," Proceedings Third IEEE int conference on automatic face and gesture recognition: IEEE., pp. 200-5, 1998.

[21] P. Lucey, J.F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, "The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression," 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops: IEEE., pp. 94-101, 2010.

[22] H. Li, J. Sun, Z. Xu, and L. Chen, "Multimodal 2D+ 3D facial expression recognition with deep fusion convolutional neural network," IEEE Transactions on Multimedia., vol. 19, pp. 2816-31, June 2017.

[23] J. Zhao, X. Mao, and L. Chen, "Learning deep features to recognise speech emotion using merged deep CNN," IET Signal Processing., vol. 12, pp. 713-21, Feb 2018.

[24] C. Zhang, H. Zhang, J. Qiao, D. Yuan, and M. Zhang, "Deep transfer learning for intelligent cellular traffic prediction based on cross-domain big data," IEEE Journal on Selected Areas in Communications., vol. 37, pp. 1389-401, March 2019.

[25] U. Côté-Allard, C.L. Fall, A. Drouin, A. Campeau-Lecours, C. Gosselin, "Deep learning for electromyographic hand gesture signal classification using transfer learning," IEEE Transactions on Neural Systems and Rehabilitation Engineering., vol. 27, pp. 760-71, January 2019.

[26] C. Deng, Y. Xue, X. Liu "Active transfer learning network: A unified deep joint spectral–spatial feature learning model for hyperspectral image classification," IEEE Transactions on Geoscience and Remote Sensing., vol. 57, pp. 1741-54, November 2018.

[27] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv., September 2014.

[28] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," Proceedings of the IEEE conference on computer vision and pattern recognition., 2016. p. 770-8.

[29] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, "Rethinking the inception architecture for computer vision," Proceedings of the IEEE conference on computer vision and pattern recognition., 2016. p. 2818-26.

[30] G. Huang, Z, Liu, L Van Der Maaten, and K.Q. Weinberger, "Densely connected convolutional networks,". Proceedings of the IEEE conference on computer vision and pattern recognition., 2017. p. 4700-8.

[31] N. Ketkar, "Deep Learning with Python," Springer, 2017.

[32] V. Nair and G.E. Hinton, "Rectified linear units improve restricted boltzmann machines," Proceedings of the 27th international conference on machine learning (ICML-10)., 2010. p. 807-14.

[33] T. Kanade, J.F. Cohn, and Y. Tian, "Comprehensive database for facial expression analysis," Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat No PR00580): IEEE., 2000. p. 46-53.

[34] F. Chollet, "Deep Learning with Python," Springer, 2018.

[35] A. Harimi, A. Shahzadi, A.R. Ahmadifard, and K. Yaghmaie, "Classification of emotional speech using spectral pattrn features," Journal of AI and data mining., vol. 2, pp. 53-61, 2014.

[36] M.W. Huang, Z.W. Wang, and Z.L. Ying, "A new method for facial expression recognition based on sparse representation plus LBP," 2010 3rd International Congress on Image and Signal Processing: IEEE., 2010. p. 1750-4.

[37] Z.L. Ying, Z.W. Wang, and M.W. Huang, "Facial expression recognition based on fusion of sparse representation," International Conference on Intelligent Computing: Springer., 2010. p. 457-64.

[38] L. Du and H. Hu, "Modified classification and regression tree for facial expression recognition with using difference expression images," Electronics Letters., vol. 53, pp.590-592, April 2017.

[39] S. Al-Sumaidaee, S. Dlay, "Facial expression recognition using local Gabor gradient code-horizontal diagonal descriptor," IET International Conference on Intelligent Signal Processing., Novomber 2015.

[40] S. Xie and H Hu, "Facial expression recognition with FRR-CNN," Electronics Letters., February 2017.