Document Type : Applied Article
Authors
1 Department of Computer and Information Technology Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran.
2 Institute for Transport Studies, University of Leeds, Leeds, LS2 9JT, UK
10.22044/jadm.2025.15658.2683
Abstract
Human Activity Recognition (HAR) using computer vision is an expanding field with diverse applications, including healthcare, transportation, and human-computer interaction. While classical approaches such as Support Vector Machines (SVM), Histogram of Oriented Gradients (HOG), and Hidden Markov Models (HMM) rely on manually extracted features and struggle with complex motion patterns, deep learning-based models (e.g., Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), Transformer-based models) have improved performance but still face challenges in handling occlusions, noisy environments, and computational efficiency. This paper introduces Attention-HAR, a novel deep neural network model designed to enhance HAR performance through three key innovations: Conv3DTranspose for spatial upsampling, ConvLSTM2D for capturing spatiotemporal patterns, and a custom attention mechanism that prioritizes critical frames within sequences. Unlike conventional attention mechanisms, our approach dynamically assigns weights to key frames, reducing the impact of redundant frames and enhancing interpretability and computational efficiency. Experimental results on the UCF-101 dataset demonstrate that Attention-HAR outperforms state-of-the-art models, achieving an accuracy of 97.61%, a precision of 97.95%, a recall of 97.49%, an F1-score of 97.64, and an AUC of 99.9%. With only 1.26 million parameters, the model is computationally efficient and well-suited for deployment on lightweight platforms. These findings suggest that integrating temporal-spatial feature learning with attention mechanisms can significantly improve HAR in dynamic and complex environments.
Keywords
- Human Activity Recognition
- Deep Neural Networks
- Attention-HAR
- Attention Mechanisms
- Video-Based Activity Recognition
Main Subjects
[1] Z. Malik and M. I. Bin Shapiai, “Human action interpretation using convolutional neural network: a survey,” Mach Vis Appl, vol. 33, no. 3, 2022.
[2] P. Khaire and P. Kumar, “Deep learning and RGB-D based human action, human–human and human–object interaction recognition: A survey,” J Vis Commun Image Represent, vol. 86, no. May, p. 103531, 2022.
[3] S. A. Khowaja and S.-L. Lee, “Semantic Image Networks for Human Action Recognition,” [Online]. http://arxiv.org/abs/1901.06792. 2019.
[4] H. B. Zhang, Y.X Zhang, B. Zhong, Q. Lei, L. Yang, J.X. Du, and D.S. Chen, “A comprehensive survey of vision-based human action recognition methods,” Sensors (Switzerland), vol. 19, no. 5, pp. 1–20, 2019.
[5] O. P. Popoola and K. Wang, “Video-based abnormal human behavior recognition a review,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42, no. 6, pp.865-878, 2012.
[6] X. Wenkai and E. J. Lee, “Continuous gesture trajectory recognition system based on computer vision,” Applied Mathematics and Information Sciences, vol. 6, no. 2 SUPPL., pp. 339–346, 2012.
[7] B. Paulson, D. Cummings, and T. Hammond, “Object interaction detection using hand posture cues in an office setting,” Int J Hum Comput Stud, vol. 69, no. 1, pp. 19–29, 2011.
[8] M. Hossein Shayesteh, B. Shahrokhzadeh, and B. Masoumi, “Game Theory Solutions in Sensor-Based Human Activity Recognition: A Review,” Journal of AI and Data Mining, vol. 11, no. 2, doi: 10.22044/jadm.2023.12538.2407. 2023.
[9] D. R. Beddiar, B. Nini, M. Sabokrou, and A. Hadid, “Vision-based human activity recognition: a survey,” Multimed Tools Appl, vol. 79, no. 41–42, pp. 30509–30555, 2020.
[10] E. M. Saoudi, J. Jaafari, and S. J. Andaloussi, “Advancing human action recognition: A hybrid approach using attention-based LSTM and 3D CNN,” Sci Afr, vol. 21, 2023.
[11] Y. Kong and Y. Fu, Human Action Recognition and Prediction: A Survey, vol. 130, no. 5. Springer US, 2022.
[12] G.V. Reddy, K. Deepika, L. Malliga, D. Hemanand, C. Senthilkumar, S. Gopalakrishnan, and Y. Farhaoui, “Human Action Recognition Using Difference of Gaussian and Difference of Wavelet,” Big Data Mining and Analytics, vol. 6, no. 3, pp. 336–346, 2023.
[13] B. S. Kumar, S. V. Raju, and H. V. Reddy, “Human Action Recognition Using a Novel Deep Learning Approach,” IOP Conf Ser Mater Sci Eng, vol. 1042, no. 1, p. 012031, 2021.
[14] R. Vrskova, P. Kamencay, R. Hudec, and P. Sykora, “A New Deep-Learning Method for Human Activity Recognition,” Sensors, vol. 23, no. 5, 2023.
[15] S. P. Sahoo, S. Ari, K. Mahapatra, and S. P. Mohanty, “HAR-Depth: A Novel Framework for Human Action Recognition Using Sequential Learning and Depth Estimated History Images,” IEEE Trans Emerg Top Comput Intell, vol. 5, no. 5, pp. 813–825, 2021.
[16] R. K. Gour and D. Rai, “Unveiling Human Actions: Vision-Based Activity Recognition Using ConvLSTM and LRCN Models,” in 2024 OPJU International Technology Conference on Smart Computing for Innovation and Advancement in Industry 4.0, OTCON 2024, Institute of Electrical and Electronics Engineers Inc., 2024.
[17] S. Li, J. Yi, Y. A. Farha, and J. Gall, “Pose Refinement Graph Convolutional Network for Skeleton-based Action Recognition,” Oct. 2020, [Online]. Available: http://arxiv.org/abs/2010.07367.
[18] M. Deyzel and R. P. Theart, “One-shot skeleton-based action recognition on strength and conditioning exercises,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5167–77. [Online]. https://github.com/michaeldeyzel/SU-EMD. 2023.
[19] H. Chen, Y. Jiang, and H. Ko, “Pose-Guided Graph Convolutional Networks for Skeleton-Based Action Recognition,” IEEE Access, vol. PP, p. 1, 2022.
[20] E. Akarsu and T. Karacalı, “Video Classification Results with Artificial Intelligence and Machine Learning,” International Journal of Innovative Research and Reviews (INJIRR), vol. 7, pp. 22–26, [Online].http://www.injirr.com/article/view/194. 2023.
[21] K. Alomar and X. Cai, “TransNet: A Transfer Learning-Based Network for Human Action Recognition,” 2023, [Online]. Available: http://arxiv.org/abs/2309.06951.
[22] B. Chen, H. Tang, Z. Zhang, G. Tong, and B. Li, “Video-based action recognition using spurious-3D residual attention networks,” IET Image Process, vol. 16, no. 11, pp. 3097–3111, 2022.
[23] X. Wei and Z. Wang, “TCN-attention-HAR: human activity recognition based on attention mechanism time convolutional network,” Sci Rep, vol. 14, no. 1, Dec. 2024.
[24] J. Wensel, H. Ullah, and A. Munir, “ViT-ReT: Vision and Recurrent Transformer Neural Networks for Human Activity Recognition in Videos,” IEEE Access, vol. 11, no. June, pp. 72227–72249, 2023.
[25] N. ur R. Malik, S. A. R. Abu-Bakar, U. U. Sheikh, A. Channa, and N. Popescu, “Cascading Pose Features with CNN-LSTM for Multiview Human Action Recognition,” Signals, vol. 4, no. 1, pp. 40–55, Mar. 2023.
[26] Z. Xing, Q. Dai, H. Hu, J. Chen, Z. Wu, and Y.-G. Jiang, “SVFormer: Semi-supervised Video Transformer for Action Recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18816–26. 2023.
[27] Y. and J. X. and H. Z. and P. Y. Mao Keming and Xiao, “KS-FuseNet: An Efficient Action Recognition Method Based on Keyframe Selection and Feature Fusion,” in Pattern Recognition and Computer Vision, M.-M. and H. R. and U. K. and S. W. and Z. H. and Z. J. and L. C.-L. Lin Zhouchen and Cheng, Ed., Singapore: Springer Nature Singapore, pp. 540–553. 2025.
[28] N. Hassan, A. S. M. Miah, and J. Shin, “A Deep Bidirectional LSTM Model Enhanced by Transfer-Learning-Based Feature Extraction for Dynamic Human Activity Recognition,” Applied Sciences (Switzerland), vol. 14, no. 2, Jan. 2024.
[29] E. Genc, M. E. Yildirim, and Y. B. Salman, “Human activity recognition with fine-tuned CNN-LSTM,” Journal of Electrical Engineering, vol. 75, no. 1, pp. 8–13, Feb. 2024.
[30] S. Uddin, T. Nawaz, J. Ferryman, N. Rashid, M. Asaduzzaman, and R. Nawaz, “Skeletal Keypoint-Based Transformer Model for Human Action Recognition in Aerial Videos,” IEEE Access, vol. 12, no. January, pp. 11095–11103, 2024.
[31] S. D. S. Dass, H. B. Barua, G. Krishnasamy, R. Paramesran, and R. C.-W. Phan, “ActNetFormer: Transformer-ResNet Hybrid Method for Semi-Supervised Action Recognition in Videos,” [Online]. Available: http://arxiv.org/abs/2404.06243. 2024.
[32] C. Shi and S. Liu, “Human action recognition with transformer based on convolutional features,” Intelligent Decision Technologies, vol. 18, no. 2, pp. 881–896, May 2024.
[33] I. U. Khan, S. Afzal, and J. W. Lee, “Human activity recognition via hybrid deep learning-based model,” Sensors, vol. 22, no. 1, 2022.
[34] S. Tomassini, H. Anbar, A. Sbrollini, M. Morettini, M. H. D. J. Mortada, and L. Burattini, “A Double-Stage 3D U-Net for On-Cloud Brain Extraction and Multi-Structure Segmentation from 7T MR Volumes,” 2023.
[35] V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” [Online]. Available: http://arxiv.org/abs/1603.07285. 2016.
[36] X. Zhang, Y. Zou, and W. Shi, “Dilated convolution neural network with LeakyReLU for environmental sound classification,” in International Conference on Digital Signal Processing, DSP, Institute of Electrical and Electronics Engineers Inc., Nov. 2017.
[37] P. Dasari, L. Zhang, Y. Yu, H. Huang, and R. Gao, “Human Action Recognition Using Hybrid Deep Evolving Neural Networks,” Proceedings of the International Joint Conference on Neural Networks, vol. 2022-July, 2022.
[38] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W. Wong, and W. Woo, “Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting,” Jun. 2015, [Online]. Available: http://arxiv.org/abs/1506.04214.
[39] Y.-L. Chang, N.B. Tatini, T.H. Chen, M.C. Wu, J.H. Chuah, Y.T. Chen, and L. Chang, “Convlstm Neural Network for Rice Field Classification from Sentinel-1A Sar Images,” in IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium, pp. 5047–5050. 2022.
[40] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, “On the Properties of Neural Machine Translation: Encoder-Decoder Approaches,” [Online]. Available: http://arxiv.org/abs/1409.1259. 2014.
[41] L. Lu and K. A. I. Cao, “A Multichannel CNN-GRU Model for Human Activity Recognition,” IEEE Access, vol. 10, no. June, pp. 66797–66810, 2022.
[42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, AN. Gomez, Ł. Kaiser, I. Polosukhin. “Attention Is All You Need,” Advances in neural information processing systems, vol 30, 2017.
[43] G. Brauwers and F. Frasincar, “A General Survey on Attention Mechanisms in Deep Learning,” IEEE Transactions on Knowledge and Data Engineering, 35(4), pp.3279-3298. 2022.
[44] M. H. Guo, T.X. Xu, J.J. Liu, Z.N. Liu, P.T. Jiang, T.J. Mu, S.H. Zhang, R.R. Martin, M.M. Cheng, and S.M. Hu, “Attention mechanisms in computer vision: A survey,” Tsinghua University. 2022.
[45] D. Kumari and R. S. Anand, “Isolated Video-Based Sign Language Recognition Using a Hybrid CNN-LSTM Framework Based on Attention Mechanism,” Electronics (Switzerland), vol. 13, no. 7, Apr. 2024.
[46] N. Jaouedi, N. Boujnah, and M. S. Bouhlel, “A new hybrid deep learning model for human action recognition,” Journal of King Saud University - Computer and Information Sciences, vol. 32, no. 4, pp. 447–453, 2020.
[47] Pan, Tian, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu. "Videomoco: Contrastive video representation learning with temporally adversarial examples." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11205-11214. 2021.
[48] D. Kumar, M., Rana, A., Ankita, Yadav, A. K., & Yadav, “Human Activity Recognition in Videos Using Deep Learning,” in In International Conference on Soft Computing and its Engineering Applications, pp. 288–299. 2022.
[49] A. Alavigharahbagh, V. Hajihashemi, and J. J. M. Machado, “Deep Learning Approach for Human Action Recognition Using a Time Saliency Map Based on Motion Features Considering Camera Movement and Shot in Video Image Sequences,”, Information 14(11), p.616, 2023.
[50] E. Dastbaravardeh, S. Askarpour, M. Saberi Anari, and K. Rezaee, “Channel Attention-Based Approach with Autoencoder Network for Human Action Recognition in Low-Resolution Frames,” International Journal of Intelligent Systems, no 1, 1052344, 2024.
)