Convolutional Neural Network Equipped with Attention Mechanism and Transfer Learning for Enhancing Performance of Sentiment Analysis

Sadr, H.; Pedram, Mir M.; Teshnehlab, M.

doi:10.22044/jadm.2021.9618.2100

Document Type : Original/Review Paper

Authors

¹ Department of Computer Engineering, Rasht Branch, Islamic Azad University, Rasht, Iran.

² Department of Electrical and Computer Engineering Faculty of Engineering, Kharazmi University, Tehran, Iran.

³ Industrial Control Center of Excellence, Faculty of Electrical and Computer Engineering, K. N. Toosi University, Tehran, Iran.

https://doi.org/10.22044/jadm.2021.9618.2100

Abstract

With the rapid development of textual information on the web, sentiment analysis is changing to an essential analytic tool rather than an academic endeavor and numerous studies have been carried out in recent years to address this issue. By the emergence of deep learning, deep neural networks have attracted a lot of attention and become mainstream in this field. Despite the remarkable success of deep learning models for sentiment analysis of text, they are in the early steps of development and their potential is yet to be fully explored. Convolutional neural network is one of the deep learning methods that has been surpassed for sentiment analysis but is confronted with some limitations. Firstly, convolutional neural network requires a large number of training data. Secondly, it assumes that all words in a sentence have an equal contribution to the polarity of a sentence. To fill these lacunas, a convolutional neural network equipped with the attention mechanism is proposed in this paper which not only takes advantage of the attention mechanism but also utilizes transfer learning to boost the performance of sentiment analysis. According to the empirical results, our proposed model achieved comparable or even better classification accuracy than the state-of-the-art methods.

Keywords

20.1001.1.23225211.2021.9.2.1.4

References

[1] S. A. Salloum, R. Khan, and K. Shaalan, "A survey of semantic analysis approaches," in Joint European-US Workshop on Applications of Invariance in Computer Vision, 2020: Springer, pp. 61-70.

[2] A. Yadav and D. K. Vishwakarma, "Sentiment analysis using deep learning architectures: a review," Artificial Intelligence Review, Vol. 53, No. 6, pp. 4335-4385, 2020.

[3] M. I. Prabha and G. U. Srikanth, "Survey of sentiment analysis using deep learning techniques," in 2019 1st International Conference on Innovations in Information and Communication Technology (ICIICT), 2019: IEEE, pp. 1-9.

[4] O. Habimana, Y. Li, R. Li, X. Gu, and G. Yu, "Sentiment analysis using deep learning approaches: an overview," Science China Information Sciences, Vol. 63, No. 1, pp. 1-36, 2020.

[5] X. Xie, S. Ge, F. Hu, M. Xie, and N. Jiang, "An improved algorithm for sentiment analysis based on maximum entropy," Soft Computing, Vol. 23, No. 2, pp. 599-611, 2019.

[6] H. Sadr, M. N. Soleimandarabi, M. Pedram, and M. Teshnelab, "Unified Topic-Based Semantic Models: A Study in Computing the Semantic Relatedness of Geographic Terms," in 2019 5th International Conference on Web Research (ICWR), 2019: IEEE, pp. 134-140.

[7] Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy, "Hierarchical attention networks for document classification," in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 1480-1489.

[8] Z. Zhang, Y. Zou, and C. Gan, "Textual sentiment analysis via three different attention convolutional neural networks and cross-modality consistent regression," Neurocomputing, Vol. 275, pp. 1407-1415, 2018.

[9] H. Sadr, M. M. Pedram, and M. Teshnelab, "Improving the Performance of Text Sentiment Analysis using Deep Convolutional Neural Network Integrated with Hierarchical Attention Layer," International Journal of Information and Communication Technology Research, Vol. 11, No. 3, pp. 57-67, 2019.

[10] R. Liu, Y. Shi, C. Ji, and M. Jia, "A survey of sentiment analysis based on transfer learning," IEEE Access, Vol. 7, pp. 85401-85412, 2019.

[11] H. Sadr and M. Nazari Solimandarabi, "Presentation of an efficient automatic short answer grading model based on combination of pseudo relevance feedback and semantic relatedness measures," Journal of Advances in Computer Research, Vol. 10, No. 2, pp. 1-10, 2019.

[12] H. Sadr, M. Nazari, M. M. Pedram, and M. Teshnehlab, "Exploring the Efficiency of Topic-Based Models in Computing Semantic Relatedness of Geographic Terms," International Journal of Web Research, Vol. 2, No. 2, pp. 23-35, 2019.

[13] M. Kuta, M. Morawiec, and J. Kitowski, "Sentiment Analysis with Tree-Structured Gated Recurrent Units," Springer International Publishing AG 2017

[14] K. S. Tai, R. Socher, and C. D. Manning, "Improved semantic representations from tree-structured long short-term memory networks," arXiv preprint arXiv:1503.00075, 2015.

[15] Y. Kim, "Convolutional neural networks for sentence classification," arXiv preprint arXiv:1408.5882, 2014.

[16] X. Zhang, J. Zhao, and Y. LeCun, "Character-level convolutional networks for text classification," in Advances in neural information processing systems, 2015, pp. 649-657.

[17] W. Yin, H. Schütze, B. Xiang, and B. Zhou, "Abcnn: Attention-based convolutional neural network for modeling sentence pairs," arXiv preprint arXiv:1512.05193, 2015.

[18] N. Kalchbrenner, E. Grefenstette, and P. Blunsom, "A convolutional neural network for modelling sentences," arXiv preprint arXiv:1404.2188, 2014.

[19] R. Socher, B. Huval, C. D. Manning, and A. Y. Ng, "Semantic Compositionality through Recursive Matrix-Vector Spaces," Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Association for Computational Linguistics., 2012.

[20] R. Socher et al., "Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank," in EMNLP, 2013.

[21] H. Sadr, M. M. Pedram, and M. Teshnehlab, "A Robust Sentiment Analysis Method based on Sequential Combination of Convolutional and Recursive Neural Networks," Neural Processing Letters, pp. 1-17, 2019.

[22] H. Sadr, M. M. Pedram, and M. Teshnehlab, "Multi-View Deep Network: A Deep Model based on Learning Features From Heterogeneous Neural Networks for Sentiment Analysis," IEEE Access, Vol. 8, pp. 86984-86997, 2020.

[23] Y. Wang, M. Huang, and L. Zhao, "Attention-based LSTM for aspect-level sentiment classification," in Proceedings of the 2016 conference on empirical methods in natural language processing, 2016, pp. 606-615.

[24] Z. Yuan, S. Wu, F. Wu, J. Liu, and Y. Huang, "Domain attention model for multi-domain sentiment classification," Knowledge-Based Systems, Vol. 155, pp. 1-10, 2018.

[25] T. Semwal, P. Yenigalla, G. Mathur, and S. B. Nair, "A practitioners' guide to transfer learning for text classification using convolutional neural networks," in Proceedings of the 2018 SIAM International Conference on Data Mining, 2018: SIAM, pp. 513-521.

[26] F. Zhuang et al., "A comprehensive survey on transfer learning," arXiv preprint arXiv:1911.02685, 2019.

[27] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Advances in neural information processing systems, 2012, pp. 1097-1105.

[28] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, "Overfeat: Integrated recognition, localization and detection using convolutional networks," arXiv preprint arXiv:1312.6229, 2013.

[29] T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," arXiv preprint arXiv:1301.3781, 2013.

[30] J. Pennington, R. Socher, and C. Manning, "Glove: Global vectors for word representation," in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532-1543.

[31] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, "Enriching word vectors with subword information,"

[32] S. Sukhbaatar, J. Weston, and R. Fergus, "End-to-end memory networks," in Advances in neural information processing systems, 2015, pp. 2440-2448.

[33] A. Kumar et al., "Ask me anything: Dynamic memory networks for natural language processing," in International conference on machine learning, 2016, pp. 1378-1387.

[34] A. Maas, R. E. Daly, P. T. P. am, D. Huang, A. Y. Ng, and C. Potts, "Learning Word Vectors for Sentiment Analysis," Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, 2011.

[35] B. Pang and L. Lee, "Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales," in Proceedings of the 43rd annual meeting on association for computational linguistics, 2005: Association for Computational Linguistics, pp. 115-124.

[36] C. Du and L. Huang, "Sentiment Classification Via Recurrent Convolutional Neural Networks," DEStech Transactions on Computer Science and Engineering, No. cii, 2017.

[37] F. Kokkinos and A. Potamianos, "Structural attention neural networks for improved sentiment analysis," arXiv preprint arXiv:1701.01811, 2017.

Journal of AI and Data Mining

Convolutional Neural Network Equipped with Attention Mechanism and Transfer Learning for Enhancing Performance of Sentiment Analysis

References

References

Volume 9, Issue 2
April 2021
Pages 141-151

Convolutional Neural Network Equipped with Attention Mechanism and Transfer Learning for Enhancing Performance of Sentiment Analysis

References

References

Volume 9, Issue 2April 2021Pages 141-151

Volume 9, Issue 2
April 2021
Pages 141-151