H.3. Artificial Intelligence
Rasoul Hosseinzadeh; Mahdi Sadeghzadeh
Abstract
The attention mechanisms have significantly advanced the field of machine learning and deep learning across various domains, including natural language processing, computer vision, and multimodal systems. This paper presents a comprehensive survey of attention mechanisms in Transformer architectures, ...
Read More
The attention mechanisms have significantly advanced the field of machine learning and deep learning across various domains, including natural language processing, computer vision, and multimodal systems. This paper presents a comprehensive survey of attention mechanisms in Transformer architectures, emphasizing their evolution, design variants, and domain-specific applications in NLP, computer vision, and multimodal learning. We categorize attention types by their goals like efficiency, scalability, and interpretability, and provide a comparative analysis of their strengths, limitations, and suitable use cases. This survey also addresses the lack of visual intuitions, offering a clearer taxonomy and discussion of hybrid approaches, such as sparse-hierarchical combinations. In addition to foundational mechanisms, we highlight hybrid approaches, theoretical underpinnings, and practical trade-offs. The paper identifies current challenges in computation, robustness, and transparency, offering a structured classification and proposing future directions. By comparing state-of-the-art techniques, this survey aims to guide researchers in selecting and designing attention mechanisms best suited for specific AI applications, ultimately fostering the development of more efficient, interpretable, and adaptable Transformer-based models.
A. Lakizadeh; E. Moradizadeh
Abstract
Text sentiment classification in aspect level is one of the hottest research topics in the field of natural language processing. The purpose of the aspect-level sentiment analysis is to determine the polarity of the text according to a particular aspect. Recently, various methods have been developed ...
Read More
Text sentiment classification in aspect level is one of the hottest research topics in the field of natural language processing. The purpose of the aspect-level sentiment analysis is to determine the polarity of the text according to a particular aspect. Recently, various methods have been developed to determine sentiment polarity of the text at the aspect level, however, these studies have not yet been able to model well complementary effects of the context and aspect in the polarization detection process. Here, we present ACTSC, a method for determining the sentiment polarity of the text based on separate embedding of aspects and context. In the first step, ACTSC deals with separate modelling of the aspects and context to extract new representation vectors. Next, by combining generative representations of aspect and context, it determines the corresponding polarity to each particular aspect using a short-term memory network and a self-attention mechanism. Experimental results in the SemEval2014 dataset in both restaurant and laptop categories show that ACTSC has been able to improve the accuracy of aspect-based sentiment classification compared to the latest proposed methods.
H. Sadr; Mir M. Pedram; M. Teshnehlab
Abstract
With the rapid development of textual information on the web, sentiment analysis is changing to an essential analytic tool rather than an academic endeavor and numerous studies have been carried out in recent years to address this issue. By the emergence of deep learning, deep neural networks have attracted ...
Read More
With the rapid development of textual information on the web, sentiment analysis is changing to an essential analytic tool rather than an academic endeavor and numerous studies have been carried out in recent years to address this issue. By the emergence of deep learning, deep neural networks have attracted a lot of attention and become mainstream in this field. Despite the remarkable success of deep learning models for sentiment analysis of text, they are in the early steps of development and their potential is yet to be fully explored. Convolutional neural network is one of the deep learning methods that has been surpassed for sentiment analysis but is confronted with some limitations. Firstly, convolutional neural network requires a large number of training data. Secondly, it assumes that all words in a sentence have an equal contribution to the polarity of a sentence. To fill these lacunas, a convolutional neural network equipped with the attention mechanism is proposed in this paper which not only takes advantage of the attention mechanism but also utilizes transfer learning to boost the performance of sentiment analysis. According to the empirical results, our proposed model achieved comparable or even better classification accuracy than the state-of-the-art methods.
