Comparative evaluation of domain-specific and general-purpose transformer models for Arabic poet classification

Transformer models vs. recurrent baselines

The experimental findings demonstrate a consistent advantage of transformer-based architectures over recurrent neural baselines for Arabic poet classification. While both LSTM and BiLSTM models are capable of modelling sequential dependencies, their performance remains considerably lower than that of transformer-based systems across both datasets (Tables 3 and 5). This result suggests that self-attention mechanisms provide a more effective framework for capturing stylistic signals in Arabic poetry.

Arabic poetic style is rarely expressed through isolated lexical items. Instead, it emerges through complex interactions between distant words, repeated rhetorical constructions, syntactic parallelism, and historically grounded poetic conventions. Transformer architectures are particularly well suited to modelling such characteristics because the self-attention mechanism enables the model to relate distant contextual cues within the same poetic line.

By contrast, recurrent models process sequences incrementally, where each step depends on the previous hidden state. This sequential processing may limit their ability to capture stylistic dependencies distributed across longer spans of text. As a result, transformer models appear better suited to modelling the stylistic structure of Arabic poetry, where authorial signals often emerge through long-range contextual relationships rather than local lexical patterns.

These findings are broadly consistent with prior research in Arabic NLP, where transformer-based architectures have demonstrated strong performance across a range of natural language understanding tasks. Previous studies on Arabic poetry analysis and meter detection have similarly reported promising results when applying transformer models, highlighting their ability to capture contextual and stylistic patterns in poetic language^10,18,24,25.

Interpretation of AraPoemBERT performance

Among all evaluated systems, AraPoemBERT achieved the strongest performance across both datasets. On the FrequentPoets dataset, the model obtained approximately 73% test accuracy, while performance increased to around 77–78% on the CrossEraPoets dataset.

These results indicate that AraPoemBERT consistently outperforms the other evaluated models across both datasets. A plausible explanation for this improvement lies in the domain-adapted pretraining of AraPoemBERT on Arabic poetic corpora. Exposure to large collections of poetic texts during pretraining may allow the model to learn representations that are better aligned with the linguistic and stylistic characteristics of Arabic poetry.

Such domain adaptation can provide the model with a stronger contextual understanding of poetic language, which may support more accurate distinctions between different authors. This advantage is particularly relevant in literary classification tasks, where authorial differences are often subtle and distributed across multiple linguistic signals.

Overall, the results suggest that domain-specific pretraining can play an important role in improving model performance in specialised literary analysis tasks.

Poet-level misclassification patterns

Further insight into model behaviour can be obtained by analysing the confusion matrices presented in Figs. 2 and 3. In both datasets, the strong diagonal dominance indicates that the model successfully captures author-specific stylistic signals. Nevertheless, several poets exhibit partial confusion with particular counterparts. For example, in the FrequentPoets dataset, verses attributed to Ibn al-Rumi are occasionally classified as those of Mihyar al-Daylami or Khalil Mutran, suggesting certain stylistic similarities among these poets. A comparable pattern is observed in the CrossEraPoets dataset, where some verses by Abu al-Ala al-Ma’arri are occasionally confused with those of Ibn al-Rumi or Khalil Mutran. Such misclassifications may reflect genuine stylistic proximity between poets rather than purely algorithmic limitations, a phenomenon commonly observed in authorship attribution studies⁵.

To further examine stylistic variation across poets, a lexical diversity analysis was conducted using the type–token ratio (TTR), which measures the proportion of unique words relative to the total number of tokens⁴⁴. The results are summarised in Tables 7 and 8. In the FrequentPoets dataset (Table 7), Mihyar al-Daylami exhibits the highest lexical diversity, followed by Ibn al-Rumi and Khalil Mutran, whereas Ahmad al-Safi al-Najafi shows the lowest lexical diversity. Similarly, in the CrossEraPoets dataset (Table 8), Khalil Mutran and Abu al-Ala al-Ma’arri display relatively high lexical diversity, while Abd al-Ghani al-Nabulsi demonstrates noticeably lower lexical variation.

Interestingly, when these observations are considered alongside the confusion matrices, a clear tendency becomes observable in the model predictions. Poets with comparatively lower lexical diversity, such as Abd al-Ghani al-Nabulsi and Ahmad al-Safi al-Najafi, tend to exhibit clearer classification behaviour. A possible explanation is that lower lexical diversity may correspond to more consistent lexical or stylistic usage patterns within a poet’s works, which can provide stronger and more stable cues for the model during authorship classification. Previous research in stylometry suggests that consistent stylistic markers can facilitate author identification, as models can more easily capture recurring linguistic patterns associated with a particular writer⁴⁵.

By contrast, poets with higher lexical diversity may employ a broader range of vocabulary and stylistic constructions, potentially leading to more distributed signals within the representation space and slightly higher levels of classification confusion. These findings suggest that lexical diversity may influence the separability of poetic styles in representation space, although it is unlikely to be the sole factor affecting classification performance. Rather, the results indicate that lexical consistency may contribute to clearer stylistic signals that transformer-based models such as AraPoemBERT can exploit during poetic authorship attribution.

Taken together, these observations suggest that lexical diversity alone does not fully explain the model’s classification behaviour. Instead, the results indicate that AraPoemBERT relies on multiple complementary signals when distinguishing between poets, including lexical diversity, recurrent stylistic patterns, token-level importance signals, and contextual representations learned through transformer-based encoding. The interaction of these factors appears to contribute to the separability of poetic styles observed in the confusion matrices and representation-space visualisations.

Token-level stylistic signals

Beyond lexical diversity, analysing the most influential tokens identified by the model provides deeper insight into how stylistic signals are captured during poet classification. To investigate this aspect, we examined the token-importance scores generated by AraPoemBERT and extracted the most influential tokens associated with each poet across both datasets.

Figure 6 presents representative examples of influential tokens identified by AraPoemBERT for two poets, illustrating how the model captures stylistic lexical signals related to rhetorical language and descriptive imagery. The importance values indicate the contribution of each token to the model’s prediction.

In the FrequentPoets dataset, distinctive lexical patterns emerge that reflect the thematic and stylistic tendencies of individual poets. For Ahmad al-Safi al-Najafi, influential tokens include terms related to poetic language and rhetorical expression, such as “isti’ara” (metaphor), “lafz” (word or expression), and “wazn” (poetic meter). These tokens are closely connected to the discourse of poetic composition and literary criticism, suggesting that Najafi’s poetry frequently engages with metapoetic reflection on language, style, and artistic expression. The prominence of such tokens indicates that the model captures stylistic cues related not only to thematic content but also to explicit references to poetic structure and rhetorical technique.

A different lexical profile appears in the poetry of Khalil Mutran. Influential tokens extracted from Mutran’s verses include terms associated with geographical imagery and descriptive narration, such as references to the Nile and directional expressions such as the south. These lexical items reflect the descriptive and narrative character of Mutran’s poetic style, where landscape imagery and spatial references frequently appear as central motifs. The presence of such tokens suggests that the model captures stylistic signals related to vivid visual description and environmental imagery.

In addition to these lexical cues, several influential tokens correspond to subword units generated through the WordPiece tokenisation used in transformer architectures. This observation indicates that stylistic information may also be encoded through morphological patterns embedded within Arabic words. Given the rich morphological structure of Arabic, where stylistic nuance often emerges through derivational forms and affixes, such subword representations allow the model to capture subtle stylistic distinctions between poets.

In addition, the token interaction heatmaps (Fig. 7) further illustrate how influential tokens co-occur within poetic contexts, providing complementary evidence to the token-level importance analysis. For example, in the case of Ahmad al-Safi al-Najafi, tokens related to rhetorical language such as “isti’ara”, “lafz”, and “wazn” exhibit strong interaction patterns, indicating that they frequently appear together and collectively contribute to the model’s prediction. Similarly, for Khalil Mutran, tokens associated with geographical imagery, such as references to the Nile and directional expressions, form coherent interaction clusters reflecting descriptive stylistic patterns. These interaction patterns indicate that the model captures not only individual lexical signals but also relationships between stylistic elements across tokens.

Taken together, these findings demonstrate that AraPoemBERT captures stylistic variation across multiple linguistic levels, including lexical choice, thematic preference, and morphological structure. Importantly, this analysis provides direct interpretability evidence that the model does not rely solely on superficial lexical frequency patterns, but instead learns meaningful stylistic representations that reflect both semantic content and rhetorical structure. This behaviour helps explain the model’s improved performance, as it can distinguish poets based on deeper stylistic patterns rather than isolated keywords.

Representation analysis of hidden features

To further investigate how stylistic information is encoded internally, we analysed the hidden representations generated by AraPoemBERT and projected them into a two-dimensional space using Principal Component Analysis (PCA). The resulting projections are illustrated in Figs. 4 and 5.

The PCA projections reveal partially separable clusters corresponding to different poets, suggesting that the model organises verses within a continuous stylistic representation space in which stylistically similar poets tend to occupy neighbouring regions. In several areas of the projection, distinct author-specific groupings emerge, indicating that the model captures meaningful stylistic signals associated with particular poets.

However, the projections also reveal areas of partial overlap between clusters. These overlaps correspond closely to the misclassification patterns observed in the confusion matrices, suggesting that poets whose verses occupy nearby regions in the representation space are also more likely to be confused during classification. This correspondence suggests that the internal representation structure learned by AraPoemBERT may reflect underlying stylistic relationships between poets rather than arbitrary separations.

Interestingly, Ibn al-Rumi and Al-Ma’arri appear relatively close in the representation space. Although they belong to different historical periods, they both emerge from closely related phases of the classical Arabic literary tradition. This temporal and cultural proximity may partly explain stylistic similarities in vocabulary usage and rhetorical constructions captured by the learned representations of the model.

Role of temporal diversity

Another important observation concerns the influence of temporal diversity in the dataset. Models consistently achieved higher accuracy on the CrossEraPoets dataset compared with the FrequentPoets dataset (see Tables 3 and 5). This pattern suggests that greater temporal diversity between poets may contribute to improved stylistic separability and facilitate more reliable classification.

When poets originate from clearly distinct historical contexts, differences in vocabulary, rhetorical style, and poetic conventions tend to become more pronounced. These distinctions provide clearer stylistic cues that allow the model to associate linguistic patterns with specific authors. Conversely, poets belonging to similar literary traditions may share broader stylistic conventions and overlapping lexical choices, which can make computational discrimination more challenging.

The observed performance differences therefore suggest that temporal variation in poetic language contributes to clearer stylistic boundaries in the representation space, ultimately supporting more accurate author classification.

General-purpose LLM limitations

In contrast to the transformer models fine-tuned in this study, GPT-4o demonstrated substantially lower performance under both zero-shot and few-shot prompting settings in the present experiments. This result highlights the limitations of applying general-purpose LLMs directly to specialised literary tasks without domain-specific training.

Arabic poetry often contains complex morphological structures, dense figurative language, and historically grounded stylistic conventions⁸. that may not be strongly represented in the general training data of multilingual LLMs. Consequently, although GPT-4o exhibits strong general language understanding capabilities, it appears less able to capture the subtle stylistic cues required for reliable poet attribution.

Impact of prompting strategies

The experiments also provide insight into how prompting strategies influence the behaviour of general-purpose large language models. When comparing zero-shot and few-shot prompting configurations for GPT-4o, a clear difference emerges between the two datasets. On the FrequentPoets dataset, the few-shot setup produced only a modest improvement over the zero-shot configuration. This limited gain suggests that providing a small number of labelled examples may not be sufficient when the poets share similar stylistic conventions and vocabulary.

By contrast, the improvement was substantially larger on the CrossEraPoets dataset. Because this dataset includes poets from different historical periods, stylistic distinctions between authors are more pronounced. In such cases, the labelled examples included in the prompt appear to provide stronger cues that help the model associate linguistic patterns with particular poets.

These findings highlight an important limitation of prompt-based inference for literary analysis tasks. Although few-shot prompting can partially improve performance, it does not replace the benefits of domain-specific training. Models that are explicitly pretrained or fine-tuned on poetic corpora remain better suited to capturing the subtle stylistic patterns required for reliable poet attribution.

Limitations

Despite the promising outcomes reported in this study, several limitations should be acknowledged. First, the experiments were conducted on datasets containing a limited number of poets. Although the selected poets represent both prolific authors and cross-era stylistic variation, the datasets may not fully capture the broader diversity of Arabic poetic traditions.

Second, the linguistic scope of the datasets is primarily restricted to classical and Modern Standard Arabic poetry. Consequently, the findings may not directly generalise to other poetic forms such as dialectal poetry, contemporary free verse, or hybrid literary styles.

Third, GPT-4o was evaluated using prompt-based inference with both zero-shot and few-shot prompting strategies. The model was used in its out-of-the-box configuration without any task-specific training on poetic corpora.

Fourth, although the experiments demonstrate strong performance for domain-adapted transformer models, the results may still depend on the characteristics of the pretraining corpus, including its stylistic and historical distribution.

Finally, the experiments were conducted on specific datasets constructed for Arabic poet classification. Additional validation across independent poetic corpora would further strengthen the generalisability of the findings.

These limitations may affect the generalisability of the results, particularly when applying the models to more diverse poetic styles, dialectal variations, or unseen literary domains.

Implications

Taken together, the findings demonstrate that domain-adapted transformer models provide a robust and effective computational framework for analysing stylistic variation in Arabic poetry. By combining comparative benchmarking with representation-level analysis, the study moves beyond a purely performance-oriented evaluation and offers insight into how stylistic and authorial signals may be encoded within modern language models.

These implications extend beyond the immediate task of poet classification. From an Arabic NLP perspective, the results highlight the importance of domain-specific pretraining when modelling linguistically rich, stylistically dense, and culturally specialised texts. More broadly, the study contributes to digital humanities research by supporting computational approaches to authorship attribution, stylistic interpretation, and the large-scale exploration of Arabic literary heritage.

The findings also suggest promising directions for future research. In particular, extending the analysis to larger and more diverse poetic corpora, incorporating contextual metadata such as historical era or poetic theme, and applying more detailed interpretability methods may further clarify how stylistic distinctions are represented within transformer-based models. Such developments could strengthen the role of computational methods not only in literary classification, but also in supporting deeper scholarly engagement with the evolution, structure, and diversity of Arabic poetic traditions.