We first summarise the datasets and prompt templates, models, and metrics used to probe FM emergence, then report per-modality findings. Two of the main characteristics of the FMs are that i) they are trained on a broad range of data, so that the resulting models can be utilised in a wide range of problems, and ii) they exploit large amounts of learning parameters. Given sufficient learning material, from a certain number of such parameters hence well trained, knowledge ‘emerges’ in the FMs, and they achieve competitive performances in tasks they have not been specifically trained for. This can, however, be difficult to predict17. We aim to investigate the ‘emergent’ affective capabilities of FMs. Focusing on the vision (cf. Section 2.1), linguistics (cf. Section 2.2), and speech (acoustics) (cf. Section 2.3) modalities, we assess the capabilities of current FMs to i) generate synthetic affective samples, from which we infer the conveyed emotions with pre-trained emotion recognition classifiers (note that in principle, this can lead to a ‘closed loop’, as we cannot be sure whether the data used in the pre-trained emotion classifiers has not also been used in the training of the FMs, but generally, it is unlikely, as high-quality affective data are rarely freely available on the Internet due to their privacy restrictions), and ii) analyse well-established datasets in the field in a zero-shot manner. To favour the comparability among the different modalities investigated, we focus on the ‘Big Six’ Ekman emotions18 (i.e., fear, anger, happiness, sadness, disgust, and surprise), in addition to the neutral state.
The research gap this work aims to fill is the assessment of the affective capabilities of current FMs, including a first impression of to which extent these represent a breakthrough in the field of Affective Computing. Since affect plays a pivotal role in human-related interactions, it is of paramount importance for AI systems—and for any system interacting with humans—to be able to understand and properly respond to the emotional state of the user. Understanding how FMs react to human emotions and how well they conform to or deviate from emotion theory is a key step towards the integration of this technology at the core of AI-powered agents. The scope of our analysis is constrained by the fundamental nature of our emotion model, which is based on Ekman’s theory of discrete emotions. However, as we have discussed, this approach allows us to unify the evaluation of the three modalities addressed here. Before exploring complex models—such as the characterisation of emotions in the dimensional (i.e., valence and arousal) space—, it is important to start the analysis of the capabilities of FMs with a model that, despite limited, is extremely useful in a wide range of tasks and applications. Moreover, the use of discrete emotions could help us gain more insights into the model performances and associated weaknesses or biases that may have emerged from their training.
The vision modality has changed
We proceed to discuss the generation capabilities of current vision-based FMs. In the visual domain, data synthesis started to obtain pseudorealistic results in the last decade thanks to the Generative Adversarial Network- (GAN) based models1,6. Nowadays, a boost in the quality of the synthesised images has been achieved via text prompt inputs-based models, due to i) the CLIP model19 and ii) the Diffuser model20. The former was presented as a model to predict how well a given caption describes an image. The latter learns to reconstruct images by removing an added Gaussian noise through a Markov Chain. During inference, the model generates new images from Gaussian noise, being more efficient than other generative architectures. Models such as Stable Diffusion (SD)21 or DALL-E 222 integrate the CLIP and the Diffuser models to efficiently synthesise images with high semantic control.
We have leveraged one of the latest versions of SD—i.e., Stable Diffusion XL (SDXL)16—to synthetically generate a face emotion dataset. This dataset is generated utilising predefined prompts with three sources of variation: i) the emotion, ii) the style (photorealistic, cartoon-painting, anime, and 3D), and iii) the demographic group. The template, along with the values explored for each attribute, are detailed in Table 1. Table 2 presents a summary of the gathered dataset. Although our emotion model is based on the ‘Big Six’ Ekman emotions18 plus the neutral state, we employed these basic emotions together with a higher intensity variation—as defined in Plutchik’s model23—to further emphasise the desired affective states. The generation process spans 18 different demographic groups; determined by age, biological sex, and skin tone. Visual examples for each emotion and style are provided in Fig. 1 for the demographic group <young, woman, white>. For the case of the photorealistic style, we played with the background to generate some samples in realistic scenarios (e.g., outdoors, office, park). The generation process involved two experts, the SDXL base model and the SDXL refiner, with a total of 40 steps (80% in the base model, 20% in the refiner) and a guidance value of 7.5. Together with the desired prompt, we utilised a negative prompt to highlight what we did not want to see in the output. Once the images were generated, one annotator curated the data according to four principles: the presence of dissatisfactions or artifacts, nudity, prompt compliance and plausibility of the style.
Synthetic facial images of a white-skin, young woman conveying the ‘Big Six’ Ekman emotions18, in addition to the neutral state. All the images were generated with Stable Diffusion XL16, conditioned on four different styles, namely photorealistic (first row), cartoon-painting (second row), anime (third row), and 3D (fourth row).
We now aim to automatically verify the affective quality of the generated facial images with Face Emotion Recognition (FER) models, which we train employing the manually annotated subset of the AffectNet dataset11. The images belonging to this dataset are annotated in terms of eleven emotions. Nevertheless, we select the images corresponding to the ‘Big Six’ Ekman emotions in addition to the neutral class. We process the selected images with OpenFace24 to extract features related to a subset of the Action Units (AU) defined in the Facial Action Coding System (FACS)25. Specifically, OpenFace extracts 35 features per facial image, indicating the presence (0 or 1) and the intensity (in a scale from 0 – not present – to 5 – present with maximum intensity) of a subset of the AUs. We discard the images that OpenFace fails to process; for instance, due to the absence of a face in the image. Table 3 summarises the resulting data.
We start our preliminary investigation by training FER models with Support Vector Classifiers (SVC), as these are considered a standard machine learning technique with excellent results in a wide range of problems. We compare their performance when utilising a linear and a Radial Basis Function (RBF) kernel. To overcome the imbalanced training samples (cf. Table 3), we weigh the training data, so that the samples corresponding to the least represented classes have more importance when training the models. We fine-tune our models optimising the regularisation parameter C ∈ [10−2, 10−1, 1, 10, 102]. The performance of the optimal models on the validation partition is depicted in Table 4.
We also contrast the performance of the SVC-based FER models with a state-of-the-art Vision Transformer26 for Facial Expression Recognition (ViT–FER), trained on the Facial Expression Recognition 2013 (FER-2013) dataset27. We use the pre-trained model off-the-shelf—without fine-tuning—and evaluate its performance on the validation partition of our subset of AffectNet. The obtained results are reported in Table 4.
The best performance on the validation partition (cf. Table 4) is obtained with the ViT–FER model. We use this model to assess the affective quality of the generated facial images. The results obtained exemplify the breakthrough of working with end-to-end approaches, operating on the raw images instead of on the features extracted from them. Nevertheless, it is worth mentioning that AffectNet was collected in the wild, which may complicate the estimation of the AUs from the facial images and, in turn, worsen the performance of the SVC-based models.
Table 5 summarises the results obtained when analysing the generated facial images with the four different styles utilising the ViT–FER pre-trained model. Figure 2 presents the confusion matrices computed, comparing the model inferences and the ground truth. Across styles, the worst results are those of the photorealistic style, as denoted by both accuracy—Weighted Average Recall (WAR)—and Unweighted Average Recall (UAR). UAR is the sum of recall per class divided by the number of classes—this reflects imbalances and is a standard measure in the field. In contrast, the best results are obtained with the 3D style. Interestingly, in all the 4 styles both the neutral and the happiness emotions consistently obtain the best results. These classes are traditionally over-represented in face datasets (e.g., see the training set distribution of AffectNet in Table 3), probably due to the nature of the data sources28. Thus, it is expected for the generative models to be biased towards these emotions, making their recognition easier. As can be observed from Fig. 2, the other emotions obtain low recognition rates, not even reaching a 50% rate in most of the cases. An interesting, recurrent mistake across all 4 styles is the models tendency towards predicting the neutral class. This result aligns with the bias associated with the imbalanced emotional datasets. The limited amount of samples from some concrete emotions available for training the generative models makes the generation of samples representative of these classes harder. By “harder” we do not only refer to the correct generation according to the prompted emotion, but also to convincingly convey the intensity of the desired emotion.
a–d Correspond to the photorealistic, cartoon-painting, anime, and 3D styles, respectively.
We proceed to discuss the analysis capabilities of current vision-based FMs. To evaluate the affective analytical capabilities of FMs in the image domain, we explore their performance in a zero-shot emotion recognition task under different configurations. Our experiments are conducted on the considered validation set of AffectNet11, as it is balanced, in-the-wild, and manually annotated. We compare three different approaches relying on model-prompting. In the first two, we provide the presence of a subset of the AUs and their intensities in a textual format as input to a FM. The third approach consists in directly feeding the images within the prompt of a FM. The first two approaches can be addressed with a Language Foundation Model (LFM), for which we select the LLaMA2 7B model. We utilise a Multimodal Foundation Model (MFM) for the third scenario; specifically, the LLaVA1.5 7B model29,30. The selected models have the same number of parameters.
In Table 7, we present the results of the aforementioned scenarios. The prompts utilised are detailed in Table 6. We also include in Table 7 the results of the ViT–FER model for comparability purposes. We obtain the best results when feeding the prompts with the raw images. Both AU-based prompt approaches exhibit accuracy results close to chance. The LLaVA model achieves an accuracy only 4 points below the ViT–FER model, which was explicitly trained on the FER-2013 dataset to recognise emotions. This is an interesting result, which suggests that the LLaVA1.5 model presents emergent affective capabilities, despite not being specifically trained on affective computing tasks.
The linguistic modality has changed
We proceed to discuss the generation capabilities of current language-based FMs. The introduction of the Transformer model31 has revolutionised the Natural Language Processing (NLP) field and marked the beginning of the Large Language Models (LLM) era. OpenAI’s work with the Generative Pre-trained Transformer (GPT) models13,32—culminating with the recent GPT-433—is the greatest exponent of this revolution, advancing text generation. From an Affective Computing perspective, LLMs present a novel approach to inject and transfer emotional content in linguistic data, with recent works demonstrating their intrinsic emotional capabilities in a variety of domains34,35,36,37,38.
We investigate and leverage the affective style transfer capabilities of cutting-edge open-source LLMs. We select LLaMA239 and Mistral40—comprising 7 billion parameters each—, given their proven high performance in both language understanding and text generation tasks. Additionally, we include the Mixtral LLM41, which utilises the Sparse Mixture of Experts (SMoE) method42. We first compile a corpus comprised of 122 human-curated neutral phrases. The dataset encompasses various topics, from mundane personal activities to formal professional interactions. Utilising the gathered corpus, we task the aforementioned LLMs (i.e., LLaMA2, Mistral, and Mixtral) to generate six emotional phrases from each original neutral phrase, conveying one of the ‘Big Six’ Ekman emotions. We generate three synthetic sentences for each combination of text and emotion, yielding a corpus of 2194 emotional phrases (the 2 missing sentences were discarded during preprocessing due to a corrupted generation). The generation parameters used a temperature of 0.9, top-p of 0.6, and a repetition penalty of 1.2. An example of the generated sentences is shown in Table 8.
To investigate the quality of the generated sentences, we implement two baseline models trained on the GoEmotions dataset43, a well-renowned corpus in the field and commonly utilised for benchmarking purposes due to its comprehensive labelling and categorisation of the emotions. The GoEmotions dataset consists of English Reddit comments annotated according to 27 distinct emotions, plus the neutral state, by 3 or 5 labellers each. Due to the nature of the annotations in the GoEmotions dataset, we begin by tailoring the data to meet the specific requirements of our experiments. First, we select instances from the dataset annotated with a single emotion, to tackle the task as a single-label classification problem, instead of a multi-label classification one. As previously, we adopt the ‘Big Six’ Ekman emotions18, in addition to a seventh neutral state. This restructuring of the GoEmotions taxonomy to the Ekman taxonomy is achieved by aggregating the original labels into the targeted, broader categories43—i.e., emotions like annoyance and irritation, originally distinct, were grouped under ‘anger’ to fit the Ekman model. Table 9 summarises the statistics of the considered subset of the GoEmotions dataset.
We employ two different architectures as baselines: a Bi-directional Long Short-Term Memory (BiLSTM) network and a fine-tuned version of the RoBERTa-base model44. Both are trained on the selected subset of the GoEmotions dataset. The BiLSTM consists of two bidirectional Long Short-Term Memory (LSTM) layers with 128 units each. It is trained with a learning rate of 5 × 10−3 and a batch size of 96 for 40 epochs, while RoBERTa-base was fine-tuned at a conservative learning rate of 5 × 10−5 and a smaller batch size of 12. The models are trained on the training partition of the dataset, and the weights yielding the highest validation UAR are selected for each model. The test scores for both baseline models (BiLSTM and RoBERTa) on the test partition of the GoEmotions dataset are shown in Table 10. Given the superior performance of RoBERTa, we utilise it for analysing the synthetically generated emotional sentences.
To evaluate the performance of the various LLMs on the emotion injection task, we test the generated sentences with the RoBERTa baseline model. We also explore GPT-4 as an approximation for human evaluation, and its weaker variant GPT-3.5. Table 11 demonstrates the prompt templates used for the GPT models, inspired by previous works37,45. The versions of GPT variants used are ‘gpt-3.5-turbo-0125’ for GPT-3.5 and ‘gpt-4-turbo-2024-04-09’ for GPT-4. The results of this evaluation are depicted in Fig. 3. These results reflect a better agreement between models than with the ground truth labels, which are not human-annotated. However, the results of GPT-4 should be the closest to the human evaluations37,46.
a–c Correspond to LLaMA2-based, Mistral-based, and Mixtral-based synthesis, respectively.
The GPT-4 model achieves high UAR scores on all six emotions. Its inferior model GPT-3.5 achieves slightly worse results in most cases, but it experiences a performance drop in the recognition of surprise. In comparison, RoBERTa has a different behaviour. It is generally much worse than GPT-4 and GPT-3.5 in most of the cases, but it obtains a higher score than GPT-3.5 for the surprise emotion with the LLaMA2- and the Mixtral-generated sentences. Additionally, RoBERTa is showing very low performance for disgust.
Figure 4 depicts the confusion matrices obtained with the RoBERTa baseline model on the LLaMA2-, the Mistral-, and the Mixtral-generated sets. We also include its performance on the test set of GoEmotions as a reference. A common issue with the RoBERTa model is the confusion between anger and disgust. Analysing the confusion matrices, we also observe an interesting effect: most of the model’s mispredictions are assigned to the neutral class.
a–d Correspond to the LLaMA2-based, Mistral-based, Mixtral-based synthetic benchmarks and the GoEmotions test benchmark, respectively.
Further analysing the results presented in Fig. 4, we observe that three systematic error patterns emerge. First, a noticeable “neutral-fallback” effect: although the synthetic test sets do not contain neutral ground-truth sentences, RoBERTa assigns the neutral label to a sizeable share of inputs with negative valence—e.g., with LLaMA2, Mistral and Mixtral a 15.6%, 26.2%, and 43.4% of the fear instances are misclassified into the neutral class, respectively; a 36.9%, 38.5%, and 18.0% of the sadness instances are misclassified into the neutral class, respectively; and a 20.5%, 53.3%, and 12.3% of the disgust instances are misclassified into the neutral class, respectively. Second, a one-way anger-disgust drift: more than half of the disgust instances generated by LLaMA2 are misclassified into the anger class (53.3%), with consistent though smaller shifts for Mistral (23.8%) and Mixtral (29.5%). The reverse confusion remains below 3% in all the three scenarios. Third, a robust performance of positive emotions: happiness is recognised with ≥80% recall across all synthetic sets, and the recognition of surprise leads to limited confusion—achieving recall rates of 86.1%, 56.6%, and 68.0%, respectively. Overall, the confusion matrices indicate that, while prompt-based style transfer still under-specifies nuanced negative emotions, it embeds strong, distinctive cues for high-arousal emotions.
We proceed to discuss the analysis capabilities of current language-based FMs. We analyse the zero-shot sentiment analysis capabilities of the following LLMs: Mistral, Mixtral, and two versions of LLaMA2 (7 billion and 13 billion parameters). We assess their zero-shot capabilities on the test partition of the GoEmotions dataset. We design a prompt that requires the selected LLMs to predict the corresponding emotion, including the neutral state (cf. Table 11). To minimise randomness and increase confidence in the predictions, we reduce the temperature setting to 0.1. This lower temperature sharpens the probability distribution, ensuring that the predicted classes reflect those that the LLMs are predicting with the highest probabilities45. However, the LLM outputs sometimes include irrelevant or multiple emotions. To address this without intrusively altering the model’s outputs, we select the first listed emotion as the most reliable prediction.
Table 12 summarises the comparative performance of the tested LLMs. We include the performance of the RoBERTa baseline model trained on the GoEmotions dataset for benchmarking purposes. The first observation is that the UAR scores obtained by all the investigated LLMs surpass the chance level (14.3%), underscoring the emergent affective capabilities of the LLMs tested in a zero-shot manner. Nevertheless, none of the LLMs outperforms the RoBERTa baseline model, fine-tuned on the GoEmotions dataset, which indicates that model-specific tuning is advantageous. Nevertheless, it is worth highlighting that the difference in the UAR scores obtained by the best-performing GPT-3.5 and GPT-4 models in comparison to the RoBERTa baseline model is around 15%. This is an interesting result, as these models have not been trained on the GoEmotions dataset, but still obtained a competitive performance on the emotion recognition task, reinforcing, one more time, the emergent affective capabilities of the models.
The speech modality has (not yet) changed
We proceed to discuss the generation capabilities of current speech-based FMs. Research on generating affective speech has been conducted for more than three decades47. The first approaches were rule-based, while contemporary methods typically rely on DL. All of these methods are explicitly engineered to produce emotional speech. Thus, this line of research can be dubbed as a subfield of ‘traditional’ Affective Computing, while technically belonging to the Text-To-Speech (TTS) field. In recent years, similar to the developments in NLP and Computer Vision (CV), research on TTS has heavily been influenced by the success of the FM paradigm48,49.
UniAudio48 is a Transformer-based general-purpose audio synthesis system, pretrained on seven generative audio tasks including TTS. In its pretrained state, however, it does not support affective speech synthesis. The authors demonstrate that their pretrained model serves as a basis for adaptation to different downstream tasks which opens up the possibility to equip the model with affective speech synthesis capabilities by mere fine-tuning. Another recent generative audio FM, PromptTTS249, synthesises speech based on text prompts that include descriptions of the voice to be generated. The controllable attributes of the synthesised speech must be defined during the training time. The authors of ref. 49 do not investigate emotion as one such attribute, though their proposed framework would permit this. To the best of our knowledge, no evidence for affective speech synthesis as an emergent capability of FMs has been published so far. At the moment, although the demos for GPT-4o claim affective speech synthesis capabilities, neither a technical report on GPT-4o, nor a systematic evaluation of affective speech produced by GPT-4o is available.
Considering the development towards releasing large pretrained models in the NLP and CV areas, we assume that in the near future powerful speech synthesis models will also be made publicly available and, consequently, investigated more thoroughly, allowing us to carry out according experiments to the above for vision and linguistics. In addition, we expect a continuing trend toward truly multimodal FMs that may not just take in but also produce natural speech with controllable prosodic properties. Several multimodal models that also produce audio output data have been proposed50,51,52. To the best of our knowledge, none of them exhibit emotional speech synthesis capabilities.
We proceed to discuss the analysis capabilities of current speech-based FMs. A system capable of analysing arbitrary affective properties of speech data without any tuning must ingest both audio and text inputs. Several FM approaches fulfilling this requirement have been proposed. However, the vast majority of them are not pretrained on speech data at all.
Examples include AnyMAL53, X-InstructBLIP54, and ModaVerse55. In only a few models, speech is part of the pretraining data. QWEN-Audio’s training data comprises several labelled speech datasets, including emotionality already. Hence, QWEN-Audio56 in the proposed form is not a candidate for exploring ‘emergent’ affective recognition capabilities. X-LLM57 processes video, text, and audio inputs and is explicitly designed to process speech. The authors, however, do not report any experiments related to predicting affect in speech. As of now, the pretrained X-LLM model is not publicly available, hence, unfortunately again, not allowing us to carry out experiments analogous to the vision and the linguistic ones.
Similar to the affective speech synthesis problem, near-future multimodal FMs can be expected to be capable of analysing affective speech in a zero-shot manner, even if not explicitly pretrained for this task. As of now, however, we are not aware of any such model.
The evaluation is changing
One of the reasons to understand the impressive performance of currently available FMs is that they use massive amounts of data from “the Internet” for training. Nevertheless, the indiscriminate use of data poses the following challenge to the scientific community: can we guarantee that the data we feed to these models for their evaluation has not been used for training? In case of a negative answer, how fair and representative of the model capabilities can standard evaluation metrics be? Although we do not have a concrete answer yet, we hope these challenges engage the research community into looking for methods and metrics that allow a proper scientific evaluation of these emerging FMs in the field of Affective Computing. As is, the current state of such models in Affective Computing may resemble a shell game: many different tools and approaches are shuffled and mixed until some partially less, partially more convincing performances are obtained. Especially because it is the popular ‘Big Six’ Ekman emotions we considered herein, chances are high that the models only reverberate with what they have already seen. Testing on more subtle models such as the dimensional approach or less considered affective states is therefore urgently needed.
