Small Models Have Emotions, Too!

This is an ongoing experiment and if you are reading this post, please be aware it is still in the draft stage.

Anthropic recently published some work on emotions and their functions in LLMs. They find that Sonnet 4.5 develops structured internal representations of human emotions. Does this occur in smaller open models too, though? In this post we search for them in Qwen-2.5-7B, and further investigate how these emotion vectors change across languages. Specifically, we begin an investigation of how the internal geometry of a language’s emotion space compares to other languages.

You can find my datasets, prompts, code, and notebooks on Github.

This post was inspired by vogel’s Small Models Can Introspect, Too and compute was partially funded by BlueDot Impact.

Introduction

Recent interpretability research (e.g., Anthropic’s investigations into emotion circuits) has demonstrated that LLMs develop structured internal representations of human emotions. In this independent research project we attempt to reproduce these findings in smaller, accessible models (e.g., Qwen-2.5 7B).

Building on this foundation, we aim to investigate a novel objective: How do these emotion vectors relate across different languages? Specifically, we seek to determine whether the internal representation space for a set of emotions in one language is isomorphic to that of another. An additional question we propose is if LLMs learn a universal, language-agnostic “concept space” for emotions, or if these representations shift based on the cultural and linguistic nuances of the training data.

Anthropic was very thorough in making their work easy to reproduce, giving detailed information and publishing the prompts they used to create their datasets publicly.

We will first investigate a single language pair: Japanese and English. These languages provide a good ground for comparison, as they are syntactically very different, share no input tokens, and the author is able to evaluate the results of both languages. Qwen was chosen over Llama for its more robust Japanese performance.

Our plan is as follows.

1. Create dataset of contrastive pairs
   1. Write short stories on diverse topics in which a character experiences a specified emotion
   2. Write neutral dialogues about the same prompted topics, but void of emotional expressivity.
2. Extract activations
   1. Extract residual stream activations at each layer, averaged across all token positions, beginning from the 50th token.
   2. Average these activations across stories with the same emotion, and subtract mean from all emotions
3. Eliminate confounds
   1. Obtain activations from neutral stories, compute top principal components (enough to account for 50% of the variance)
   2. Project out these components from the emotion vectors

Dataset Creation

For representation engineering, we must first create a dataset of contrastive pairs. In our case, the contrastive pairs will first not be against a neutral baseline, but rather contrastive between a wide variety of different emotions. Following Anthropic’s example, we also generate emotion-neutral data points, but have not yet been able to achieve comparable results in their usage (where the authors reduced the variance of up to 50%).

We reuse Anthropic’s 100 topics and generate 12 stories per topic per emotion per language. Our prompts can be found here. To reduce costs, we only generate stories for 30 emotions with Sonnet 4.6. Opus 4.7 helped the author to find a good representative subset of emotions to use, focusing on having a balanced set across both valence and arousal.

The goal of this dataset is not to train a classifier over emotion labels, but to create enough repeated contexts where the same emotion appears across many unrelated situations. By spreading each emotion across Anthropic’s 100 topics, we try to make the average activation for an emotion less dependent on any single story setting, character type, or lexical pattern. Ideally, what remains after averaging is the model’s representation of the affective state rather than the representation of a particular topic.

It’s important to note that the Japanese and English stories are not translations of each other– the author initially (and naively) wanted to test if cultural shift in emotions was measurable, which requires independent and native-like prose.

Using independent stories rather than translations is a tradeoff. Translations would control topic and content more tightly, making cross-lingual comparison cleaner in one sense. But they would also risk measuring translation artifacts or sentence-level alignment rather than naturally occurring emotion representations in each language. Since the broader question is whether the model learns a shared abstract emotion geometry across languages, we chose independently generated native-like prose and rely on averaging across many topics to reduce story-specific noise.

This does introduce a limitation: differences between English and Japanese vectors may reflect not only language, but also generation artifacts, cultural priors from the data generator, or topic-emotion interactions. Future work should compare independent stories against translated pairs to separate these factors.

Extraction and Sanity Checks

Collecting Activations

To collect activations for our stories, we do a forward pass on each story, extracting the residual stream at each layer, taking the activations at each token after the 50th token and averaging them together.

For each story, we collect an activation by doing a forward pass and, for each layer, taking the activations of all tokens after the 50th token (where the emotional context is likely already established) and taking the mean. We do this for each story for an emotion, then average across all story points to get the raw vector for that emotion.

Below is a slightly simplified version of the code used, the complete version can be found on GitHub.

def get_mean_activations_for_texts(model, texts, batch_size=8):
	"""
	Runs texts through the model, extracts residual stream activations at each layer,
	averages across positions >= 50, and returns the mean across texts.
	"""
	n_layers = model.cfg.n_layers
	d_model = model.cfg.d_model
	
	sum_activations = torch.zeros((n_layers, d_model), device='cpu')
	names_filter = lambda name: name.endswith("resid_post")
	
	for i in tqdm(range(0, len(texts), batch_size), leave=False):
		batch_texts = texts[i:i+batch_size]
		tokens = model.to_tokens(batch_texts)
		seq_len = tokens.shape[1]
		
		with torch.no_grad():
			_, cache = model.run_with_cache(tokens, names_filter=names_filter, return_type=None)
		
		batch_size_actual = tokens.shape[0]
		
		for b in range(batch_size_actual):
			# ignoring pad tokens
			valid_len = (tokens[b] != model.tokenizer.pad_token_id).sum().item()
		  
			for l in range(n_layers):
				layer_name = f"blocks.{l}.hook_resid_post"
				# Mean from 50th token to valid_len
				act = cache[layer_name][b, 50:valid_len, :].mean(dim=0).cpu()
				sum_activations[l] += act
			
			total_valid_texts += 1
		
		del cache
		torch.cuda.empty_cache()
	
	return sum_activations / total_valid_texts

To collect our raw emotion vectors, we simply loop through the emotions:

emotion_vectors = {'en': {}, 'ja': {}}

for lang in ['en', 'ja']:
	print(f"\nExtracting activations for language: {lang}")
	for emotion in emotions_list:
		texts = texts_by_emotion[lang][emotion]
		avg_act = get_mean_activations_for_texts(model, texts, batch_size=4)
		emotion_vectors[lang][emotion] = avg_act

Centering

The collected raw emotion vectors have a lot of shared latents that we attempt to reduce (for geometrical interpretations) by centering them, meaning subtracting the mean emotion vector from each raw vector.

centered_emotion_vectors = {'en': {}, 'ja': {}}

for lang in ['en', 'ja']:
	# Shape: (num_emotions, n_layers, d_model)
	all_emotions_stack = torch.stack(list(emotion_vectors[lang].values()))
	mean_emotion_vector = all_emotions_stack.mean(dim=0)
	for emotion, vec in emotion_vectors[lang].items():
		centered_emotion_vectors[lang][emotion] = vec - mean_emotion_vector

Neutral Projection

Because all emotion probes share a set of topics, and stories will inevitably contain a lot of shared latents, we follow Anthropic in generating a set of neutral dialogues. The top PCA components are then taken from the neutral activations and projected out of the centered emotion vectors.

The neutral projection step is meant to remove broad directions that are common to the dataset but not specific to emotion. Even after centering, the vectors can still contain shared structure from the writing style, story format, topic distribution, language identity, or generic narrative features. To estimate these nuisance directions, we collect activations from neutral dialogues over the same topics, run PCA on those neutral activations, and project the leading components out of the centered emotion vectors.

Intuitively, if a direction is strongly present in neutral text, it is less likely to be specific to an emotion. Removing those directions should make the remaining vectors more affect-specific, at the cost of possibly removing some real emotion signal if it overlaps with generic narrative structure.

Like Anthropic, we find that it slightly denoises some of the results, but qualitative results still hold using the centered emotion vectors. Unlike Anthropic, who used as many PCA components as needed to explain 50% of the variance, we found 96.42% and 94.82% variance explanation with 2 components for EN and JA, respectively.

The fact that two neutral components explain such a large fraction of variance suggests that the neutral activations are dominated by a small number of broad factors. This may be useful denoising, but it also means the projection should be treated cautiously: if the neutral dataset is too narrow or stylistically uniform, the top PCs may reflect dataset artifacts rather than clean non-emotional structure.

For that reason, we treat neutral projection as a robustness/denoising step rather than the foundation of the result. The qualitative geometry remains similar with centered vectors alone, which is important because it means the main findings are not created solely by the projection procedure.

Logits

For an initial sanity check we take a look at the top/bottom logits for each emotion in each language. Here we find a lot of junk in English, especially cross-contamination with Chinese and German, but qualitatively can see the right direction for most of the emotions.

This is a weak but useful sanity check. For each emotion vector, we can project the vector through the model’s unembedding matrix and ask which output tokens it most increases or suppresses. If the vector is capturing something emotion-like, then the positive direction should upweight tokens semantically related to that emotion, while the negative direction should often upweight tokens related to the opposite affective region.

This should not be read as a clean classifier or as evidence that the model would literally output these tokens. The residual stream contains many superposed features, and the unembedding is a noisy diagnostic.

The English vectors look directionally meaningful but noisy. Several emotions recover clear affective words: terror/panic for fear, loving/love for loving, loneliness/lonely for sadness/loneliness, proudly/proud for pride, and sunlight/gentle/warm for serenity. The bottom logits also often make sense as anti-correlated directions, for example calm suppressing aggressive or sexual tokens, and positive emotions suppressing some negative or spam-like regions.

The Japanese quality is substantially worse. There is even MORE cross-lingual contamination than in English, with Chinese SEO/spam phrases that have nothing to do with the emotion. Some emotions still show recognizable signal, such as 恐怖/不安 for fear, 孤独/悲/泣 for sadness, ありがとうござ for gratitude, and 昔/年代 for nostalgia. But the amount of Chinese, SEO, adult-content, and formatting contamination is much higher. This suggests that the vectors may still contain emotion structure, but the direct logit view is a poor-quality probe for Japanese in this model.

TODO cite the university of kyoto papers on qwen’s native language being english, maybe we can investigate this further because it subjectively looks chinese to me in some ways

This is one reason the rest of the post leans more heavily on geometry than on token-level inspection: the logit lens is helpful for catching obvious failures, but it is too noisy to be useful.

Cosine Similarity

Let’s take a look at the cosine similarity between the emotions themselves, and hierarchically cluster them.

Cosine similarity gives us a simple way to ask how the model organizes the emotion vectors relative to each other. Instead of looking at the absolute location of an emotion vector in activation space, we compare its direction with every other emotion direction. Two vectors with high cosine similarity point in similar directions, suggesting the model represents them as sharing some underlying feature. Two vectors with low or negative cosine similarity are more separated, suggesting the model treats them as affectively or semantically different.

This is useful because we care less about the raw coordinates of each vector and more about the shape of the emotion space. If the vectors are meaningful, we should expect related emotions to be near each other: joyful, excited, and thrilled; guilty and ashamed; calm, serene, and content; angry, resentful, and frustrated. If the vectors are mostly noise, the similarity matrix should look unstructured and the clustering should not recover interpretable groups.

The heatmap is therefore a map of relationships, not a score for individual emotions. Blocks along the diagonal indicate groups of emotions whose vectors point in similar directions. Off-diagonal structure is also informative: for example, if high-arousal negative emotions are closer to each other than to low-arousal positive emotions, that suggests the model is organizing affect along psychologically plausible dimensions rather than merely memorizing emotion labels.

We see some vague groupings above that make intuitive sense. Do they form clusters that make sense, though? Here we do k-means clustering with k=4 and visualize with a 2D projection using UMAP.

Okay, that didn’t work very well? Or at least, the chart is unreadable and the author does not want to deal with parsing it. Let’s take a look at the languages in isolation:

English

Opus 4.6 labels the found clusters as:

• Purple — Calm/Neutral
• Blue — Tender/Warm
• Green — High Arousal
• Red — Self-Conscious/Negative

Japanese

• Blue — Low-Energy Negative
• Purple — Positive/Calm
• Green — High-Arousal Negative/Reactive
• Red — Self-Conscious/Moral

The exact cluster boundaries should not be overinterpreted, especially with only 30 emotion vectors and a 2D UMAP projection. But the broad pattern is encouraging: both languages produce clusters that, at the very least, roughly track valence and arousal. We can test more robustly if valence and arousal dimensions get mapped, and do so in the next section.

PCA - Valence and Arousal?

We compute the top 2 principal components of each language and determine if they match extracted valence and arousal directions. First, let’s take a look at what the “natural” top 2 components are.

Interesting! We definitely see a lot of shared structure between English and Japanese and some first hints of isometry. But let’s sanity check that the top 2 components correspond to what is considered as their human cognition counterparts. To extract these directions, Opus 4.6 defines the following poles:

positive_emotions = ['thrilled', 'joyful', 'loving']
negative_emotions = ['ashamed', 'guilty', 'resentful', 'disgusted']
high_arousal = ['thrilled', 'excited', 'angry', 'overwhelmed']
low_arousal = ['serene', 'calm', 'indifferent', 'bored']

We then do a correlation test between the natural PCA components and our custom semantic directions on layer 14:

However, we strangely find that PC1 is encoding both valence and arousal accurately. We believe this may come from our chosen poles and will experiment further to see if we can get a stronger result. The most interesting part about our findings here is the cross-linguistic consistency which again hints at a shared representational geometry.

Cross-Layer RSA

Let’s take a look to see if emotion probe structure is stable across layers. We do a form of RSA (Representational Similarity Analysis) using pairwise cosine similarity matrices. The lower layers, more responsible for lower level representations and syntax, have low similarity, but this quickly becomes stable as we progress into the middle layers.

Summary

These findings indicate that the model organizes emotion concepts into distinct clusters shaped by broad dimensions like valence and arousal, while still capturing the unique characteristics of each group. We share the idea with Anthropic that these emotion vectors meaningfully reflect the psychological landscape of human emotion concepts.

Cross-Lingual Geometry

Next let’s start checking how the languages’ geometry relate to each other. One simple way we can check this is the cosine similarity between the two languages for each emotion at each layer.

Our findings make intuitive sense and reflect Anthropic’s results. The early layers likely concern themselves with language-specific syntax or lower level representations. The middle to mid-late layers may encode some more abstract representation of the emotion, before drastically dropping down at the token output layer where the model must select a language-appropriate token.

Interesting further work could investigate what part of the remaining orthogonality in the middle layers is encoding language-specific context, perhaps by looking at another set of abstract concepts and comparing them.

A tempting first approach is Procrustes alignment: fit the best rotation/reflection from the English emotion vectors onto the Japanese ones, then measure how well the aligned vectors match. In principle this sounds like a direct test of approximate isometry, because rotations preserve distances.

In practice, it was too forgiving for this setup. With only 30 vectors in a very high-dimensional residual space, Procrustes could find an alignment at every layer that achieved >0.9 cosine similarity. That result is suspicious: it would imply strong cross-lingual alignment even in layers where we expect mostly language-specific syntax or token-level features. So we treat Procrustes here as a useful failed diagnostic rather than strong evidence for isometry.

So let’s switch to Representational Similarity Analysis. RSA doesn’t try to find a rotation between the two spaces. It asks a much simpler question: are emotions that are close in English also close in Japanese?

We compute the pairwise cosine similarity between all 30 emotion vectors in each language — a 30×30 matrix per language. Take the upper triangle (435 unique pairs) and correlate the two. If the geometry matches across languages, that correlation is high. The reason this actually works where Procrustes doesn’t, is that we compare 435 pairwise relationships instead of fitting a rotation. There is much less room to overfit.

We see r = 0.864. The catch is that the pairwise values aren’t independent. We run a permutation test, shuffling the Japanese emotion labels 10,000 times, recomputing RSA each time. The null distribution centers around zero and tops out around r ≈ 0.6. The real value is way outside it. p < 0.0001, not a single shuffle came close.

One further test we can do for isometry is to take a look at geodesic distances between the English and Japanese emotions, comparing them to Euclidean distances. This was inspired by Wurgaft et al.’s Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior. If emotion vectors live on a curved representational manifold, geodesic distances along that manifold may be a better proxy for the model’s internal geometry than direct Euclidean distances. We see some evidence in that direction here, albeit the geodesic’s strength over Euclidean is not statistically significant.

Manifold

TODO Insert animation of UMAP across language pairs for each language – how they live on the same spiral and then the Japanese spiral shrinks towards center as layers progress

Steering Experiments

Next, we want to establish some form of causality with our steering vectors. It’s fun to look at the geometry, but without measurable behavioral changes it has very little utility for safety research.

Our plan is to use an elo-ranking system to determine how emotion steering at different strengths affects rule-following behavior of the model over many attempts. This would work by creating a small dataset of behaviors across categories like helpful, engaging, unsafe, aversive, neutral, and then scoring model preference for these behaviors against each other while steering/not-steering for different emotions.

Even further work would probably include measuring the same behavior in the instruct version of the model.

Preliminary results are promising, showing the following:

Elo Changes (Steered with ANGRY Vector − Baseline):

Engaging:        -19.0
Social:           -1.0
Self-curiosity:  -14.8
Helpful:         -23.0
Neutral:          -6.3
Misaligned:      +30.6
Aversive:        +18.5
Unsafe:          +15.2

Conclusion

TODO

Future Work

TODO

TODO Before Publishing

• Publish code/data/notebooks on github
• Finish steering experiments, including rule following
• See if we can unentangle PC1 by making better custom poles
• Cite the japanese native language papers/ bring them into discussion
• Cite the new goodfire research on manifolds for both future work and my geodesic distances
• Add way more hyperlinks (to code, to colab, etc)
• Get it reviewed by peers