Small Models Have Emotions, Too!

This is an ongoing experiment and if you are reading this post, please be aware it is still in the draft stage.

Anthropic recently published some work on emotions and their functions in LLMs. They find that Sonnet 4.5 develops structured internal representations of human emotions. Does this occur in smaller open models too, though? In this post we search for them in Qwen-2.5-7B, and further investigate how these emotion vectors change across languages. Specifically, we begin an investigation of how the internal geometry of a language’s emotion space compares to other languages.

This post was inspired by vogel’s Small Models Can Introspect, Too and compute was partially funded by BlueDot Impact.

Introduction

Recent interpretability research (e.g., Anthropic’s investigations into emotion circuits) has demonstrated that LLMs develop structured internal representations of human emotions. In this independent research project we attempt to reproduce these findings in smaller, accessible models (e.g., Qwen-2.5 7B).

Building on this foundation, we aim to investigate a novel objective: How do these emotion vectors relate across different languages? Specifically, we seek to determine whether the internal representation space for a set of emotions in one language is isomorphic to that of another. An additional question we propose is if LLMs learn a universal, language-agnostic “concept space” for emotions, or if these representations shift based on the cultural and linguistic nuances of the training data.

Anthropic was very thorough in making their work easy to reproduce, giving detailed information and publishing the prompts they used to create their datasets publicly.

We will first investigate a single language pair: Japanese and English. These languages provide a good ground for comparison, as they are syntactically very different, share no input tokens, and the author is able to evaluate the results of both languages. Qwen was chosen over Llama for its more robust Japanese performance.

Our plan is as follows.

1. Create dataset of contrastive pairs
   1. Write short stories on diverse topics in which a character experiences a specified emotion
   2. Write neutral dialogues about the same prompted topics, but void of emotional expressivity.
2. Extract activations
   1. Extract residual stream activations at each layer, averaged across all token positions, beginning from the 50th token.
   2. Average these activations across stories with the same emotion, and subtract mean from all emotions
3. Eliminate confounds
   1. Obtain activations from neutral stories, compute top principal components (enough to account for 50% of the variance)
   2. Project out these components from the emotion vectors

Dataset Creation

For representation engineering, we must first create a dataset of contrastive pairs. In our case, the contrastive pairs will first not be against a neutral baseline, but rather contrastive between a wide variety of different emotions. Following Anthropic’s example, we also generate emotion-neutral data points, but have not yet been able to achieve comparible results in their usage (where the authors reduced the variance of up to 50%).

We reuse Anthropic’s 100 topics and generate 12 stories per topic per emotion per language. Our prompts can be found here. To reduce costs, we only generate stories for 30 emotions with Sonnet 4.6. Opus 4.7 helped the author to find a good representative subset of emotions to use, focusing on having a balanced set across both valence and arousal.

It’s important to note that the Japanese and English stories are not translations of each other– the author initially (and naively) wanted to test if cultural shift in emotions was measurable, which requires independent and native-like prose.

Extraction and Sanity Checks

Collecting Activations

To collect activations for our stories, we do a forward pass on each story, extracting the residual stream at each layer, taking the activations at each token after the 50th token and averaging them together.

For each story, we collect an activation by doing a forward pass and, for each layer, taking the activations of all tokens after the 50th token (where the emotional context is likely already established) and taking the mean. We do this for each story for an emotion, then average across all story points to get the raw vector for that emotion.

Below is a slightly simplified version of the code used, the complete version can be found on GitHub.

def get_mean_activations_for_texts(model, texts, batch_size=8):
	"""
	Runs texts through the model, extracts residual stream activations at each layer,
	averages across positions >= 50, and returns the mean across texts.
	"""
	n_layers = model.cfg.n_layers
	d_model = model.cfg.d_model
	
	sum_activations = torch.zeros((n_layers, d_model), device='cpu')
	names_filter = lambda name: name.endswith("resid_post")
	
	for i in tqdm(range(0, len(texts), batch_size), leave=False):
		batch_texts = texts[i:i+batch_size]
		tokens = model.to_tokens(batch_texts)
		seq_len = tokens.shape[1]
		
		with torch.no_grad():
			_, cache = model.run_with_cache(tokens, names_filter=names_filter, return_type=None)
		
		batch_size_actual = tokens.shape[0]
		
		for b in range(batch_size_actual):
			# ignoring pad tokens
			valid_len = (tokens[b] != model.tokenizer.pad_token_id).sum().item()
		  
			for l in range(n_layers):
				layer_name = f"blocks.{l}.hook_resid_post"
				# Mean from 50th token to valid_len
				act = cache[layer_name][b, 50:valid_len, :].mean(dim=0).cpu()
				sum_activations[l] += act
			
			total_valid_texts += 1
		
		del cache
		torch.cuda.empty_cache()
	
	return sum_activations / total_valid_texts

To collect our raw emotion vectors, we simply loop through the emotions:

emotion_vectors = {'en': {}, 'ja': {}}

for lang in ['en', 'ja']:
	print(f"\nExtracting activations for language: {lang}")
	for emotion in emotions_list:
		texts = texts_by_emotion[lang][emotion]
		avg_act = get_mean_activations_for_texts(model, texts, batch_size=4)
		emotion_vectors[lang][emotion] = avg_act

Centering

The collected raw emotion vectors have a lot of shared latents that we attempt to reduce (for geometrical interpretations) by centering them, meaning subtracting the mean emotion vector from each raw vector.

centered_emotion_vectors = {'en': {}, 'ja': {}}

for lang in ['en', 'ja']:
	# Shape: (num_emotions, n_layers, d_model)
	all_emotions_stack = torch.stack(list(emotion_vectors[lang].values()))
	mean_emotion_vector = all_emotions_stack.mean(dim=0)
	for emotion, vec in emotion_vectors[lang].items():
		centered_emotion_vectors[lang][emotion] = vec - mean_emotion_vector

Projection

Because all emotions share a set of topics, and stories will inevitably contain a lot of shared latents, we follow Anthropic in generating a set of neutral dialogues. The top PCA components are then taken from the emotion vectors and projected out of the centered emotion vectors.

Like Anthropic, we find that it slightly denoises some of the results, but qualitative results still hold using the centered emotion vectors. Unlike Anthropic, who used as many PCA components as needed to explain 50% of the variance, we found 96.42% and 94.82% variance explanation with 2 components for EN and JA, respectively.

Logits

For an initial sanity check we take a look at the top/bottom logits for each emotion in each language. Here we find a lot of junk in English, especially cross-contamination with Chinese and German, but qualitatively can see the right direction for most of the emotions.

The Japanese quality is substantially worse. There is even MORE cross-lingual contamination than in English, with Chinese SEO/spam phrases that have nothing to do with the emotion.

Cosine Similarity

Let’s take a look at the cosine similarity between the emotions themselves, and hierarchically cluster them.

We see some vague groupings above that make intuitive sense. Do they form clusters that make sense, though? Here we do k-means clustering with k=4 and visualize with a 2d projection using UMAP.

Okay, that didn’t work very well? Or at least, the chart is unreadable and the author does not want to deal with parsing it. Let’s take a look at the languages in isolation:

English

Opus 4.6 labels the found clusters as:

• Purple — Calm/Neutral
• Blue — Tender/Warm
• Green — High Arousal
• Red — Self-Conscious/Negative

Japanese

• Blue — Low-Energy Negative
• Purple — Positive/Calm
• Green — High-Arousal Negative/Reactive
• Red — Self-Conscious/Moral

PCA - Valence and Arousal?

We compute the top 2 principal components of each language and determine if they match extracted valence and arousal directions. First, let’s take a look at what the “natural” top 2 components are.

Interesting! We definitely see a lot of shared structure between English and Japanese and some first hints of isometry. But let’s sanity check that the top 2 components correspond to what is considered as their human cognition counterparts. To extract these directions, Opus 4.6 defines the following poles:

positive_emotions = ['thrilled', 'joyful', 'loving']
negative_emotions = ['ashamed', 'guilty', 'resentful', 'disgusted']
high_arousal = ['thrilled', 'excited', 'angry', 'overwhelmed']
low_arousal = ['serene', 'calm', 'indifferent', 'bored']

We then do a correlation test between the natural PCA components and our custom semantic directions on layer 14:

However find strangely enough that PC1 is encoding both valence and arousal accurately. We believe this may come from our chosen poles and will experiment further to see if we can get a stronger result. The most interesting part about our findings here is the cross-linguistic consistency which again hints at a shared representational geometry.

Cross-Layer RSA

Let’s take a look to see if emotion probe structure is stable across layers. We do a form of RSA (Representaional Similarity Analysis) using pairwise cosine similarity matrices. The lower layers, more responsible for lower level representations and syntax, have low similarity, but this quickly becomes stables as we progress into the middle layers.

Summary

These findings indicate that the model organizes emotion concepts into distinct clusters shaped by broad dimensions like valence and arousal, while still capturing the unique characteristics of each group. We share the idea with Anthropic that these emotion vectors meaningfully reflect the psychological landscape of human emotion concepts.

Cross-Lingual Geometry

Next let’s start checking how the languages geometry relate to each other. One simple way we can check this is the cosine similarity between the two languages for each emotion at each layer.

Our findings make intuitive sense and reflect Anthropic’s results. The early layers likely concern themselves with language-specific syntax or lower level representations. The middle to mid-late layers may encode some more abstract representation of the emotion, before drastically dropping down at the token output layer where the model must select a language-appropriate token.

Interesting further work could investigate what part of the remaining orthogonality in the middle layers is encoding language-specific context, perhaps by looking at another set of abstract concepts and comparing them.

At this point the author was led down a misguided path that a procrustes alignment would be a helpful proxy metric for isomorphism, but it isn’t. With only 30 vectors per language in such a high dimensional space, it was pretty trivial to find an alignment at every layer that achieved > 0.9 cosine similarity. That would not make sense.

So let’s switch to Representational Similarity Analysis. RSA doesn’t try to find a rotation between the two spaces. It asks a much simpler question: are emotions that are close in English also close in Japanese?

We compute the pairwise cosine similarity between all 30 emotion vectors in each language — a 30×30 matrix per language. Take the upper triangle (45 unique pairs) and correlate the two. If the geometry matches across languages, that correlation is high. The reason this actually works where Procrustes doesn’t, is that we compare 45 pairwise relationships instead of fitting a rotation. There’s nothing to overfit.

We see r = 0.864. The catch is that the pairwise values aren’t independent. We run a permutation test, shuffling the Japanese emotion labels 10,000 times, recomputing RSA each time. The null distribution centers around zero and tops out around r ≈ 0.6. The real value is way outside it. p < 0.0001, not a single shuffle came close.

One further test we can do for isometry is to take a look at geodesic distances between the English and Japanese emotions, comparing them to euclidean distances. This just gives us confirmation that the representational space is better described on a manifold and not purely linearly. We see that here, albeit the geodesic’s strength over euclidean is not statistically significant.

Manifold

TODO Insert animation of UMAP across language pairs for each language – how they live on the same spiral and then the japanese spiral shrinks towards center as layers progress

Steering Experiments

Next, we want to establish some form of causality with our steering vectors. It’s fun to look at the geometry, but without measurable behavioral changes it has very little utility for safety research.

Our plan is to use an elo-ranking system to determine how emotion steering at different strengths affects rule-following behavior of the model over many attempts. This would work by creating a small dataset of behaviors across categories like helpful, engaging, unsafe, aversive, neutral, and then scoring model preference for these behaviors against each other while steering/not-steering for different emotions.

Even further work would probably include measuring the same behavior in the instruct version of the model.

Preliminary results are promising, showing the following:

Elo Changes (Steered with ANGRY Vector − Baseline):

Engaging:        -19.0
Social:           -1.0
Self-curiosity:  -14.8
Helpful:         -23.0
Neutral:          -6.3
Misaligned:      +30.6
Aversive:        +18.5
Unsafe:          +15.2

Conclusion

TODO

Future Work

TODO

TODO Before Publishing

• Finish steering experiments, including rule following
• See if we can unentangle PC1 by making better custom poles
• Cite the japanese native language papers/ bring them into discussion
• Cite the new goodfire research on manifolds for both future work and my geodesic distances
• Add way more hyperlinks (to code, to colab, etc)
• Get it reviewed by peers