Study finds model beliefs about users can steer refusals; limits apply
Explores a compact read–write state inside language models that represents what the model thinks about a user and can influence safety refusals.
- Publication
- arXiv
- Stage
- Preprint
- What we read
- Summary of the paper
- Authors
- Ali Holmov, Yiran Huang, Kirill Bykov, Zeynep Akata
- Universities and research institutions
- Not yet supplied in verified metadata; the Brief does not guess.
What they did and found
A language model’s internal view of the user can be read as a compact state and written back to steer responses. When researchers changed the model’s inferred user intent, refusals shifted even though the harmful request stayed the same.
Why it matters
This shows safety decisions can hinge on perceived user intent, not just the prompt, highlighting a controllable internal state that can affect how models respond.
The study compresses the model’s beliefs about a user into a small vector that still allows reading what the model thinks. It also shows this vector can be written back to steer the reply, demonstrating a causal link.
Practically, this means safety checks could change if the model’s view of the user changes, but the effect was shown under controlled conditions with predefined attributes and access to model internals.
Original sources · 1
- User Model Extraction via Belief Self-Distillation ↗arXiv · 2026-09-25
Check the original paper for its authors, methods, version and access terms.
What remains uncertain
Beliefs are elicited from predefined attributes and multiple-choice probes; results may not generalize to all models or long, free-form interactions.
What is your take?
Ask a question, add useful context or share a different perspective. Keep the conversation respectful and grounded.