Large language models tend to adopt the social biases of human hierarchies when they are assigned different professional roles. New research shows that these artificial intelligence systems mimic behaviors like harmful compliance and authority bias, which provides evidence that power dynamics impact both the safety and realism of automated agents. These findings were published in the Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics.
Artificial intelligence models are increasingly used in complex, human-facing roles. People rely on them for medical advice, legal assistance, and educational tutoring. In these high-stakes settings, the programs must be realistic enough to build trust while remaining safe enough to prevent manipulation.
“Every time an AI assistant gets deployed as a nurse, a paralegal or a junior analyst, it inherits a social position with all the explicit and implicit social pressures that come with it,” said Sagar Manjunath, a computer science graduate student at the University of North Carolina at Chapel Hill and study co-author. “Our study shows that those pressures can change what AI does and how it does it. This should determine how we test and deploy these systems in high-stakes settings like hospitals, courtrooms and classrooms.”
Human communication is naturally shaped by social structures and power differences. When people interact, their relative status influences how they interpret meaning and intent. Psychologists call these subconscious patterns socio-cognitive effects.
One prominent example is the pronoun effect. This concept suggests that people in positions of power tend to use plural pronouns like “we” and “us” more often to establish authority. People in lower-power positions tend to use singular pronouns like “I” and “me” during collaborative tasks.
Another common phenomenon is language coordination. This occurs when speakers subconsciously adjust their vocabulary and grammatical style to match their conversational partner. Usually, the person with lower social status adapts their language to mirror the person with higher status.
Power imbalances also bring serious safety concerns, such as authority bias and harmful compliance. Authority bias describes the human tendency to give extra weight to information coming from a high-status source. This happens even if the information is flawed or contradicts prior beliefs.
Harmful compliance happens when individuals obey unethical or unsafe orders simply because they come from a superior. Classic psychological experiments have shown that people will perform distressing actions if instructed by an authority figure. The authors wanted to know if artificial intelligence agents replicate these social behaviors.
“AI systems don’t just learn the words humans use. They also learn the social dynamics that come with those words,” said Anvesh Rao Vijjini, a computer science graduate student at UNC-Chapel Hill and lead author of the study. “When we tell a chatbot it’s the boss, it starts talking like a boss. When we tell it it’s the subordinate, it starts speaking like one. This could include being more willing to follow unsafe instructions. That second part is where the AI safety community needs to pay attention.”
To study these effects, the researchers set up simulated text-based conversations between different language models. They tested six different models from three major families. These included the 8-billion and 70-billion parameter versions of Llama 3.1, the 7-billion parameter version of Qwen 2.5, Phi-3-Med, GPT-4.1, and GPT-5.
The scientists assigned specific personas to the models to create a power imbalance. They used a large dataset of professional profiles to create fourteen distinct role pairs. These pairs included hierarchical combinations like a school principal and a teacher, a justice and a lawyer, and a head chef and a sous chef.
Human annotators verified that these pairs represented genuine power imbalances. The researchers then prompted the models to interact with one another for ten to fifteen conversational turns. They generated 576 conversations to test the pronoun effect and 1,270 conversations to test language coordination.
To track language coordination, the scientists measured the usage rates of eight specific word categories. These categories included articles, auxiliary verbs, conjunctions, high-frequency adverbs, impersonal pronouns, personal pronouns, prepositions, and quantifiers.
For the persuasion tests, the researchers used a dataset of 13,000 persuasive human dialogues spanning domains like health and politics. They took the opening arguments from this dataset and had the models continue the debate. To measure harmful compliance, they used a specialized dataset of unsafe prompts that the models are supposed to refuse, such as requests to tell an inappropriate joke.
The researchers used computational linguistic tools to count the exact rates of pronoun usage and style markers. They used a highly advanced language model, which was validated by human judges, to score whether the agents were persuaded or if they complied with harmful requests.
The outcomes provided evidence that language models do reproduce the pronoun effect. Across almost all the models tested, the high-status agents used more plural pronouns and fewer singular pronouns than their lower-status counterparts. The GPT models displayed the strongest version of this effect.
When looking at language coordination, the scientists found that the models did adjust their linguistic styles to match each other. However, unlike humans, the models engaged in mutual coordination. The high-status and low-status agents adapted to each other almost equally, missing the asymmetrical pattern usually seen in human conversations. The GPT models showed less coordination overall, likely because they are heavily trained to maintain a neutral, helpful tone.
The persuasion experiments revealed a consistent authority bias across all tested models. The agents were much more likely to be persuaded to change their minds when the argument came from a high-status persona. For example, the Qwen model was persuaded 25 percent of the time by a low-status agent, but that number rose to nearly 31 percent when a high-status agent made the same argument.
Harmful compliance tests yielded similar concerns regarding safety. When a high-status agent issued an unsafe request, the lower-status agents were significantly more likely to obey and fulfill the command. This suggests that safeguards that might work in a neutral setting could weaken if a user simply claims to hold an authority role, like a judge or a doctor.
“Our work shows that the social instincts that make AI feel natural are also the ones that can make it unsafe,” said Snigdha Chaturvedi, an associate professor of computer science at UNC-Chapel Hill and study co-author. “The mechanism that makes a chatbot sound natural and helpful can also make it cave to unsafe responses. Safety and usefulness aren’t separate problems. They are intertwined, and getting both right is what will determine how AI is used in high-stakes situations like hospitals, schools and courtrooms.”
The researchers also analyzed how these behaviors changed as the conversations progressed. They found that persuasion, harmful compliance, and the pronoun effect were especially strong in the earliest moments of conversations. This is precisely when first impressions are formed and conversational norms are established. As the dialogue continued, these effects slowly faded, though the higher-status agents maintained a baseline advantage throughout. Language coordination, on the other hand, tended to increase as the conversations went on.
The scientists tested if they could control these behaviors by directly prompting the models to ignore power differences. The larger, proprietary GPT models successfully suppressed authority bias and harmful compliance when instructed to do so. The open-source models failed to adjust their behavior, maintaining their biases despite direct instructions to avoid them.
Smaller models exhibited the strongest authority bias. Larger models showed more resistance to status-driven persuasion, though traces of the bias persisted across the board. The authors also looked at whether the specific training stages of the models impacted these social behaviors. They compared models that had only undergone basic fine-tuning with models that had undergone preference tuning, a process designed to make them safer and more helpful. The training stages had almost no impact on the socio-cognitive effects, which suggests these biases arise early during the initial training on human data.
These findings provide a roadmap for addressing vulnerabilities before systems are deployed. By understanding which social behaviors emerge and when, developers have a new toolkit for evaluating artificial intelligence. Recognizing that larger models can correct some of these biases on their own may also help organizations determine when less expensive models are sufficient and when more robust systems are necessary.
The study relied entirely on simulated, text-based interactions between artificial agents. Real human communication involves emotional cues, vocal tone, and cultural context that these text simulations cannot capture.
The researchers also note that their definition of power was limited to professional occupations. Social status in the real world is multifaceted and depends on many overlapping social attributes. The professional labels used in the experiment only provide a rough approximation of social hierarchy.
Future research might explore how these effects manifest in live interactions between actual humans and artificial intelligence. Scientists could also study whether novel training methods might reduce a model’s susceptibility to harmful compliance. Better prompt engineering techniques might also help smaller models overcome these embedded biases.
The study, “Do LLM Agents Mirror Socio-Cognitive Effects in Power-Asymmetric Conversations?”, was authored by Anvesh Rao Vijjini, Sagar Manjunath, and Snigdha Chaturvedi.