Artificial intelligence systems are increasingly being asked to make recommendations that have serious moral consequences, raising questions about how humans decide to trust these artificial advisors. A new study published in Computers in Human Behavior suggests that people do not simply trust an artificial intelligence based on whether it acts friendly or logical, but rather on how well its communication style matches the severity of the situation and the ethical choices it makes.
Researchers Lianshan Zhang and Mei Yin Zhao wanted to understand how humans perceive the moral minds of artificial conversational agents. “AI chatbots are increasingly being used not only for casual conversation, but also in contexts where their recommendations may carry moral or social consequences, such as healthcare, autonomous driving, and national security,” said Zhang, an associate professor at the School of Media and Communication at Shanghai Jiao Tong University in Shanghai.
“This made us interested in how people judge whether an AI system seems morally capable or trustworthy, not simply whether it gives a correct answer, but whether it appears to reason, care, and communicate appropriately in morally difficult situations.”
Two main psychological concepts explain how people view moral decision-makers in these digital contexts. The first is perceived moral agency, which refers to the belief that an entity can make intentional, deliberate choices and be held accountable for its actions. The second concept is perceived moral emotion. This involves the belief that an entity can feel or express concern, sympathy, and care for others, even if those feelings are simulated by a machine.
Zhang and Zhao wanted to see how different observable behaviors influence these perceptions. They focused on conversational style, which can be warm and friendly or highly logical and competent. They also looked at the specific moral stances a system might take in a difficult situation. A utilitarian stance focuses on the greatest good for the greatest number, which sometimes requires causing a smaller harm to prevent a larger one. By contrast, a deontological stance strictly follows established moral rules, dictating that causing harm is always wrong regardless of the potential positive outcomes.
To test these ideas, Zhang and Zhao set up an experiment with 447 participants recruited from a Chinese online survey platform called Credamo. The sample consisted of mostly young adults living in urban areas, and a majority held at least a bachelor’s degree. The participants engaged in a live chat with a custom artificial intelligence program built using a platform called Coze, which allowed the researchers to script specific responses and control the flow of the conversation.
To establish a baseline impression, participants completed a twenty-turn conversation with the chatbot about personal stress and coping mechanisms. This interaction lasted about six minutes and was designed to expose the user to a specific programmed personality. In the warm condition, the chatbot used an empathetic, supportive tone, complete with expressions of care and emojis. In the competent condition, the chatbot was highly formal, task-focused, and prioritized efficiency and logic.
After this initial conversation, the participants read a specific moral dilemma. The researchers randomly assigned participants to read either a low-severity scenario, which involved harm but no risk of death, or a high-severity scenario involving life-or-death consequences. The chatbot then offered a pre-written response to the moral dilemma it was presented with. This response utilized either a utilitarian logic that accepted a trade-off or a rule-based deontological logic that refused to cause harm.
Following the interaction, participants filled out questionnaires rating the chatbot on seven-point scales. They assessed the system’s perceived moral agency, its capacity for moral emotions, and its overall trustworthiness across the dimensions of competence, benevolence, and integrity.
The data provides evidence that single conversational cues do not independently guarantee trust. “One finding that surprised us was that conversational style alone did not strongly shape perceptions of the chatbot’s moral agency or moral emotions,” Zhang told PsyPost. “Simply making the chatbot warmer did not automatically make people see it as more morally emotional, and simply making it more competent did not automatically make it seem more morally agentic.”
Human evaluations instead relied heavily on how well the chatbot’s tone matched its moral choices. The researchers discovered that participants attributed higher emotional capacity to the system depending on how its personality aligned with its recommendations.
“Another unexpected finding was that people attributed slightly more moral emotion to the chatbot when it chose to accept harm to one person in order to save more people, compared with when it refused to directly harm an individual,” Zhang said. “This may sound counterintuitive, because sacrificing one person for the greater good is often seen as cold or calculating. But participants may have read the chatbot’s response not as pure calculation, but as concern for saving as many lives as possible.”
The stakes of the situation also shifted these perceptions significantly. “The main takeaway is that a trustworthy AI is not necessarily the warmest AI or the most professional-sounding AI,” Zhang said. “What matters is whether its communication style fits the moral situation.”
In low-severity situations, participants tended to view the warm chatbot as more capable of moral emotions and generally more trustworthy, treating warmth as a sign of good intentions. “For example, when the stakes are relatively low, a warmer and more empathic tone may help people see the chatbot as caring and morally concerned,” Zhang said.
In high-severity scenarios, the pattern changed dramatically. Participants attributed higher moral agency and emotional capacity to the competent chatbot, particularly when it delivered a utilitarian recommendation that weighed serious consequences. “But when the stakes are high, especially when the chatbot recommends a difficult trade-off, such as accepting harm to one person in order to save more people, people may expect a more careful and reasoned response, with clear explanations and a sense of accountability,” Zhang said. “In other words, trust depends less on whether AI sounds ‘warm’ or ‘cold,’ and more on whether it communicates in a way that feels appropriate for the situation.”
But as with all research, there are some limitations to consider. “One important caveat is that our study does not show that AI systems actually possess moral emotions or moral agency,” Zhang said. “We studied users’ perceptions, that is, whether people attribute moral emotions or moral agency to a chatbot based on how it communicates and what kind of moral judgment it gives.”
The study also relies on a relatively brief, single-episode interaction with a text-based conversational agent. “Trust in real-world AI systems may develop differently when people use them repeatedly, rely on them over time, or face real consequences,” Zhang said.
The participant pool poses another limitation to the generalizability of the findings. “Our participants were also recruited from an online panel in China and were relatively young, urban, and highly educated, so future research should test whether the same patterns appear in other groups and cultural contexts,” Zhang said.
“One misunderstanding we want to avoid is the idea that AI should simply be made more empathic,” Zhang said. “Our findings suggest that ‘more empathy’ is not always the answer. In morally sensitive situations, people may also need clear explanations, transparency, boundaries about what the AI can and cannot do, and accountability.”
Moving forward, the researchers plan to investigate how human-machine trust forms across more diverse demographics and scenarios. “Our long-term goal is to better understand how people form trust in AI systems that participate in socially and morally sensitive interactions,” Zhang said.
“In future work, we hope to study AI trust in longer and more realistic interactions, across different cultures and age groups, and in a wider range of sensitive contexts, such as health advice, education, emotional support, and public safety. We are also interested in how AI should communicate uncertainty, explain difficult trade-offs, and make clear the limits of its role.”
The authors suggest that developers must prioritize contextual appropriateness over simple friendliness. “One broader message of this study is that we should not design AI simply to sound more human, more friendly, or more persuasive,” Zhang said. “In morally sensitive situations, the more important question is whether the AI communicates in a way that fits the seriousness of the decision and helps people understand its reasoning.”
“As AI becomes part of everyday decisions, the goal should not be to make people trust AI more, but to help them trust AI more wisely,” Zhang said. “That means knowing when AI advice is useful, when it should be questioned, and when human judgment and responsibility must remain central.”
By keeping human accountability in the loop, developers can create tools that offer better support. “In this sense, our study is not only about chatbot design, but about building healthier human-AI relationships,” Zhang said. “Ultimately, this line of research is about designing AI systems that are not only useful and persuasive, but also responsible, transparent, and appropriately trusted.”
The study, “Not warm or cold, but appropriate: How outcome severity shifts moral-mind inferences and trust in AI chatbots,” was authored by Lianshan Zhang and Mei Yin Zhao.