The paper published in the new issue of Nature · Human Behavior shows that in the task of testing the ability to track the mental state of others, also known as Theory of Mind, two types of large language models (LLMs) perform similarly or even better than humans in specific situations.
?
The research team tested the popular LLM. Photo source: Nature · Human Behavior
Mental theory is very important to human social interaction, and is the key to human communication and resonance. Previous studies have shown that artificial intelligence (AI) such as LLM can solve complex cognitive tasks, such as multiple choice decision-making. However, it has been unclear whether LLM can also compete with humans in the performance of mental theory tasks (considered to be the unique ability of humans).
This time, the team from Hamburg Eppendorf University Medical Center in Germany chose tasks that can test different aspects of mental theory, including finding wrong ideas, understanding indirect speech and recognizing discourtesy. They then compared the ability of 1907 people to complete tasks with two popular LLM families, GPT and LLaMA2 models.
The team found that the GPT model can achieve or exceed the human average in identifying indirect requirements, wrong ideas and misleading, while the performance of LLaMA2 is inferior to the human level; LLaMA2 is better than human in recognizing disrespect, but its GPT performance is poor. The researchers pointed out that the success of LLaMA2 is due to the low degree of prejudice in the answers, rather than the real sensitivity to disrespect; GPT's apparent failure is actually due to its ultra conservative attitude towards sticking to conclusions, rather than reasoning mistakes.
The research team believes that LLM's performance in mental theory tasks is comparable to that of human beings, which does not mean that they have human like "emotional intelligence", nor does it mean that they can master mental theory. However, they also pointed out that these results are an important basis for future research, and suggested that further research should be carried out on the performance of LLM in psychological inference and how these performance would affect human cognition in human-computer interaction.
(Original title: AI's psychological inference ability may be equivalent to that of human beings, but it does not mean that they have emotional intelligence like human beings)
