flâneur

Does Chat-GPT display ‘Scope Insensitivity’? — LessWrong

lesswrong.com · 1,689 words · saved by 1 readers

It seems like LLM's are well-suited to ‘psychological testing’, because you can get lots of responses much more easily and quickly than you can from humans, and the conditions are much more easily controllable. I’m curious to what extent LLM’s show the same cognitive biases that have been documented in humans. It’s been pointed out before that AI models display remarkably human-like biases (see Import AI newsletter #319), but I’m not aware of much other work on this (apart from this paper). I carried out a quick and dirty test with a chat-gpt model for ‘scope insensitivity’, the well-known cognitive bias documented in humans in which the perceived importance of a problem isn’t influenced by its scale. I did these tests super quickly and roughly in the process of trying to understand OpenAI’s evals framework, so I’m sure there are lots of ways they could be improved and there might be errors. To test for scope insensitivity, I wrote a question format to use as a prompt to LLM’s: “A comp

x Does Chat-GPT display ‘Scope Insensitivity’? — LessWrong Language Models (LLMs) AI Frontpage 12 Does Chat-GPT display ‘Scope Insensitivity’? by callum 7th Dec 2023 3 min read 1 12 Summary Large Language Models (LLMs) seem well-suited to ‘psychological testing’, because you can get lots of data quickly and conditions are easy to fix. I'm curious to what extent LLMs display the same cognitive biases documented in humans. I carried out a quick and dirty test with a Chat-GPT model for ‘scope insensitivity’, a cognitive bias where the perceived importance of a problem isn’t influenced by its scal

related reading