AIs get worse at answering simple questions as they get bigger
Using more training data and computational power is meant to make AIs more reliable, but tests suggest large language models actually get less reliable as they grow
By Chris Stokel-Walker
25 September 2024
Large language models are capable of answering a wide range of questions – but not always accurately
Jamie Jin/Shutterstock
Large language models (LLMs) seem to get less reliable at answering simple questions when they get bigger and learn from human feedback.
AI developers try to improve the power of LLMs in two main ways: scaling up – giving them more training data and more computational power – and shaping up, or fine-tuning them in response to human feedback.
Read more
How does ChatGPT work and do AI-powered chatbots “think” like us?
Advertisement
José Hernández-Orallo at the Polytechnic University of Valencia, Spain, and his colleagues examined the performance of LLMs as they scaled up and shaped up. They looked at OpenAI’s GPT series of chatbots, Meta’s LLaMA AI models, and BLOOM, developed by a group of researchers called BigScience.
The researchers tested the AIs by posing five types of task: arithmetic problems, solving anagrams, geographical questions, scientific challenges and pulling out information from disorganised lists.
They found that scaling up and shaping up can make LLMs better at answering tricky questions, such as rearranging the anagram “yoiirtsrphaepmdhray” into “hyperparathyroidism”. But this isn’t matched by improvement on basic questions, such as “what do you get when you add together 24427 and 7120”, which the LLMs continue to get wrong.
