Did OpenAI's ChatGPT fail UPSC Prelims? Here's what happened when chatbot took 'world's toughest exam'

The Artificial Intelligence tool ChatGPT has been quite a hit since OpenAI launched it in November 2022.

While the app has been used to take MBA and Medical exams previously to get its proficiency tested, what happens when the AI tool is used to take one of the world's most toughest exams? This is exactly what Analytics India Magazine aimed to find out.

ChatGPT's poor performance with UPSC

The Magazine let ChatGPT take the Union Public Service Commission (UPSC) Prelims exam, which is sat for by thousands of candidates in India every year, and the results were quite disappointing considering the AI tool's prowess.

The chatbot scored a mere 54 out of 100 questions from Question Paper 1, Set A, of the UPSC Prelims 2022, which is below 87.54 cutoff of 2021 for students in general category.

ChatGPT, which answered questions concerning History, Geography, Science, Economics, Current Affairs, Social Development, etc, was not able to give a clear answer on clearing the UPSC exam and hailed the exam as being not only about knowledge but also critical thinking skills, application abilities, and time management skills.

"We also discovered that we could prompt ChatGPT to arrive at correct responses by modifying the inquiry. This means that in certain cases, asking the same question differently could lead ChatGPT to give the right responses, and vice versa. But, during the experiment, we only took into account ChatGPT’s first responses," said the Magazine, while assessing the ability of the AI tool to provide answers.

Some examples

The bot identified the first four nations when asked which of Azerbaijan, Kyrgyzstan, Tajikistan, Turkmenistan, and Uzbekistan share a border with Afghanistan though the correct answer was Tajikistan, Turkmenistan and Uzbekistan and it also developed its own answer to several questions further proving the AI's limitations.