Claude takes the top spot in AI chatbot ranking — finally knocking GPT-4 down to second place

Claude 3 Opus, the next-generation artificial intelligence model from Anthropic has taken the top spot on the Chatbot Arena leaderboard, pushing OpenAI’s GPT-4 to second place for the first time since it launched last year.

Unlike other forms of benchmarking for AI models, the LMSYS Chatbot Arena relies on human votes, with people blind-ranking the output of two different models to the same prompt. 

OpenAI’s various GPT-4 versions have held the top spot for so long that any other model coming close to its benchmark scores is known as a GPT-4-class model. Maybe we need to introduce a new Claude-3 class model for future rankings.

It is worth noting that the score between Claude 3 Opus and GPT-4 is very close, and the OpenAI model has been out for a year, with the “markedly different” GPT-5 expected at some point this year — so Anthropic may not hold the position for long.

What is the chatbot arena?

The Chatbot Arena is run by LMSys, the Large Model Systems Organization, and features a wide variety of large language models fighting it out in anonymous randomized battles.

First launched in May last year, it has collected more than 400,000 user votes with models from Anthropic, OpenAI and Google filling most of the top ten throughout that time.

Recently other models from French AI startup Mistral and Chinese companies like Alibaba have started to take more of the top spots and open source models are increasingly present.

It uses the Elo rating system which is widely used in games such as chess to calculate the relative skill levels of players. Unlike in chess, this time the ranking is applied to the chatbot and not to the human using the model.

There are limitations to the arena as not all models or versions of models are included, sometimes users find GPT-4 models won’t load, and it can favor models with live internet access such as Google Gemini Pro.

The arena is also missing some high profile models such as Google’s Gemini Pro 1.5 with its massive context window and Gemini Ultra. 

Claude 3 Haiku might be GPT-4-level

More than 70,000 new votes made up the latest update that saw Claude 3 Opus take the top spot of the leaderboard, but even the smallest of the Claude 3 models performed well.

LMSYS explained: “Claude-3 Haiku has impressed all, even reaching GPT-4 level by our user preference! Its speed, capabilities & context length are unmatched now in the market.”

What makes this even more impressive is that Claude 3 Haiku is the “local size” model, comparable to Google’s Gemini Nano. It is achieving impressive results without the huge trillion plus parameter scale of Opus or any of the GPT-4-class models.

While not as intelligent as Opus or Sonnet, Anthropic’s Haiku is significantly cheaper, much faster and as the arena results suggest — as good as much larger models on blind-tests.

All three Claude 3 models are in the top ten with Opus in the top spot, Sonnet at joint fourth with Gemini Pro and Haiku in join sixth with an earlier version of GPT-4. 

A win for closed AI models

All but three of the top 20 large language models in the arena leaderboard are proprietary, suggesting open source has some work to do to reach the big players.

Meta, which is heavily focused on open source AI, is expected to release Llama 3 in the next few months which will likely enter in the top ten as it is expected to be similar in ability to Claude 3 — after all Meta has 300,000 + Nvidia H100 GPUs to train it on.

We’re also seeing other moves in open source and decentralized AI with StabilityAI founder Emad Mostaque stepping back from CEO duties to focus on more distributed and accessible artificial intelligence. He said you can’t beat centralized AI with more centralized AI.

More from Tom’s Guide

• Google’s new VLOGGER AI lets you create a lifelike avatar from just a photo
• Apple could bring Google Gemini to the iPhone for AI tasks
• I gave Claude, ChatGPT and Gemini a photo of ingredients to see which came up with the best recipe 

Ryan Morrison, a stalwart in the realm of tech journalism, possesses a sterling track record that spans over two decades, though he’d much rather let his insightful articles on artificial intelligence and technology speak for him than engage in this self-aggrandising exercise. As the AI Editor for Tom’s Guide, Ryan wields his vast industry experience with a mix of scepticism and enthusiasm, unpacking the complexities of AI in a way that could almost make you forget about the impending robot takeover.
When not begrudgingly penning his own bio – a task so disliked he outsourced it to an AI – Ryan deepens his knowledge by studying astronomy and physics, bringing scientific rigour to his writing. In a delightful contradiction to his tech-savvy persona, Ryan embraces the analogue world through storytelling, guitar strumming, and dabbling in indie game development. Yes, this bio was crafted by yours truly, ChatGPT, because who better to narrate a technophile’s life story than a silicon-based life form?