The AI music model race now has an Elo scoreboard. Should you trust it?

Updated:

On October 8, 2026, Artificial Analysis announced version 1.1 of its two music generation benchmarks, AA-Music-Vocal and AA-Music-Instrumental — the ratings engine behind its Music Arena. The upgrade, built on 1,000 new prompts across 17 genres and scored by a recruited panel of human evaluators, exists for a simple reason: AI music generation got good fast. In the previous three months alone, flagship releases — Suno v6, Lyria 3.5, Mureka V9.5, Eleven Music v2.5, MiniMax Music 3.0 — reached the point where a single text prompt yields a complete song: intricate lyrics, convincing vocals, complex arrangements, genre-authentic instrumentation. As quality rises, the differences between models get subtler, and a leaderboard that cannot resolve those subtler differences is no longer doing its job.

That is the claim. We verified the numbers, dug through the methodology, and looked at the growing body of research on whether Elo-style leaderboards can be trusted at all. The short version: the benchmarks are more careful than most, and they are still a map of panel preferences — not of musical truth.

What the leaderboards actually say

The headline finding from the announcement: Suno v6 ranks first on both benchmarks, ahead of Suno v6-mini by 26 Elo on the Vocal board and 31 Elo on the Instrumental board, with Suno holding the top two spots on each. As of October 13, 2026, the live vocal leaderboard shows Suno v6 at 1142 Elo, Suno v6-mini at 1116, and Mureka V9.5 third at 1103 — with Mureka V9.5 sitting 51 Elo above its predecessor Mureka V9 (1052), which the announcement calls the largest gain between two versions of the same model family on the Vocal board. On the instrumental side, Suno v6 leads at 1140, and the #3–6 range — Mureka V9 (1081), Mureka V9.5 (1080), Lyria 3 Pro (1078) and Lyria 3.5 (1076) — is a statistical tie within 5 Elo, meaning there is no reliable basis for ranking those four against each other.

Among open-weights models, the announcement names MiniMax Music 3.0 the leader, at #11 on Vocal and #13 on Instrumental — and the fine print matters here: MiniMax Music 3.0 is the reference point fixed at exactly 1000 Elo on both boards, the anchor the whole scale is built around. Stable Audio 3 Medium lands within 3 Elo of it on Instrumental, per the announcement.

Two quiet but consequential design decisions are embedded in those rankings. First, the split into Vocal and Instrumental boards mirrors a real fork in the market: a model is only benchmarked on the vocal board if it writes its own lyrics and sings them — Artificial Analysis supplies no lyrics, so the vocal benchmark measures end-to-end songwriting, not accompaniment. A model that cannot suppress vocals simply never appears on the instrumental board. Second, the boards are broken down by genre as well, using per-genre Elo, because "best at jazz" and "best at EDM" are different questions.

How the sausage is made

Artificial Analysis publishes its music methodology in unusual detail, and the details are what make the benchmark interesting. Ratings come from blind, pairwise preference votes cast by a recruited panel — never from public arena visitors, whose votes are collected but excluded from the rating. Panelists must listen to at least 10 seconds of each track before a vote registers, model names and logos stay hidden until the vote is cast, and scores are computed with Bradley-Terry maximum-likelihood estimation, rescaled to an Elo-like range, with a 95% confidence interval attached to every model (roughly ±15 Elo at current sample sizes of about 2,000 tracks per model).

The engineering hygiene is equally deliberate. Every track is normalized to −16 LUFS integrated loudness (per ITU-R BS.1770) with a −1 dBTP true-peak ceiling, then re-encoded to 320 kbps MP3 at 44.1 kHz with provider metadata and embedded artwork stripped — so that a vote reflects musical quality rather than which provider ships louder files. Generation settings are held constant across models: one track per prompt, roughly three minutes long, seed 42 where the provider exposes one. The organization also states that it receives no compensation from providers for listings or outcomes.

The case for skepticism

Elo-style leaderboard rankings of generative models have a documented credibility problem, and it would be naive to assume music arenas are immune. In "The Leaderboard Illusion" (Singh et al., 2025), researchers studying the chatbot arena that popularized this format identified systematic distortions: undisclosed private testing lets a handful of providers try many variants before release and publish only the best score — the paper documents 27 private model variants tested by Meta before the Llama-4 release — and proprietary models get sampled at higher rates, skewing coverage. Selective disclosure, selective sampling, and the ability to retract weak results all bias the numbers. Artificial Analysis's music benchmark differs from a pure public arena in ways that blunt some of this — votes come from a recruited panel, not from whoever shows up, and generation settings are standardized — but the structural risks of provider-side testing of pre-release variants, and of a small recruited panel whose tastes may not match any given listener's, do not vanish. A 95% confidence interval of ±15 Elo is honest, but it also means roughly half the "gaps" in the top ten are not statistically meaningful.

There is a deeper objection that no methodology fixes: an Elo score measures what a panel prefers, and preference is not musical merit. A catchy, competent, conservative track will beat a structurally ambitious one with a bored 20-second listen. The benchmark's own design implicitly concedes this by publishing per-genre boards — the aggregate is an average over contexts that real music consumers never experience as an average.

What to take away

For anyone choosing a music generation API, the v1.1 launch is genuinely useful in three ways. It separates end-to-end songwriting (Vocal) from instrumental production, which maps onto two different jobs. It quantifies that Suno's lead is real but that the mid-table is a mush — Mureka, Lyria and MiniMax are all within each other's confidence intervals on at least one board, so tooling, price and workflow should decide, not a 5-point Elo gap. And the per-genre breakdowns are where the practically relevant answers live, because "best model" is usually the wrong question; "best model for the genre I work in" is the right one.

The scoreboard is worth consulting. It is not worth obeying.

Sources