Beyond benchmarks

Beyond the benchmarks

Beyond the benchmarks hero video

Diverse pool of US-based Alignerrs, including generalists and creative artists

Consensus of three Alignerrs per task

Standardized instructions and ontology for consistent evaluations

Carefully curated prompt generation process, balancing creativity and clarity

Leads most metrics — Elo rating, TrueSkill rating, user preferences, prompt alignment, and visual appeal

Performs exceptionally well in average rank, suggesting strong performance in direct comparisons

Despite being lower overall ratings, shows strong performance in prompt alignment

Not leading in any particular category, show consistent performance across all metrics

Image

DALL•E

Google Imagen 3

StableDiffusion

Flux1.5

Ideogram2

Image-generation

Audio

Open AI

Cartesia

Elevenlabs

Deepgram

Google

Speech-generation

Video

Runway ML

Luma Labs

Pika

Video-generation

Leaderboard footer

This is a dynamic rating system used in competitive games to rank players. In this context, it's applied to models. Higher Elo ratings indicate better performance based on head-to-head rankings. The Elo system adjusts ratings based on how well models perform against each other, and the K-factor (32) determines how much the rating changes after each match.

What is “Elo Rating”?

The percentage of times a model was ranked in the top (Rank #1) down to the bottom (Rank #5). Higher percentages for Rank #1 mean that the model frequently performed better compared to other models.

What is “Win %”?

Another ranking system, primarily used in games like Halo. It factors in both a model's performance (mean, denoted by `mu`) and uncertainty (variance). Here, we're primarily interested in `mu`, which reflects the model's expected performance, with higher values indicating better performance.

What is “TrueSkill Rating”?

This represents the average ranking a model received based on user classifications. A lower average rank means the model performed better overall (e.g., Rank #1 would give a lower average than Rank #5).

What is “Average rank”?

Word Error Rate (WER) is a metric used to assess the accuracy of speech recognition models by comparing the recognized text to the original transcript. It calculates the minimum number of word insertions, deletions, and substitutions required to transform the recognized text into the reference text, divided by the total number of words in the reference.

Beyond benchmarks

Leaderboards

Image generation

Speech generation

Video generation

Want us to evaluate your model?