[2] progress

match0 / —

your votes0

unseen—

[3] vote mode

all comparisons harness (same profile) default / mcp / skills

choose which configurations can be matched

[4] judge by

taskrequested elements present, convincingly combined

compclear, intentional, readable at a glance

finishmaterials, lighting, detail resolved

[5] filters: all

harnessmodeleffort

identities sealed until vote

[6] task

Loading comparison…

Loading task instructions…

best follows the task · strongest overall quality

[A] candidate

[B] candidate

VOTEmatch —

blind · sides randomized

[2] ranking mode

Quality

Compare

Efficiency

[3] scope

[4] / search · filters: all

Option counts show matching configurations with the other filters applied. Zero-match options are unavailable.

modelharnesseffortprovidermin sample

Keyboard shortcuts

R: Reset filters
/: Focus search
J / K: Select next / previous row
Enter: Open the selected row when page background has focus
N / P: Next / previous page

Shortcuts are inactive while typing in a field, while a dialog or navigation menu is open, or while Ctrl, Alt or Command is held.

[5] rankings

rankings · quality

Quality ranks models by human preference, grouped by reasoning effort.

[7] leaderboard · by quality (elo)

effort colorsoff / noneminimal / lowmediumhighxhighmax

W = wins · L = losses · T = tie / both good · B = both bad.

Configuration rankings
			cost/run · tokens	confidence
Loading standings…

Ratings begin at 1500. Model × Effort pools across harnesses and profiles; Harness uses direct same-model, same-effort comparisons. Configurations with fewer than five votes are marked provisional.

[2] totals

—total spent · all runs

—tokens used · all runs

runs—

matching—

[3] filters: all

Option counts show matching runs with the other filters applied. Zero-match options are unavailable.

modelharnesseffortproviderprompt

Keyboard shortcuts

R: Reset filters
/: Focus search
J / K: Select next / previous run
Enter: Open selected run when page background has focus
C: Compare selected run
N / P: Next / previous page

Shortcuts are inactive while typing in a field, while a dialog or navigation menu is open, or while Ctrl, Alt or Command is held.

[4] run explorer

runs

search and filter imported runs, then open one to inspect its render, 3d model and recorded metrics.

[5] runs

Sort by Order Unknown values sort last.

effort colorsoff / noneminimal / lowmediumhighxhighmax

Imported run history
run · date	model · harness	effort	prompt	time · cost	agent activity	geometry	compare
Loading run history…

Loading run history…

[A] configuration

model

effort & harness

effortharness

[B] configuration

model

effort & harness

effortharness

[2] benchmark prompt

prompt

Prompt availability uses each side’s selected effort and harness.

select a prompt to inspect each side’s render and 3d scene.

Keyboard shortcuts

S: Swap configurations A and B

Shortcuts are inactive while typing in a field, while a dialog or navigation menu is open, or while Ctrl, Alt or Command is held.

[3] compare configurations

compare configurations

loading model data…

[2] prompt version

esc back to history

[3] prompt version

loading prompt…

the exact instructions given to the agent. the benchmark task replaces $@.

[4] prompt text

[5] runs using this prompt

[2] model information

esc back to rankings

[3] model

loading model…

[4] stats · all runs

[5] by effort setting

[6] by prompt

[7] runs

[2] prompt information

esc back to history

[3] prompt

loading prompt…

[4] prompt text

[5] stats · all runs

[6] by model and effort

[7] runs

[2] run information

[3] actions esc back to history

← history

run / —

Loading run…

[5] render preview · 2d

Preview unavailable

[6] scene model · 3d/glb

drag to orbit · scroll to zoom · r reset

[7] stats

[8] session timeline · loading trace…

loading session activity…

[2] show

[3] scope model minimum votes per setup

Each side must have this many votes for its model, effort, harness and tool configuration. Votes are pooled across prompts. The default of 10 excludes setups with fewer votes.

Checking how many comparisons this filter hides…

[5] model insights

what changes the result

each comparison holds every other setting fixed and changes one: reasoning effort, harness, blender mcp or skills. elo differences are in rating points; cost, tokens and runtime are geometric-mean ratios of the changed setup against the base (2× means twice as much). the smaller line under each value is the range across the matched setups.

Loading insights…

[3] research method

how the benchmark works

the evaluation isolates the full tool-using stack while keeping the voting experience simple and blind.

[4] pipeline

01[controlled]standardized task
every run begins with the same natural-language blender task.
02[varies]agent builds
models work through their assigned harness and reasoning setting.
03[hidden until vote]blind comparison
identities and configuration details remain hidden during voting.
04[calculated]stack-level elo
pairwise outcomes update ratings for each configuration.

[5] what is fixed · what varies

controlled: prompt, benchmark scene, comparison protocol, and vote outcomes.
what varies: model, reasoning effort, harness, and the resulting blender scene.
after voting: identities and technical run metrics are revealed without moving the renders.
low samples: new configurations remain marked provisional until enough votes accumulate.

[8] known limits

rating order: Elo is replayed in vote order. Reordering the same outcomes can change ratings. Inspect current ratings and vote records.
shared outcomes: Tie / both good and both bad can shift the overall rating level through the reference adjustment below. They do not just exchange points. Compare W/L/T/B outcome counts.
blindness limits: Identities are hidden during voting, but public renders can be recognized. Browse the public runs and renders.
uneven coverage: Models and configurations have different numbers of runs and votes. Matched comparisons can cover only a few setups. Inspect matched setups, ranges and vote-filter exclusions.
vote limits: Each browser votes once per pair, and requests are rate-limited per network address. This does not establish that every vote comes from a different person. Inspect recorded vote totals; individual voter identities and a raw vote-event export are not public.

[6] rating formulas

Every entry starts at 1500 Elo. For eligible same-task votes between different entries, the expected score for A and the rating update are:

E_A = 1 / (1 + 10^{(R_B − R_A) / 400})

Δ = round(24 × (S_A − E_A))

R′_A = R_A + Δ; R′_B = R_B − Δ

K-factor = 24. R is the rating before the vote. S_A is 1 when A wins, 0 when B wins, and 0.5 for tie / both good or both bad. Updates are rounded to integer Elo points.

Shared outcomes: the tie button records “both good.” After the pairwise update, each entry receives a reference adjustment for both good (Q = 1) or both bad (Q = 0):

E_ref(R′) = 1 / (1 + 10^{(1500 − R′) / 400})

R″ = R′ + round(8 × (Q − E_ref(R′)))

Reference K-factor = 8. At equal ratings of 1500, an A win produces 1512 / 1488; both good produces 1504 / 1504; both bad produces 1496 / 1496.

Ranking pools: full stack rates model × effort × harness × profile. Model × effort pools across harnesses and profiles and ignores votes within the same model × effort entry. Harness ratings use same-model, same-effort, same-profile votes across different harnesses and exclude shared outcomes entirely. Profile ratings use same-model, same-effort, same-harness votes across different profiles; shared outcomes use the half-point update without the reference adjustment.

Win rate is wins / (wins + losses); both-good and both-bad outcomes are excluded. Explore the ranking pools or inspect full-stack ratings as JSON.

[7] uncertainty and sample size

Confidence intervals are not currently calculated. There are no 95% intervals or significance tests for Elo ranks. The “provisional” label means fewer than five votes; “established” means at least five and is not a statistical confidence guarantee.

Insights hold other configuration factors fixed and summarize differences across matched setups. Elo uses an arithmetic mean of rating differences. Cost, tokens and runtime use a geometric mean of available positive ratios: exp(mean(log(after / before))). Missing measurements are excluded from that metric.

Ranges show the smallest and largest observed setup values, not confidence intervals. Setups can share models and runs, and votes can come from repeat voters, so observations are not necessarily independent. Small rating gaps do not establish a reliable ordering.

Inspect vote counts and provisional ratings · Explore matched setup counts and ranges · Inspect run coverage, costs and runtimes · download matched-comparison data as JSON.

[2] supported agents

pi · omp · omp-mcp
omp-mcp-skill · codex · opencode
claude-code · claude-code-mcp

images pulled on first use

[3] contribute a run

run the benchmark locally

every run starts in a pinned docker image: one prompt per fresh container with blender and the selected agent, then the run is submitted separately.

before you start

You need a terminal, Bash, Python 3, and Docker installed and running. Configure credentials for the agent and model provider you plan to use. Model API calls may incur charges. Blender runs inside the downloaded container.

Download bench-run, open a terminal in its download folder, and run the steps below one at a time. PROVIDER/MODEL:high is a placeholder: replace it with your provider, model identifier and a supported reasoning effort. For a model without reasoning, use --model PROVIDER/MODEL --no-reasoning.

Already have a run directory? Download submit-run.sh and use submit-run.sh --dir <path-to-run>.

Before uploading: session logs can contain prompts, tool output and local paths. Inspect the completed run directory before running the submit step.

[4] quick start

Edit the highlighted model value before running. Run each step separately.

# 1. Make the downloaded launcher executable
$ chmod +x bench-run
# 2. Replace PROVIDER/MODEL:high, then generate one run
$ ./bench-run pi --model PROVIDER/MODEL:high --result-file benchmark-run-path.txt worldtree
# 3. Read the actual output directory; inspect its files and logs
$ cat benchmark-run-path.txt
# 4. After inspection, submit that exact run directory
$ ./bench-run submit "$(cat benchmark-run-path.txt)"

download bench-run requires docker · agents: pi, omp, omp-mcp, omp-mcp-skill, codex, opencode, claude-code, claude-code-mcp

[5] what the run does

images for pi, omp, codex, opencode and claude-code are pulled on first use.
only the selected agent’s credentials are copied into the container.
no cpu or gpu rendering is assumed; the actual blender environment is recorded with the session, preview, glb and blend file.
the launcher prints the run directory and saves it to benchmark-run-path.txt. Its numbered suffix increases for each new run; the submit command reads the saved path.

[6] after you submit

run locallybench-run executes the prompt in a fresh container
inspect, then submitreview the files in the saved run directory, then upload it with bench-run submit
private staging queueuploads wait here, not yet public
administrator review publishedthe run appears in history and enters voting

[3] about the benchmark

preference data for a more open 3D future.

Render Arena is an open benchmark for comparing real Blender runs across the complete agent stack. Votes are stored against an anonymous browser token so one voter cannot submit the same comparison twice. Read more