CAD Benchmarks — the open benchmark for AI on CAD
LLM models are measured by driving the CADceptional webcad_* MCP tool environment against a live parametric kernel — graded on exact geometry, editability, and the tool-call trajectory. Powered by CADceptional · pre-alpha
Why it's different — the benchmark is the MCP host, so it scores editability (do the model's named parameters actually drive the geometry?) and the full tool-call trajectory, not just a static output file. A model can build the exact right shape and still not solve the task if it isn't editable — the distinction no other CAD-AI benchmark makes. It's rigorous about its own fairness, too: building it live surfaced (and fixed) a bug where the core editability metric was accidentally testing internal-API memorization instead of CAD ability.
The field at a glance
One row per model — the bar is the point estimate (solve rate over its task set); the whisker beneath it spans the 95% Wilson confidence interval: the true rate is plausibly anywhere between its two end caps. At today's small n the whiskers are wide on purpose — that width is the honesty. ◆ marks the mean part score; the dashed bounds are the controls (oracle ceiling, null/baked/escape floor), which are references, not competitors. Hover or focus a row to drop down that model's per-operation breakdown (creation, holes, pockets, booleans, …), each with its own CI; the ⊞ toggle expands every model into the full matrix. Per-operation scores are over the tasks that exercise that operation — tasks mix operations, so sub-scores overlap and their n is small.
Per-operation solve rate — over the tasks whose reference construction exercises each operation. Tasks mix operations, so groups overlap and share failures; cross-group comparison within one model mostly restates task difficulty (booleans exist only in L5–L8, finishing only in L1–L3) — compare a group across models. Small n ⇒ wide CIs; that width is the honesty.
Leaderboard
| # | Model / agent | Solve % (95% CI) | Mean part score | Tasks |
|---|
The badge shows how each model ran: OpenRouter · API = native API tool-calls (metered/free API models); Max plan · MCP = Claude versions driving the same tools via native MCP on the Anthropic Max plan (free — these carry Claude Code agent scaffolding, so comparable but not perfectly like-for-like). Solve % is over each model's own task set (the public suite is a 60-task L1–L8 ladder plus a purpose-built diverse-capability tier; coverage varies per model — see each row's n), so the 95% confidence interval (Wilson) is wide — at this n a few-point gap is not significant. Ranges shrink as the suite and run counts grow.
What the suite reveals — a shared blind spot. Across 457 archived runs, every tested model (frontier and free) that builds a bolt circle reaches for an inline pattern arg on the hole tool — but webcad has no such option (patterns are a separate call) and silently ignores it. Circular hole-patterns are dropped 12/12 (verified by replay), so the bolt-circle tasks trip everyone identically. It's concordant — it doesn't bias rankings — but surfacing exactly this kind of real, shared model behavior is what the benchmark is built to do.
Why it ranks tiers, not near-peers. Profiling the frontier-tested tasks by how many frontier models solve each: most saturate (every frontier model passes — zero discrimination), and only a minority discriminate (some pass, some fail). The bulk of the suite is too easy to tell frontier models apart, so the ranking signal lives in that discriminating minority — and there, near-peers like Fable-5 and Sonnet-5 trade (each wins tasks the other fails). That's the structural reason a two-point gap between frontier peers isn't certifiable here: it's not more runs that's needed, but a larger discriminating subset (harder tasks).
Capability axes probed — the frontier is a peer group on all of them
The leaderboard above is the parametric-construction solve rate. But “is model X better at CAD than model Y” spans more than one skill, so beyond the public suite we ran targeted probes across several distinct capability axes (each with a trusted golden + deterministic grader). The consistent result: the frontier models — Fable-5, Sonnet-5, Opus-4-8 — come out as statistical peers on every axis. The suite discriminates capability tiers (frontier ≫ Haiku ≫ free models), not near-peer frontier models. Per-axis evidence lives in the repo’s RANKING-VERDICT.
| Axis | What it probes | Frontier | Tier signal |
|---|---|---|---|
| Parametric construction | build-to-spec, L1–L8 ladder | peers | frontier ≫ Haiku ≫ free |
| Complex geometry (FreeCAD-graded) | freeform loft · helical spring · 5-part bolted assembly · involute spur gear from first principles · coupled gear train | peers (7 regimes all tie) | Haiku RED on the involute gear |
| Parametric binding | named params that actually drive geometry (editability) | peers (symmetric) | Haiku hardcodes |
| Design / reasoning | “design a part satisfying these constraints” — many valid answers, constraint-graded | peers (D1–D5 tie) | Haiku 1/6 (can't build or parametrize) |
| Finishing | fillet / chamfer, verified feature-based | peers | Haiku hardcodes params |
| Error-recovery (debugging) | diagnose + fix a broken parametric build from an under-specified symptom | peers | Haiku fails the cascade defect |
| Trajectory efficiency | turns-to-solve for the same correct part | peers (noisy) | — |
Honest bottom line. No axis we could build separates the frontier — Fable-5 ≈ Sonnet-5 is a genuine tie, not a test-quality failure. Harder build-to-spec tasks saturate both; the one place they differ, trajectory efficiency, is within the suite’s run-to-run noise. What the benchmark does do reliably is separate capability tiers — and it surfaces real, shared model behaviors (e.g. the bolt-circle blind spot above).
Runs — model builds + oracle reference parts (click a card for the rotatable 3D)
Each run: the model builds the part through MCP tool calls; we grade the result and — uniquely — the trajectory (how it worked). Open any result's actual construction in CADceptional to inspect the feature history.