CAD Benchmarks — the open benchmark for AI on CAD
LLM models are measured by driving the CADceptional webcad_* MCP tool environment against a live parametric kernel — graded on exact geometry, editability, and the tool-call trajectory. Powered by CADceptional · pre-alpha
Why it's different — the benchmark is the MCP host, so it scores editability (do the model's named parameters actually drive the geometry?) and the full tool-call trajectory, not just a static output file. A model can build the exact right shape and still not solve the task if it isn't editable — the distinction no other CAD-AI benchmark makes. It's rigorous about its own fairness, too: building it live surfaced (and fixed) a bug where the core editability metric was accidentally testing internal-API memorization instead of CAD ability.
The field at a glance
One bar per model — length is the solve rate over its task set, the whisker is the 95% Wilson interval. Hover or focus a row: the confidence band expands to show how wide the uncertainty really is at small n, with the full readout in a tooltip. ◆ marks the mean part score; the dashed bounds are the controls (oracle ceiling, null/baked/escape floor), which are references, not competitors.
Leaderboard
| # | Model / agent | Solve % (95% CI) | Mean part score | Tasks |
|---|
The badge shows how each model ran: OpenRouter · API = native API tool-calls (metered/free API models); Max plan · MCP = Claude versions driving the same tools via native MCP on the Anthropic Max plan (free — these carry Claude Code agent scaffolding, so comparable but not perfectly like-for-like). Solve % is over 24 public tasks, so the 95% confidence interval (Wilson) is wide — at this n a few-point gap is not significant. Ranges shrink as the suite and run counts grow.
Runs — model builds + oracle reference parts (click a card for the rotatable 3D)
Each run: the model builds the part through MCP tool calls; we grade the result and — uniquely — the trajectory (how it worked). Open any result's actual construction in CADceptional to inspect the feature history.