CivilCADBench
Benchmarking Multimodal LLMs for Civil Engineering Drawing Sets
Real infrastructure drawing sets and source CAD files containing over 250,000 entities. CivilCADBench evaluates whether frontier models can read a set, rebuild the editable CAD file hidden behind a monochrome print, and design a new sheet under updated parameters.
Models redraw the print. They lose the file.
We evaluate seven frontier MLLMs at high reasoning effort. The strongest model reads and traces printed content accurately, yet recovers far less of the hidden structure that printing discards.
Where frontier models actually fail
Inspecting generated DXF files side by side with the designers' source files exposes three distinct failure modes that pixel metrics miss.
L1 Single-Page Reconstruction: Geometry F1 100, Layer F1 12
On the bar schedule (top), GPT-6 Astra reproduces every cell accurately while GPT-5.6 Sol mis-scales the table and loses most text. On the pier detail (bottom), both GPT-6 models trace every line (geometry F1 100), yet place reinforcement on the wrong layer (layer-aware F1 12 and 11).
Three task families, in rising difficulty
Each family starts from the printed sheet and demands one more layer of engineering structure that plotting discarded.
Tie every printed value to the object it describes
Six-option multi-select questions (1–4 correct, chance 1/56) reviewed by three civil engineering experts. Every wrong option encodes one named misbinding refuted by the drawing.
- Single-Sheet1,800 items over one page across six binding types: instance, cross-view, table, datum, scope, and view.
- Few-Sheet383 multi-sheet parent questions given evidence pages plus two look-alike decoy sheets.
- Full-Set330 parents posed over the whole ~30-page unit, where the model searches pages with a zoomable viewer.
- ProbesShort-answer read & bind probes (1,953 / 1,989) separate reading individual facts from composing them.
Rebuild the editable CAD file behind the pixels
Given only a 200 dpi monochrome raster with no vector or text layer, the agent writes Python / ezdxf code in a sandbox to output a layered DXF with live dimensions.
- L1 · Page232 bridge and tunnel sheets across small (108), medium (96), and large (28) entity-count tiers.
- L2 · Set28 complete drawing sets (5–14 pages each, 230 pages total) requiring cross-sheet consistency.
L1 input · monochrome raster
Target · 9-layer DXF file
Derive a new sheet under updated parameters
Mirroring real design-office workflows, a derived sibling sheet is withheld as ground truth while the agent receives the reference DXF, context rasters, and new member parameters.
- L3 · Design19 parametric design tasks where bar counts, lengths, geometry, dimensions, and schedule totals must all update coherently.
Given · reference sibling DXF
Withheld · designers' target sheetSandboxed agent loop, deterministic file-to-file scoring
Agents write and execute code inside an isolated container, preview their rendered DXF against the input, and submit once. Scoring uses zero LLM or VLM judges.
ezdxf tools.- Geometry F1curves sampled at 0.25 mm, matched within 1 mm on paper0.25
- Layer-aware F1matched geometry points on the exact same target layer0.30
- Layer sharefraction of entities placed on the nine allowed layers0.10
- Text F1strings within 6 mm, similarity ≥ 0.8, identical numbers0.20
- Dimension F1dimension labels matched by value and position0.15
Inspect the benchmark data yourself
Browse a fixed-seed, stratified sample drawn without looking at any model result, complete with Chinese originals, English translations, wrong-option explanations, layer plots, and fault-injection tables.