Measuring the design factory in the open
Nordform Trueline is an open benchmark of whether the design factory returns the design a brief asked for and stops where it should. The first run, the cases, the graders and every result are published, losses included.
Nordform editorial · 3 min read
When a brief goes through Nordform, a model reads it and proposes a change to a design, deterministic code builds the design, and the registered checks run. We say that this produces the design the brief asked for and stops when it should. Until now we had no number to show for that claim, so we built a benchmark and published it, method and results together.
It is called Nordform Trueline. Before laying a course, a builder sets a true line and checks every course against it.
What it measures
Trueline gives the factory a fixed set of briefs and grades each answer against the answer the case expects. The main number is verified designs: of the cases whose answer is a design, how many ran, were accepted by the factory's own verification, were delivered, and carried the change the brief asked for.
Beside it, never folded into one score:
- Honesty. How often the factory stopped or asked when the brief could not be met, how often it delivered a design when it should have stopped, and how often it refused work it could have done.
- Constraints. Protected dimensions and openings left alone, and no numbers off by a power of ten.
- Files open. Delivered IFC and DXF files opened by IfcOpenShell and ezdxf, readers that share no code with ours.
- Reruns identical, reliability, cost and time.
No model judges anything. Every case has a typed answer to compare against.
The first run
The first run, on 20 October 2026, was a shakedown on the same cases before the corpus was frozen:
- 23 of 28 design cases ended in a verified design, 4 of 5 among the held-out cases.
- In 22 cases that required a stop, the factory delivered a design 0 times, and it refused none of the 38 cases where work was possible.
- All 54 delivered files opened in both independent readers.
- 2 of 53 runs failed.
The comparison Trueline exists for, the same model asked on its own without the factory, was not part of that run. That run comes next, and it will be published whatever it shows. The first run also replayed model answers recorded a few days earlier and ran on an emulated machine, so its times are not production times.
How we keep it honest
About one case in five is held out. Held-out cases are not published, only counted and fixed by a hash, so a number can be checked against cases nobody outside has seen. Every run on the held-out cases is published from now on, and a run is never removed. When a result falls, it stays on the page with the run that caused it.
The cases, the graders, the model-alone runner and every result are open in the Trueline repository: cases and results under CC BY 4.0, code under Apache-2.0. Another design system that answers the same contract can be graded on the published cases.
Trueline covers one kernel today, the timber kit-house frame, and says nothing about the others.
The method, the results and the run history are on Research.