The local test rigMac Studio · Apple M4 Max · 64 GB unified memoryView the test rig →
Evaluation lab v2
Foundation · No published v2 results yet.
The archived pilot remains a separate historical evidence set.
Model + harness
Artifacts, screenshots, A/B comparisons and explicit harness configuration.
- H01 · Brand board · 1.0.0
- H02 · Precision edit (multi-step) · 1.0.0
- H03 · Aurora / mesh gradient art · 1.0.0
- H04 · Animated hero · 1.0.0
- H05 · Interactive micro-app · 1.0.0
- H06 · Data visualization dashboard with insights · 1.0.0
Task-level comparison
Planned views: same model at different effort settings; provider defaults; configured maxima; API versus local runtime. No global ranking.
- Result
- Validator evidence, components, completion status, replicates (3 or 5)
- Effort
- Provider-native labels; separate local reasoning configuration
- Measurements
- Latency, wall-clock, input/output/reasoning tokens, API cost
- Runtime
- Backend/version, weights revision, quantization, dtype, chat template, sampling, context, chip, memory, OS
- Local performance
- TTFT, tokens/s, peak unified memory where measured. API cost 0; hardware and energy costs not yet assessed.
Effort curves are comparable only within the same provider and model. Unavailable measurements remain unknown.