The local test rigMac Studio · Apple M4 Max · 64 GB unified memoryView the test rig →
Evaluation lab v2
Foundation · No published v2 results yet.
The archived pilot remains a separate historical evidence set.
Model only
Identical tasks and validators for managed APIs and self-hosted models. Results are shown per task and replicate.
- M01 · Resource scheduling · 1.0.0
- M02 · Twelve-day operations planning · 1.0.0
- M03 · Code debugging and repair · 1.0.0
- M04 · Lossless precision transformation · 1.0.0
- M05 · Evidence audit · 1.0.0
- M06 · Multi-source synthesis · 1.0.0
- M07 · Rule discovery and generalization · 1.0.0
- M08 · Visual route planning · 1.0.0
Task-level comparison
Planned views: same model at different effort settings; provider defaults; configured maxima; API versus local runtime. No global ranking.
- Result
- Validator evidence, components, completion status, replicates (3 or 5)
- Effort
- Provider-native labels; separate local reasoning configuration
- Measurements
- Latency, wall-clock, input/output/reasoning tokens, API cost
- Runtime
- Backend/version, weights revision, quantization, dtype, chat template, sampling, context, chip, memory, OS
- Local performance
- TTFT, tokens/s, peak unified memory where measured. API cost 0; hardware and energy costs not yet assessed.
Effort curves are comparable only within the same provider and model. Unavailable measurements remain unknown.