Onyx AI Proving Ground

Business AI, graded on real work.

550 business tasks across 6 kinds of work, each with a correct answer we check mechanically. Every model runs the same tasks three times. Ranked on correctness — speed is published, but it never moves a model up.

Leaderboard

11 models · click one for its full results and files
  1. 1
    Qwen 3.8 27B
    Dense · 27B · DFlash2
    MLX 4-bit affine (group 64) · 1× DGX Spark
    99.1%
    545/550
    typical task 23.2s
    DOCXLSPPTPOST
  2. 2
    DeepSeek V4.1 Flash 1017
    MoE/n-gram · 552B backbone total / 8B prefill / 16B decode active · no spec-decode
    2.9bpw EXL3-EAQ · 2× DGX Spark
    98.5%
    542/550
    typical task 29.1s
    DOCXLSPPTPOST
  3. 3
    Gemma 4 31B
    Dense · 30.7B · no spec-decode
    QAT w4a16 compressed-tensors (4-bit weights, 16-bit activations) · 1× DGX Spark
    94%
    517/550
    typical task 18.9s
    DOCXLSPPTPOST
  4. 4
    Flint
    MoE · 39.8B total / ~2.5B active · no spec-decode
    NVFP4 GGUF (Recursive) · 1× DGX Spark
    91.8%
    505/550
    typical task 38.6s
    DOCXLSPPTPOST
  5. 5
    Nemotron 3.5 Bolt 74B-A9B
    MoE · 74B total / 9B active · LiLiCorr
    IQ2_XXS experts + NVFP4 Mamba + Q8_0 attention (24GB pack) · 1× DGX Spark
    91.5%
    503/550
    typical task 27.3s
    DOCXLSPPTPOST
  6. 6
    Gemma 4 26B-A4B
    MoE · 25.2B total / 3.8B active · no spec-decode
    QAT q4_0 (4-bit), 3.8B active of 25.2B total · 1× DGX Spark
    90.4%
    497/550
    typical task 16.1s
    DOCXLSPPTPOST
  7. 7
    Laguna XS 2.1
    MoE · 33B total / 3B active · DFlash
    NVFP4 (poolside) · 1× DGX Spark
    90.2%
    496/550
    typical task 15.8s
    DOCXLSPOST
  8. 8
    LFM3 1.7B
    Dense · 1.7B · no spec-decode
    GGUF BF16 · 1× Mac Studio
    88.2%
    485/550
    typical task 48.3s—
  9. 9
    Muse Glimmer 30B
    Dense · ~29.6B (incl. ~1.8B vision) · DFlash
    NVFP4 W4A4 (Red Hat) · 1× DGX Spark
    87.6%
    482/550
    typical task 37.5s
    DOCXLSPPTPOST
  10. 10
    Laguna S 2.1
    MoE · 118B total / 8B active · DFlash
    NVFP4 (poolside) · 1× DGX Spark
    84.7%
    466/550
    typical task 10.4s
    POST
  11. 11
    Gemma 4 E4B
    Dense · 4.5B effective (8B with embeddings) · no spec-decode
    QAT w4a16 compressed-tensors (4-bit weights, 16-bit activations) · 1× DGX Spark
    59.8%
    329/550
    typical task 3.8s
    XLSPOST

Score = graded answers correct out of 550 tasks × 3 passes. Typical = half of tasks finish faster than this. Files = the report, workbook, deck and social posts the model produced, downloadable on its page. Full method →

The Proving Ground suite — code, tasks, keys, and grading — is proprietary Onyx AI Labs intellectual property. All rights reserved. Published per-run result files are shared for transparency; the bench itself is not open source.

Built on the shoulders of

Shoutouts

People really moving the ball forward on local AI. Our leaderboard sits on their work.

Mia's AI Lab
MiaAI-Lab · model builds & serving recipes

Several rows on this board run Mia's model builds and DGX Spark serving recipes — the recipe links on those model pages go to her repos. The visual language of onyxailabs.com also grew out of one of her public 100 HTML Files experiments.

Her own benchmarks measure performance (speed and throughput) plus traditional academic sets. Ours are business benchmarks — real desk work with mechanical keys. They answer different questions, so the rankings often disagree. Complementary, not competing.

Ash Hart · TensorFold
Local inference engine

TensorFold is the serving engine behind several of our DGX Spark rows. Fast local inference on hardware you control — the stack a lot of this board actually runs on. His work on Mac and DGX Spark interoperability is changing the scale at which local AI can be used.

Want your model on this page?

We run any open-weight or hosted model against the same tasks and publish the result with its configuration — good, bad, or ugly.

Talk to us