Qwen3.8-27B-Hermes-oQ4
A ready-to-serve bundle: the Nous Research Hermes adapter fused into Ash Hart's TensorFold oQ4 conversion of Qwen3.8-27B, served with the DFlash2 drafter for lossless, ~3–6× faster decode on Apple Silicon. ~234 tok/s on code, ~82 tok/s on prose — a 250-word paragraph in about four seconds. No fusing required at install time — pull, serve, point your agent at it.
Three upstream works, fused into one serving-ready checkpoint
Dense 27B vision-language model, Apache-2.0. 64 layers (48 linear-attention, 16 full-attention), 248k vocab, native 262k context.
ConversionMixed-precision MLX build: 4-bit affine base with 160 sensitive modules at 5-bit and one at 6-bit. Fits a 64 GB Mac with room for the KV cache.
AdapterRL-trained on the Hermes Agent harness (r=32, α=64), merged into the oQ4 weights at serving time so no LoRA is needed to run it.
The Hermes adapter is RL-trained on the Hermes Agent harness. With thinking off, it reaches the base model's thinking-at-max-effort accuracy while generating 3.5–4.4× fewer tokens per task — thinking-level capability at a quarter of the work. Measured by harness-hub / Nous Research on the base weights under the harness (100-step budget, 131k context, T=0.8):
| Benchmark | Qwen3.8-27B base | + Hermes adapter (thinking off) | Δ |
|---|---|---|---|
| SWE-bench Pro (hard 51) | 29.4% | 43.1% | +13.7 |
| Terminal-Bench 2.1 (85 graded) | 44.7% | 58.8% | +14.1 |
Honest scope: these are the adapter's gains on BF16 base weights. oQ4 quantization is lossy, so expect a few percent of divergence, and this bundle's quality was validated by coherence and tool-call spot-checks, not re-benchmarked on SWE-bench/Terminal-Bench. Validate on your own workload before production.
Measured on an M5 Ultra, 256 GB — TensorFold + DFlash2
| Workload | Decode, tokens/s |
|---|---|
| Code, greedy (900-token sustained) | ~234 |
| Code, sampled (T=0.7) | ~181 |
| Prose, greedy | ~90 (88–105) |
| Prose, sampled (T=0.7) | ~82 |
| Plain oMLX oQ6, no drafter (reference) | ~40 |
~5.8× on code, ~2.2× on prose, versus plain decode of the same model. Lossless: greedy output is byte-identical to the target, and Hermes tool-calling fires clean through the drafter.
Why prose is slower, honestly: DFlash2 speeds decode by predicting tokens. Code is highly predictable, so draft acceptance is high; quality prose is less predictable, so acceptance is lower. A cruder uniform-4-bit build posted higher prose numbers partly because its output was more predictable — speed partly inversely tracks quality on prose under speculation. These are one M5 Ultra's numbers, not a cross-platform benchmark; M5 Max/Pro and M1–M4 are slower (bandwidth-bound).
Every choice is a deliberate trade-off, not a default
Quality stays near the base at ~16.6 GB — the sweet spot between oQ6 (22 GB) and a crude uniform 4-bit, which decoded faster on prose at a real quality cost. TensorFold's drafter requires a 4-bit target.
Lossless block-diffusion speculation, 7 tokens per pass. The big decode lever on predictable text. Apache-2.0, so it bundles cleanly.
Ash Hart's native Metal engine with DFlash2 support. oMLX 0.8.0 has classic draft-model speculation but not the dflash method.
The adapter is trained thinking-off. Thinking-on generates ~3× more tokens with no significant accuracy gain, and exhausts context under an agent harness.
The model's native max. Agentic sessions grow long, and a 256 GB box has RAM for the full KV. On smaller Macs, start lower and watch memory pressure.
Single-stream daily driver: same decode speed as auto, less RAM. Use --parallel auto when serving concurrent callers.
Local-only by default — a private mind, no LAN exposure. Bind wider only deliberately.
The standard fuse tool fails on this combination; the weight-level path preserves Ash's per-module precision, the vision tower, and the original sharding (below).
Pull, serve, done
brew install ashhart/tensorfold/tensorfold # TensorFold 1.3.0 native engine tensorfold pull OnyxAILabs/Qwen3.8-27B-Hermes-oQ4 # this bundle (pre-fused) tensorfold pull z-lab/Qwen3.8-27B-DFlash2 # drafter (Apache-2.0) tensorfold serve OnyxAILabs/Qwen3.8-27B-Hermes-oQ4 \ --drafter z-lab/Qwen3.8-27B-DFlash2 \ --host 127.0.0.1 --port 8080 \ --no-thinking --context 262144 --parallel 1
# ~/.hermes/config.yaml — point Hermes Agent at it
model:
default: Qwen3.8-27B-Hermes-oQ4
provider: qwen-hermes-local
providers:
qwen-hermes-local:
api: http://127.0.0.1:8080/v1
api_key: not-needed
context_length: 262144
transport: chat_completionsOpenAI-compatible /v1/chat/completions (also /v1/responses and Anthropic /v1/messages); /health exposes live decode tokens/s, memory, backend, and device. Hermes Agent needs ≥64k context. Keep thinking off.
The one real gotcha, and the fix
The Hermes adapter is a LoRA, so it can be merged into the base weights to produce a standalone checkpoint. The standard tool (mlx_lm.fuse) fails on this combination for two reasons: harness-hub ships a PEFT-format adapter while mlx_lm.fuse expects mlx-format (on this VL model it raises AttributeError: num_layers), and even a manual module-level fuse followed by save drops all 333 vision-tower tensors, yielding a broken text-only checkpoint whose config still declares a vision model. So we fuse at the weight level, touching only the LoRA-target tensors:
- 1
Load every base oQ4 tensor and the PEFT LoRA (lora_A:[r,in], lora_B:[out,r], fan_in_fan_out=false).
- 2
For each of the 256 target modules, map the PEFT key to the model tensor path: the 16 full-attention layers' q/k/v/o plus every layer's MLP gate/up/down. The 48 linear-attention layers have no q_proj, so they are not LoRA targets.
- 3
Dequantize that module's base weight at its own bits and group size, read from the config quantization map (4-bit base, 5/6-bit overrides).
- 4
Add the LoRA delta: W' = W + (alpha/r)·(lora_B @ lora_A), with alpha/r = 2.0. No transpose.
- 5
Requantize W' at the same bits and group size, so the mixed precision is preserved exactly — no global re-quant, no precision flattening.
- 6
Write the tensors back with the base's original sharding, index, and config. Non-target tensors (vision tower, embeddings, norms) are copied untouched.
Verified end-to-end on Apple Silicon: coherent correct code generation, and Hermes tool-calling through the drafter (get_weather({"city":"Reykjavik"}), finish_reason=tool_calls). The reproducible script (fuse-hermes-weights.py, MLX) is included in the bundle and works for any oQ precision.
Limitations, honestly
- —Thinking off is the trained configuration. Thinking-on generates ~3× tokens with no significant accuracy gain.
- —Speed is workload-dependent: speculative decoding accelerates predictable text (code) far more than prose.
- —The drafter was trained at 4096 context; acceptance may drop on very long generations (prefix caching mitigates long prompts).
- —Quantization is lossy. Near-ties can flip; validate quality on your workload, and use a higher-precision variant when quality matters more than memory.
- —The vision tower is present in the checkpoint but not served by the text-focused TensorFold serve path; route images to a vision-capable helper if needed.
- —Not safety-tuned by us; inherits base-model biases and training-data opacity. Not for safety-critical use.
Built on published work — every author credited
Authored by Mike Gibbs + Iris Vale — Onyx AI Labs (X @CMichaelGibbs). Integration, weight-level fuse, Apple-silicon serving validation, and packaging by us. Built following MiaAI-Lab's contributor-attribution playbook: every author credited, every upstream traced, nothing relicensed, no invented identities. Design, training, and benchmark credit belongs to the authors below.
| Component | Source | Credit | License |
|---|---|---|---|
| Integration (this bundle) | this page / repository | Mike Gibbs + Iris Vale — Onyx AI Labs (X @CMichaelGibbs): weight-level fuse, M5 Ultra validation, serving kit, card | Apache-2.0 |
| Base model | Qwen/Qwen3.8-27B | Qwen team, Alibaba — model design, training, benchmarks | Apache-2.0 |
| oQ4 conversion | TensorFold/Qwen3.8-27B-oQ4 | Ash Hart / TensorFold — mixed-precision MLX conversion | Apache-2.0 |
| Hermes adapter | harness-hub/Qwen3.8-27B-hermes-adapted | Nous Research / harness-hub (Hermes team) — AsyncGRPO RL on the Hermes Agent harness | Apache-2.0 |
| DFlash2 drafter | z-lab/Qwen3.8-27B-DFlash2 (mirror of incoai) | Inco AI / z-lab — DFlash2 block-diffusion drafter; DFlash paper: Chen, Liang, Liu (ICML 2026) | Apache-2.0 |
| Inference engine | ashhart/TensorFold | Ash Hart — native Metal/CUDA engine, DFlash2 speculative support | Apache-2.0 |
| Attribution method | MiaAI-Lab contributor-attribution playbook | Mia / MiaAI-Lab — every author credited, every upstream traced, nothing relicensed | — |
All components are Apache-2.0 and ungated; a copy of each upstream license is preserved in the bundle. Please also cite the upstream works — Qwen3.8-27B, the Hermes adapter (harness-hub release), DFlash2 (Inco AI), and the DFlash paper (Chen, Liang, Liu, ICML 2026) — and review upstream licenses before redistribution or commercial use.
Run it. Read the code. Cite the people who built it.
The bundle, the fuse script, and this recipe are Apache-2.0. If you improve something, send it upstream — that is how this stack got built.