Preferred profile · EAGLE MTP · SM121 · 2× GB10
Independent NVFP4 conversion of XiaomiMiMo/MiMo-V2.6-Flash-RL. The preferred serve profile is EAGLE MTP, 3 steps and 4 draft tokens, advertised context 524288. DFlash is an earlier recorded lane, not this profile.
Preferred profile, systems decode
17.31tok/s
EAGLE MTP · think-off · context 524288 · c1 median
Method. Systems throughput lane, 2,048 requested output tokens. Not the 1024-token harness, which measured decode 24.1, 23.6, and 15.6 tok/s. The 69.1 figure is an earlier 8-wide DFlash comparison against the vendor MXFP4 median 56.9. It is not this profile.
NLL, 32,768 tokens
0.0027Δ
1.3387 vs vendor 1.3360 · gate 0.05
Method. Teacher-forced negative log likelihood on calibration rows. Completions endpoint, echo on, one logprob, null BOS logprob dropped, mean over the token logprobs. 32,768 tokens. Lower is closer to the data. The gate is an absolute delta under 0.05.
Weights converted from the vendor MXFP4 checkpoint with NVIDIA ModelOpt public APIs. No third-party quantization code or configs.
Method. QDQ scoring compares candidate activation scales on the runtime quantize-dequantize path, not on a proxy MSE of the weights alone. The conversion check is a block-level exactness count against the expected NVFP4 packing, then a separate shard walk that hashes every quantized linear.
SGLang nightly 20260922-582389ce on two GB10 nodes. The preferred image is the public MTP tag. The parent is that nightly, plus torchcodec, plus one Marlin skip.
Preferred image, public
ghcr.io/r0b0tlab/sglang-mimo26-env:20260922-582389ce-mtp-mm
sha256:84857252a1a9b4196702154ae38eb3cfafdf4cd832795eba18ac2772f8a83f1e
Build. Parent tag 20260922-582389ce-marlin-skip is the nightly plus torchcodec plus the Marlin skip, digest sha256:42737e9dfd3731072c8fd3d65d479ba03381e0e0cb5e171cbab632a8bddeb507. The preferred tag copies mimo_v2.py, mimo_v2_nextn.py, and triton_backend.py onto that parent. The published MTP systems suite ran on the parent plus those three files bind-mounted, not inside the baked tag. Engine pin: SGLang 0.0.0.dev1+g582389cec, torch 2.13.0+cu130, flashinfer 0.6.18.
What it changes. On the NVFP4 MoE path, SGLang allocates w13_blockscale_swizzled and w2_blockscale_swizzled even when the runner is Marlin. Marlin never reads them. It repacks w13_weight_scale and w2_weight_scale in prepare_moe_nvfp4_layer_for_marlin. The patch adds is_marlin() to the same skip already used for TRT-LLM and MegaMoE, and sets those two tensors to null.
Why it is the SM121 path. Packed MiMo experts are served with --moe-runner-backend marlin. Skipping the unused swizzle frees about 9 GiB per rank (1/16 byte per expert weight element at EP=2 on this 309B-A15B MoE). That memory is what the KV pool uses. DFlash still loads through the stock stacked q/k/v map. No second loader image.
Vendor MXFP4 is the control. This NVFP4 build is the candidate. Flags matched. Lanes booted fresh.
| Check | Vendor MXFP4 | This NVFP4 | How it was scored |
|---|---|---|---|
| NLL, 32,768 tok | 1.3360 | 1.3387 | Teacher-forced mean NLL. Echo and logprobs on the completions endpoint. Null BOS dropped. Gate is delta under 0.05. Observed delta 0.0027. |
| GSM8K-200, think-off | 0.970 | 0.965 | Chat endpoint, enable_thinking false, temperature 0, 512 max tokens. Prompt asks for a trailing #### number. Score is exact numeric match of the last number after that marker. Model text is never executed. 200 cached items. |
| Decode, bs=8 | 56.9 median | 69.1 median | Eight concurrent 600-token probes, steady-state decode, identical prompts. Range 63.7–73.5 versus 52.0–60.8. Median ratio is +21%. |
| DFlash accept | 0.95–1.00 | 0.95–1.00 | Accept rate and accept length 1.95–2.00 on both freshly booted lanes, same flags. Speculative decoding was active on the control and on this checkpoint. |
A later 8-wide probe, not the comparison above. On the FINAL3 serve, eight concurrent completions, 400 tokens each, ignore_eos, greedy, prompt def binary_search(arr, x):. Aggregate samples 78.0, 73.8, 95.4 tok/s, median 78.0. That number is aggregate across eight streams. Per-request decode on those reps was 12.1–15.3 tok/s. Do not read 78.0 as single-stream speed, and do not substitute it for the 69.1 comparison.
Main serve profile. One r0b0bench 1.0.0rc2 systems run, think-off, advertised context 524288. EAGLE, 3 steps, 4 draft tokens, top-k 1, draft window 4096, mem-fraction 0.90. Draft-extend CUDA graphs on. Recorded 2026-09-27. Entry mimo26-nvfp4-mtp-500k-mm-systems-20260927.
C1 decode
17.31
tok/s median
Prefill proxy
17,941
tok/s median
TTFT
276.5
ms mean
NIAH
3/3
exact
C1 decode method. OpenAI portable single-stream throughput. Five reps, first dropped, four stable. Requested 2,048 output tokens. Client-side output tokens per second. Median 17.31, mean 17.26. This is not the 8-wide vendor comparison and not the 1024-token harness below.
Prefill method. Same throughput lane. Two stable reps. Median prompt length 22,771 tokens. Median server prompt tok/s 17,941. The harness records this as an end-to-end wall proxy that includes the short decode, not a pure prefill-kernel timer.
Latency method. Streaming chat, one client. Requested 128 output tokens. Five reps, first dropped, four stable. Mean TTFT 276.5 ms. Mean inter-token latency 139.6 ms, p50 139.6 ms, p95 147.4 ms. Mean end-to-end 4.12 s.
1024-token harness, same boot, not C1. Think-off, greedy, one stream, 1,024 new tokens, two repeats. Decode is content tokens divided by time after the first token. End-to-end is completion tokens divided by full request wall time, so prefill is included. Figures are medians of the two repeats.
| Workload | Decode | End-to-end | TTFT | Prompt tokens |
|---|---|---|---|---|
| Short code | 24.1 | 23.8 | 0.59 s | 71 |
| Medium code | 23.6 | 20.5 | 6.59 s | 4,405 |
| Prose | 15.6 | 14.6 | 4.52 s | 3,248 |
| 8k prefill | — | — | 14.2 s | 9,713 |
Decode and end-to-end are tok/s. The 8k row requested one completion token, so it is a prefill measurement: median server prefill 684 tok/s, not a decode rate. Systems c1 above remains 17.31 tok/s at 2,048 requested output tokens.
| Lane | Result | Method |
|---|---|---|
| Canary | 5/5 | Identity, arithmetic, structured output, tool call, and a short needle. Tool-call parser mimo. All five checks passed. |
| BFCL multi-turn | 117/200 | Official BFCL v4 multi_turn_base accuracy, 0.585. Think-off. Two hundred cases. Score is the official evaluator, not a hand grade. |
| BFCL AST | 215/600 | Official score-file headers after an exact filename match. multiple 70/200, parallel 103/200, parallel_multiple 42/200. Micro is the sum, 0.358. Think-off. |
| Concurrency | 26.7 → 93.2 | OpenAI portable ladder. Levels 1, 2, 4, 6. 512 output tokens. Three reps, warmup dropped. Aggregate output tok/s: 26.7, 43.5, 66.6, 93.2. |
| NIAH | 3/3 | 25, 50, and 90 percent of advertised 524288 minus a 256-token reserve. Depths 131008, 262016, and 471628. Exact return of the planted code. Finish reason stop. |
The measured process used the parent image with the loader and draft-extend window-index files bind-mounted. Tag ghcr.io/r0b0tlab/sglang-mimo26-env:20260922-582389ce-mtp-mm is public and copies those same three files. It was not the process that served this suite.
Not the preferred profile. Recorded 2026-09-26. Entry mimo26-nvfp4-final3-500k-systems-20260925. Think-off. Tool parser mimo. Advertised context 524288.
C1 decode
15.48
tok/s median
Prefill proxy
16,535
tok/s median
TTFT
253.5
ms mean
NIAH
3/3
exact
C1 decode method. openai portable single-stream throughput. Five reps, first dropped, four stable. Requested 2,048 output tokens. Client-side output tokens per second. Median 15.477, mean 15.489. One recorded completion was 1,767 tokens from a 40-token prompt. This is not the 8-wide aggregate.
Prefill method. Three reps, first dropped, two stable. Prompt target 14,000 tokens. Measured median prompt length 22,771. Sixteen output tokens. Median server prompt tok/s 16,535. The harness records this as an end-to-end wall proxy that includes the short decode, not a pure prefill-kernel timer.
Latency method. Streaming chat, one client. Requested 128 output tokens. Five reps, first dropped, four stable. Prompt was 26 tokens. Mean TTFT 253.5 ms. Mean inter-token latency 159.2 ms, p50 159.1 ms, p95 172.9 ms. Mean end-to-end 5,824 ms. Finish reason stop on every stable rep. Reasoning tokens 0.
| Lane | Result | Method |
|---|---|---|
| Canary | 5/5 | Identity, arithmetic, structured output, tool call, and a short needle. Tool-call parser mimo. All five checks passed. |
| BFCL multi-turn | 114/200 | Official BFCL v4 multi_turn_base accuracy, 0.570. Think-off. Two hundred cases. Score is the official evaluator, not a hand grade. |
| BFCL AST | 211/600 | Official non-live score-file headers. multiple 71/200 (0.355), parallel 101/200 (0.505), parallel_multiple 39/200 (0.195). Micro is the sum, 0.352. Think-off. |
| Concurrency | 50.2 → 159.6 | OpenAI portable ladder. Levels 1, 2, 4, 6. 512 output tokens. Three reps, warmup dropped. Aggregate output tok/s: 50.18, 66.04, 91.65, 159.65. |
Functional probes on the graph-on MTP boot, think-off. Not a benchmark score. The vision tower is still BF16. Encoders ran: the requests carried modality tokens that a text-only prompt does not.
| Probe | Result | Tokens | What was checked |
|---|---|---|---|
| Image | colors matched | 220 image | 640 by 360 white canvas: green word SEVEN, red circle on the left, blue square on the right. Answer: the word is green, the circle is red, and the square is blue. It did not name SEVEN. 253 prompt tokens, 17 completion tokens. |
| Audio | line matched | 21 audio | 3.26 second clip. Spoken line: The access code is blue heron forty two. The answer repeated that line. 39 prompt tokens, 11 completion tokens. |
| Video | ending matched | 240 video | 2.25 seconds, 12 frames. A circle moves right and changes from red to blue. The on-screen word changes from LEFT to RIGHT. Answer: the circle at the end is blue, and the word on screen is RIGHT. 301 prompt tokens, 18 completion tokens. |
Method. Same MTP process as the systems run. Triton vision attention, torchvision, multimodal enabled, reasoning tokens 0. Controlled media, not a scored suite.
180 questions, all finished with finish reason stop. Longest completion 5,844 tokens. Ceiling 8,192. Think-off. Measured on the earlier FINAL3 serve, not re-run on the 524288 profile.
GSM8K
78/80
97.5%
HumanEval
38/40
95.0%
IFEval
35/40
87.5%
Total
168/180
includes hard reasoning
Method. Kit grader on the Q200 text-180 set for GSM8K, HumanEval, and IFEval. Chat template kwargs enable_thinking false and thinking false. Configured max tokens 8,192. All 180 rows stopped. No row hit the ceiling. Those three families are 80, 40, and 40 items, 151 correct. The 20 hard_reasoning answers were graded by independent review of the recorded responses, 17 correct. The three failures are hard-04, hard-12, and hard-16. Total 168/180. Responses were not regenerated. This is not a core-subset claim, and it is not the MTP systems run.
Throughput on this run. Workers 1. 51,977 completion tokens in 2,254.2 seconds, 23.06 client tok/s. Elapsed includes prefill. A separate 1024-token harness on the same boot, truncated at the token cap, measured concurrency-1 aggregate 27.25 and 26.02 tok/s, per-request decode 27.51 and 26.25, and concurrency-2 aggregate 32.39 and 34.08 tok/s. That harness is not this quality-run rate.
Memory on this boot. Weights 84.436 GB. KV cache 6.575 GB. Startup available 9.166 GB. Full-token pool 1,044,581. SWA pool 20,891. Graph reservations: target-verify 1.495 GB, draft-decode 0.134 GB. Prefill, decode, and draft-extend graphs were 0. KV is the calibrated FP8 pool. Host available at admission was 4.84 GiB and 8.71 GiB, above a 4 GiB floor.
Two probes, two profiles. Both returned the planted values exactly. They are not the same measurement.
3/3
Method. Depths are 25, 50, and 90 percent of max_model_len minus a 256-token generation reserve. That is 131,008, 262,016, and 471,628 tokens. Marker R0B0BENCH_NIAH_SLOT. Pass is an exact return of the code R0B0-NIAH-7K3M. The needle sat near the middle of each constructed prompt (effective fraction about 0.50). Usage confirmed the constructed prompt length on every depth. Finish reason stop. Thirteen completion tokens each.
3/3
Method. Advertised-window needle plus three keys at 33, 66, and 99 percent. Constructed prompt 480,655 tokens. Keys NEEDLE-A ni4h7q2x9, NEEDLE-B k7d2m8v4, NEEDLE-C t3r9w5y1. All three retrieved. Forty completion tokens. Estimated prefill 342.8 tok/s over 1,402 seconds. This probe is not the r0b0bench lane above.
Preferred profile is EAGLE MTP. Licenses below are the ones those projects publish. This page does not relicense their code.