Massive hybrid MoE fashions like Nemotron-3-Tremendous are correct however costly to serve. Their energetic parameters, KV cache, and Mamba state cap what number of customers a node can maintain at a given per-user token price. NVIDIA AI staff has launched Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Tremendous. The father or mother mannequin has 120.7B complete and 12.8B energetic parameters. The compressed mannequin has 75.3B complete and 9.3B energetic parameters.
The deployment goal was mounted earlier than the structure search started. Goal one was 2x server throughput at 100 tokens per second per person. Goal two was 8 concurrent 1M-token requests on a single H100. Three checkpoints on Hugging Face: BF16, FP8, and NVFP4.
TL;DR
- 120.7B/12.8B energetic compresses to 75.3B/9.3B energetic, with the 88-block hybrid format preserved.
- 8xB200 complete throughput rises 1.60x to 2.14x over Tremendous at matched NVFP4 and matched person throughput.
- Single-H100 1M-token concurrency goes 1 to eight, pushed by a 70 GB to 44.5 GB weight drop.
- Iterative Puzzle beats single-step Puzzle by 0.57 common factors on the identical compression goal.
- Area-Onerous-V2 (-4.2) and SWE-Bench (-2.6) are the actual prices; RULER and AA-LCR barely transfer.
Nemotron-Labs-3-Puzzle-75B-A9B
Nemotron-3-Tremendous is a hybrid Mamba-Transformer MoE mannequin. Puzzle-75B-A9B preserves the father or mother’s block format precisely. It has 88 blocks: 40 Mamba, 40 MoE, and eight consideration blocks.
What modified is capability inside these blocks:
| Amount | Tremendous | Puzzle-75B-A9B | Ratio |
|---|---|---|---|
| Complete parameters | 120.7B | 75.3B | 62.4% |
| Lively parameters | 12.8B | 9.3B | 73.1% |
| Mamba SSM state dimension | 128 | 96 | 75% |
| MoE routed skilled intermediate dimension | 2688 | 1280-2688 | Imply 59.9% |
| Activated routed specialists per token | 22 | 4-18 | Imply 50% |
| Lively routed skilled capability (relative) | 100% | 8.7%-62.3% | Imply 30.9% |
The variety of routed specialists, the shared skilled dimension, and the MoE latent dimension are unchanged. Consideration layers had been left untouched. The proposed analysis’s acknowledged purpose is that Nemotron-3-Tremendous is already very KV-cache environment friendly. Mamba layers had been pruned uniformly, as a result of inference frameworks don’t help a distinct SSM state dimension per layer.
The consequence will not be a uniformly scaled-down trainer. The above determine exhibits the allocation throughout depth. Puzzle preserved capability in chosen center and late layers, and minimize laborious elsewhere.
Benchmark and Efficiency
The beneath desk reviews Pareto-optimal complete throughput on a single 8xB200 node, with single-step decoding.
| State of affairs (in/out) | UT flooring | Tremendous (tok/s) | Puzzle-75B-A9B (tok/s) | Increase |
|---|---|---|---|---|
| 50K / 2K | >= 100 | 5,128 | 8,210 | 1.60x |
| 50K / 2K | >= 125 | 3,784 | 6,412 | 1.69x |
| 50K / 2K | >= 150 | 2,532 | 4,523 | 1.79x |
| 8K / 64K | >= 100 | 20,939 | 42,601 | 2.03x |
| 8K / 64K | >= 125 | 13,074 | 27,918 | 2.14x |
| 8K / 64K | >= 150 | 8,522 | 18,047 | 2.12x |
Each fashions had been served at matched NVFP4 weights, FP8 KV cache, and FP16 Mamba state. The hole subsequently displays compression, not a change in numeric format. The prefill-heavy 50K/2K regime good points least. The decode-heavy 8K/64K regime good points most.
On a single 8xH100 node at UT = 100, the good points are smaller. They’re 1.91x on 50K/2K and 1.82x on 8K/64K. Each fashions there use FP8 weights, FP8 KV cache, and FP32 Mamba state.
On a single H100 at 1M context, the binding constraint flips from compute to reminiscence. Tremendous’s NVFP4 weights occupy about 70 GB of the 80 GB HBM finances. Every 1M-token request provides about 4 GB of KV cache. Efficient concurrency is subsequently 1.
Puzzle-75B-A9B’s NVFP4 weights occupy about 44.5 GB. Consideration format is unchanged, so per-request KV value is unchanged. Concurrency at 1M rises to eight. Mixture decode throughput at that concurrency is roughly 4x Tremendous’s single-request throughput. Prefill of a 990K-token immediate is about 1.2x sooner.
How Iterative Puzzle Works
Puzzle is a decomposed neural structure search framework, applied right here as Puzzletron. It defines a discrete search area of other layer implementations. Every various will get a top quality rating. A mixed-integer program then selects one various per layer below a deployment constraint.
Three pruning methods type the search area:
- Intermediate channel pruning: Channels inside every routed skilled are ranked by contribution to the skilled’s output. All specialists inside one MoE layer are pruned to a uniform dimension, for kernel compatibility.
- High-k discount: The variety of specialists a token is routed to varies per layer, as much as the father or mother’s ok=22.
- Mamba SSM pruning: The SSM state dimension drops from 128 to 96 channels.
The SSM result’s measured. Dropping 128 channels to 96 speeds the SSM kernel 1.2x to 1.3x throughout decode. This holds at batch sizes between 8 and 512. Channels had been ranked by estimated contribution to the Mamba layer output. The estimate averaged over 67M tokens of validation information. Appendix A exhibits this beats random channel choice below aggressive pruning.
The unique formulation assumes alternative high quality impacts are roughly additive. Every candidate block is scored contained in the unmodified father or mother. That ignores higher-order interactions between replacements.
Iterative Puzzle alternates bounded compression with quick data distillation restoration. It builds a sequence M0, M1, … MR as a substitute of leaping to the goal. Scores are recomputed in opposition to the present compressed mannequin, not the unique father or mother.
Three phases had been used:
- MoE weights to 75% of trainer capability, Mamba SSM state to 75%. Healed for 24B tokens.
- MoE weights to 60% of trainer capability. Healed for 43.2B tokens.
- Activated routed-expert finances to 50%, allotted heterogeneously. Healed for 52.8B tokens.
The above desk compares this in opposition to a single-step Puzzle baseline on the identical goal. The three-step process averages 69.05 throughout ten benchmarks, in opposition to 68.48. Positive factors seem on MMLU-Professional, GPQA, HLE, AA-LCR, LiveCodeBench, SciCode, and RULER-256K. IFBench-Instruction fell 0.2 factors and IFBench-Immediate fell 0.5.
Restoration: Distillation, RL, and Verbosity
Information distillation ran on 30% pretraining information and 70% SFT information from Nemotron-3-Nano. In the course of the Puzzle part, KD used a 32K sequence size. Restoration then educated at 128K, and scaled to 512K. The finances was as much as 100B tokens, with a 16M-token world batch, in Megatron-LM.
RL post-training adopted Stage 2 of the Nemotron-3-Tremendous RL pipeline, targeted on software program engineering. Part 2.1 did single-step tool-use comparability. Part 2.2 moved to end-to-end sandbox RL, the place brokers run as much as 200 turns. Each phases used a KL penalty of 0. The staff swept studying charges, then averaged the ensuing weights.
The above Determine 4 exhibits what every stage contributed. Quick-context KD recovers most classes to over 97% of Nemotron-3-Tremendous. Lengthy-context KD then lifts long-input and long-generation benchmarks particularly. The analysis staff states that RL’s influence in these experiments was small.
Verbosity is the quiet element. After the final Puzzle iteration, the mannequin generated 132% of Tremendous’s token rely. That fell to 99% after the total restoration pipeline.
Deployment: Quantization and Multi-Token Prediction
Two post-training quantization recipes had been produced: FP8 W8A8 targets Hopper and NVFP4 W4A4 targets Blackwell.
| Element | BF16 baseline | FP8 checkpoint | NVFP4 checkpoint |
|---|---|---|---|
| Sparse and shared MoE GEMMs | BF16 | FP8 | NVFP4 |
| Mamba GEMMs | BF16 | FP8 | FP8 |
| Mamba SSM cache | FP32 | FP32 | FP16+SR |
| KV cache | FP8 | FP8 | FP8 |
| Router | FP32 | FP32 | FP32 |
| Consideration QKV/output, MoE latent projections, LM head | BF16 | BF16 | BF16 |
Each recipes calibrated on 256 post-training SFT samples. NVFP4 used max calibration, not the AutoQuantize sensitivity search used for Tremendous. The ensuing checkpoint is barely extra aggressively quantized, and carried out equally.
NVFP4 will not be natively supported on Hopper. It’s nonetheless used for the 1M-context H100 goal, as a result of HBM capability binds there.
Puzzle-75B-A9B inherits a shared MTP head from Tremendous. Parameters are shared throughout MTP steps, so one head applies recursively at inference. Transferring Tremendous’s educated head instantly gave comparable acceptance lengths.
The analysis staff then identifies a training-inference mismatch. Trainer-forced MTP coaching feeds the total shifted hidden-state sequence. Autoregressive drafting as a substitute feeds a combination of target-model and MTP-generated hidden states. Acceptance charges fall at deeper draft positions.
Continued coaching on the transferred head addresses this. On SPEED-Bench at draft size 7, common acceptance size rose from 3.45 to 4.34. That’s roughly 25% to 30%, concentrated at later draft positions. In contrast to Tremendous, the NVFP4 checkpoint barely degrades: 4.31 in opposition to 4.34.
The place Compression Helps and The place It Hurts
| Benchmark (BF16) | Tremendous | Puzzle-75B-A9B | Delta |
|---|---|---|---|
| MMLU-Professional | 83.8 | 82.4 | -1.4 |
| AIME25 (no instruments) | 92.2 | 89.7 | -2.5 |
| GPQA (no instruments) | 80.5 | 78.6 | -1.9 |
| LiveCodeBench | 82.1 | 81.1 | -1.0 |
| SciCode (subtask) | 42.3 | 40.6 | -1.7 |
| SWE-Bench (OpenHands) | 59.5 | 56.9 | -2.6 |
| Area-Onerous-V2 | 72.8 | 68.6 | -4.2 |
| AA-LCR | 56.8 | 56.9 | +0.1 |
| RULER 1M | 93.9 | 92.2 | -1.7 |
| MMLU-ProX | 79.5 | 77.5 | -2.0 |
The analysis paper’s personal abstract is that instruction-following and agentic evaluations lose most. Area-Onerous-V2 is the worst case, at -4.2 factors. RULER stays inside roughly 1 to 2 factors at 256K, 512K, and 1M.
Three BF16 outcomes don’t regress. AA-LCR good points 0.1, Scale AI Multi-Problem ties at 56.6, and TauBench Telecom good points 0.4.
NVFP4 prices little on high of compression. On RULER 1M the NVFP4 checkpoint scores 93.2, above BF16’s 92.2. HLE is the clearest NVFP4 value, dropping from 16.5 to fifteen.7. FP8 outcomes sit in Appendix E, and monitor BF16 carefully. SWE-Bench will not be reported for the FP8 checkpoint.
Use Instances
- Extremely-long-context RAG on one GPU: A doc evaluation service at 1M context goes from 1 concurrent request to eight. Mixture decode throughput at that concurrency is roughly 4x.
- Interactive coding assistants: At UT >= 100 tok/s within the 8K/64K regime, one node serves 2.03x the tokens. Adjusted for verbosity, that’s 2.16x the finished requests per minute.
- Prefill-heavy doc pipelines: The 50K/2K regime good points just one.60x. Compression helps much less when immediate processing dominates compute.
- Agentic SWE loops: Verify the two.6-point SWE-Bench hole in opposition to your activity combine. RL restoration focused this functionality, and solely partly restored it.
Deployment Explorer
