NVIDIA Shrinks Nemotron 3.5 Lightning to 22 GB

NVIDIA researchers have successfully compressed the NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 model down to a 22 GB NVFP4 checkpoint using a technique called quantization-aware distillation. By shrinking the model from its original 66 GB size, developers achieved up to four times higher throughput. This process, executed via the NVIDIA Model Optimizer, relies on a two-stage pipeline that aggressively quantizes weights to a W4A16 format while keeping accuracy remarkably close to the 16-bit brain floating-point baseline.

The optimization pipeline begins with post-training quantization to generate a low-precision student model from the full-precision teacher. In the second stage, the student model undergoes distillation training against the frozen teacher using Kullback-Leibler divergence loss. During this phase, every forward pass runs through simulated quantization, allowing the student to adapt to quantization noise. Developers can choose between dynamic scale quantization for max-calibrated checkpoints or frozen scale quantization for search-based methods like mean squared error and four-over-six configurations.

Experiments on intermediate checkpoints demonstrated the superiority of this distillation method over standard post-training quantization. On an early supervised fine-tuning checkpoint, standard quantization yielded a 96.33 percent median score recovery, whereas distillation achieved 99.72 percent recovery, reclaiming lost ground on reasoning and coding tests. On an intermediate reinforcement learning checkpoint, distillation boosted the median recovery from 95.84 percent to 98.53 percent, raising the AA v4.1 Index score by 3.45 points to reach 23.48.

For the final, more conservative Nemotron-3.5-Lightning-30B-A3B/lightning_w4a16_nvfp4_4o6 checkpoint, standard quantization already sat close to the baseline with a 99.24 percent median recovery. However, distillation still delivered critical gains on agentic and coding benchmarks. It outperformed standard quantization by 3.79 points on Terminal-Bench v2.1, 1.07 points on SWE-Bench Multilingual, and 0.65 points on HLE. The entire workflow is reproducible using the NVIDIA Model Optimizer and Megatron-Bridge, utilizing datasets like Nemotron-Post-Training v2 and hardware such as the NVIDIA DGX B300.