Pruning MoE LLMs: Half Experts Removed, Full Coding Performance

Mixture-of-Experts (MoE) large language models are powerful but often too large for typical developer machines due to their extensive pools of "expert" subnetworks. This research explores whether these models can be pruned for domain-specific tasks, specifically coding, without sacrificing performance. The study investigated two open-weight MoE models, Qwen3.6-35B-A3B and Gemma-4-26B-A4B, using five expert selection strategies. The findings are significant: up to half of the experts can be removed from either model with no statistically detectable loss on primary code benchmarks. The performance degradation primarily affects abilities outside the intended coding domain. Interestingly, the optimal pruning strategy varied between the two model families, indicating that a universal recipe does not exist. The research also highlighted that perplexity, a common metric in pruning literature, can be misleading, sometimes rating broken models higher than intact ones. Further analysis showed that lightweight fine-tuning could recover about half of the performance lost from aggressive pruning. While pruning offers memory savings, it only outperforms quantizing the full model when quantization would drop below 3 bits per weight. Attempts to overcome this crossover point, even with advanced evaluation methods, were unsuccessful. However, the study also found that single-shot benchmarks might overstate compression penalties, as a single repair turn from execution feedback could entirely erase the penalty of 2-bit quantization. This suggests expert pruning is effective but requires task-specific validation.