building a bigger qwen out of two qwen models

a crash course in model chimerism

tl;dr: Take a giant Qwen3 MoE model, collapse its experts into a dense stack that matches Qwen3-8B’s width, then interleave those dense MoE blocks with the 8B blocks to create a deeper hybrid. Validate that the transplanted blocks actually matter, fix the ‘hot’ ones, and end with a model that’s both larger and reasonably sane. the layers of the final model Why does this work at all? Qwen3-8B and Qwen3-235B-A22 share the same hidden size and attention head dimension. If hidden size matches, you can line up linear layers across both networks without inventing sketchy adapters. The other…

related reading

• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• An Alternative to Test-Time Scalingrentry.org
• Papers I’ve read this week, Mixture of Experts editionfinbarrtimbers.substack.com
• Quantization from the ground upngrok.com
• [2101.03961] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsityarxiv.org
• ali (@waterloo_intern) on Xx.com
• DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0mlcommons.org
• Mixture of Experts Explainedhuggingface.co
• MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensatorsbeichenhuang.github.io
• Very Simple MoE Intro1a3orn.com