[2605.29101] Model Merging by Output-Space Projection

arxiv.org · 6,827 words · saved by 1 readers

Abstract:Model merging combines fine-tuned checkpoints into a single multi-task model without retraining. Existing methods - such as task arithmetic, model soups, TIES, and DARE - are computationally efficient and empirically successful, but rely on heuristic design choices and lack formal optimality guarantees. We show that merging can be formulated as a convex quadratic programme over residual updates, yielding weights that minimise a squared-output calibration objective using calibration inputs and fine-tuned model outputs, and subsuming existing methods as special cases. Our framework yields a closed-form diagnostic - the fraction of residual energy captured by a chosen basis - that predicts downstream merge quality using only the calibration set. Empirically, the QP matches or outperforms existing methods in the single-layer setting, and we characterise when the optimal basis provides significant gains over the cheaper diagonal QP. We extend to multi-layer merging via a sequential layer-wise algorithm and demonstrate consistent gains across language and vision benchmarks.

Model Merging by Output-Space Projection Bethan Evans 1 Benjamin Etheridge 2 Stephen Roberts 2 Jared Tanner 1 Abstract a precise mathematical notion or loss function for which their method is optimal. Model merging combines fine-tuned checkpoints…

[2605.29101] Model Merging by Output-Space Projection

related reading