[2110.04274] Kernel Interpolation as a Bayes Point Machine

Abstract:A Bayes point machine is a single classifier that approximates the majority decision of an ensemble of classifiers. This paper observes that kernel interpolation is a Bayes point machine for Gaussian process classification. This observation facilitates the transfer of results from both ensemble theory as well as an area of convex geometry known as Brunn-Minkowski theory to derive PAC-Bayes risk bounds for kernel interpolation. Since large margin, infinite width neural networks are kernel interpolators, the paper's findings may help to explain generalisation in neural networks more broadly. Supporting this idea, the paper finds evidence that large margin, finite width neural networks behave like Bayes point machines too.

Kernel Interpolation as a Bayes Point Machine Jeremy Bernstein Alex Farhang Yisong Yue bernstein@caltech.edu afarhang@caltech.edu yyue@caltech.edu Abstract 1.2. Knowledge vs. Belief A Bayes point machine is a single classifier that Kernel…

related reading

• Bayesian Neural Networkscs.toronto.edu
• arxiv.org/pdf/1805.08522arxiv.org
• [2007.12826] The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingarxiv.org
• Support vector machine - Wikipediaen.wikipedia.org
• catoni_draft2.pdfocatoni.perso.math.cnrs.fr
• Kernel methoden.wikipedia.org
• Understanding Deep Learningudlbook.github.io
• Gregory Gundersengregorygundersen.com
• Support vector machine - Wikipediaen.wikipedia.org
• Maybe I was too harsh on deep learning theory (three days ago) — LessWronglesswrong.com
• [2605.01172] A Theory of Generalization in Deep Learningarxiv.org
• Elon Litman | Elements of a Vector Spaceelonlit.com