# Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance

@article{Jean2018SemisupervisedDK, title={Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance}, author={Neal Jean and Sang Michael Xie and Stefano Ermon}, journal={ArXiv}, year={2018}, volume={abs/1805.10407} }

Large amounts of labeled data are typically required to train deep learning models. For many real-world problems, however, acquiring additional data can be expensive or even impossible. We present semi-supervised deep kernel learning (SSDKL), a semi-supervised regression model based on minimizing predictive variance in the posterior regularization framework. SSDKL combines the hierarchical representation learning of neural networks with the probabilistic modeling capabilities of Gaussian… Expand

#### Supplemental Code

Github Repo

Via Papers with Code

Code that accompanies the paper Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance

#### Figures, Tables, and Topics from this paper

#### 37 Citations

Metric-Based Semi-Supervised Regression

- Computer Science
- IEEE Access
- 2020

The experimental results indicate that the proposed method to cope with semi-supervised regression problems achieves promising results and could capture the trend of a non-linear function and normally predict well even though this dataset comprises extreme outliers. Expand

Deep Low-Density Separation for Semi-supervised Classification

- Computer Science
- ICCS
- 2020

A novel hybrid method that applies low-density separation to the embedded features of neural network-based embeddings and effectively classifies thousands of unlabeled users from a relatively small number of hand-classified examples is introduced. Expand

Weakly-supervised Multi-output Regression via Correlated Gaussian Processes

- Mathematics, Computer Science
- ArXiv
- 2020

This paper considers multi-output regression under a weakly-supervised setting where a subset of data points from multiple groups are unlabeled and derives a modified Variational bound for fast and stable convergence in model inference and a scalable variational bound that is amenable to stochastic optimization. Expand

Lautum Regularization for Semi-Supervised Transfer Learning

- Computer Science, Mathematics
- 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)
- 2019

The theory suggests that one may improve the transferability of a deep neural network by imposing a Lautum information based regularization that relates the network weights to the target data. Expand

High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learning

- Computer Science
- ArXiv
- 2021

A method combining variational autoencoders (VAEs) and deep metric learning to perform Bayesian optimisation (BO) over high-dimensional and structured input spaces is introduced, using label guidance from the blackbox function to structure the VAE latent space, facilitating the Gaussian process fit and yielding improved BO performance. Expand

A Simple yet Effective Baseline for Robust Deep Learning with Noisy Labels

- Computer Science, Mathematics
- ArXiv
- 2019

This work proposes a simple but effective baseline that is robust to noisy labels, even with severe noise, and involves a variance regularization term that implicitly penalizes the Jacobian norm of the neural network on the whole training set (including the noisy-labeled data), which encourages generalization and prevents overfitting to the corrupted labels. Expand

Twin Neural Network Regression is a Semi-Supervised Regression Algorithm

- Computer Science
- ArXiv
- 2021

Semi-supervised training of twin neural network regression improves TNNR performance, which is already state of the art, significantly. Expand

Efficient Statistical Tests: A Neural Tangent Kernel Approach

- Computer Science
- ICML
- 2021

This work proposes a shift-invariant convolutional neural tangent kernel (SCNTK) based outlier detector and two-sample tests with maximum mean discrepancy (MMD) that isO(n) in the number of samples due to using the random feature approximation. Expand

Learning Data-adaptive Nonparametric Kernels

- Computer Science, Mathematics
- J. Mach. Learn. Res.
- 2020

A Data-Adaptive Nonparametric Kernel (DANK) learning framework by imposing an adaptive matrix on the kernel/Gram matrix in an entry-wise strategy that outperforms other representative kernel learning based algorithms on various classification and regression benchmark datasets. Expand

Implicit Kernel Learning

- Computer Science, Mathematics
- AISTATS
- 2019

This paper explores learning the spectral distribution of kernel via implicit generative models parametrized by deep neural networks via Implicit Kernel Learning (IKL), which is simple to train and inference is performed via sampling random Fourier features. Expand

#### References

SHOWING 1-10 OF 45 REFERENCES

Semi-supervised Learning by Entropy Minimization

- Computer Science, Mathematics
- CAP
- 2004

This framework, which motivates minimum entropy regularization, enables to incorporate unlabeled data in the standard supervised learning, and includes other approaches to the semi-supervised problem as particular or limiting cases. Expand

Semi-Supervised Regression with Co-Training

- Computer Science
- IJCAI
- 2005

Experiments show that COREG can effectively exploit unlabeled data to improve regression estimates and is proposed as a co-training style semi-supervised regression algorithm. Expand

Stochastic Variational Deep Kernel Learning

- Computer Science, Mathematics
- NIPS
- 2016

An efficient form of stochastic variational inference is derived which leverages local kernel interpolation, inducing points, and structure exploiting algebra within this framework to enable classification, multi-task learning, additive covariance structures, and Stochastic gradient training. Expand

Realistic Evaluation of Semi-Supervised Learning Algorithms

- Computer Science
- ICLR
- 2018

This work creates a unified reimplemention and evaluation platform of various widelyused SSL techniques and finds that the performance of simple baselines which do not use unlabeled data is often underreported, that SSL methods differ in sensitivity to the amount of labeled and unlabeling data, and that performance can degrade substantially when the unlabelED dataset contains out-of-class examples. Expand

Temporal Ensembling for Semi-Supervised Learning

- Computer Science
- ICLR
- 2017

Self-ensembling is introduced, where it is shown that this ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Expand

Deep Hybrid Models: Bridging Discriminative and Generative Approaches

- 2017

Most methods in machine learning are described as either discriminative or generative. The former often attain higher predictive accuracy, while the latter are more strongly regularized and can deal… Expand

Minimum variance semi-supervised boosting for multi-label classification

- Computer Science
- 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP)
- 2015

The experiments show that the proposed algorithm outperforms its supervised counterpart as well as the existing information theoretic based semi-supervised methods, and its performance is steadily improving as more unlabeled data is available. Expand

Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

- Computer Science, Mathematics
- NIPS
- 2017

The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks, but it becomes unwieldy when learning large datasets, so Mean Teacher, a method that averages model weights instead of label predictions, is proposed. Expand

Auxiliary Deep Generative Models

- Computer Science, Mathematics
- ICML
- 2016

This work extends deep generative models with auxiliary variables which improves the variational approximation and proposes a model with two stochastic layers and skip connections which shows state-of-the-art performance within semi-supervised learning on MNIST, SVHN and NORB datasets. Expand

Introduction to Semi-Supervised Learning

- Computer Science
- Introduction to Semi-Supervised Learning
- 2009

This introductory book presents some popular semi-supervised learning models, including self-training, mixture models, co-training and multiview learning, graph-based methods, and semi- supervised support vector machines, and discusses their basic mathematical formulation. Expand