Bayesian Classification with Gaussian Process

Despite prowess of the support vector machine, it is not specifically designed to extract features relevant to the prediction. For example, in network intrusion detection, we need to learn relevant network statistics for the network defense. In consumer credit rating, we would like to determine relevant financial records for the credit score. As for medical genetics research, we aim to identify genes relevant to the illness.

Bayesian learning has the Automatic Relevance Determination (ARD) capability built-in for this purpose. A particularly effective implementation is the variational Bayes approximation algorithm adopted in the R package vbmp. Using a Gaussian process prior on the function space, it is able to predict the posterior probability much more economically than plain MCMC.

Example 1

For illustration, we begin with a toy example based on the rvbm.sample.train data set in rpud. The data set has two components, namely X and t.class. The first component X contains data points in a six dimensional Euclidean space, and the second component t.class classifies the data points of X into 3 different categories according to the squared sum of the first two coordinates of the data points. The other four coordinates in X serve only as noise dimensions. It is created with R code in the vbmp vignette.

We can visualize the data set is in a scatter plot as follows. The innermost layer is plotted in green triangles, the middle one is in blue solid dots, and the outermost layer is in red hollow dots.

We will perform Gaussian process classification with the data set using the vbmp package. As it is in the Bioconductor repository, we have to install it with the BiocManager.

Then we apply the vmbp method with the bThetaEstimate option for Automatic Relevance Determination. The training process takes about 200 seconds on an AMD X4 CPU.

We then print out the covariance parameters with the covParams method, and note that the first two parameters are less than one, while the rest are all much larger than one. It thus indicates that only the first two parameters are relevant.

Applying the predError method in vbmp, we found the error ratio to be 3.8%. Similarly, the predLik method shows that the posterior log likelihood is -0.3483.

We can run the same training process in much shorter time with the rvbm method in the rpudplus package. It takes about 20 seconds on an NVIDIA GTX 460 GPU.

Example 2

A more practical example is the BRCA12 data set in vmbp. It contains the genetic test results of 30 cancer patients. Because of its large feature space with 8,080 genes and small sample size of 30, determining relevant genes is difficult with the support vector machine. With Bayesian methods, we do not have such constraint.

Let us load the data set into the workspace.

As the data set is in the Bioconductor format, we need to install the Biobase package in order to extract the gene expression values. We then save the values in a new matrix brca.x. The matrix has 30 rows, each containing 8,080 gene expression values of a patient.

Next, we save the individual case categories of the patients in a vector brca.y.

Then we apply the rvbm method in rpudplus for the Gaussian process classification. Since the data lies in a high-dimensional Euclidean space, a linear kernel, instead of the usual Gaussian one, is more appropriate. On an GTX 460 GPU, the task takes about 2 minutes and a half to finish. Performing similar task with vbmp using the equivalent iprod kernel would take hours.

The following indicates no extreme value in the kernel parameters, and confirms that all genes in the BRCA12 data set are relevant in the study. The summary also shows that the posterior log likelihood is -0.0338, and the prediction error rate is zero.

Lastly we can plot the training history, and visually check the convergence of the lower bound estimate of marginal likelihood.

References

1. Zoubin Ghahramani: A Tutorial on Gaussian Processes. (or Why I Don’t Use SVMs) URL http://mlss2011.comp.nus.edu.sg/uploads/Site/lect1gp.pdf
2. Mark Girolami and Simon Rogers: Variational Bayesian Multinomial Probit Regression with Gaussian Process Priors. Neural Computation, 18:1790–1817, 2006.
3. Zsofia Kote-Jarai, et al: Accurate Prediction of BRCA1 and BRCA2 Heterozygous Genotype Using Expression Profiling After Induced DNA Damage. Clinical Cancer Research, 12 (13):3896–3901, Jul 2006.
4. Nicola Lama and Mark Girolami: Variational Bayesian Multinomial Probit Regression for Gaussian Process Multi-class Classification. URL http://www.bioconductor.org/packages/release/bioc/html/vbmp.html