RUser4512 gave the correct answer: RBF kernel works well in practice and it is relatively easy to tune. It's the SVM equivalent to "no one's ever been fired for estimating an OLS regression:" it's accepted as a reasonable default method. Clearly OLS isn't perfect in every (or even many) scenarios, but it's a well-studied method, and widely understood. Likewise, the RBF kernel is well-studied and widely understood, and many SVM packages include it as a default method.

But the RBF kernel has a number of other properties. In these types of questions, when someone is asking about "why do we do things this way", I think it's important to also draw contrasts to other methods to develop context.

It is a stationary kernel, which means that it is invariant to translation. Suppose you are computing A stationary kernel will yield the same value for , where may be vector-valued of dimension to match the inputs. For the RBF, this is accomplished by working on the difference of the two vectors. For contrast, note that the linear kernel does not have the stationarity property.

The single-parameter version of the RBF kernel has the property that it is isotropic, i.e. the scaling by occurs the same amount in all directions. This can be easily generalized, though, by slightly tweaking the RBF kernel to where is a p.s.d. matrix.

Another property of the RBF kernel is that it is infinitely smooth. This is aesthetically pleasing, and somewhat satisfying visually, but perhaps it is not the most important property. Compare the RBF kernel to the Matern kernel and you'll see that there some kernels are quite a bit more jagged!

The moral of the story is that kernel-based methods are very rich, and with a little bit of work, it's very practical to develop a kernel suited to your particular needs. But if one is using an RBF kernel as a default, you'll have a reasonable benchmark for comparison.

(so an rbf is the correct choice?)

It depends. RBF is very simple, generic kernel which might be used, but there are dozens of others. Take a look for example at the ones included in pykernels https://github.com/gmum/pykernels

When the SVM is trained it will plot a hyperplane(which I think is like a plane in 3d but with more dimensions?) that best separates the data.

Lets avoid some weird confusions. Nothing is plotted here. SVM will look for d-dimensional hyperplane defined by v (normal vector) and b (bias, distance from the origin), which is simply set of points x such that <v, x> = b. In 2D hyperplane is a line, in 3D hyperplane is plane, in d+1 dimensions it is d dimensional object, always one dimension lower than the space (line is 1D, plane is 2D).

When tuning, changing the value of gamma changes the surface of the of the hyperplane (also called the decision boundary?).

Now this is an often mistake. Decision boundary is not a hyperplane. Decision boundary is a projection of the hyperplane onto input space. You cannot observe actual hyperplane as it is often of very high dimension. You can express this hyperplane as a functional equation, but nothing more. Decision boundary on the other hand "lives" in your input space, if input is low-dimensional, you can even plot this object. But this is not a hyperplane, it is just the way this hyperplane intersects with your input space. This is why decision boundary is often curved or even discontinous even though hyperplane is always linear and continuous - because you just see a nonlinear section through it. Now what is gamma doing? RBF kernel leads to the optimization in the space of continous functions. There are plenty ot these (there is continuum of such objects). However, SVM can express only a tiny fraction of these guys - linear combinations of kernel values in training points. Fixing particular gamma limits set of functions to consider - bigger the gamma, more narrow the kernels, thus functions that are being considered consists of linear combinations of such "spiky" distributions. So gamma itself does not change the surface, it changes the space of considered hypotheses.

So an increase in the value of gamma, results in a Gaussian which is narrower. Is this like saying that the bumps on the plane (if plotted in 3d) that can be plotted are allowed to be narrower to fit the training data better? Or in 2D is this like saying gamma defines how bendy the line that separtates the data can be?

I think I answered with previous point - high gamma means that you only consider hyperplanes of form

<v, x> - b = SUM_i alpha_i K_gamma(x_i, x) - b

where K_gamma(x_i, x) = exp(-gamma ||x_i-x||^2), thus you will get very "spiky" elements of your basis. this will lead to very tight fit to your training data. Exact shape of the decision boundary is hard to estimate, as this depends on optimal lagrange multipliers alpha_i selected during training.

I'm also very confused about about how this can lead to an infinite dimensional representation from a finite number of features? Any good analogies would help me greatly.

The "infinite representation" comes from the fact, that in order to work with vectors and hyperplanes, each of your point is actually mapped to a continuous function. So SVM, internally, is not really working with d-dimensional points anymore, it is working with functions. Consider 2d case, you have points [0,0] and [1,1]. This is a simple 2d problem. When you apply SVM with rbf kernel here - you will instead work with an unnormalized gaussian distribution centered in [0, 0] and another one in [1,1]. Each such gaussian is a function from R^2 to R, which expresses its probability density function (pdf). It is a bit confusing because kernel looks like a gaussian too, but this is only because dot product of two functions is usually defined as an integral of their product, and integral of product of two gaussians is .... a gaussian too! So where is this infinity? Remember that you are supposed to work with vectors. How to write down a function as a vector? You would have to list all its values, thus if you have a function f(x) = 1/sqrt(2*pi(sigma^2) exp(-||x-m||^2 / (2*sigma^2)) you will have to list infinite number of such values to fully define it. And this is this concept of infinite dimension - you are mapping points to functions, functions are infinite dimensional in terms of vector spaces, thus your representation is infinitely dimensional.

One good example might be different mapping. Consider a 1D dataset of numbers 1,2,3,4,5,6,7,8,9,10. Lets assign odd numbers different label than even ones. You cannot linearly separate these guys. But you can instead map each point (number) to a kind of characteristic function, function of the form

f_x(y) = 1 iff x e [y-0.5, y+0.5]

now, in space of all such functions I can easily linearly separate the ones created from odd x's from the rest, by simply building hyperplane of equation

<v, x> = SUM_[v_odd] <f_v_odd, f_x(y)> = INTEGRAL (f_v * f_x) (y) dy

And this will equal 1 iff x is odd, as only this integral will be non zero. Obviously I am just using finite amount of training points (v_odd here) but the representation itself is infinite dimensional. Where is this additional "information" coming from? From my assumptions - the way I defined the mapping introduces a particular structure in the space I am considering. Similarly with RBF - you get infinite dimension, but it does not mean you are actually considering every continouus function - you are limiting yourself to the linear combinations of gaussians centered in training points. Similarly you could use sinusoidal kernel which limits you to the combinations of sinusoidal functions. The choice of a particular, "best" kernel is the whole other story, complex and without clear answers. Hope this helps a bit.

1) Could anyone explain why the number of feature space after mapping is corresponding to the derivatives of the kernel? I am not clear on this part.

It has nothing to do with being differentiable, linear kernel is also infinitely differentiable and does not map to any higher dimensional space, whoever told you that it is the reason -- lied or did not understand the math behind it. The infinite dimension comes from the mapping

phi(x) = Nor(x, sigma^2)

in other words you are mapping your point into function being a Gaussian distribution, which is an element of L^2 space, infinitely dimension space of continuous function, where scalar product is defined as an integral of multiplication of functions, so

<f,g> = int f(a)g(a) da

and as such

<phi(x),phi(y)> = int Nor(x,sigma^2)(a)Nor(y,sigma^2)(a) da 
                = X exp(-(x-y)^2 / (4sigma^2) )

for some normalising constant X (which is completely unimportant). In other words, Gaussian kernel is a scalar product between two functions, which have infinite dimensions.

2) There are many non-linear kernels, such as polynomial kernel, and I believe they are also able to map the data from a low dimensional space to an infinite dimensional space. But why the RBF kernel is more popular then them?

Polynomial kernel maps into feature space with O(d^p) dimensions, where d is input space dimension and p is polynomial degree, so it is far from being infinite. Why is Gaussian popular? Because it works, and is quite easy to use and fast to compute. From theoretical point of view it also has guarantees of learning any arbitrary set of points (with small enough variances used).

The kernel function is a measure of similarity between two sets of features. So in this case, $x'$ and $x$ will both be $5\times 1$ feature vectors (not necessarily the same). $K(x,x')$ is a scalar that represents the similarity between $x$ and $x'$, and the kernel matrix $[K(x,x')]_{x\in X, x'\in X}$ is a $100\times 100$ matrix which represents the pairwise similarities.

The kernel function can be thought of as a cheap way of computing an infinite dimensional inner product - this 'kernel trick' is described in more detail in these notes. This lets the algorithm learn arbitrarily complex functions (though it may take an infinite number of samples for it to learn).

You can possibly start by looking at one of my answers here:
Non-linear SVM classification with RBF kernel

In that answer, I attempt to explain what a kernel function is attempting to do. Once you get a grasp of what it attempts to do, as a follow up, you can read my answer to a question on Quora : https://www.quora.com/Machine-Learning/Why-does-the-RBF-radial-basis-function-kernel-map-into-infinite-dimensional-space/answer/Arun-Iyer-1

Reproducing the content of the answer on Quora, in case you don't have a Quora account.

Question: Why does the RBF (radial basis function) kernel map into infinite dimensional space? Answer : Consider the polynomial kernel of degree 2 defined by, $$k(x, y) = (x^Ty)^2$$ where $x, y \in \mathbb{R}^2$ and $x = (x_1, x_2), y = (y_1, y_2)$.

Thereby, the kernel function can be written as, $$k(x, y) = (x_1y_1 + x_2y_2)^2 = x_{1}^2y_{1}^2 + 2x_1x_2y_1y_2 + x_{2}^2y_{2}^2$$ Now, let us try to come up with a feature map $\Phi$ such that the kernel function can be written as $k(x, y) = \Phi(x)^T\Phi(y)$.

Consider the following feature map, $$\Phi(x) = (x_1^2, \sqrt{2}x_1x_2, x_2^2)$$ Basically, this feature map is mapping the points in $\mathbb{R}^2$ to points in $\mathbb{R}^3$. Also, notice that, $$\Phi(x)^T\Phi(y) = x_1^2y_1^2 + 2x_1x_2y_1y_2 + x_2^2y_2^2$$ which is essentially our kernel function.

This means that our kernel function is actually computing the inner/dot product of points in $\mathbb{R}^3$. That is, it is implicitly mapping our points from $\mathbb{R}^2$ to $\mathbb{R}^3$.

Exercise Question : If your points are in $\mathbb{R}^n$, a polynomial kernel of degree 2 will map implicitly map it to some vector space F. What is the dimension of this vector space F? Hint: Everything I did above is a clue.

Now, coming to RBF.

Let us consider the RBF kernel again for points in $\mathbb{R}^2$. Then, the kernel can be written as $$k(x, y) = \exp(-\|x - y\|^2) = \exp(- (x_1 - y_1)^2 - (x_2 - y_2)^2)$$ $$= \exp(- x_1^2 + 2x_1y_1 - y_1^2 - x_2^2 + 2x_2y_2 - y_2^2) $$ $$ = \exp(-\|x\|^2) \exp(-\|y\|^2) \exp(2x^Ty)$$ (assuming gamma = 1). Using the taylor series you can write this as, $$k(x, y) = \exp(-\|x\|^2) \exp(-\|y\|^2) \sum_{n = 0}^{\infty} \frac{(2x^Ty)^n}{n!}$$ Now, if we were to come up with a feature map $\Phi$ just like we did for the polynomial kernel, you would realize that the feature map would map every point in our $\mathbb{R}^2$ to an infinite vector. Thus, RBF implicitly maps every point to an infinite dimensional space.

Exercise Question : Get the first few vector elements of the feature map for RBF for the above case?

Now, from the above answer, we can conclude something:

• It may be quite hard to predict in general what the mapping function $\Phi$ looks like for arbitrary kernel. Though, for some cases like polynomial and RBF we can see what it looks like.
• Even when we know the mapping function, the exact effect that kernel will have on our set of points may be hard to predict. However, in certain cases we can say some things. For example, look at the $\Phi$ map given above for degree 2 polynomial kernel for $\mathbb{R}^2$. It looks like $\Phi(x) = (x_1^2, \sqrt{2}x_1x_2, x_2^2)$. From this we can determine that this map collapses diametrically opposite quadrants i.e first and third quadrant are mapped to same set of points and second and fourth quadrant are mapped to the same set of points. Therefore, this kernel allows us to solve XOR problem! In general, however, it might be harder to predict such behaviour for multidimensional spaces. And it gets harder in the case of RBF kernels.