Videos
I don't fully understand your code, but let's go through the example in the documentation page you referenced:
import numpy as np
X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]])
y = np.array([1, 1, 2, 2])
from sklearn.svm import SVC
clf = SVC()
clf.fit(X, y)
Now let's apply both the decision_function() and predict() to the samples:
clf.decision_function(X)
clf.predict(X)
The output we get is:
array([[-1.00052254],
[-1.00006594],
[ 1.00029424],
[ 1.00029424]])
array([1, 1, 2, 2])
And that is easy to interpret: The decision function tells us on which side of the hyperplane generated by the classifier we are (and how far we are away from it). Based on that information, the estimator then labels the examples with the corresponding label.
For those interested, I'll post a quick example of the predict function translated from C++ (here) to python:
# I've only implemented the linear and rbf kernels
def kernel(params, sv, X):
if params.kernel == 'linear':
return [np.dot(vi, X) for vi in sv]
elif params.kernel == 'rbf':
return [math.exp(-params.gamma * np.dot(vi - X, vi - X)) for vi in sv]
# This replicates clf.decision_function(X)
def decision_function(params, sv, nv, a, b, X):
# calculate the kernels
k = kernel(params, sv, X)
# define the start and end index for support vectors for each class
start = [sum(nv[:i]) for i in range(len(nv))]
end = [start[i] + nv[i] for i in range(len(nv))]
# calculate: sum(a_p * k(x_p, x)) between every 2 classes
c = [ sum(a[ i ][p] * k[p] for p in range(start[j], end[j])) +
sum(a[j-1][p] * k[p] for p in range(start[i], end[i]))
for i in range(len(nv)) for j in range(i+1,len(nv))]
# add the intercept
return [sum(x) for x in zip(c, b)]
# This replicates clf.predict(X)
def predict(params, sv, nv, a, b, cs, X):
''' params = model parameters
sv = support vectors
nv = # of support vectors per class
a = dual coefficients
b = intercepts
cs = list of class names
X = feature to predict
'''
decision = decision_function(params, sv, nv, a, b, X)
votes = [(i if decision[p] > 0 else j) for p,(i,j) in enumerate((i,j)
for i in range(len(cs))
for j in range(i+1,len(cs)))]
return cs[max(set(votes), key=votes.count)]
There are a lot of input arguments for predict and decision_function, but note that these are all used internally in by the model when calling predict(X). In fact, all of the arguments are accessible to you inside the model after fitting:
# Create model
clf = svm.SVC(gamma=0.001, C=100.)
# Fit model using features, X, and labels, Y.
clf.fit(X, y)
# Get parameters from model
params = clf.get_params()
sv = clf.support_vectors_ #added missing underscore
nv = clf.n_support_
#a = clf.dual_coef_
a = clf._dual_coef_ #use complementary dual coefficients
b = clf._intercept_
cs = clf.classes_
# Use the functions to predict
print(predict(params, sv, nv, a, b, cs, X))
# Compare with the builtin predict
print(clf.predict(X))