SVM is a support-vector machine which is a special linear-model. From a theoretical view it's a convex-optimization problem and we can get the global-optimum in polynomial-time. There are many different optimization-approaches.

In the past people used general Quadratic Programming solvers. Nowadays specialized approaches like SMO and others are used.

sklearn's specialized SVM-optimizers are based on liblinear and libsvm. There are many documents and research papers if you are interested in the algorithms.

Keep in mind, that SVC (libsvm) and LinearSVC (liblinear) make different assumptions in regards to the optimization-problem, which results in different performances on the same task (linear-kernel: LinearSVC much more efficient than SVC in general; but some tasks can't be tackled by LinearSVC).

SGD is an Stochastic Gradient Descent-based (this is a general optimization method!) optimizer which can optimize many different convex-optimization problems (actually: this is more or less the same method used in all those Deep-Learning approaches; so people use it in the non-convex setting too; throwing away theoretical-guarantees).

sklearn says: Stochastic Gradient Descent (SGD) is a simple yet very efficient approach to discriminative learning of linear classifiers under convex loss functions. Now it's actually even more versatile, but here it's enough to note that it subsumes (some) SVMs, logistic-regression and others.

Now SGD-based optimization is very different from QP and others. If one would take QP for example, there are no hyper-parameters to tune. This is a bit simplified, as there can be tuning, but it's not needed to guarantee convergence and performance! (theory of QP-solvers, e.g. Interior-point method is much more robust)

SGD-based optimizers (or general first-order methods) are very very hard to tune! And they need tuning! Learning-rates or learning-schedules in general are parameters to look at as convergence depends on these (theory and practice)!

It's a very complex topic, but some simplified rules:

• Specialized SVM-methods

  • scale worse with the number of samples
  • do not need hyper-parameter tuning

• SGD-based methods

  • scale better for huge-data in general
  • need hyper-parameter tuning
  • solve only a subset of the tasks approachable by the the above (no kernel-methods!)

My opinion: use (the easier to use) LinearSVC as long as it's working, given your time-budget!

Just to make it clear: i highly recommend grabbing some dataset (e.g. from within sklearn) and do some comparisons between those candidates. The need for param-tuning is not a theoretical-problem! You will see non-optimal (objective / loss) results in the SGD-case quite easily!

And always remember: Stochastic Gradient Descent is sensitive to feature scaling docs. This is more or less a consequence of first-order methods.

You may have some mis-understanding of SVM types. There is no SGD SVM. See this post.

Difference between the types of SVM

Stochastic gradient descent (SGD) is an algorithm to train the model. According to the documentation, SGD algorithm can be used to train many models. SVM is just one special case. More information about SGD, can be found here.

How could stochastic gradient descent save time comparing to standard gradient descent?

Therefore, SGD will not affect what is inside of SVM. If you are using SVC (C-Support Vector Classification) and use SGD for learning, you should still have everything about SVC. The $\alpha$ will tell you where are the "support vectors" as usual.

In python, it is possible they use some object oriented design that hide some fields if you are using SGD classifiers. But that should be a stack overflow question.

The first two always use the full data and solve a convex optimization problem with respect to these data points.

The latter can treat the data in batches and performs a gradient descent aiming to minimize expected loss with respect to the sample distribution, assuming that the examples are iid samples of that distribution.

The latter is typically used when the number of samples is very big or not ending. Observe that you can call the partial_fit function and feed it chunks of data.

Hope this helps?

When SVMs and SGD can't be combined

SVMs are often used in combination with the kernel trick, which enables classification of non-linearly separable data. This answer explains why you wouldn't use stochastic gradient descent to solve a kernelised SVM: https://stats.stackexchange.com/questions/215524/is-gradient-descent-possible-for-kernelized-svms-if-so-why-do-people-use-quadr

Linear SVMs

If we stick to Linear SVMs, then we can run an experiment using sklearn, as it provides wrappers over libsvm (SVC), liblinear (LinearSVC) and also offers the SGDClassifier. Recommend reading the linked documentation of libsvm and liblinear to understand what is happening under the hood.

Comparison on example dataset

Below is a comparison of computational performance and accuracy over a randomly generated dataset (which may not be representative of your problem). You should alter the problem to fit your requirements.

import time
import numpy as np
import matplotlib.pyplot as plt

from sklearn.svm import SVC, LinearSVC
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

# Randomly generated dataset
# Linear function + noise
np.random.seed(0)
X = np.random.normal(size=(50000, 10))
coefs = np.random.normal(size=10)
epsilon = np.random.normal(size=50000)
y = (X @ coefs + epsilon) > 0

# Classifiers to compare
algos = {
    'LibSVM': {
        'model': SVC(),
        'max_n': 4000,
        'time': [],
        'error': []
    },
    'LibLinear': {
        'model': LinearSVC(dual=False),
        'max_n': np.inf,
        'time': [],
        'error': []
    },
    'SGD': {
        'model': SGDClassifier(max_iter=1000, tol=1e-3),
        'max_n': np.inf,
        'time': [],
        'error': []
    }
}

splits = list(range(100, 1000, 100)) + \
         list(range(1500, 5000, 500)) + \
         list(range(6000, 50000, 1000))
for i in splits:
    X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                        test_size=1-i/50000,
                                                        random_state=0)
    for k, v in algos.items():
        if i < v['max_n']:
            model = v['model']
            t0 = time.time()
            model.fit(X_train, y_train)
            t1 = time.time()
            v['time'].append(t1 - t0)
            preds = model.predict(X_test)
            e = (preds != y_test).sum() / len(y_test)
            v['error'].append(e)

Plotting the results, we see that the traditional libsvm solver cannot be used on large n, while the liblinear and SGD implementations scale well computationally.

plt.figure()
for k, v in algos.items():
    plt.plot(splits[:len(v['time'])], v['time'], label='{} time'.format(k))
plt.legend()
plt.semilogx()
plt.title('Time comparison')
plt.show()

Plotting the error, we see that SGD is worse than LibSVM for the same training set, but if you have a large training set this becomes a minor point. The liblinear algorithm performs best on this dataset:

plt.figure()
for k, v in algos.items():
    plt.plot(splits[:len(v['error'])], v['error'], label='{} error'.format(k))
plt.legend()
plt.semilogx()
plt.title('Error comparison')
plt.xlabel('Number of training examples')
plt.ylabel('Error')
plt.show()