Answering my own question after some investigation:

• warm_start=True and calling .fit() sequentially should not be used for incremental learning on new datasets with potential concept drift. It simply uses the previously fitted model's parameters to initialize a new fit, and will likely be overwritten if the new data is sufficiently different (i.e. signals are different). After a few mini-batches with large enough sample size (datasets in my case), the overall performance converges to exactly that of simply re-initializing the model. My guess is that this method should be used for the primary purpose of reducing training time when fitting the same dataset, or when there is no significant concept drift in new data.
• partial_fit on the other hand has an effect and can be used for incremental learning (especially for datasets too large to fit into memory and feeding in mini-batches). However, in datasets with potential concept drift or high noise, it performs worse than disregarding past observations and simply fitting on each dataset without any incremental learning.
• For SGDClassifier, calling partial_fit repeatedly makes a difference.

Edit (2022)
This post/answer has gotten a lot more views than expected, and I thought I'd expand a bit on my previous answer.

Let's say your ML model is a very simple linear regression, where are the weights and biases, and the input/features.

And let's say that you've trained the model so that you've obtained some estimates for on some initial dataset . Now you've obtained another dataset .

Using warm_start=True and .fit() simply uses as an initialization for the parameters to be optimized on . This can reduce the training time, especially if the datasets and are assumed to be generated from the same underlying data generating process ("more-or-less constant" in the docs).

On the other hand,partial_fit is for incrementally updating the parameters. So if you trained the model on , and then partial_fit on , this would be conceptually similar to training a fresh model on a combined dataset.

The distinction can be pretty subtle so here's an example. Let's say you are training a classifier on the Iris dataset. Now suppose you went out and collected more data on the flowers. If you think there is concept drift (the flowers have evolved and are slightly different) or perhaps only care about the new data, then using warm_start lets you train the model on the new data faster than training from scratch with random initialization.

On the other hand, let's say you are building a music recommendation system where users provide feedback on whether the recommendation was good or not. Then you can use partial_fit to incrementally update the model as the live data comes in.

I have finally found the answer. You need to shuffle the training data between each iteration, as setting shuffle=True when instantiating the model will NOT shuffle the data when using partial_fit (it only applies to fit). Note: it would have been helpful to find this information on the sklearn.linear_model.SGDClassifier page.

The amended code reads as follows:

from sklearn.linear_model import SGDClassifier
import random
clf2 = SGDClassifier(loss='log') # shuffle=True is useless here
shuffledRange = range(len(X))
n_iter = 5
for n in range(n_iter):
    random.shuffle(shuffledRange)
    shuffledX = [X[i] for i in shuffledRange]
    shuffledY = [Y[i] for i in shuffledRange]
    for batch in batches(range(len(shuffledX)), 10000):
        clf2.partial_fit(shuffledX[batch[0]:batch[-1]+1], shuffledY[batch[0]:batch[-1]+1], classes=numpy.unique(Y))

I don't know about the Passive Aggressor, but at least when using the SGDRegressor, partial_fit will only fit for 1 epoch, whereas fit will fit for multiple epochs (until the loss converges or max_iter is reached). Therefore, when fitting new data to your model, partial_fit will only correct the model one step towards the new data, but with fit and warm_start it will act as if you would combine your old data and your new data together and fit the model once until convergence.

Example:

from sklearn.linear_model import SGDRegressor
import numpy as np

np.random.seed(0)
X = np.linspace(-1, 1, num=50).reshape(-1, 1)
Y = (X * 1.5 + 2).reshape(50,)

modelFit = SGDRegressor(learning_rate="adaptive", eta0=0.01, random_state=0, verbose=1,
                     shuffle=True, max_iter=2000, tol=1e-3, warm_start=True)
modelPartialFit = SGDRegressor(learning_rate="adaptive", eta0=0.01, random_state=0, verbose=1,
                     shuffle=True, max_iter=2000, tol=1e-3, warm_start=False)
# first fit some data
modelFit.fit(X, Y)
modelPartialFit.fit(X, Y)
# for both: Convergence after 50 epochs, Norm: 1.46, NNZs: 1, Bias: 2.000027, T: 2500, Avg. loss: 0.000237
print(modelFit.coef_, modelPartialFit.coef_) # for both: [1.46303288]

# now fit new data (zeros)
newX = X
newY = 0 * Y

# fits only for 1 epoch, Norm: 1.23, NNZs: 1, Bias: 1.208630, T: 50, Avg. loss: 1.595492:
modelPartialFit.partial_fit(newX, newY)

# Convergence after 49 epochs, Norm: 0.04, NNZs: 1, Bias: 0.000077, T: 2450, Avg. loss: 0.000313:
modelFit.fit(newX, newY)

print(modelFit.coef_, modelPartialFit.coef_) # [0.04245779] vs. [1.22919864]
newX = np.reshape([2], (-1, 1))
print(modelFit.predict(newX), modelPartialFit.predict(newX)) # [0.08499296] vs. [3.66702685]

fit(), always initializes the parameters like a new object, and trains the model with the dataset passed in fit() method.

Whereas partial_fit(), works on top of the initialize parameter and tries to improve the existing weights with the new dataset passed in partial_fit().

It is always good to save the model in persistent storage (say pickle file), for later use or for further training.