Yes you are right. In Keras batch_size refers to the batch size in Mini-batch Gradient Descent. If you want to run a Batch Gradient Descent, you need to set the batch_size to the number of training samples. Your code looks perfect except that I don't understand why you store the model.fit function to an object history.

This happens for two reasons:

• First, when the data is not shuffled, the train/validation split is inappropriate.
• Second, full gradient descent performs a single update per epoch, so more training epochs might be required to converge.

Why doesn't your model match the wave?

From model.fit:

• validation_split: Float between 0 and 1. Fraction of the training data to be used as validation data. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch. The validation data is selected from the last samples in the x and y data provided, before shuffling.

Which means that your validation set consists of the last 20% training samples. Because you are using a log scale for your independent variable (x_train), it turns out that your train/validation split is:

split_point = int(0.2*N)
x_val = x_train[-split_point:]
y_val = y_train[-split_point:]
x_train_ = x_train[:-split_point]
y_train_ = y_train[:-split_point]
plt.scatter(x_train_, y_train_, c='g')
plt.scatter(x_val, y_val, c='r')
plt.show()

In the previous plot, training and validation data are represented by green and red points, respectively. Note that your training dataset is not representative of the whole population.

Why does it still not match the training dataset?

In addition to an inappropriate train/test split, full gradient descent might require more training epochs to converge (the gradients are less noisy, but it only performs a single gradient update per epoch). If, instead, you train your model for ~1500 epochs (or use mini-batch gradient descent with a batch size of, say, 32), you end up getting:

There are actually three (3) cases:

• batch_size = 1 means indeed stochastic gradient descent (SGD)
• A batch_size equal to the whole of the training data is (batch) gradient descent (GD)
• Intermediate cases (which are actually used in practice) are usually referred to as mini-batch gradient descent

See A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size for more details and references. Truth is, in practice, when we say "SGD" we usually mean "mini-batch SGD".

These definitions are in fact fully compliant with what you report from your experiments:

• With batch_size=len(train_data) (GD case), only one update is indeed expected per epoch (since there is only one batch), hence the 1/1 indication in Keras output.

• In contrast, with batch_size = 1 (SGD case), you expect as many updates as samples in your training data (since this is now the number of your batches), i.e. 90000, hence the 90000/90000 indication in Keras output.

i.e. the number of updates per epoch (which Keras indicates) is equal to the number of batches used (and not to the batch size).

Although the other answer basically already gives you the correct result, I would like to clarify on a few points you made in your post, and correct it.
The (commonly accepted) definitions of the different terms are as follows.

• Gradient Descent (GD): Iterative method to find a (local or global) optimum in your function. Default Gradient Descent will go through all examples (one epoch), then update once.
• Stochastic Gradient Descent (SGD): Unlike regular GD, it will go through one example, then immediately update. This way, you get a way higher update rate.
• Mini Batching: Since the frequent updates of SGD are quite costly (updating the gradients is kind of tedious to perform), and can lead to worse results in certain circumstances, it is helpful to aggregate multiple (but not all) examples into one update. This means, you would go through n examples (where n is your batch size), and then update. This will still result in multiple updates within one epoch, but not necessarily as many as with SGD.
• Epoch: One epoch simply refers to a pass through all of your training data. You can generally perform as many epochs as you would like.

One another note, you are correct about ADAM. It is generally seen as a more powerful variant of vanilla gradient descent, since it uses more sophisticated heuristics (first order derivatives) to speed up and stabilize convergence.

It works just as you suggest. batch_size parameter does exactly what you would expect: it sets the size of the batch:

• batch_size: Integer or None. Number of samples per gradient update. If unspecified, batch_size will default to 32.

From programming point of view, Keras decouples the weight update formula parameters specific to each optimizer (learning rate, momentum, etc.) from the global training properties (batch size, training length, etc.) that are share between methods. It is matter of convenience—there is no point in having optimizers SGD, MBGD, BGD that all do the same thing just with different batch size.

Keras has an inbuilt batch_size argument to its model.fit method (since you tagged this question with keras I assume that you're using it). I believe that this will probably be the best optimised method to achieve what you're looking for.