SHORT ANSWER According to other answers Multinomial Logistic Loss and Cross Entropy Loss are the same.

Cross Entropy Loss is an alternative cost function for NN with sigmoids activation function introduced artificially to eliminate the dependency on on the update equations. Some times this term slows down the learning process. Alternative methods are regularised cost function.

In these type of networks one might want to have probabilities as output but this does not happen with the sigmoids in a multinomial network. The softmax function normalizes the outputs and force them in the range . This can be useful for example in MNIST classification.

LONG ANSWER WITH SOME INSIGHTS

The answer is quite long but I'll try to summarise.

The first modern artificial neurons that have been used are the sigmoids whose function is:

which has the following shape:

The curve is nice because it guarantees the output is in the range .

Regarding the choice of a cost function, a natural choice is the quadratic cost function, whose derivative is guaranteed to exist and we know it has a minimum.

Now consider a NN with sigmoids trained with the quadratic cost function, with layers.

We define the cost function as the sum of the squared errors in the output layer for a set of inputs :

where is the j-th neuron in the output layer , the desired output and is the number of training examples.

For simplicity let's consider the error for a single input:

Now an activation output of for the neuron in the layer, is:

Most of the times (if not always) NN are trained with one of the gradient descent techniques, which basically consists updating the weights and biases by small steps towards the direction of minimization. The goal is to a apply a small change in the weights and biases towards the direction that minimizes the cost function.

For small steps the following holds:

Our are the weights and biases. Being it a cost function we want to minimise, i.e., find the proper value . Suppose we choose , then:

which means the change in the parameter decreased the cost function by .

Consider the -th output neuron:

Suppose we want to update the weight which is the weight from the neuron in the layer to the -th neuron in the \ell layer. Then we have:

Taking the derivatives using the chain rule:

You see the dependency on the derivative of the sigmoid (in the first is w.r.t. in the second w.r.t actually, but it does not change a lot since both are exponents).

Now the derivative for a generic single variable sigmoid is:

Now consider a single output neuron and suppose you that neuron should output instead it is outputting a value close to : you'll see both from the graph that the sigmoid for values close to is flat, i.e. its derivative is close to , i.e. updates of the parameter are very slow (since the update equations depend on .

Motivation of the cross-entropy function

To see how cross-entropy has been originally derived, suppose one has just found out that the term is slowing down the learning process. We might wonder if it is possible to choose a cost function to make the term disappear. Basically one might want:

\begin{equation} \begin{aligned} \frac{\partial C}{\partial w} & =x \left( a - y\right)\\ \frac{\partial C}{\partial b} =\left( a - y\right) \end{aligned} \end{equation} From the chain-rule we have: \begin{equation} \frac{\partial C}{\partial b} =\frac{\partial C}{\partial a} \frac{\partial a}{\partial b } =\frac{\partial C}{\partial a}\sigma'(z) = \frac{\partial C}{\partial a} \sigma(1-\sigma) \end{equation} Comparing the desired equation with the one of the chain rule, one gets \begin{equation} \frac{\partial C}{\partial a} = \frac{a-y}{a(1-a)} \end{equation} Using the cover-up method : \begin{equation} \frac{\partial C}{\partial a} = -\left[ y\ln a + (1-y)\ln(1-a)\right]+const \end{equation} To get the full cost function, we must average over all the training samples \begin{equation} \frac{\partial C}{\partial a} = -\frac{1}{n}\sum_x\left[y\ln a + (1-y)\ln(1-a)\right]+const \end{equation} where the constant here is the average of the individual constants for each training example.

There is a standard way of interpreting the cross-entropy that comes from the field of information theory. Roughly speaking, the idea is that the cross-entropy is a measure of surprise. We get low surprise if the output is what we expect (), and high surprise if the output is unexpected.

Softmax

For a binary classification cross-entropy resembles the definition in information theory and the values can still be interpreted as probabilities.

With multinomial classification this does not hold true anymore: the outputs do note sum up to .

If you want them to sum up to you use softmax function, which normalize the outputs so that the sum is .

Also if the output layer is made up of softmax functions, the slowing down term is not present. If you use log-likelihood cost function with a softmax output layer, the result you will obtain a form of the partial derivatives, and in turn of the update equations, similar to the one found for a cross-entropy function with sigmoid neurons

However