They are (mostly) equivalent. Binary cross entropy is equivalent to categorical cross entropy where the logit score for the second class is fixed to zero (which can be shown easily by plugging in a zero in the cross entropy formula). Fixing the last class to zero does not reduce the representational power of the model, because classes probabilities will be normalized to one anyway. It is technically true that this will affect the learning dynamics during training, since cross-entropy with two classes will have more parameters than the a binary-cross entropy model, but generally that is not expected to make a difference in performance. More on reddit.com

There are at least three reasons that are in favour of the multiclass cross-entropy: coherence, the weak likelihood principle, and the concept of proper scoring rules. Coherence: when you encode a categorical outcome into multiple binary outcomes and model probabilities for each binary outcome then your model assigns probability to events that cannot happen (e.g. class 1 and 2 being true at the same time). Such statements are incoherent because you can simultaneously believe things which cannot be true together. Weak likelihood principle: this principle from statistics says that any conclusions obtained from data and a model must be obtained as a result of the likelihood function. There are good reasons for this principle, see e.g. Berger's book on the likelihood principle. Now, when using a sum of binary cross entropies for a categorical outcome you do not use the likelihood of a single probabilistic model. The normal multiclass cross-entropy is a function of the likelihood. Proper scoring rules: when estimating parameters of a probabilistic model by minimising a function the function being minimised should be what is called a proper scoring rule. Proper scoring rules guarantee that you could recover true probabilities with enough data and a sufficiently rich model, see Gneiting and Raftery, JASA 2007. Now, the binary cross entropy is indeed a proper scoring rule for binary outcomes, but the sum of binary cross entropies is not a proper scoring rule for categorical outcomes which are merely encoded as one hot vectors. More on reddit.com

This seems to be a good explanation More on reddit.com

BCE is a loss function not an evaluation metric of the model. The loss function is used during training to tune parameters through gradient descent. Minimizing BCE is how your model “learns” but you need something after to evaluate how well your model learned. So Accuracy on the other hand is just the percentage of correct v incorrect. Much more intuitive when evaluating a model but not the only option! Precision, recall, F1 score are all other examples More on reddit.com