Why in the convolutional neural network are low losses possible, but also very low accuracy?

Question

I am new to computer training and am currently trying to train a convolutional neural network with 3 convolutional layers and 1 fully connected layer. I use a dropout probability of 25% and a learning speed of 0.0001. I have 6000 150x200 training images and 13 classes of lessons. I am using shadoworflow. I notice a trend where my loss is steadily decreasing, but my accuracy is slightly increasing, and then falling again. My training images are blue lines, and my verification images are orange lines. The x axis is the steps.

I am wondering if there is something that I do not understand or what could be the possible causes of this phenomenon? From the material I read, I believed that low loss means high accuracy. Here is my loss function.

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, y)) 

Answer 1

This is because Loss and Accuracy are two completely different things (at least logically)!

Consider an example where you define loss as:

 loss = (1-accuracy) 

In this case, when you try to minimize loss , accuracy automatically increases.

Now consider another example in which you define loss as:

 loss = average(prediction_probabilities) 

Although this does not make any sense, it is technically still a valid loss function, and your weights is still set up to minimize such loss .

But, as you can see, in this case there is no relationship between loss and accuracy , so you cannot expect a simultaneous increase / decrease.

Note: loss will always be minimized (thus, your loss reduced after each iteration)!

PS: update your question with the loss function that you are trying to minimize.

Answer 2

softmax_cross_entropy_with_logits (), and accuracy are two different concepts with different definitions of formulas. In normal cases, we could expect to get better accuracy by minimizing the cross-entropy of softmax, but they are calculated differently, so we could not expect them to always increase or decrease synchronously.

We use softmax cross-entropy in CNN because it is effective for training a neural network. If we use the loss function = (1-accuracy) as a function of loss, it is very difficult to achieve a better result by adjusting weights for our CNN nervous system using our modern backprogation recovery preparation programs, I really did it and confirmed this conclusion, you also you can try it yourself. Maybe this is due to our current decision on poor learning back, maybe it is due to the definition of our neurons (do we need to change it to some other types of neurons?), But in any case, at present, using the accuracy of the loss function is not effective way for a neural network, so just use softmax_cross_entropy_with_logits (), because those scientists from AI told us that they already confirmed that this method is effective, for other ways we still don’t know them.