This is a branch from another question / answer
I need a function equivalent to this:
def softmax(x, tau): """ Returns softmax probabilities with temperature tau Input: x -- 1-dimensional array Output: s -- 1-dimensional array """ e_x = np.exp(x / tau) return e_x / e_x.sum()
which is stable and reliable, i.e. it does not overflow at small values of tau , nor at large x . Since this will be used to calculate probabilities, the output should be 1.
In other words, I pass in some values (and temperature), and I want to output an array of probabilities "scaled" with input and tau.
Examples:
In [3]: softmax(np.array([2,1,1,3]), 1) Out[3]: array([ 0.22451524, 0.08259454, 0.08259454, 0.61029569]) In [5]: softmax(np.array([2,1,1,3]), 0.1) Out[5]: array([ 4.53978685e-05, 2.06106004e-09, 2.06106004e-09, 99954598e-01]) In [7]: softmax(np.array([2,1,1,3]), 5) Out[7]: array([ 0.25914361, 0.21216884, 0.21216884, 0.31651871])
Since tau goes to 0, the highest probability in the output is at the position of the highest element. As tau grows larger, probabilistic ones become closer to each other.
Optionally, questions about a related answer. There Neil gives the following alternative:
def nat_to_exp(q): max_q = max(0.0, np.max(q)) rebased_q = q - max_q return np.exp(rebased_q - np.logaddexp(-max_q, np.logaddexp.reduce(rebased_q)))
However, this conclusion cannot be summed with 1, and the explanation is that the function returns a categorical distribution that has only N-1 free parameters, the last of which 1 - sum(others) . But at startup, I notice that for a vector of length 3, it returns a vector of length 3. So, where is it missing? Can I make it equivalent to the above example?
Why is this answer stable? How to get from a simple softmax formula to this?
Perhaps a related question: general softmax , but no temperature