Logic / sigmoid function implementation numerical accuracy

Question

c scipy.special.expit, the logistic function is implemented as follows:

if x < 0
    a = exp(x) 
    a / (1 + a) 
else 
    1 / (1 + exp(-x))

However, I have seen implementations in other languages ​​/ frameworks that simply execute

1 / (1 + exp(-x))

I am wondering what benefit the meager version will bring.

For very small xresults close to 0. It works even if it exp(-x)overflows to Inf.

Answer 1

It is really just for stability. Inserting values ​​that are very large can otherwise return unexpected results.

expit , 1 / (1 + exp(-x)), -710 nan, -709 , , . , exp(710) , .

, .

. .

Answer 2

, :

if x < -709
  sigmoid = 0.0
else
  sigmoid = 1.0 / (1.0 + exp(-x))

10 ^ -309 (. ), !

>>> 1 / (1 + math.exp(709.78))
5.5777796105262746e-309

Answer 3

:

python np.where(x > 0, 1. / (1. + np.exp(-x)), np.exp(x) / (np.exp(x) + np.exp(0)))

np.exp(x) / (np.exp(x) + np.exp(0)) 1. / (1. + np.exp(-x)),