Overflow error in the implementation of neural networks

Question

I am trying to create my own implementation of a neural network feedback propagation algorithm. The code I wrote for training so far,

def train(x,labels,n): lam = 0.5 w1 = np.random.uniform(0,0.01,(20,120)) #weights w2 = np.random.uniform(0,0.01,20) for i in xrange(n): w1 = w1/np.linalg.norm(w1) w2 = w2/np.linalg.norm(w2) for j in xrange(x.shape[0]): y1 = np.zeros((600)) #output d1 = np.zeros((20)) p = np.mat(x[j,:]) a = np.dot(w1,pT) #activation z = 1/(1 + np.exp((-1)*a)) y1[j] = np.dot(w2,z) for k in xrange(20): d1[k] = z[k]*(1 - z[k])*(y1[j] - labels[j])*np.sum(w2) #delta update rule w1[k,:] = w1[k,:] - lam*d1[k]*x[j,:] #weight update w2[k] = w2[k] - lam*(y1[j]-labels[j])*z[k] E = 1/2*pow((y1[j]-labels[j]),2) #mean squared error print E return 0 

No input blocks - 120, No hidden units - 20, No output blocks - 1, No training samples - 600

x - training set 600 * 120 with zero average and single change with a maximum value of 3.28 and a minimum value of -4.07. The first 200 samples belong to class 1, the second 200 to class 2 and from 200 to 3 classes. Labels are class labels assigned to each instance, n is the number of iterations required for convergence. Each sample has 120 functions.

I initialized the weights from 0 to 0.01, and the input data is scaled to have a unit variance and a zero mean value, and still the code throws an overflow warning, resulting in the values ​​"a", i.e. NaN values ​​are NaN values. I can’t understand what the problem is.

Each sample has 120 elements. Example line x:

 [ 0.80145231 1.29567936 0.91474224 1.37541992 1.16183938 1.43947296 1.32440357 1.43449479 1.32742415 1.40533852 1.28817561 1.37977183 1.2290933 1.34720161 1.15877069 1.29699635 1.05428735 1.21923531 0.92312685 1.1061345 0.66647463 1.00044203 0.34270708 1.05589558 0.28770958 1.21639524 0.31522575 1.32862243 0.42135899 1.3997094 0.5780146 1.44444501 0.75872771 1.47334256 0.95372771 1.48878048 1.13968139 1.49119962 1.33121905 1.47326017 1.47548571 1.4450047 1.58272343 1.39327328 1.62929132 1.31126604 1.62705274 1.21790335 1.59951034 1.12756958 1.56253815 1.04096709 1.52651382 0.95942134 1.48875633 0.87746762 1.45248623 0.78782313 1.40446404 0.68370011 

Answer 1

overflow

The logistic sigmoid function tends to overflow in NumPy with increasing signal power. Try adding the following line of code:

 np.clip( signal, -500, 500 ) 

This will limit the values ​​in the NumPy matrices within the specified interval. This, in turn, will prevent overflow of precision in the sigmoid function.

 >>> arr array([[-900, -600, -300], [ 0, 300, 600]]) >>> np.clip( arr, -500, 500) array([[-500, -500, -300], [ 0, 300, 500]]) 

Implementation

This is a snippet that I use in my projects:

 def sigmoid_function( signal ): # Prevent overflow. signal = np.clip( signal, -500, 500 ) # Calculate activation signal signal = 1.0/( 1 + np.exp( -signal )) return signal #end 

Why is the sigmoid function overflowing?

As learning progresses, the network improves its accuracy. As this accuracy approaches perfection, the sigmoidal signal will approach 1 from below or 0 from above. For example: 0.99999999999 ... or 0.00000000000000001 ...

Because NumPy is focused on performing high-precision numerical operations, it will try to maintain the highest possible accuracy and thus cause an overflow error. Note. This error message can be ignored by setting:

 np.seterr( over='ignore' )