Neural network cost function in MATLAB

Question

How to implement this neural network cost function in Matlab:

Here is what the symbols represent:

% m is the number of training examples. [a scalar number] % K is the number of output nodes. [a scalar number] % Y is the matrix of training outputs. [an m by k matrix] % y^{(i)}_{k} is the ith training output (target) for the kth output node. [a scalar number] % x^{(i)} is the ith training input. [a column vector for all the input nodes] % h_{\theta}(x^{(i)})_{k} is the value of the hypothesis at output k, with weights theta, and training input i. [a scalar number] %note: h_{\theta}(x^{(i)}) will be a column vector with K rows. 

I am having problems with nested sums, offset nodes, and the overall complexity of this equation. I also struggle because there are 2 weight matrices, one of which connects the inputs to the hidden layer and connects the hidden layer to the outputs. Here is my attempt.

Define Variables

 m = 100 %number of training examples K = 2 %number of output nodes E = 2 %number of input nodes A = 2 %number of nodes in each hidden layer L = 1 %number of hidden layers Y = [2.2, 3.5 %targets for y1 and y2 (see picture at bottom of page) 1.7, 2.1 1.9, 3.6 . . %this is filled out in the actual code but to save space I have used ellipsis. there will be m rows. . . . . 2.8, 1.6] X = [1.1, 1.8 %training inputs. there will be m rows 8.5, 1.0 9.5, 1.8 . . . . . . 1.4, 0.8] W1 = [1.3, . . 0.4 %this is just an E by A matrix of random numbers. this is the matrix of initial weights. . . . - 2 . . . 3.1 . . . - 1 2.1, -8, 1.2, 2.1] W2 = [1.3, . . 0.4 %this is an A by K matrix of random numbers. this is the matrix of initial weights. . . . - 2 . . . 3.1 . . . - 1 2.1, -8, 1.2, 2.1] 

The hypothesis using these weights is ...

 Htheta = sigmf( dot(W2 , sigmf(dot(W1 , X))) ) %This will be a column vector with K rows. 

The cost function using these weights is ... (That's where I'm afraid)

  sum1 = 0 for i = 1:K sum1 = sum1 + Y(k,i) *log(Htheta(k)) + (1 - Y(k,i))*log(1-Htheta(k)) 

I just keep writing such things, and then I realize that all this is wrong. I can’t let my life determine how to make invested amounts, or include an input matrix, or do any of this. All this is very complicated.

How to create this equation in matlab?

Thank you very much!

Two-layer neural network with 2 inputs, 2 outputs, 2 hidden nodes and 2 offset units http://imagizer.imageshack.us/v2/320x240q90/40/92bn.jpg

Note. The code has weird colors since stackoverflow doesn't know what I'm programming in MATLAB. I also wrote the code directly in stackoverflow, so it may have syntax errors. I'm more interested in the general idea of ​​how I should do this, and not just copy and paste the code. It is for this reason that I did not worry about half columns, etc.

Answer 1

I implemented neural networks using the same error function as above. Unfortunately, I have not worked with Matlab for quite some time, but I am pretty good at Octave, which I hope you will find useful anyway, since many of the Octave functions are similar to Matlab functions.

@sashkello provided a nice piece of code to calculate the cost function. However, this code is written using a loop structure, and I would like to suggest a vectorized implementation.

To evaluate the current theta values, we need to forward forward propagation throughout the network. I assume that you know how to write live code because you only care about J(theta) errors. Let the vector representing the results of your direct distribution be F

Once you have completed the forward function, you will need to execute the equation. Notice, I implement this in vector form.

 J = (-1/m) * sum(sum(Y .* log(F) + (1-Y) .* log(1-F),2)); 

This will compute the summation part related to:

Now we need to add a regularization member that:

As a rule, we will have an arbitrary number of theta matrices, but in this case we have 2, so we can just do a few sums to get:

 J =J + (lambda/(2*m)) * (sum(sum(theta_1(:,2:end).^2,2)) + sum(sum(theta_2(:,2:end).^2,2))); 

Please note that in each amount I work only from the second column. This is because the first column will correspond to the theta values ​​that we prepared for the offset blocks.

So, there is a vectorized implementation of computing J.

Hope this helps!

Answer 2

I think that Htheta is an array of K * 2. Note that you need to add the offset ( x0 and a0 ) when calculating the direct cost function. I showed you the size of the array at each step, assuming that you have two nodes at the input, hidden and output levels as comments in the code.

 m = size(X, 1); X = [ones(m,1) X]; % m*3 in your case % W1 2*3, W2 3*2 a2 = sigmf(W1 * X'); % 2*m a2 = [ones(m,1) a2']; % m*3 Htheta = sigmf(a2 * W2); % m*2 J = (1/m) * sum ( sum ( (-Y) .* log(Htheta) - (1-Y) .* log(1-Htheta) )); t1 = W1(:,2:size(W1,2)); W2 = W2'; t2 = W2(:,2:size(W2,2)); % regularization formula Reg = lambda * (sum( sum ( t1.^ 2 )) + sum( sum ( t2.^ 2 ))) / (2*m); 

Answer 3

Ok, since I understand that your question has nothing to do with neural networks, but basically asks how to make a nested sum in matlab. I really do not want to introduce the whole equation above, but, that is, the first part of the first sum will look like this:

 Jtheta = 0 for i=1:m, for j=1:K, Jtheta = Jtheta + Y(i, j) * log(Htheta(X(i))(j)) end end 

where Jtheta is your result.

Answer 4

This works for any number of hidden layers:

 % Allow arbitrary network architectures. Create cell array of all Theta parameters Theta={Theta1; Theta2}; % Compute unregularised cost (J) J = 1/m * sum(sum((-y .* log(hX) - (1 - y) .* log(1 - hX)))); % Add regularisation for i = 1:length(Theta) J += lambda / 2 / m * sum(sum(Theta{i}(:,2:end) .^ 2)); end