Summation over the list of tensors in the tensor flow

Question

I have a deep neural network where weights between layers are stored in a list.

layers[j].weights I want to include a comb fine in my cost function. I need then to use something like tf.nn.l2_loss(layers[j].weights**2 for j in range(self.n_layers)) ie the square of the sum of all weights.

In particular, weights are defined as:

 >>> avs.layers [<neural_network.Layer object at 0x10a4b2a90>, <neural_network.Layer object at 0x10ac85080>, <neural_network.Layer object at 0x10b0f3278>, <neural_network.Layer object at 0x10b0eacf8>, <neural_network.Layer object at 0x10b145588>, <neural_network.Layer object at 0x10b165048>, <neural_network.Layer object at 0x10b155ba8>] >>> >>> avs.layers[0].weights <tensorflow.python.ops.variables.Variable object at 0x10b026748> >>> 

How to do it in tensor flow?

Answer 1

The standard way to summarize the list of tensors is to use the tf.add_n() operation, which takes a list of tensors (each of which has the same size and shape) and creates one tensor containing the sum.

For the specific problem you are having, I assume that each layers[j].weights may have a different size. Therefore, before summing, you will need to reduce each element to a scalar, for example, using the tf.nn.l2_loss() function tf.nn.l2_loss() :

 weights = [layers[j].weights for j in range(self.n_layers)] losses = [tf.nn.l2_loss(w) for w in weights] total_loss = tf.add_n(losses) 

(However, note that when the values ​​to be added are large, it may be more efficient to calculate the tf.add() sequence of operations, since TensorFlow stores the values ​​of each of the add_n arguments in memory until all of them are in the add chain of operations allows some calculations to happen earlier.)

Answer 2

The tf.nn.l2_loss() function returns a tensor with 0 dimensions.

But I don’t need to manually apply this to every weight tensor, so storing weight tensors in the list is one way to solve the problem (as @mrry noted).

But instead of writing every time what you can do, use the following function

 def l2_loss_sum(list_o_tensors): return tf.add_n([tf.nn.l2_loss(t) for t in list_o_tensors]) 

In your case, it will look like this:

 total_loss = l2_loss_sum([layers[j].weights for j in range(self.n_layers)]) 

In addition, tf.nn.l2_loss() implicitly applies the squared operation to the values, as well as multiplying all the quadratic values ​​by 1/2, so you used something like tf.nn.l2_loss(layers[j].weights**2 for j in range(self.n_layers)) , which you would tf.nn.l2_loss(layers[j].weights**2 for j in range(self.n_layers)) weight to the 4th degree. As a result, your derivative of this term loss would be strange: it would not cancel 1/2 to 1 (but implicitly double your β), and the weights would be cubed.