Where should I apply dropouts to a convolutional layer?

Question

Since the word “layer” often means different things when applied to a convolutional layer (some process everything by combining into one layer, others consider convolution, non-linearity and combining as separate “layers”; see Fig. 9.7 ) it’s not clear where to apply screening in the convolutional layer.

Does non-linearity and unification arise?

---

For example, in TensorFlow it would be something like:

kernel_logits = tf.nn.conv2d(input_tensor, ...) + biases activations = tf.nn.relu(kernel_logits) kept_activations = tf.nn.dropout(activations, keep_prob) output = pool_fn(kept_activations, ...) 

Answer 1

You might be trying to apply the drop in different places, but from the point of view of preventing retraining, you are not sure that you will see most of the problem before the collection. What I saw for CNN is that tensorflow.nn.dropout applies AFTER non-linearity and union:

  # Create a convolution + maxpool layer for each filter size pooled_outputs = [] for i, filter_size in enumerate(filters): with tf.name_scope("conv-maxpool-%s" % filter_size): # Convolution Layer filter_shape = [filter_size, embedding_size, 1, num_filters] W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name="W") b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name="b") conv = tf.nn.conv2d( self.embedded_chars_expanded, W, strides=[1, 1, 1, 1], padding="VALID", name="conv") # Apply nonlinearity h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu") # Maxpooling over the outputs pooled = tf.nn.max_pool( h, ksize=[1, sequence_length - filter_size + 1, 1, 1], strides=[1, 1, 1, 1], padding='VALID', name="pool") pooled_outputs.append(pooled) # Combine all the pooled features num_filters_total = num_filters * len(filters) self.h_pool = tf.concat(3, pooled_outputs) self.h_pool_flat = tf.reshape(self.h_pool, [-1, num_filters_total]) # Add dropout with tf.name_scope("dropout"): self.h_drop = tf.nn.dropout(self.h_pool_flat, self.dropout_keep_prob)