TensorFlow: How to set epoch-based learning speed decay?

Question

The decay function of the learning speed tf.train.exponential_decay accepts the decay_steps parameter. To reduce the learning speed every num_epochs , you should set decay_steps = num_epochs * num_train_examples / batch_size . However, when reading data from .tfrecords files .tfrecords you do not know how many training examples there are.

To get num_train_examples , you can:

• Configure tf.string_input_producer with num_epochs=1 .
• Run this via tf.TFRecordReader / tf.parse_single_example .
• Loop and count how many times it produces some result before stopping.

However, it is not very elegant.

Is there an easier way to get the number of training examples from a .tfrecords file or to set the breakdown of learning speed based on eras instead of steps?

Answer 1

You can use the following code to get the number of entries in the .tfrecords file:

 def get_num_records(tf_record_file): return len([x for x in tf.python_io.tf_record_iterator(tf_record_file)]) 

Answer 2

In the tutorial below,

 learning_rate = tf.train.exponential_decay(starter_learning_rate, global_step, 100000, 0.96, staircase=True) 

starter_learning_rate can be changed after the desired eras by defining a function such as:

 def initial_learning_rate(epoch): if (epoch >= 0) and (epoch < 100): return 0.1 if (epoch >= 100) and (epoch < 200): return 0.05 if (epoch >= 200) and (epoch < 500): return 0.001 

And then you can initialize your starter_learning_rate inside a for loop (iterating over epochs) as follows:

 for epoch in range(epochs): #epochs is the total number of epochs starter_learning_rate = initial_learning_rate(epoch) ... 

Note

The global_step variable does not change:

 decayed_learning_rate = starter_learning_rate * decay_rate ^ (global_step / decay_steps) 

Answer 3

I recommend that you set the breakdown of your learning speed according to changes in your studies or loss of grade. If the loss fluctuates, you can reduce the learning speed. It is unlikely that you can predict from which era or step you should reduce it before starting training.