Tensorflow fail with "Unable to retrieve item from feed as bytes." when trying to restore a breakpoint

Question

I am using Tensorflow r0.12.

I use google-cloud-ml locally to run two different training exercises. In the first assignment, I find good starting values ​​for my variables. I store them at checkpoint V2.

When I try to restore my variables for use in the second task:

import tensorflow as tf sess = tf.Session() new_saver = tf.train.import_meta_graph('../variables_pred/model.ckpt-10151.meta', clear_devices=True) new_saver.restore(sess, tf.train.latest_checkpoint('../variables_pred/')) all_vars = tf.trainable_variables() for v in all_vars: print(v.name) 

The following error message appeared:

 tensorflow.python.framework.errors_impl.InternalError: Unable to get element from the feed as bytes. 

A breakpoint is created with these lines in the first job:

 saver = tf.train.Saver() saver.export_meta_graph(filename=os.path.join(output_dir, 'export.meta')) saver.save(sess, os.path.join(output_dir, 'export'), write_meta_graph=False) 

According to this answer, this may be due to the lack of a metadata file, but I am uploading a metadata file.

PS: I use the argument clear_devices=True , because the device specifications created at startup in google-cloud-ml are quite complicated, and I don’t need to receive the same send.

Answer 1

The error message was due to the absence of a file named "breakpoint" by mistake.

After re-entering this file in the appropriate folder, it appears that the load of the checkpoint is working.

Sorry to miss this key point.

Answer 2

I think the problem may be that when you save the model, you set write_meta_graph = False. As a result, I don’t think that you really save the schedule, so when you try to recover, there is no schedule for recovery. Try setting write_meta_graph = True

Answer 3

The error message was also due to errors in the file labeled "breakpoint" by inadvertency.

For example, the folder containing the models was moved, but the value of "model_checkpoint_path:" at the "breakpoint" is still the old path.