Embed an early stop in tf.estimator.DNNRegressor using the available training hooks

Question

I'm new to tensorflow and want to implement an early stop in tf.estimator.DNNRegressorwith the available Training Hooks for the MNIST dataset. An early stop stops the workout if the loss does not improve for a certain number of steps. The Tensorflow documentation provides only an example of Logging hooks . Can someone write a code snippet to implement it?

Answer 1

Here is an example implementation EarlyStoppingHook:

import numpy as np
import tensorflow as tf
import logging
from tensorflow.python.training import session_run_hook


class EarlyStoppingHook(session_run_hook.SessionRunHook):
    """Hook that requests stop at a specified step."""

    def __init__(self, monitor='val_loss', min_delta=0, patience=0,
                 mode='auto'):
        """
        """
        self.monitor = monitor
        self.patience = patience
        self.min_delta = min_delta
        self.wait = 0
        if mode not in ['auto', 'min', 'max']:
            logging.warning('EarlyStopping mode %s is unknown, '
                            'fallback to auto mode.', mode, RuntimeWarning)
            mode = 'auto'

        if mode == 'min':
            self.monitor_op = np.less
        elif mode == 'max':
            self.monitor_op = np.greater
        else:
            if 'acc' in self.monitor:
                self.monitor_op = np.greater
            else:
                self.monitor_op = np.less

        if self.monitor_op == np.greater:
            self.min_delta *= 1
        else:
            self.min_delta *= -1

        self.best = np.Inf if self.monitor_op == np.less else -np.Inf

    def begin(self):
        # Convert names to tensors if given
        graph = tf.get_default_graph()
        self.monitor = graph.as_graph_element(self.monitor)
        if isinstance(self.monitor, tf.Operation):
            self.monitor = self.monitor.outputs[0]

    def before_run(self, run_context):  # pylint: disable=unused-argument
        return session_run_hook.SessionRunArgs(self.monitor)

    def after_run(self, run_context, run_values):
        current = run_values.results

        if self.monitor_op(current - self.min_delta, self.best):
            self.best = current
            self.wait = 0
        else:
            self.wait += 1
            if self.wait >= self.patience:
                run_context.request_stop()

This implementation is based on the Keras implementation .

CNN MNIST example, hook train.

early_stopping_hook = EarlyStoppingHook(monitor='sparse_softmax_cross_entropy_loss/value', patience=10)

mnist_classifier.train(
  input_fn=train_input_fn,
  steps=20000,
  hooks=[logging_hook, early_stopping_hook])

sparse_softmax_cross_entropy_loss/value - .

1:

, "" node ( ).

DNNRegressor node dnn/head/weighted_loss/Sum.

:

• . , :
  WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpInj8SC
  :

  tensorboard --logdir /tmp/tmpInj8SC
  

• GRAPHS.

• . : dnn → head → weighted_loss Sum node ( , node loss).

• , , - node, monitor pf EarlyStoppingHook.

node DNNClassifier . DNNClassifier DNNRegressor loss_reduction, node ( - losses.Reduction.SUM).

2:

, .
GraphKeys.LOSSES, . . , .

, monitor EarlyStoppingHook begin, :

self.monitor = tf.get_default_graph().get_collection(tf.GraphKeys.LOSSES)[0]

, , , .