Geek Answers Handbook

Keras autoencoder doesn't converge

Can someone explain to me why the auto-encoder does not fit? For me, the results of the two networks below should be the same. However, the autocoders below do not converge, while the network is below it.

# autoencoder implementation, does not converge autoencoder = Sequential() encoder = containers.Sequential([Dense(32,16,activation='tanh')]) decoder = containers.Sequential([Dense(16,32)]) autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=True)) rms = RMSprop() autoencoder.compile(loss='mean_squared_error', optimizer=rms) autoencoder.fit(trainData,trainData, nb_epoch=20, batch_size=64, validation_data=(testData, testData), show_accuracy=False) 
  # non-autoencoder implementation, converges model = Sequential() model.add(Dense(32,16,activation='tanh')) model.add(Dense(16,32)) model.compile(loss='mean_squared_error', optimizer=rms) model.fit(trainData,trainData, nb_epoch=numEpochs, batch_size=batch_size, validation_data=(testData, testData), show_accuracy=False)
+6
source share
2 answers

I think that the Keras Autoencoder implementation binds the weights of the encoder and decoder, while in your implementation the encoder and decoder have separate weights. If your implementation results in significantly better test data performance, this may indicate that unrelated scales may be required for your problem.

+2
source

The new version (0.3.0) of Keras no longer tied weights in AutoEncoder, and it still shows different convergence. This is because weights are initialized differently.

In an example other than AE, weight (32.16) weights are first initialized, then Dense (16.32). In the AE example, the weight (32.16) weights are first initialized, then Dense (16.32), and then when you create an AutoEncoder instance, the Dense (32.16) weights are initialized again (self.encoder.set_previous (node) will call build () to initialize the weights).

Now the following two NNs converge in exactly the same way:

 autoencoder = Sequential() encoder = containers.Sequential([Dense(32,16,activation='tanh')]) decoder = containers.Sequential([Dense(16,32)]) autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=True)) rms = RMSprop() autoencoder.compile(loss='mean_squared_error', optimizer=rms) np.random.seed(0) autoencoder.fit(trainData,trainData, nb_epoch=20, batch_size=64, validation_data=(testData, testData), show_accuracy=False) 
 # non-autoencoder model = Sequential() model.add(Dense(32,16,activation='tanh')) model.add(Dense(16,32)) model.set_weights(autoencoder.get_weights()) model.compile(loss='mean_squared_error', optimizer=rms) np.random.seed(0) model.fit(trainData,trainData, nb_epoch=numEpochs, batch_size=batch_size, validation_data=(testData, testData), show_accuracy=False)
+2
source

More articles:


All Articles