Matrix determinant differentiation in a tensor flow

Question

I am interested in calculating the derivative of the matrix determinant using TensorFlow. From experiments, I see that TensorFlow did not implement the differentiation method through the determinant:

LookupError: No gradient defined for operation 'MatrixDeterminant' (op type: MatrixDeterminant) 

Further research showed that it is actually possible to calculate the derivative; see, for example, the Jacobi formula . I decided that to implement this means of differentiation through the determinant that I need to use a function decorator,

 @tf.RegisterGradient("MatrixDeterminant") def _sub_grad(op, grad): ... 

However, I am not familiar with tensor flow to understand how this can be done. Does anyone have an idea about this issue?

Here is an example where I ran into this problem:

 x = tf.Variable(tf.ones(shape=[1])) y = tf.Variable(tf.ones(shape=[1])) A = tf.reshape( tf.pack([tf.sin(x), tf.zeros([1, ]), tf.zeros([1, ]), tf.cos(y)]), (2,2) ) loss = tf.square(tf.matrix_determinant(A)) optimizer = tf.train.GradientDescentOptimizer(0.001) train = optimizer.minimize(loss) init = tf.initialize_all_variables() sess = tf.Session() sess.run(init) for step in xrange(100): sess.run(train) print sess.run(x) 

Answer 1

Please check the box "Implement Gradient in Python" here

In particular, you can implement it as follows

 @ops.RegisterGradient("MatrixDeterminant") def _MatrixDeterminantGrad(op, grad): """Gradient for MatrixDeterminant. Use formula from 2.2.4 from An extended collection of matrix derivative results for forward and reverse mode algorithmic differentiation by Mike Giles -- http://eprints.maths.ox.ac.uk/1079/1/NA-08-01.pdf """ A = op.inputs[0] C = op.outputs[0] Ainv = tf.matrix_inverse(A) return grad*C*tf.transpose(Ainv) 

Then a simple workout cycle to verify that it works:

 a0 = np.array([[1,2],[3,4]]).astype(np.float32) a = tf.Variable(a0) b = tf.square(tf.matrix_determinant(a)) init_op = tf.initialize_all_variables() sess = tf.InteractiveSession() init_op.run() minimization_steps = 50 learning_rate = 0.001 optimizer = tf.train.GradientDescentOptimizer(learning_rate) train_op = optimizer.minimize(b) losses = [] for i in range(minimization_steps): train_op.run() losses.append(b.eval()) 

Then you can visualize your loss over time.

 import matplotlib.pyplot as plt plt.ylabel("Determinant Squared") plt.xlabel("Iterations") plt.plot(losses) 

Gotta see something like this

Answer 2

I think you are embarrassed at being a derivative of the matrix determinant .

The matrix determinant is a function that is calculated by the elements of the matrix according to some formula. Therefore, if all elements of the matrix are numbers, you are the determinant, you will receive only one number, and the derivative will be 0 . When some of the elements are variables, you will get an expression of these variables. For example:

 x, x^2 1, sin(x) 

The determinant will be x*sin(x) - x^2 , and the derivative 2x + sin(x) + x*cos(x) . The Jacobi formula simply associates the determinant with the adjunct matrix.

---

In your example, your matrix A consists only of numbers, so the determinant is just a number, and loss is just a number. GradientDescentOptimizer needs to have some free variables to minimize and not have them, because your loss is just a number.

Answer 3

For those interested, I found a solution that works on my problems:

 @tf.RegisterGradient("MatrixDeterminant") def _MatrixDeterminant(op, grad): """Gradient for MatrixDeterminant.""" return op.outputs[0] * tf.transpose(tf.matrix_inverse(op.inputs[0]))