Scipy.optimize.fmin_bfgs function evaluates both f and fprime

Question

I use scipy.optimize.fmin_bfgs(f, init_theta, fprime) to minimize f that has a fprime gradient. I calculate f and fprime in one function because most of the calculations are the same, so there is no need to do this twice.

Is it possible to call fmin_bfgs() , specifying a single function that returns both f and fprime ?

Answer 1

If you are trying to save the calculation time, and not just combine the calculations of f and f' for the convenience of the code, it seems that you need an additional shell around the function for caching values, since fmin_bfgs does not seem to allow you to pass such a function (unlike some other optimization functions).

Here is one way to do this by supporting the 10 most recently rated points in a small cache. (I'm not sure if calls to this function should be thread safe: maybe not, but if so, you probably need to add some locks here.)

 def func_wrapper(f, cache_size=10): evals = {} last_points = collections.deque() def get(pt, which): s = pt.tostring() # get binary string of numpy array, to make it hashable if s not in evals: evals[s] = f(pt) last_points.append(s) if len(last_points) >= cache_size: del evals[last_points.popleft()] return evals[s][which] return functools.partial(get, which=0), functools.partial(get, which=1) 

If we then do

 >>> def f(x): ... print "evaluating", x ... return (x-3)**2, 2*(x-3) >>> f_, fprime = func_wrapper(f) >>> optimize.fmin_bfgs(f_, 1000, fprime) evaluating [ 994.93480441] evaluating [ 974.67402207] evaluating [ 893.63089268] evaluating [ 665.93446894] evaluating [ 126.99931561] evaluating [ 3.] Optimization terminated successfully. Current function value: 0.000000 Iterations: 4 Function evaluations: 7 Gradient evaluations: 7 array([ 3.]) 

we can see that we are not repeating any ratings.

Answer 2

Suppose you have a Python function f(x) that returns both the value of the function and the gradient:

 In [20]: def f(x): ....: return (x-3)**2, 2*(x-3) 

Then just pass the outputs separately like this:

 In [21]: optimize.fmin_bfgs(lambda x: f(x)[0], 1000, lambda x: f(x)[1]) Optimization terminated successfully. Current function value: 0.000000 Iterations: 4 Function evaluations: 7 Gradient evaluations: 7 Out[21]: array([ 3.]) 

Answer 3

There seems to be no way to do this directly. But scipy.optimize.minimize allows you to do this. You can pass True to fprime instead of a function. This means that f returns a tuple of function value and gradient. You can invoke minimize with the = 'BFGS' method to get the desired effect.

minimize enlightening, let's see the source code for minimize . Both he and fmin_bfgs end up calling _minimize_bfgs, which takes f and fprime as separate function arguments. When fprime is logical, minimize skillfully constructs fprime as an object that remembers the last value returned by f and caches the gradient information.