How to local gae golang urlfetch local test through proxy?

Question

My urlfetch client works great when deployed to appspot. But local testing (dev_appserver.py) through a proxy server has a problem. I cannot find a way to set proxies for urlfetch.Transport.

How do you check the url behind a local proxy?

Answer 1

http.DefaultTransport and http.DefaultClient are not available in App Engine. See https://developers.google.com/appengine/docs/go/urlfetch/overview

This error message was received while testing PayPal OAuth on GAE dev_appserver.py (works when creating at compilation)

const url string = "https://api.sandbox.paypal.com/v1/oauth2/token" const username string = "EOJ2S-Z6OoN_le_KS1d75wsZ6y0SFdVsY9183IvxFyZp" const password string = "EClusMEUk8e9ihI7ZdVLF5cZ6y0SFdVsY9183IvxFyZp" client := &http.Client{} req, _ := http.NewRequest("POST", url, strings.NewReader("grant_type=client_credentials")) req.SetBasicAuth(username, password) req.Header.Set("Accept", "application/json") req.Header.Set("Accept-Language", "en_US") req.Header.Set("Content-Type", "application/x-www-form-urlencoded") resp, err := client.Do(req) 

As you can see, the Go App Engine breaks http.DefaultTransport (GAE_SDK / goroot / src / pkg / appengine_internal / internal.go, line 142, GAE 1.7.5)

 type failingTransport struct{} func (failingTransport) RoundTrip(*http.Request) (*http.Response, error) { return nil, errors.New("http.DefaultTransport and http.DefaultClient are not available in App Engine. " + "See https://developers.google.com/appengine/docs/go/urlfetch/overview") } func init() { // http.DefaultTransport doesn't work in production so break it // explicitly so it fails the same way in both dev and prod // (and with a useful error message) http.DefaultTransport = failingTransport{} } 

It helped me with Go App Engine 1.7.5

  transport := http.Transport{} client := &http.Client{ Transport: &transport, } req, _ := http.NewRequest("POST", url, strings.NewReader("grant_type=client_credentials")) req.SetBasicAuth(username, password) req.Header.Set("Accept", "application/json") req.Header.Set("Accept-Language", "en_US") req.Header.Set("Content-Type", "application/x-www-form-urlencoded") 

Answer 2

This is just an assumption, but you tried setting proxy variables

On a Unix or Windows environment, set the http_proxy or ftp_proxy environment variables for the URL that identifies the proxy server before starting with the Python interpreter. For example ("%" is a line command):

% http_proxy = "http://www.someproxy.comhaps128"

% export http_proxy

Answer 3

if you use the default proxy, then the transport is implemented as

 var DefaultTransport RoundTripper = &Transport{Proxy: ProxyFromEnvironment} 

setting the environment variable when go starts should solve the problem.

See also this other question: How do I configure Go to use a proxy server?

Answer 4

The urlfetch package itself does not support proxy settings even in development, since it does not actually select the URL: it sends a request to the application server (possibly development) and a request for its selection. I do not have the source dev_appserver.py , but it must respect the standard proxy variables:

 export http_proxy='http://user: pass@1.2.3.4 :3210/' 

If you do this before dev_appserver.py , this will probably just work.

If the above does not work, you should delete the file , and then use the following workaround:

 func client(ctx *appengine.Context) *http.Client { if appengine.IsDevAppServer() { return http.DefaultClient } return urlfetch.Client(ctx) } 

This will use the urlfetch API on the production application server, but use the standard net/http client otherwise, executing the http_proxy environment variable.