Geek Answers Handbook

Go http.Get, concurrency and "Connection reset by peer"

I have between 1000-2000 web pages to download from a single server, and I use routines and channels to achieve high performance. The problem is that every time I run my program, up to 400 requests with the error "connection reset by peer". Rarely (possibly 1 out of 10 times), requests are not executed.

What can I do to prevent this?

The interesting thing is that when I ran this program on a server in the same country as the server hosting the site, 0 requests were unsuccessful, so I assume there is some problem with the delay (as it is now working on a server on a different continent) .

The code I'm using is just a simple http.Get (url) request, no additional parameters, or a custom client.

+10
concurrency go
source share
4 answers

A connection reset by peer message indicates that the remote server sent RST to force the connection to close, either intentionally as a mechanism to limit connections, or as a result of a lack of resources. In any case, you are probably opening too many connections or reconnecting too quickly.

Starting 1000-2000 connections is parallel - this is rarely the most efficient way to load many pages, especially if most or all of them come from the same server. If you check the bandwidth, you will find the optimal level of concurrency, which is much lower.

You will also want to set Transport.MaxIdleConnsPerHost according to the concurrency level. If MaxIdleConnsPerHost less than the expected number of concurrent connections, connections to the server are often closed after a request, and only to be opened immediately again - this will significantly slow down your progress and possibly reach the connection restrictions imposed by the server.

+18
source

Still new to golangs, hope this helps.

 var netClient = &http.Client{} func init() { tr := &http.Transport{ MaxIdleConns: 20, MaxIdleConnsPerHost: 20, } netClient = &http.Client{Transport: tr} } func foo() { resp, err := netClient.Get("http://www.example.com/") }
+16
source

Perhaps the server from which you load web pages has some kind of throttling mechanism that prevents more than a certain number of requests per second / (or similar) from a specific ip ?. Try to limit perhaps 100 requests per second or add sleep between requests. A peer-to-peer reset connection is basically a server that denies your service. ( What does “reset by peer connection” mean? )

+1
source

I achieved good results by setting the MaxConnsPerHost option on transport ...

 cl := &http.Client{ Transport: &http.Transport{MaxConnsPerHost: 50} }

MaxConnsPerHost optionally limits the total number of connections per host, including dial-up, active, and idle connections. If the limit is violated, the set will be blocked.

https://golang.org/pkg/net/http/#Transport.MaxConnsPerHost

UPDATE: To clarify, this option was released in Go 1.11, which was not available during answers @ AG1 or @JimB above, so I posted this.

0
source

More articles:


All Articles