Convert HTTP proxies to HTTPS proxies in Twisted

Question

I recently played with HTTP proxies in twisted. After much trial and error, I think that finally something works for me. However, I want to know how, if possible, to expand this proxy server so that it can handle HTTPS pages? Here is what I got so far:

from twisted.internet import reactor from twisted.web import http from twisted.web.proxy import Proxy, ProxyRequest, ProxyClientFactory, ProxyClient class HTTPProxyClient(ProxyClient): def handleHeader(self, key, value): print "%s : %s" % (key, value) ProxyClient.handleHeader(self, key, value) def handleResponsePart(self, buffer): print buffer ProxyClient.handleResponsePart(self, buffer) class HTTPProxyFactory(ProxyClientFactory): protocol = HTTPProxyClient class HTTPProxyRequest(ProxyRequest): protocols = {'http' : HTTPProxyFactory} def process(self): print self.method for k,v in self.requestHeaders.getAllRawHeaders(): print "%s : %s" % (k,v) print "\n \n" ProxyRequest.process(self) class HTTPProxy(Proxy): requestFactory = HTTPProxyRequest factory = http.HTTPFactory() factory.protocol = HTTPProxy reactor.listenSSL(8001, factory) reactor.run() 

As this code shows, for an example at the moment I am just printing everything that happens through the connection. Is it possible to handle HTTPS with the same classes? If not, how should I implement such a thing?

Answer 1

If you want to connect to an HTTPS site through an HTTP proxy, you need to use the CONNECT HTTP verb (because this is how the proxy works for HTTPS). In this case, the proxy server simply connects to the target server and sends everything that the server sends back to the client socket (and vice versa). In this case, caching is not used (but you can register the hosts to which you are connecting).

The exchange will look like this (client for proxy):

 C->P: CONNECT target.host:443 HTTP/1.0 C->P: P->C: 200 OK P->C: 

After this, the proxy simply opens a simple socket on the target server (not yet HTTP or SSL / TLS) and does not transfer everything between the original client and the target server (including the acknowledgment in TLS that the client initiates). The client updates the existing socket that it has for the proxy server to use TLS / SSL (by starting the SSL / TLS handshake). As soon as the client has read the status bar “200”, as far as the client is concerned, it is as if it is directly connected to the target server.

Answer 2

I'm not sure what is twisted, but I want to warn you that if you implement the HTTPS proxy, the web browser expects the SSL server certificate to match the domain name in the URL (address bar). Otherwise, the web browser will issue security warnings.

There are ways around this, for example, generating certificates on the fly, but you need a root certificate that you need to trust in the browser.