Using http-conduit browser

I am trying to clear data from a site using HTTPS. I was able to complete basic requests using Network.HTTP.Conduit successfully (placing credentials, etc.), but was unable to extract cookie information from the response headers (Set-Cookie). It seems that http-conduit has its own mechanism for processing cookies, which I did not understand.

Network.HTTP.Conduit.Browser seems to process cookies automatically (this is good for me), but I could not get it to work due to lack of documentation .

Can anyone with a lot of experience with the http-conduit browser module show me how:

  • Work with self-signed certificates (I managed to do this with managerCheckCerts in the base module)
  • Send a POST request with URL-encoded parameters in the body, without following any redirects (I used urlEncodedBody from the base module for this)
  • Use the cookie from step 2. in a simple GET request and read the answer as (lazy) ByteString (I would use httpLbs for this)

It seems to me that the abstraction level of Network.HTTP.Conduit.Browser is more suitable for my application compared to Network.HTTP.Conduit, so I would like to make a switch, even if I could handle cookies manually using the latter.

+8
session-cookies haskell web-scraping
source share
1 answer

I have never used a browser, but I used an http channel. I read the source code to answer these questions, I apologize if I am wrong.

  • Do the same as you. When you created Manager with the right managerCheckCerts , go to browse :: Manager -> BrowserAction a -> ResourceT IO a .

  • makeRequest :: Request IO -> BrowserAction (Response (Source IO BS.ByteString)) takes the value Request IO ; use urlEncodedBody as before to create a POST request with parameters in the body and pass it to makeRequest . Set redirectCount to 0 to disable redirection, as I assume.

  • I believe that you just need to use getCookieJar :: BrowserAction CookieJar ; BrowserAction comes from getBrowserState :: BrowserAction BrowserState .

The way http-conduit manages cookies outside the browser module is that it is not. Cookies are returned in the HTTP response; what you can do is parse the answer and save the cookies in the cookie jar. In fact, this is all a browser.

+2
source

All Articles