Is Haskell Network.Browser a module like Perl LWP or Python?

Question

The Network.Browser documentation says that the module supports the following:

• HTTP authentication processing
• Transparent redirection processing
• Cookies + transmission.
• Transaction Logging Proxy-mediated connections.

For me, this sounds like the beginning of a browser that allows me to crawl web pages, handle authentication on websites, cookies, etc.

However, the module comes with zero sample code, instructions or tutorials. I can’t figure out how to use it.

Can someone give an example of how it can be used to 1) go to a website, 2) enter it and 3) upload a file that requires you to be logged in?

Answer 1

I would also like to take a look at Network.Curl.

To answer your question, here is an example (taken from http://haskell.pastebin.com/9kPiGxiH ):

import Data.IORef
import Network.HTTP
import Network.Browser
import Network.URI
import Data.Maybe
import Control.Monad
import Data.List
import Text.Regex.TDFA
import Control.Concurrent

pageUrl off = URI "http:" (Just $ URIAuth "" "www.interpals.net" "") "/dosearch.php" ("?todo=search&sec=adv&age1=15&age2=18&sex[]=FEMALE&lfor[]=lfor_email&lfor[]=lfor_snail&lfor[]=lfor_langex&lfor[]=lfor_friend&lfor[]=lfor_flirt&lfor[]=lfor_relation&countries[]=AT&countries[]=DE&countries[]=CH&state=&languages[]=any&keywords=&sort=p.last_login+DESC&offset="++(show off)) ""

getPage     :: URI -> BrowserAction (HandleStream [Char]) String
getPage uri = do
    setErrHandler $ const $ return ()
    setOutHandler $ const $ return ()
    (_,s) <- request $ Request (uri) GET
        [Header HdrCookie "__ubic1=MTE3ODM0NDM0MTRjN2RkYTA1OTAzMmU4LjkxODE1Njk2; __utma=46363135.421215970.1283316265.1283538085.1283541700.10; __utmz=46363135.1283316265.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); __utmc=46363135; PHPSESSID=59a130c66d4853f85289852f15cefa3a; resolution=1920x1080; ip_auto_login[login]=cap11235; ip_auto_login[password_md5]=NDM0NWM0NDlkZTg4MjRkMWVhZmJmZWNiZTQwOWQ4YTE%3D; __utmb=46363135"] ""
    return $ rspBody s

getPeople :: Int -> BrowserAction (HandleStream [Char]) ([String], Int)
getPeople off = do
    s <- getPage (pageUrl off)
    let t = (s=~"<a href='/([^?.]+)\\?")::[[String]]
    let next = if length t > 0 then off+10 else 0
    return (nub $ map (!!1) t, next)

personUrl :: String -> URI
personUrl name = fromJust $ parseURI ("http://www.interpals.net/"++name)

viewPerson :: String -> BrowserAction (HandleStream [Char]) ()
viewPerson name = do
    _ <- getPage $ personUrl name
    return ()

doCycle :: IORef (Int, Int) -> IO ()
doCycle r = do
    (count, off) <- readIORef r
    (people, newOff) <- browse $  getPeople off
    mapM_ (forkIO . browse . viewPerson) people
    let newCount = count + (length people)
    writeIORef r (newCount, if newOff<2000 then newOff else 0)
    print newCount
    doCycle r

main = do
    t <- newIORef (0,0)
    doCycle t