Screen scraping a site using an asp.net form in C #?

Question

Is it possible to write a screen scraper for a site that is protected by registration form. I have access to the site, of course, but I have no idea how to enter the site and save my credentials in C #.

Also, any good screen examples in C # would be greatly appreciated.

Is it already done?

Answer 1

It is pretty simple. You need your custom login method (HttpPost).

You can come up with something like this (this way you will get all the necessary cookies after logging in, and you just need to pass them to the following HttpWebRequest):

public static HttpWebResponse HttpPost(String url, String referer, String userAgent, ref CookieCollection cookies, String postData, out WebHeaderCollection headers, WebProxy proxy) { try { HttpWebRequest http = WebRequest.Create(url) as HttpWebRequest; http.Proxy = proxy; http.AllowAutoRedirect = true; http.Method = "POST"; http.ContentType = "application/x-www-form-urlencoded"; http.UserAgent = userAgent; http.CookieContainer = new CookieContainer(); http.CookieContainer.Add(cookies); http.Referer = referer; byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData); http.ContentLength = dataBytes.Length; using (Stream postStream = http.GetRequestStream()) { postStream.Write(dataBytes, 0, dataBytes.Length); } HttpWebResponse httpResponse = http.GetResponse() as HttpWebResponse; headers = http.Headers; cookies.Add(httpResponse.Cookies); return httpResponse; } catch { } headers = null; return null; } 

Answer 2

Of course, this has already been done. I have done this a couple of times. This is (generally) called Screen-scraping or web scraper.

You should take a look at this question (as well as view the questions under the screen-scraping tag.) Please note that Scraping not only refers to retrieving data from a web resource, but also includes transferring data to online forms to simulate user actions when sending input, for example, in the login form.