Optimized loading of multiple web pages. WITH#

Question

I am developing an application where I need to load a bunch of web pages, preferably as quickly as possible. What I'm doing now is that I have several threads (100) that have their own System.Net.HttpWebRequest. Such works, but I do not get the performance that I would like. Currently, I have a promising 600+ Mbps connection for work, and this is used no more than 10% (at peaks). I suppose my strategy is flawed, but I cannot find another good way to do this.

Also: if use is HttpWebRequestnot a good way to load web pages, say so :) The code has been semi-automatically converted from java.

Thank:)

Update:

public String getPage(String link){
   myURL = new System.Uri(link);
   myHttpConn = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(myURL);
   myStreamReader = new System.IO.StreamReader(new System.IO.StreamReader(myHttpConn.GetResponse().GetResponseStream(),
            System.Text.Encoding.Default).BaseStream,
                new System.IO.StreamReader(myHttpConn.GetResponse().GetResponseStream(),
                    System.Text.Encoding.Default).CurrentEncoding);

        System.Text.StringBuilder buffer = new System.Text.StringBuilder();

        //myLineBuff is a String
        while ((myLineBuff = myStreamReader.ReadLine()) != null)
        {
            buffer.Append(myLineBuff);
        }
   return buffer.toString();
}

Answer 1

, , , :

myStreamReader = new System.IO.StreamReader(
    new System.IO.StreamReader(
        myHttpConn.GetResponse().GetResponseStream(),
        System.Text.Encoding.Default).BaseStream,
             new System.IO.StreamReader(myHttpConn.GetResponse().GetResponseStream(),
                 System.Text.Encoding.Default).CurrentEncoding);

GetResponse. , , . , ...

var response = (HttpWebResponse)myHttpCon.GetResponse();
myStreamReader = new StreamReader(response.GetResponseStream(), Encoding.Default)

.

, , , , . , , . response.Close(). . http://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.close.aspx

Answer 2

, ,

• Windows XP Vista

tcpip.sys 10 . , , , 10/. Microsoft, /. , .

, , -. Windows "" ( , ) 10/s. , , 20 , , , - , , , .

, , "". :

EventID 4226: TCP/IP , TCP.

, . , P2P (Torrent), , .

1200 ( ) 5- . ( WinXP) 20 , SQL. , - , , 10 .

, , . , , , , - ( ). , , , , . - "-" , .

HEX tcpip.sys, speedguide.net. ( SP3, SP2), . , , SP3, .

, , Windows 7 , Windows 7 .

Answer 3

, , XML . , , , , , , .

, . , 20 50 . , , .

ThreaderEngine , . , while, . .

, (.NET 4.0):

public static string FileDownload(string _ip, int _port, string _file, int Timeout, int ReadWriteTimeout, NetworkCredential _cred = null)
{
    string uri = String.Format("http://{0}:{1}/{2}", _ip, _port, _file);
    string Data = String.Empty;
    try
    {
        HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(uri);
        if (_cred != null) Request.Credentials = _cred;
        Request.Timeout = Timeout; // applies to .GetResponse()
        Request.ReadWriteTimeout = ReadWriteTimeout; // applies to .GetResponseStream()
        Request.Proxy = null;
        Request.CachePolicy = new System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.NoCacheNoStore);
        using (HttpWebResponse Response = (HttpWebResponse)Request.GetResponse())
        {
            using (Stream dataStream = Response.GetResponseStream())
            {
                if (dataStream != null)
                    using (BufferedStream buffer = new BufferedStream(dataStream))
                    using (StreamReader reader = new StreamReader(buffer))
                    {
                        Data = reader.ReadToEnd();
                    }
            }
            return Data;
        }
    }
    catch (AccessViolationException ave)
    {
        // ...
    }
    catch (Exception exc)
    {
        // ...
    }
}

Using this, I can download about 60 KB each of 1200+ remote computers (72 MB) in less than 5 minutes. The machine is a Core 2 Quad with 2 GB of RAM and uses four connected T1 connections (~ 6 Mbit / s).