Download large files (WebException: connection was unexpectedly closed)

UPDATED

See message # 3 below.

You must upload the file to the network automatically (without a browser). Host - Mini File Host v1.2 (if that matters). The specific api could not be found in the documentation, so I first sniffed the browser requests in Firebug as follows:

Params : do Value : verify POST /upload.php?do=verify HTTP/1.1 Host: webfile.ukrwest.net User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; ru; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8 (.NET CLR 4.0.20506) Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: ru,en-us;q=0.7,en;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive Referer: http://filehoster.awardspace.com/index.php Content-Type: multipart/form-data; boundary=---------------------------27368237179714 Content-Length: 445 -----------------------------27368237179714 Content-Disposition: form-data; name="upfile"; filename="Test.file" Content-Type: application/octet-stream 12345678901011121314151617sample text -----------------------------27368237179714 Content-Disposition: form-data; name="descr" -----------------------------27368237179714 Content-Disposition: form-data; name="pprotect" -----------------------------27368237179714-- 

Here we can see the parameters, headers, content type and pieces of information (1 - file name and type, 2 - file content, 3 - additional parameters - description and password, not necessarily used). So I created a class that mimics this behavior in stages: HttpWebRequest on the url, applies the necessary parameters for the request, forms the request strings using StringBuilder and converts them into byte arrays, reads the file using FileStream, putting it all into a MemoryStream and then writing it for the request (took most of the code from the article in CodeProject, where it uploads the file to the Rapidshare host). Neat and tidy, but ... It seems that it does not work :( As a result, it returns the start page of the download, and not the results page with links that I can analyze and present to the user ... Here are the main methods of the Uploader class:

 // Step 1 - request creation private HttpWebRequest GetWebrequest(string boundary) { Uri uri = new Uri("http://filehoster.awardspace.com/index.php?do=verify"); System.Net.HttpWebRequest httpWebRequest = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(uri); httpWebRequest.CookieContainer = _cookies; httpWebRequest.ContentType = "multipart/form-data; boundary=" + boundary; httpWebRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.2; ru; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8 (.NET CLR 4.0.20506)"; httpWebRequest.Referer = "http://filehoster.awardspace.com/index.php"; httpWebRequest.Method = "POST"; httpWebRequest.KeepAlive = true; httpWebRequest.Timeout = -1; //httpWebRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"; httpWebRequest.Headers.Add("Accept-Charset", "windows-1251,utf-8;q=0.7,*;q=0.7"); httpWebRequest.Headers.Add("Accept-Encoding", "gzip,deflate"); httpWebRequest.Headers.Add("Accept-Language", "ru,en-us;q=0.7,en;q=0.3"); //httpWebRequest.AllowAutoRedirect = true; //httpWebRequest.ProtocolVersion = new Version(1,1); //httpWebRequest.SendChunked = true; //httpWebRequest.Headers.Add("Cache-Control", "no-cache"); //httpWebRequest.ServicePoint.Expect100Continue = false; return httpWebRequest; } // Step 2 - first message part (before file contents) private string GetRequestMessage(string boundary, string FName, string description, string password) { System.Text.StringBuilder stringBuilder = new StringBuilder(); stringBuilder.Append("--"); stringBuilder.Append(boundary); stringBuilder.Append("\r\n"); stringBuilder.Append("Content-Disposition: form-data; name=\""); stringBuilder.Append("upfile"); stringBuilder.Append("\"; filename=\""); stringBuilder.Append(FName); stringBuilder.Append("\""); stringBuilder.Append("\r\n"); stringBuilder.Append("Content-Type: application/octet-stream"); stringBuilder.Append("\r\n"); return stringBuilder.ToString(); } // Step 4 - additional request parameters. Step 3 - reading file is in method below private string GetRequestMessageEnd(string boundary) { System.Text.StringBuilder stringBuilder = new StringBuilder(); stringBuilder.Append(boundary); stringBuilder.Append("\r\n"); stringBuilder.Append("Content-Disposition: form-data; name=\"descr\""); stringBuilder.Append("\r\n"); stringBuilder.Append("\r\n"); stringBuilder.Append("Default description"); stringBuilder.Append("\r\n"); stringBuilder.Append(boundary); stringBuilder.Append("\r\n"); stringBuilder.Append("Content-Disposition: form-data; name=\"pprotect\""); stringBuilder.Append("\r\n"); stringBuilder.Append("\r\n"); stringBuilder.Append(""); stringBuilder.Append("\r\n"); stringBuilder.Append(boundary); stringBuilder.Append("--"); //stringBuilder.Append("\r\n"); //stringBuilder.Append(boundary); //stringBuilder.Append("\r\n"); return stringBuilder.ToString(); } // Main method public string ProcessUpload(string FilePath, string description, string password) { // Chosen file information FileSystemInfo _file = new FileInfo(FilePath); // Random boundary generation DateTime dateTime2 = DateTime.Now; long l2 = dateTime2.Ticks; string _generatedBoundary = "----------" + l2.ToString("x"); // Web request creation System.Net.HttpWebRequest httpWebRequest = GetWebrequest(_generatedBoundary); // Main app block - form and send request using (System.IO.FileStream fileStream = new FileStream(_file.FullName, FileMode.Open, FileAccess.Read, FileShare.Read)) { byte[] bArr1 = Encoding.ASCII.GetBytes("\r\n--" + _generatedBoundary + "\r\n"); // Generating pre-content post message string firstPostMessagePart = GetRequestMessage(_generatedBoundary, _file.Name, description, password); // Writing first part of request byte[] bArr2 = Encoding.UTF8.GetBytes(firstPostMessagePart); Stream memStream = new MemoryStream(); memStream.Write(bArr1, 0, bArr1.Length); memStream.Write(bArr2, 0, bArr2.Length); // Writing file byte[] buffer = new byte[1024]; int bytesRead = 0; while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0) { memStream.Write(buffer, 0, bytesRead); } // Generating end of a post message string secondPostMessagePart = GetRequestMessageEnd(_generatedBoundary); byte[] bArr3 = Encoding.UTF8.GetBytes(secondPostMessagePart); memStream.Write(bArr3, 0, bArr3.Length); // Preparing to send httpWebRequest.ContentLength = memStream.Length; fileStream.Close(); Stream requestStream = httpWebRequest.GetRequestStream(); memStream.Position = 0; byte[] tempBuffer = new byte[memStream.Length]; memStream.Read(tempBuffer, 0, tempBuffer.Length); memStream.Close(); // Sending request requestStream.Write(tempBuffer, 0, tempBuffer.Length); requestStream.Close(); } // Delay (?) System.Threading.Thread.Sleep(5000); // Getting response string strResponse = ""; using (Stream stream = httpWebRequest.GetResponse().GetResponseStream()) using (StreamReader streamReader = new StreamReader(stream/*, Encoding.GetEncoding(1251)*/)) { strResponse = streamReader.ReadToEnd(); } return strResponse; } 

Playback using the Version (1.0, 1.1) protocol, AllowAutoRedirect (true / false), even the well-known ServicePoint.Expect100Continue (false) does not fix the problem. Even a 5 second timeout before getting an answer (the thought in the case of a large file that it does not load so quickly) did not help. The octet-stream content type was selected for the purpose of downloading any file (some kind of switch can be used for the most popular jpg / zip / rar / doc, etc., but this one seems universal). The border is randomly generated from ticks of the timer, and not a big deal. What else?: / I could refuse this and forget about it, but I feel pretty close to the solution, and then forget about it: P. If you need the whole application to run and debug - here it is (70kb, zipped C # 2.0 VS2k8, no ads, no viruses):

@Mediafire @FileQube @FileDropper

+3
c # file upload
source share
4 answers

Update: no, no redirection.

screenshot

Read RFC2388 several times, rewriting the code, and it finally worked (I think the problem was in the utf-read trailing border instead of the correct 7 bit ascii). Hooray? Nope: (only small files are transferred, large ones throw "Connection unexpectedly closed."

 System.Net.WebException was unhandled by user code Message="The underlying connection was closed: The connection was closed unexpectedly." Source="Uploader" StackTrace: at Uploader.Upload.ProcessUpload(String FilePath, String description, String password) in F:\MyDocuments\Visual Studio 2008\Projects\Uploader\Uploader.cs:line 96 at Uploader.Form1.backgroundWorker1_DoWork(Object sender, DoWorkEventArgs e) in F:\MyDocuments\Visual Studio 2008\Projects\Uploader\Form1.cs:line 45 at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument) 

I know that there is an error with the .net stack and several solutions:

1) increase both the Timeout and ReadWriteTimeout of the request

2) set request.KeepAlive = false and System.Net.ServicePointManager.Expect100Continue = false

3) set ProtocolVersion to 1.0 But neither of them, nor all of them help me in this matter. Any ideas?

EDIT - Source Code:

 // .. request created, required params applied httpWebRequest.ProtocolVersion = HttpVersion.Version10; // fix 1 httpWebRequest.KeepAlive = false; // fix 2 httpWebRequest.Timeout = 1000000000; // fix 3 httpWebRequest.ReadWriteTimeout = 1000000000; // fix 4 // .. request processed, data written to request stream string strResponse = ""; try { using (WebResponse httpResponse = httpWebRequest.GetResponse()) // error here { using (Stream responseStream = httpResponse.GetResponseStream()) { using (StreamReader streamReader = new StreamReader(responseStream)) { strResponse = streamReader.ReadToEnd(); } } } } catch (WebException exception) { throw exception; } 
+8
source share

"As a result, it returns the start page of the download, not the results page with links that I could parse and present to the user ..."

Perhaps this is just the behavior of the download function: after the download is complete, can you upload another file? I think you need to call a different url to โ€œview the fileโ€ of the page (I assume that the one you need).


Change In fact, if the server sends a "redirect" (http 3xx), something that the browser should handle, so if you are working with your own client application instead of the browser, you will have to implement the redirection yourself. Here rfc for more information.

0
source share

Try setting the maxRequestLength property of the httpRuntime element in the Web.config file.

0
source share

In my case, duplicates the file name causing the problem. I save the file parameters in an XML file, but the name parameter duplicates each other.

  <field name="StillImage"> <prefix>isp_main_</prefix> <suffix>308</suffix> <width>1080</width> <height>1080</height> </field> <field name="ThumbnailImage"> <prefix>isp_thumb_</prefix> // pay attention to this <suffix>308</suffix> <width>506</width> <height>506</height> </field> <field name="Logo"> <prefix>isp_thumb_</prefix> // and this <suffix>306</suffix> <width>506</width> <height>506</height> </field> 

And, in another case, I had , the problem is in the file length . Check the allowed file size on your server. In the script, just check this part:

 dataStream.Write(filesBytesArray, 0, filesBytesArray.Length); dataStream.Close(); 

And if you do not know, just limit the size of the downloaded file in your section of the interface, i.e. HTML <input type="file"> upload element, this is a good link to limit file size and other filter .

0
source share

All Articles