Geek Answers Handbook

The best way to handle very large HTTP downloads and server-side downloads

I have an upcoming project where I will need to handle very large downloads from browsers (classic input type = "file" or Java applet) and look for the best tool for completing the task on the server.

Abstracts are what I need:

  • low memory consumption on the server
  • the ability to save the file at the final destination on the server (without copying the file)
  • no blocking of other important tasks performed by the web server
  • good file processing up to 2 GB
  • file authorization (permissions will be specified in the application)

I still have a breadth of technology use, so I would like to get some recommendations in order to be able to choose the best technologies on the server to solve this problem:

  • ASP.NET?
  • Java?
  • Amazon S3?
  • Other options?

I'm more used to Microsoft Stack, but I’m ready to change if necessary: ​​as mentioned above, I'm just looking for the best tool for the job.

Thanks!

Update: The server side is the part that is really interesting to me for this issue , and not on the client side.

It seems like this may be trivial, but when you start to nod a bit, you see 4 MB restrictions with .NET loaded with a lot of memory that MAY block other threads (when you have a limit on the number of threads, and if the thread can be done within 2 GB of uploading / downloading files over the Internet: well, it will not scale very well, right?) etc.

+4
source share
7 answers

You will need:

  • Client code (Java Applet, Silverlight, etc.) for splitting files in small fragments
  • Server code (ASP.NET, Java, does not matter) to create these files.

I just finished the application this way; I would use Silverlight ( WebRequest async), ASP.NET ( IHttpHandler / IHttpAsyncHandler ) and SQL Server 2005 ( UPDATETEXT / READTEXT ) to store files.

UPDATE : About ASP.NET Server Code:

The default configuration of ASP.NET will allow 100 threads per processor; IHttpAsyncHandler will not block your process, and you can directly write the contents of your file to context.Response.OutputStream .

For upload, you also send several data blocks, but in several HTTP connections; while this can lead to overheating due to HTTP headers, it works very well in my tests.

+6
source
  • low memory consumption on the server

Writing input directly to disk, not to memory.
In Java terms, use FileOutputStream / BufferedOutputStream.

  • the ability to save the file at the final destination on the server (without copying the file)

See above.

  • no blocking of other important tasks performed by the web server

Each request is launched in its own thread, so there is nothing to worry about. It just depends on how you code everything.

  • good file processing up to 2 GB

Non-issue when writing a file to disk directly.
In terms of Java, you can use the Apache Commons FileUpload API to do this.

  • file authorization (permissions will be specified in the application)

Not sure what level of authorization you are talking about. File system level? Web application tier? Customer level?

+4
source

On the client side, input type="file" via HTTP POST has its drawbacks - in particular, it cannot compress downloads (it may not be a problem), and cannot resume transferring (this can be painful when a 1000 MB download fails 990 MB). SWFUpload, although it is great in other aspects, also relies on the implementation of the HTTP POST browser.

I would probably go with the Java applet on the client - this would allow me to establish a connection and check the necessary permissions before downloading; although this path also has its problems:

  • FS permissions (signed applet?)
  • write native http downloader
  • Proxy management

also provide an opportunity to return to plain old HTTP mail.

The server side can be written in almost all cases if you can process the data as it arrives (i.e. do not wait until you have the entire file).

+1
source

You can do this using asynchronous file upload in asp.net. Using flash to present it to the user.

http://swfupload.org/node/47

0
source

You can use uploadify . I used this several times before, and it always matched my needs. This is an asynchronous file downloader that uses flash to load multiple files if necessary.

0
source

A few other quick notes based on my experience ...

  • Input type = file does not work reliably for large files at all (due to memory)
  • Some components for downloading files will solve memory problems, but they still have problems when bytes are lost when transferring from the client to the server.
  • You should look at a java applet that supports data splitting.

The way it should work is a java applet that splits a file into manageable pieces and creates a hash from bytes. As soon as the server receives the piece, it should compare the hash of the bytes received with the hash provided by the java applet.

If the hashes do not match, try again. If they match, skip to the next snippet. Then use the tool to combine all the pieces.

0
source

More articles:


All Articles