Jersey multi-threaded streaming without disk buffering on the receiving server

Question

I am trying to transfer (large) files via HTTP to a database. I use Tomcat and Jersey as a Webframework. I noticed that if I send the file to my resource, the file is first buffered to disk (in temp \ MIME * .tmp) before it is processed in my doPOST method.

This is really an undesirable behavior because it doubles disk I / O and also leads to incorrect UX, because if the browser is already loaded, the user needs to wait a few minutes (depending on the file size, of course) until he receives an HTTP response.

I know that this is probably not the best implementation of a large file download (since you don’t even have the ability to resume), but also the requirements.: /

So my questions are: if there is a way to disable (disk) buffering for MULTIPART POST. Meme buffering is obviously too expensive, but I don’t see the need for disk buffering at all? (Explain please) How do large sites such as YouTube handle this situation? Or, at least, is there a chance to give the user immediate feedback if the file is sent? (It should be bad, as there might be something like SQLException)

Answer 1

It is best to take full control and write your own servlet, which simply captures request.getInputStream (or request.getWriter if you consume text) and the stream itself. Most frameworks make your life "easy" by handling all downloads, temporary storage, etc. For you, and often make it difficult to work with threads. It is very easy to capture the stream yourself and do whatever you want.

Answer 2

I am sure that Jersey writes files to disk so that the memory is not flooded. Since you know exactly what you need to do with the incoming data stream → to the database, you probably have to write your own MessageBodyReader and get a Jersey to use to process your incoming multi-page data.

Answer 3

Ok, so after several days of reading and testing different things, I came across an HTTPServletRequest. At first I didn’t even want to try, since it removes all the convenient methods from @FormDataParam, but since I did not know what else to do ...

It turns out that helped. When I use @Context HTTPServletRequest request and request.getInputStream() , I do not get disk buffering at all.

Now I just need to figure out how to get to the FormDataContentDisposition form without @FormDataParam

Edit:

Ok Probably MultiPartFormData should be buffered on disk to parse the InputStream request. Therefore, it seems to me that I should manually parse it myself if I want to prevent any buffering :(

Answer 4

If anyone is still interested, I solved the same problem using the Apache Commons Streaming api

The sample code on this page worked great for me.