Serving dynamic zip files through Apache

Question

One of the responsibilities of my Rails application is to create and maintain signed xmls. Any signed xml, after its creation, never changes. Therefore, I store each xml in the public folder and redirect the client appropriately to avoid unnecessary processing from the controller.

Now I want a new function: each xml is associated with a date, and I would like to implement the ability to serve a compressed file containing each xml whose date is in the period specified by the client. However, this period cannot be limited to less than one month for this feature to be useful, and this means that some mail files that will be served will be as large as 50M.

My application is deployed as a Passenger Apache module. Thus, it is completely unacceptable to serve the file with send_data , since the client will have to wait until the entire compressed file is created before the actual download begins. Although I have an idea on how to implement this function in Rails, so a compressed file is created during maintenance, I feel that my server will be poorer in resources, as soon as some long Ruby / Passenger processes are allocated to serve large zip files.

I read the best solution for serving static files through Apache, but not for dynamic ones.

So what is the solution to the problem? Do I need something like a custom Apache handler? How do I tell Apache from my application how to process a request, compress files and transfer the result at the same time?

Answer 1

Check out my mod_zip module for Nginx:

http://wiki.nginx.org/NgxZip

You can have a backend script tell Nginx which URL locations will be included in the archive, and Nginx will dynamically transfer the ZIP file to the client containing these files. The module uses a single-threaded Nginx proxy code and has a very light weight.

The module was first released in 2008 and is currently quite mature. From your description, I think this will suit your needs.

Answer 2

You just need to use any API you have to create a zip file and write it back, periodically flushing the output. If this serves large zip files or is often requested, try running it in a separate process with a nice / ionice / low priority high.

In the worst case scenario, you can run the zip line in a low priority process and periodically transmit the result.

Answer 3

See Rails answer : streaming data on the fly in zip format?

Use a zip stream.

Answer 4

It's hard to do, but I created a gem called zipline ( http://github.com/fringd/zipline ) that makes me work. I want to update it so that it can support simple file descriptors or paths, right now it assumes you are using a supporting structure ...

Also, you probably can't pass the answer on to the passenger ... I had to use a unicorn for the streaming to work properly ... and some middleware in the middle layer might even screw this up (calling response.to_s interrupts it)

If someone still needs this, it bothers me on the github page