How can I submit a large file for upload using Perl?

Question

I need to serve a large file (500+ MB) to download from a location not accessible by the web server. I found the question Serving large files with PHP , which is the same as my situation, but I use Perl instead of PHP.

I tried just printing the file line by line, but this does not cause the browser to request a download before capturing the entire file:

use Tie::File; open my $fh, '<', '/path/to/file.txt'; tie my @file, 'Tie::File', $fh or die 'Could not open file: $!'; my $size_in_bytes = -s $fh; print "Content-type: text/plain\n"; print "Content-Length: $size_in_bytes\n"; print "Content-Disposition: attachment; filename=file.txt\n\n"; for my $line (@file) { print $line; } untie @file; close $fh; exit; 

Does Perl have the equivalent of the PHP readfile() function (as suggested with PHP), or is there a way to accomplish what I'm trying to do here?

Answer 1

If you just want to output input to output, this should do the trick.

 use Carp (); { #Lexical For FileHandle and $/ open my $fh, '<' , '/path/to/file.txt' or Carp::croak("File Open Failed"); local $/ = undef; print scalar <$fh>; close $fh or Carp::carp("File Close Failed"); } 

I think, in response to the question: "Does Perl have PHP ReadFile Equivelant", and I assume that my answer will be "But it really is not needed."

I used PHP File IO files to manage the files and it hurts, Perls is just so easy to use compared to the fact that the workaround for a function of the same size is suitable for everyone, which seems excessive.

Alternatively, you can watch the X-SendFile and basically send the header to your web server to tell it which file to send: http://john.guen.in/past/2007/4/17/send_files_faster_with_xsendfile/ ( assuming it has enough permissions to access the file, but the file is simply NOT accessible by the regular URI)

Edit Noted, it is better to do this in a loop, I checked the above code with a hard drive and it will implicitly try to save all this in an invisible temporary variable and eat your whole ram.

Alternative use blocks

The following improved code reads the given file in blocks of 8192 characters, which is much more efficient in terms of memory and provides bandwidth comparable to my disk read speed. (I also pointed to / dev / full for fits and giggles and got a useful bandwidth of 500 MB / s, and he didn’t eat all my rams, so that should be fine)

 { open my $fh , '<', '/dev/sda' ; local $/ = \8192; # this tells IO to use 8192 char chunks. print $_ while defined ( $_ = scalar <$fh> ); close $fh; } 

Application of jrockways offers

 { open my $fh , '<', '/dev/sda5' ; print $_ while ( sysread $fh, $_ , 8192 ); close $fh; } 

This literally doubles the performance ... and in some cases gives me better bandwidth than DD does O_o.

Answer 2

The readline function is called readline (and can also be written as <> ).

I am not sure what the problem is. It is possible that for loops are not lazily evaluated (and this is not so). Or maybe Tie :: File is screwing something? Anyway, the idiomatic Perl to read the file line at a time:

 open my $fh, '<', $filename or die ...; while(my $line = <$fh>){ # process $line } 

No need to use Tie :: File.

Finally, you should not do this business yourself. This is a job for a web framework. If you used Catalyst (or HTTP :: Engine ), you would just say:

 open my $fh, '<', $filename ... $c->res->body( $fh ); 

and the structure will automatically serve the data in the file efficiently. (Using stdio via readline is not a good idea here; it’s better to read the file in blocks from disk. But who cares, it's allotted!)

Answer 3

You can use my Sys :: Sendfile module. It should be very efficient (since it uses sendfile under the hood), but not fully portable (currently only Linux, FreeBSD and Solaris are supported).

Answer 4

Answering the (original) question (“Does Perl have an equivalent PHP function readline() ...?”), The answer is “angle bracket syntax”:

 open my $fh, '<', '/path/to/file.txt'; while (my $line = <file>) { print $line; } 

Getting the length of the content using this method is not always easy, so I recommend staying with Tie::File .

---

Note

Using:

 for my $line (<$filehandle>) { ... } 

(as I wrote earlier) copies the contents of the file to the list and iterates over it. Using

 while (my $line = <$filehandle>) { ... } 

not. When working with small files, the difference is small, but when working with large files, this definitely can be.

---

Answering the (updated) question ("Does Perl have the equivalent of the PHP readfile() ... function?"), The answer is slurping . There are a couple of syntax pairs , but Perl6::Slurp seems to be the current module of choice.

The incomprehensible question (“why does the browser not request downloading before capturing the entire file?”) Has absolutely nothing to do with how you read in the file, and everything that the browser considers is good form. I would suggest that the browser sees the mime type and decides that it knows how to display plain text.

---

Studying the Content-Disposition problem more closely, I remember that I have similar problems with IE, ignoring Content-Disposition. Unfortunately, I cannot remember a workaround. IE has a long history of problems here (the old page refers to IE 5.0, 5.5, and 6.0). However, for clarification, I would like to know:

• What link do you use to point to this large file (i.e. do you use the regular link a href="perl_script.cgi?filename.txt or use some kind of Javascript)?

• What system do you use to actually maintain the file? For example, does the web server make its own connection to another computer without a web server, and then copy the file to the web server and then send the file to the end user, or does the user make a connection directly to the computer without the web server

• In the original question, you wrote "this does not make the browser request downloading before capturing the entire file" and in the comment you wrote: "I still do not receive the invitation to download the file until all this has been downloaded." Does this mean that the file is displayed in the browser (since it is just text), that after the browser has downloaded the entire file, you will receive a prompt “where do you want to save this file” or something else?

I have the feeling that at some point the HTTP headers get dropped out or that the Cache-control header is being added (which seems to cause problems).

Answer 5

When you say "this does not make the browser ask for download" - what is a "browser"?

Different browsers behave differently, and IE is especially intentional, it ignores the headers and decides what to do based on reading the first few kilobytes of the file.

In other words, I think that your problem may be on the client side, and not on the server.

Try to quit the browser and report that the file is of type application / octet-stream. Or why not just archive the file, especially since it is so huge.

Answer 6

Do not use for/foreach (<$input>) because it reads the entire file at once and then iterates over it. Use while (<$input>) instead. The sysread solution sysread good, but sendfile is the best in performance.

Answer 7

I successfully did this by telling the browser that this is type application / octet-stream instead of type text / plain. Apparently, most browsers prefer to display a text / plain line, rather than letting the user use the download dialog.

It technically lies in the browser, but it does the job.

Answer 8

The most efficient way to upload a large file to upload depends on the web server you are using.

In addition to the @Kent Fredric X-Sendfile Sendfile suggestion:

Downloading Done Right files contains links that describe how to do this for Apache , lighttpd (mod_secdownload: URL generation security), nginx . There are examples in PHP, Ruby (Rails), Python that can be adopted for Perl.

It basically boils down to:

• Configure paths and permissions for your web server.
• Create valid redirect headers in your Perl application ( Content-Type , Content-Disposition , Content-length ^? X-Sendfile or X-Accel-Redirect , etc.).

There are probably CPAN modules, web framework plugins that do just that, for example, @Leon Timmermans mentions Sys::Sendfile in his answer .