Download using curl using multiple connections

Question

I am trying to transfer a file with several gb from one server to another; RTT problem is 150ms +. Ive already tried using aria2, but limited to 16 connections, lftp has no protection against stopped transfers.

I am wondering if it is possible to upload a single file with multiple connections using curl cli.

Answer 1

Here the script is called pcurl , which wraps the curl to make several simultaneous downloads, and then combine them at the end.

By default, 10 connections are established, but this can be changed in MAX_SEGMENTS on line 81.

It is called as follows:

./pcurl.sh http://myurl.to/a/very/largefile.zip 

Answer 2

No, the curl tool does not have such capabilities.

(oh and btw, big RTT very rarely explains why simple TCP transmission is slow)

Answer 3

It is possible. Retrieve the total file size using the -I option in curl.

Then you can develop many processes in the shell, each curl connection with a different Content-Length header to load a different part of the file.

After completing all the tasks, merge all the boot fragments into a large file.

I wrote a simple script and it is available here mcurl.sh , with the -s option you can specify how many tasks you create to download a large file.