What is the fastest way to transfer files with high latency and high bandwidth?

Question

I am trying to transfer files with high latency and high bandwidth. Unfortunately, when I use rsync, my transfer rate uses only part of my available bandwidth. My total transfer time takes much longer than expected (i.e., transfer time = byte / byte width per second)!

What is the fastest way to transfer files with high latency and high bandwidth?

So for example:

• latency exceeds 900 ms latency (round trip)
• 512 kbps bandwidth

[1], that is, use most of the available bandwidth

Answer 1

When used rsyncin high latency and high bandwidth environments, your data transfer speed will be slower [1] than your available bandwidth. In the above example, the expected transfer rate will be 56.25 KB or less than 10% of the available bandwidth.

One solution is the parallel execution of N rsyncprocesses :

#!/bin/bash

# tar up the files
tar -cvzf x.tar ${list_of_files}

# [optional]
# compute the md5sum
md5sum x.tar > x.tar.md5sum

# break the large tar file into N files (i.e. x.tar would become x.tar.1 ... x.tar.N)
# TODO

# start N `rsync` processes in parallel 
for ((i=1;i<=N;i++)); do rsync -avzh x.tar.${i} ${destination} & done

# wait for the transfers to finish
wait && echo "success" || echo "fail" && exit 1

# stitch the N files back together into x.tar
TODO

# [optional... but gives everyone a nice warm and fuzzy]
# copy the md5sum and verify your files (even though `rsync` already did so)
scp x.tar.md5sum ${destination}
ssh ${destination_machine} "cd ${path} && md5sum -c x.tar.md5sum && echo 'PASS (files verified with md5sum)' || echo 'FAIL (file verification failed md5sum)' && exit 1"
# done!

---

[1] Why is your transfer rate slow in this example?

In a word: product with bandwidth delay (actually three words)

. rsync . rsync ( - , TCP TCP), .

, TCP- ( TCP- ), ACK, , . . , .

, 56,25 KiB 10% .

. rsync .

1:

, TCP- , - ( Google - - uftp, UDP TCP). , uftp - .

2:

rsync TCP , , .

3:

rsync , .