Performance issues with Ruby and Net :: SCP (sockets)

Question

SCP boot speeds are apparently very limited in the library the command line scp utility is capable of. I understand that this is Ruby (1.9.2-p0), but Net :: SCP is about 8 times slower than the Linux utility (seen from large files ... see below). I am curious to know (I quickly looked through the code) if this is how the sockets are in Ruby, or if it's possible to better connect Net :: SCP sockets?

I noticed that no matter what loading style I tried (sequential loading, asynchronous channels using multiple instances of the scp object), I could never get more than 9 megabytes per second transfer rate when loading SCP. Now ... let me explain the details of my investigation:

1) Tried various encryption algorithms

I used different types of encryption without a significant change in speed. Example: I could send my test 1GB file using the scp command line (encryption algorithm = arcfour128) and get a transfer rate of 73.3 megabytes per second on my internal gigabit connection. I never received more than 9 megabytes / s on my internal gigabit connection using the Net :: SCP.upload library.

2) Tried different hosts / OS

I found that Linux -> Linux downloads were the fastest. The SUA ssh server (Windows) could provide me with a maximum download speed of 13.5 megabytes / s (Linux → Windows using the arcfour algorithm with the scp command line), while Linux → Linux (using the arcfour command line, w / scp) was blazing 73.3 megabytes / s. I should mention that these Windows and Linux machines are the same model, hardware, etc.

3) Tried various ways of loading SCP

-> 2 synchronous downloads are used! the calls, one by one, were completed. → 2 asynchronous download calls were used, one after another it was launched → 2 Net :: SCP objects were used and the file was sent to the non-blocking / asynchronous version of the download (therefore they were executed in parallel). None of these different methods provide any significant increase in performance, which is disappointing.

Here are the test results (the text is expanded for readability, but is similar to the output code of the provided code):

Net::SCP Done creating channels Starting transfer of /home/seth/afpcases/systeme.afp # two upload! calls, one after another Finished transfer of /home/seth/afpcases/systeme.afp --> Duration: 126.07707 seconds (8.7168903909331 megabytes/s) should show transfer speed of serial uploads Starting transfer of /home/seth/afpcases/systeme.afp # two upload calls, one after another, with a wait on both channels after both have started Finished transfer of /home/seth/afpcases/systeme.afp --> Duration: 122.588784 seconds (8.964931082112699 megabytes/s) should show transfer speed of simultaneous async channels. Starting transfer of /home/seth/afpcases/systeme.afp # two upload calls on two separate Net::SCP objects, one after another, with a wait on both channels after both have started Finished transfer of /home/seth/afpcases/systeme.afp --> Duration: 122.822663 seconds (8.947860054133495 megabytes/s) should show transfer speed of simultaneous SCP instances Finished in 371.761262 seconds 

If you have a large file (I used a ~ 1 GB file), you can use these rspec tests (in scp_spec.rb) or change them to any test harnesses that you are familiar with in order to see this performance degradation.

If you don’t know how this performance can be improved in the library, do you have any ideas on how to open an additional parallel SCP transfer rate besides just calling the scp utility through a subshell?

Rspec test here: https://gist.github.com/703966

Answer 1

Instead, you can try Net-sftp . Sftp is a newer protocol, and the linux scp utility actually uses the sftp protocol, if available. I don’t know if net-scp uses sftp, but I won’t be surprised if it is not.

You can also try rsync , but for this you will also need to install rsync on the remote host. Rsync is by far the king of speed with remote file transfers, although I cannot use pennies with six rsync.