UNIX scp Copy Performance Tuning

By Laurent Schneider

Copying a terabyte over ssh is very slow. The question was how to fastcopy in Unix without installing Software and without the root privilege?

I have not gone too far in recompiling the SSH, and I did not try plain FTP.

Ok, let�s try first to transfer 10 files of 100M from srv001 to srv002 with scp :

time scp 100M* srv002:
100M1    100%   95MB   4.5MB/s   00:21 100M10   100%   95MB   6.4MB/s   00:15 100M2    100%   95MB   6.0MB/s   00:16 100M3    100%   95MB   4.2MB/s   00:23 100M4    100%   95MB   3.4MB/s   00:28 100M5    100%   95MB   4.2MB/s   00:23 100M6    100%   95MB   6.4MB/s   00:15 100M7    100%   95MB   6.8MB/s   00:14 100M8    100%   95MB   6.8MB/s   00:14 100M9    100%   95MB   6.4MB/s   00:15

real    3m4.50s
user    0m27.07s
sys     0m21.56s

more than 3 minutes for 1G.

I got hints about the buffer size, about SFTP, about the cipher algorithm, and about parallelizing. I did not install new software and I have a pretty old openssh client (3.8). Thanks to all my contributors and the ones the will answer after the writting of this blog post�

Ok, let�s try a faster algorithm, with sftp (instead of scp), a higher buffer and in parallel: 

$ cat batch.ksh
echo "progress\nput 100M1" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M2" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M3" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M4" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M5" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M6" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M7" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M8" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M9" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M10" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&

wait
$ time batch.ksh

real    0m19.07s
user    0m12.08s
sys     0m5.86s

This results in a 1000% speed enhancement.