Very slow SFTP download from HP Server

[RichardBlake]

For months I've seen very slow downloads of large files from a server running HP-UX 11.23 to a PC running WinXP Pro SP3, thinking something had been changed on the server related to SFTP. (Things were fine in early BC3 builds except for the occasional null at 0x7FFF.)

Today I happened to see the server admin, who told me nothing has changed recently. I started composing this thread, doing some test downloads (10MB file in 4.5 minutes), and noticed that the problem seems to be limited to ASCII files having suffix .inp. (Well, all the big (to 90MB) files I download are .inp suffix. Small files transfer fine, or at least are finished before I notice they are slow.) Start of the log:
6/10/2009 6:20:46 PM Username: xxxxxxxx
6/10/2009 6:20:46 PM Connecting to xxxxxxx.xxx.xxx
6/10/2009 6:20:46 PM Server key [ssh-rsa 0000 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00]
6/10/2009 6:20:47 PM Authorization successful.
6/10/2009 6:20:47 PM Connected to xxxxxxxx.xxx.xxx port 22
6/10/2009 6:20:47 PM Server software: OpenSSH_3.7
6/10/2009 6:20:47 PM Using SFTP version 3.
6/10/2009 6:20:47 PM Compression: none
6/10/2009 6:20:47 PM Encryption: aes256-ctr
6/10/2009 6:20:47 PM MAC algorithm: hmac-md5
Settings include:
Transfer type: Auto; Compress transfers is not checked; Force faster uploads is not checked, both limits are set to 0; 1 simultaneous connection; PASV is not checked. Currently running BC3 3.1.3.

I removed *.inp from the ASCII list then tried downloading an 85MB file with .inp suffix. It completed in seconds!

I put *.inp back into the list (in alphabetical order) and tried the download again. Problem returns.
I click the (X) icon and it hangs on Cancelling with the animated marching squares. Click the [X] on title bar and I'm warned this will cancel running file operations. OK; hourglass. Have to kill the program.

Then I added *.ipp to the ASCII list and renamed a big .inp file to that suffix. Slow again. So BC3 isn't prejudiced against the .inp suffix. This time I clicked the close button. After the warning it did the 'Cancelling' thing as before.

I have 41 items in my ASCII list - not too many, I hope.

One more try: after changing .ipp back to .inp in the ASCII list, I renamed my big .inp file to be a .txt file. That's slow, too. Killing it with '40 Minutes Remaining'. So it looks like the problem is with any large file that is auto-detected to be ASCII.

All file transfers are fast using the SFTP included with SSH Secure Shell Client 3.2.9 (last free).

(BTW, it would be convenient if the log message at completion of a transfer showed the transfer rate in KB/sec or whatever, as well as elapsed time; would make it easier to compare to other programs that show that value.)

[Zoë]

Hi Richard,

OpenSSH doesn't actually support ASCII transfers (it was introduced in SFTP v4), so BC fakes it by translating the files locally as it's sending it. I believe a side effect of the current implementation is that it also disables pipelining (sending chunks in parallel), and significantly reduces the size of the buffer it sends. Normally it has ~4MB transferring at a time, but the faked ASCII transfer only does 32KB and has to wait for the server to respond before sending the next chunk.

So, it's a known issue, and it's unlikely to change. You can work around it by switching entirely to binary transfers, or by using a SFTP server that supports a newer version of the spec. Unfortunately, AFAIK, the latter option excludes all versions of OpenSSH.

[RichardBlake]

OK, I've switched to binary mode, and follow the file transfer by filtering through a CR <> CR-LF converter. Each step runs rapidly. Seems like BC could do it this way, rather than doing conversion during the file transfer and taking the big performance hit. Only downside is the need for temporary diskspace for the translated copy of the file. Separate thread for the translation step would allow that to run in parallel with other file transfers. Is my site the rare exception in using OpenSSH as an SFTP server?

[Zoë]

I've fixed this for the next release. I changed where the translation takes place, so it's now as fast as the binary transfer, and it still occurs on the fly as it's reading/writing, so it doesn't use any extra disk space. The same changes also let me fix a long-standing issue with ASCII transfers in our Linux build.