MD5 Support for FTP?

[davenovak]

As far as I know, Beyond Compare does not contain support for MD5s. Thus far (for local file/folder moves/syncs), I've been OK with that as CRC and BINARY comparisons have been functionally suitable replacements for MD5.

Now, however, I have a real need for MD5 that I can't seem to get around: FTP sites that use extensions supporting MD5 hash checks. For example, we use Akamai Netstorge, which allows you to retrieve the MD5 of a file on the server through the site command. Here's the syntax:

site shohash <filename>

Unfortunately, Akamai does not support CRC (or XCRC) commands.

Has any thought been given to supporting this FTP extension? It would go a long ways towards making comparisons between our build machine and Akami fast and accurate. As is, the CRC checking in BC3 forces each file to be read in its entirety (since Akamai does not support XCRC).

[Zoë]

Hi Dave,

I'm assuming you have a typo, and the command is actually "site shahash <filename>".

In any case, no I haven't considered it because this is the first I've heard of it and I can't find any documentation on that command. Do you have any links I could see? Have you contacted Akamai and asked them to support XMD5 or XCRC? Does the FTP site show that command in the FEAT response?

[davenovak]

No, that is actually typed correctly, and the command works. I have a tool that I've written that makes use of that feature all the time. My tool uses this for validating the handful of uploads/downloads it performs. Now I find myself with a with a rather large directory (that changes all the time) and would really like to use Beyond Compare to sync contents between source and FTP server (as this is clearly beyond the scope of my little tool). But given the lack of CRC support at Akamai (and MD5 support in BC3), the BC3 sync is quite slow.

You can download documentation for the Akamai NetStorage service here: http://pigdogslow.dyndns.org/NetStorage_UserGuide.pdf. Page 17 documents all of their "FTP Extensions".

Here are some of the responses I get when querying the Akamai server for its capabilities:

ftp> quote feat
211-Features:
 MDTM
 REST STREAM
 SIZE
 AUTH TLS
 PBSZ
 PROT
211 End
ftp> quote help
214-The following commands are recognized (* =>'s unimplemented):
 CWD     XCWD    CDUP    XCUP    SMNT*   QUIT    PORT    PASV
 EPRT    EPSV    ALLO*   RNFR    RNTO    DELE    MDTM    RMD
 XRMD    MKD     XMKD    PWD     XPWD    SIZE    SYST    HELP
 NOOP    FEAT    OPTS    AUTH    CCC*    CONF*   ENC*    MIC*
 PBSZ    PROT    TYPE    STRU    MODE    RETR    STOR    STOU
 APPE    REST    ABOR    USER    PASS    ACCT*   REIN*   LIST
214 Direct comments to [email protected]
ftp> quote help site
214-The following SITE commands are recognized (* =>'s unimplemented).
 HELP    COPY    SYMLINK SHOHASH CHKSIZE CHKHASH SETTIME EXTNAME
 AZ2Z
214 Direct comments to [email protected]
ftp> quote help xcrc
502 Unknown command 'XCRC'.
ftp> quote help xmd5
502 Unknown command 'XMD5'.
ftp>

Given that so many people likely use Akamai, having this MD5 support built-in to BC3 would be really great. Having said that, it's only fair that I ask Akamai to start supporting XCRC and XMD5 as well, which I'll do. But Akamai doesn't do a very good job of listening (from my experience).

Another option (supported by both Akamai and BC3) is to use SFTP. Though the Akamai SFTP server does not support any sort of hash-checking, I've been told that SFTP transfers are generally more reliable and possibly even self-validating (with hash checks occurring with each packet transfer). Can you make any comment about the reliability of SFTP transfers? Would this be a better way to go?

[Aaron]

Hello,

Thanks for the information. I've added it in our wishlist entry on MD5 and hashing. If/When we add MD5 support, we will look into adding this command.

I would recommend asking Akamai for xCRC support. It is a bit surprising that they cover MD5 but not CRC (since CRC is built into zips).

I do not know if SFTP transfers are more reliable. They are probably close/comparable if they do differ. SFTP has the secure layer/SSH, and can sometimes be a little bit slower to transfer. We are working on improving the SFTP speed.

[davenovak]

Yeah, it really is surprising that Akamai doesn't support both CRC and MD5. I emailed them yesterday about this and am awaiting a reply.

Thanks for the info about SFTP. Thus far, the speed BC3 accomplishes with SFTP is pretty good and certainly much faster than the SFTP library I am using with my tool.

Keep up the good work!

[davenovak]

Quick Update --

I found the following relevant blurb about SFTP file transfer integrity:

As a file is transferred between client and server, it is broken up into smaller chunks called "packets." For example, suppose each packet is 32KB. The SFTP protocol does a checksum on each 32KB file as it is sent, and includes that checksum along with that packet. The receiver gets that packet and decrypts the data, and then verifies the checksum. The checksum itself is "stronger" than the CRC32 checksum. (Because SFTP uses a 128-bit or higher checksum, such as MD5 or SHA, and because this is done on each and every packet, there is a very granular integrity checking that is accomplished as part of the transfer.) Thus, the protocol itself is slower (because of the additional overhead), but the successful completion of a transfer means, de facto, that it has be transferred integrally and there is no need for an additional check.

If that's the case, I think that I should be quite safe to just use Akamai SFTP together with BC3 for my large folder sync operations. Plus, BC3 will at least do the size and date/time comparison under SFTP.

Still, I find it strange that none of the SFTP servers (at least none that I've seen) support an explicit CRC or MD5 file check. The paranoid part of me would like to see SFTP servers support that.

[Zoë]

To clarify some things:

BC's SFTP speed for ASCII transfers on SFTP v3 and earlier servers (mostly OpenSSH) was slow until 3.1.4. Both it and binary transfers are now as fast as I expect them to get, barring tweaks for specific servers.

As for server-side CRC/MD5 checks, they were added to the SFTP spec in one of the later versions (v4 or higher). I'm not aware of any server (and I've tested lots) that supports them. VanDyke's is actually one of the few that supports anything higher than SFTP v3, so it might be worth asking them to support it. Theoretically BC will use them if a server supports them.

[SacredSoundSystems]

Do you have any links I could see?

http://tools.ietf.org/html/draft-ietf-ftpext2-hash-00

[snidely.too]

http://tools.ietf.org/html/draft-ietf-ftpext2-hash-00

This looks like a good reference, but it differs from the description of the Akamai implementation ("site shohash fn", instead of "hash fn", for instance). Akamai may be what part of the introduction refers to:

In the past, several attempts have been made to add commands to obtain checksums and hashes, however none have been formally specified, leading to non-interoperability and confusion.

("shohash" is clippish from "show hash", perhaps.)

davenovak may need to get a link from Akamai's support to enable Scooter Software to work with their implementation.

/dps

[dr_barnowl]

SSH hash support

It occurs to me that hashes could just be supported on SSH connections just by running the server-local md5sum or sha1sum utility ; you could even do them in bulk by addressing more than one file in the parameters.