Comparison With NAS (FTP Server) Slow - Any Way To Speed It UP?

[tackyjan]

Hello All!!

I have an external USB hard drive that I sync with an NAS drive using BC3. The NAS drive (Synology 112j) has an FTP server and so I setup my compare session to use FTP.

The comparison seems to run very slow. Is there anything I can do to speed up the compare process when the "right-side" is an FTP server?

Also, I have the ability to map my NAS drive to a drive letter would it be faster to compare to a mapped drive instead?

Thank you!!

Jan

[Aaron]

Hello,

Are you running a content comparison? If so, BC3 needs to download the files from remote locations before scanning them. The exception would be if the FTP server supports the xCRC command; xCRC allows the generation of the code on the server, which speeds up the overall comparison.

[luckman212]

Does anyone know of a linux SFTP server that supports this XCRC command?

[Zoë]

Hi luckman212,

Are you interested in SFTP (SSH) or FTP over SSL?

SFTP is a binary protocol, so the server-side CRC calculation isn't called XCRC. I don't know of any servers (Windows or Linux) that actually support it, since it's a relatively recent addition to the protocol. BC will use it if it's available, but I honestly haven't been able to test it.

If you are actually asking about FTP over SSL, it looks like Serv-U has a Linux version now. There's also an add-on for ProFTPD called mod_digest available on GitHub at https://github.com/smartftp/mod_digest.

[tackyjan]

Hello,

Are you running a content comparison? If so, BC3 needs to download the files from remote locations before scanning them. The exception would be if the FTP server supports the xCRC command; xCRC allows the generation of the code on the server, which speeds up the overall comparison.

I am not running content comparison. In the "Comparison" tab of my session I have the following checked:

- Compare file size
- Compare filename case
- Override quick test results

In general should I be using content comparison? It seems like this would be more robust albeit slower. I would like to see if my NAS has xCRC. Will get back with you on that.

Thanks!!

[tackyjan]

How To Check If My NAS Supports xCRC?

Anyone know how I might go about checking if my NAS supports xCRC?

Here is the model that I have:

http://www.synology.com/products/pro...DS112j&lang=us


Thanks!

[Zoë]

tackyjan,

When you connect to the FTP site using BC, look in the green log in the bottom for "Sent> FEAT" followed by a list of mostly four-letter capitalized words. If your server supports it you should see "XCRC" listed there.

It doesn't look like Aaron mentioned it previously, but the biggest way to speed up FTP connections is to enable multiple connections. Use the Tools-FTP Profiles dialog to modify your settings. On the "Connection" tab increase the "Simultaneous connections" setting to 3, 5, or 10 and see if that improves things.

Since the NAS supports it, you should try mounting it as a drive too, and compare the speed. It doesn't support simultaneous connections like FTP does, but the directory listing transfers are faster to begin with, so it's quite possible it will be faster.

[tackyjan]

tackyjan,

When you connect to the FTP site using BC, look in the green log in the bottom for "Sent> FEAT" followed by a list of mostly four-letter capitalized words. If your server supports it you should see "XCRC" listed there.

It doesn't look like Aaron mentioned it previously, but the biggest way to speed up FTP connections is to enable multiple connections. Use the Tools-FTP Profiles dialog to modify your settings. On the "Connection" tab increase the "Simultaneous connections" setting to 3, 5, or 10 and see if that improves things.

Since the NAS supports it, you should try mounting it as a drive too, and compare the speed. It doesn't support simultaneous connections like FTP does, but the directory listing transfers are faster to begin with, so it's quite possible it will be faster.

Thanks!!

I will play with the FTP connection as well as using a mapped drive and see which is faster.

I don't recall if BC reports the total time at the end of a session, if not that would be a nice feature!

Thanks again!

[tackyjan]

...increase the "Simultaneous connections" setting to 3, 5, or 10 and see if that improves things....

I changed my FTP profile to use 10 connection threads and now I get the following error about 50% of the time:

10/30/2012 11:22:20 AM Unable to load ftp://[email protected]/shared/_b...Backup_Misc/My Documents/My PSP Files/Palettes: Can't open passive connection: Address already in use.


Note that the error does not always occur on the same file/directory. Seems to be random.

Is this some kind of threading issue? Perhaps one thread is trying to access a connection already owned by another?

[Aaron]

Hello,

If you lower the total number of connections, how does this work for you? You might be hitting a max number of connections, and everything over that is throwing this error. You may also need to restart BC3 to get the FTP Profile settings to apply.

[tackyjan]

Hello,

If you lower the total number of connections, how does this work for you? You might be hitting a max number of connections, and everything over that is throwing this error. You may also need to restart BC3 to get the FTP Profile settings to apply.

It still happens with only 5 connections. I will try less and report back. Thanks!

Note. Whenever I get that error some of the folders remain yellow. I am assuming this is because the comparison failed and that yellow means not processed (?).

If I do a Refresh then it finishes the comparison so there is a workaround.

[Aaron]

Hello,

Yellow does mean not processed. Usually there would be a secondary mark (a small red 'x') if the folder had an error and wasn't accessed.

[tackyjan]

Hello,

Yellow does mean not processed. Usually there would be a secondary mark (a small red 'x') if the folder had an error and wasn't accessed.

Ok gotcha! Then what does the yellow mean? And why do folders turn yellow only at times when I get that error and then turn green when I refresh?

Thanks!

[Aaron]

Hello,

Yellow means that it is unscanned. It does have the error 'x' mark, right? You can view a legend of all the different colors in the View menu -> Legend.

We do not have a 'green' by default, so you might have customized one of your colors. If so, the Legend will be updated to show what that is. If you are referring to "Teal", that is for filtered files/folders. It likely means you have Suppressed Filters enabled, and if it were disabled the items would be hidden from view due to a filter that is currently set.

A full screen screenshot might help. If you would like to email privately, please do so to [email protected] and include a link back to this forum thread for our reference.

[tackyjan]

Hello,

Yellow means that it is unscanned. It does have the error 'x' mark, right? You can view a legend of all the different colors in the View menu -> Legend.

We do not have a 'green' by default, so you might have customized one of your colors. If so, the Legend will be updated to show what that is. If you are referring to "Teal", that is for filtered files/folders. It likely means you have Suppressed Filters enabled, and if it were disabled the items would be hidden from view due to a filter that is currently set.

A full screen screenshot might help. If you would like to email privately, please do so to [email protected] and include a link back to this forum thread for our reference.

It doesn't matter to me if this discussion is public. Unless you want it to be I am fine with staying on the forums.

Will post a screenshot soon.