optimization request for comparing large trees via FTP

[b.steffen]

Hello support team,

I have an optimization request concerning folder comparison via FTP/SFTP:

setting is:
- comparing two large directory trees (some 40.000 files on each side)
- using FTP (or SFTP)
- one large subpart is only available in the remote server

I've configured BC to use CRC algorithm, which speeds up FTP comparison considerably.

Comparing works pretty well in general, but I found one downside, which is very time consuming in the above mentioned scenario:

BC still tries to get all CRC values even for the files on the remote server, which are definitely not on the local server. This is no problem for some tenth of files, but gets very time consuming for 10.000's of files, especially large ones. (I'm talking of 15-30 minutes in my setting alone for this branch.)

Can this behavior be optimized?

TIA
Bernhard

[Zoë]

Beyond Compare is already optimized for cases like this, and only calculates CRCs if it needs them. It won't calculate CRCs for files that are only on one side, and it won't calculate them for files that have matches on the opposite side with different sizes. If you have the CRC column visible then it will calculate them for orphans as they're scrolled into view.

[Michael Bulgrien]

BC still tries to get all CRC values even for the files on the remote server, which are definitely not on the local server. This is no problem for some tenth of files, but gets very time consuming for 10.000's of files, especially large ones.

Do you have the CRC column visible in the BC3 folder compare? If so, then BC3 needs to calculate the CRC to display it. Try hiding the column so that BC3 only needs to calculate the CRC when its needed for the compare.

[b.steffen]

Do you have the CRC column visible in the BC3 folder compare? If so, then BC3 needs to calculate the CRC to display it. Try hiding the column so that BC3 only needs to calculate the CRC when its needed for the compare.

no, it's standard columns only (name, size, date/time). I use CRC for detecting changes (instead of having to download each and every file).

setting is:
- app. 50.000 files on each side (15 GB of data)
- comparision (using CRC) takes about 1 hour (rough estimate)
- local network (100 MBit)
- relatively fast machines (dual core on each side)
- (nearly) no other network traffic at the same time
- plenty of mem available (RAM / HDD) on both machines

I'm using this for backing up my production machine (every couple of days). Besides consuming much time (which I cannot justify or disagree with at the moment) I get crashes of BC once in a while (yesterday twice, one with error reporting, one without any comment, BC just disappearing suddenly), but not reproducable

[Aaron]

Hello,

Were there any messages in your System log (Right click on My Computer, Manage: Event Viewer: System (or other folders))?

The crash you submitted indicates you were deleting files. Did you delete files/folders while the background comparison scan was still running?

[b.steffen]

Hello,

Were there any messages in your System log (Right click on My Computer, Manage: Event Viewer: System (or other folders))?

The crash you submitted indicates you were deleting files. Did you delete files/folders while the background comparison scan was still running?

Thanks for looking into this.

I've checked system and application log - there were no specific error messages related to ftp, network connection etc. on that day (only about docking / undocking etc.).

Yes, I was deleting some files and directories while background comparision was still running. That's been files / dirs I didn't need any more and therefore deleted them on both sides (at the same time). I'm not sure any more, but I think, those files were still grey (meaning: no crc calculated so far).

Kind regards
Bernhard