No announcement yet.

Performance of folder compares (non-binary; TrueCrypt)

  • Filter
  • Time
  • Show
Clear All
new posts

  • Performance of folder compares (non-binary; TrueCrypt)

    I read the forum topic "BC3 44% slower than BC2 on folder compares", but I'm starting a separate topic because (1) I'm not doing binary comparisons and (2) I'm using encryption.

    I'm using TrueCrypt's "System-Drive Encryption" to encrypt my entire C: hard drive (including the Windows XP operating system). I know that encryption will affect the performance of BC's folder comparisons, but I don't know how much. After all, I'm not comparing contents, just the filename, size, & timestamp. I have only a handful of exclusion filters.

    I know there are many factors that affect performance, so rather than list all of my software and hardware, I'll just say that my computer was above average when I bought it in 2003, and has a 2.8 Ghz CPU (the bottleneck during comparisons is the CPU, not the hard drive).

    Before the test, I do a full refresh twice, so all the file metadata should be cached in memory.

    I have 28,000 files in 3,600 folders (mirrored on the left and right sides). The folder comparison in BC takes 36 seconds.

    I wrote a very naive, non-GUI C++ program that loads both the left and right folder trees into memory. For each tree, it recursively walks through each folder, and for each file, does fifty string comparisons of 100-byte strings to simulate comparing filenames, timestamps, exclude filters, etc. My test took 7 seconds total for all 28,000 files.

    I know it's an extremely over-simplified test that ignores a lot of the complex processing and comparisons that BC has to do behind the scenes, not to mention that BC has to constantly repaint the screen, handle Windows events, and on and on. I know that it's not a fair comparison and please know that I think BC is an amazing piece of software and I use it several times every day.

    Nonetheless, the difference between 7 seconds for my over-simplified test to 36 seconds for BC seems too big to ignore. I could more easily understand if BC took 2 times as long or even 3 times, but 5 times seems a bit unreasonable. Is there some kind of conflict between TrueCrypt's I/O drivers and the way that BC interacts with the file system (and that my little test somehow avoids)? Or is it unrelated to TrueCrypt and I'm seeing about the same performance as everyone else?

    Note that of my 28,000 files, only about 200 of them change per day. So for 99% of my folders, all files in them should already be identical on the left and right sides. From my perspective, it seems that BC only has to verify this, add each file to the GUI list, and move to the next folder. However, it seems that BC might be over-complicating this and spending too much time with these already mirrored folders.

    Another odd thing I noticed is how BC "jumps all over the place" in the GUI during folder comparisons. Of course I have no idea what BC is doing, but as a user who watches this happen several times per day on his screen, it doesn't *seem* very efficient. Instead, I would expect to see BC process the first folder at the top level, then all of its subfolders, then move to the second folder at the top level, and so on.

    I would also prefer such consistent top-down visual behavior when doing the sync operation after the folder compare. Sometimes the BC window is "stuck" with unprocessed files in view at the top while BC is processing other files that are out of view. I would much rather see top-down processing so that I can always see the files that BC is currently syncing.

    Here are the "Folder Handling" options I use:

    [X] Automatically scan subfolders in background
    ...[ ] Automatically scan top-level orphan subfolders
    [X] Expand subfolders when loading session
    ...[X] Only expand subfolders with differences

    I can reduce the comparison time by five seconds if I disable the last two options, but that's still 31 seconds.

    I realize that BC3 was just released and that the Scooter team probably hasn't had much (if any) time to do optimizations, but in the meantime I wanted to post my (unfair) test results.

    I know that everyone has different hardware speeds, but I'm curious if BC's performance on my computer (36 seconds for 28,000 files) is approximately the same as other BC users, and also if anyone else is using TrueCrypt's system-wide encryption with BC.

  • #2

    My first thought is that this is related to the 44% slower issue; you can confirm that easily enough by seeing how BC2 performs in the same situation. v3.0.5 will have a fix for the problem, so I'd suggest testing that when it's released and let us know if it's still a problem.

    If the file size or timestamp is guaranteed to change whenever a file is updated you can consider turning on "Skip if quick tests indicate files are the same". That's really the only way BC can not spend time on files you've already mirrored.

    As for the jumping all over the place, it's doing a breadth-first comparison instead of a depth-first one. For a variety of BC-specific reasons that's both more efficient and easier to implement, though I can't say what kind of effect it has on Windows' caching. I agree that it's a bit disconcerting, but it'll take time to switch it around, and there's quite a few things that have higher priority right now.
    Zoë P Scooter Software


    • #3
      In regard to the 37 seconds for my folder compare, I'm happy to report that build 3.0.4 (and perhaps non-related factors) have reduced this to 29 seconds for the 28,000 files.

      Even better, I've discovered a way to chop the 29 seconds down to 11 seconds by immediately clicking the Mirror-to-Right button right after I click the Full Refresh button. (Of course this doesn't start the mirror operation but it shows the modal summary dialog on the screen while the folder compare runs.)

      Eleven seconds is awesome! That's only about 1/3 of the duration and only 4 seconds more than my simple test program. It seems quite amazing that BC3 can do all that it does during a folder compare and only take 4 seconds more. I'm very surprised yet very happy about this.

      However, now I wonder how I can achieve this huge performance gain without clicking on the Mirror-to-Right button until after the folder compare completes. I want to wait to click it because I usually follow this workflow: start/complete the folder compare showing only orphans, manually examine the orphans for problems, then change the filter to show differences before I start the sync (so I can watch each file as it's copied).

      I'm hoping that this is one of the improvements that you're planning.

      I've read other posts that say BC3 does more in the compare and less in the sync than BC2, so I'm also wondering if clicking the Mirror button during the compare bypasses this extra processing or is this performance gain unrelated?

      [...] you can consider turning on "Skip if quick tests indicate files are the same".
      I'm pretty sure that I'm not doing any kind of content comparisons (and the "Skip if quick tests" checkbox is grayed out along with those other content options). I compare just timestamp and size.


      • #4
        By the way, if I temporarily disable the last two options:

        [X] Automatically scan subfolders in background
        ...[ ] Automatically scan top-level orphan subfolders
        [ ] Expand subfolders when loading session
        ...[ ] Only expand subfolders with differences

        and click the Mirror button right after the folder compare starts, then BC completes the folder compare in a mere 8 seconds (versus 7 seconds for the bare-bones DOS program). I had to post these test results because I'm just thoroughly impressed with such performance from a full-featured Windows program.


        • #5
          The folder compare completes in about 11 seconds now using BC 3.0.7 (versus 29 seconds in BC 3.0.4). Thanks!