No announcement yet.

Large folder volume comparison techniques

  • Filter
  • Time
  • Show
Clear All
new posts

  • Large folder volume comparison techniques

    I am trying to see if there are any threads discussing tricks, tips and wisdom regarding how to handle very large folder comparisons. I was thinking of scripting the output to text files and filtering a bunch of the output (with findstr via cmd shell), but, wanted to see if there were other, smarter ways to approach this. I am looking at something in the order of 10,000 folders.

  • #2

    Have you had a chance to try out Folder Compare natively? We have successfully tested up to 2.5 million files on each side with it:

    Outputting a report to a text file is also perfectly valid (our Text Compare has been tested up to file sizes of 500 megs).

    Both compare types have various display filters to help filter the view.

    If it is over FTP and you are running a Content Comparison, then it will require downloading each file. Use Timestamp and Size only, or run an FTP server that supports the xCRC command to use CRC comparisons.

    If you have not tried it directly, I recommend trying out trial to see how it works for you. Please be sure to give it enough time to complete the large comparison, and that your system has adequate free resources:
    Aaron P Scooter Software


    • #3
      Yeah, I use the Folder Compare feature. I am not sure if I am overlooking something regarding the native functionality. I am running over a 3Mb pipe between two SAN's, so, running CRC and file size checks has proven to be pretty slow, but, it works well. Typically, we are examining about 10-12,000 files at a time with a total size of about 50 Gb at a go, but, I've not seen any issues with comparisons. Most of my issue is folder structuring and variables. My hope in posting was to see if anyone had walked through a process automating this yet.


      • #4
        If you only want to compare certain files or folders, you can limit the comparison using filename filters. Right click on a file or folder, then select "Exclude". You can also specify filters in the "Name Filters" tab of "Session > Session Settings".

        If you want to automate the comparison instead of running it interactively, BC does support scripting. See the scripting section of our discussion forums or the "Using Beyond Compare > Automating With Scripts" topic in the help file.
        Chris K Scooter Software