Out of Memory

[LeonIoannou]

I am trying to find the differences between 2 folder structures but there are 41,000 folders in the parent and eventually, BC is running out of memory. Is there a way to increase/remove this cap or even a different functionality I am missing?

[Chris]

Beyond Compare 3 for Windows is a 32-bit application. It will run out of memory when the total number of files being compared reaches 2.5 million files on each side on 32-bit Windows, or 5 million files on 64-bit Windows.

Beyond Compare 4 is a 64-bit application on 64-bit Windows so upgrading to version 4 is the best way to handle large comparisons. If you can only use version 3, then you'll need to split the comparison up into smaller pieces.

See also the max number of files article on our website.
https://www.scootersoftware.com/supp...z=kb_maxfilev3

[kratzbuerste]

I have the same problem -but I am using the 64bit version of BC4 (V4.3.7) on a Windows machine with 24GB of RAM.

Now when I try to find the differences between two folders ("C:" = my internal 500GB SSD and "D:\backup\C" = my external 5TB USB-drive), I am always running into the situation that BC4 hangs.

Symptom is that I see the BC4 task in Windows' taskmanager increasing its used memory while it scans both folders. When the amount of allocated memory exceeds 1GB, the "status" of the BC4 task in the task manager suddenly changes to "inactive" and BC4 doesn't react anymore. Also, the "page faults" in the Windows task manager jump quickly up to ~1 million. Nevertheless, the allocated memory of the BC4 keeps still growing.

Note that with the previous V4.0.7 (=a 32bit version perhaps?), it grew until it finally reached 4GB, then BC4 presented an "out of memory" error message.

With the V4.3.7, it just keeps increasing the RAM ever more, even beyond that 2^32 = 4GB of RAM. But it slows down in the speed of further allocations. After a few hours, I then manually killed the "inactive" BC4 task (memory allocation at that point was ~10GB RAM).

Again: My PC has 24GB RAM, so I doubt this is due to physical RAM shortage. Seems more like a software error in BC that is trashing the system at ~1GB of memory allocation(?)

Any clue on that?

[Aaron]

Hello,

Your analysis of 4.0.7 is right; it was a 32bit application and would hit that 4gb limit. BC4's limitation is significantly higher and shouldn't just slow down after about a gig.

There are a few main things we hit with very large comparisons:
1) Number of items. BC4 can binary scan extremely large files, but the entire display (the tree of files) takes resources, so how many files are you loading?
2) Rules-based scan hitting a problem file. Rules-based scanning has to open the file (and some use an External Conversion process), and if the scan happens to be running into a problem file (that happens to be around the 1 gig amount) that might be the issue. I'd suggest using File Name Filters or smaller base folders to cut the task into smaller parts, and see if there's a specific part that completes without issue (narrowing down onto a specific problem file pair). What kind of scan is configured in the Session Settings?
3) Remote locations or anti-virus. Could either of these be factors in the comparison? It sounds like you have C:\ vs D:\, so I suspect 'remote' isn't an issue like it is with FTP or Network share syntax, but just confirming you aren't accessing the external D:\ as a bound network location. If you whitelist bcompare.exe or bcompare (process), does this impact the performance issue you are seeing?

[kratzbuerste]

Hi Aaron,

guess I came a bit closer to the root cause: I have a subdirectory..

C:\Users\<myname>\AppData\Local\X1 Search\Temp\

If I just try to sync this very directory, it hangs BC4 as I did post before. The other way round, if I exclude just that directory, the sync works.

That directory is very (very very) sparsely populated with files: All in all, it may be just ~100 files in it. But: It is a huge amount of subdirectories therein -to be exact: 2'549'050 (yes: 2.5 million subdirectories).

-Can it be that this large amount of directories is killing BC4?

Note that everything else is truly nothing special from my side:

• simple "folder sync" job
• "include files" = "*"
• "exclude files" = none
• "include folders" = ".\Users\<myname>\AppData\Local\X1 Search\Temp"
• "exclude folders" = none
• all directories are local, no network drives involved here
• the handful of files in those subdirectories are nothing special either: a few PDF, DOCX, XLSX, PPTX, TXT, PNG, JPG, -nothing large or special

Thanks!

P.S.: Just in case you ask: The aforementioned subdirectory belongs to a desktop-search engine application named "X1 Search" (https://www.x1.com/)

[Aaron]

If you've narrowed down to that specific Temp folder, I'd suggest further cutting it into parts to see if there's actually specific trouble items within it.

First, if you set that as the base folder, does the issue reproduce?

You can then start to cut the comparison into parts with filters like:
[A-Ja-j]*\
-.\*.*;[K-Mk-m]*\
-.\*.*;[^A-Ma-m]*\