Process stops responding when comparing large text files

[cowwoc]

Hi,

When comparing large text files (1GB) in Beyond Compare 3.3.13 (build 18981) or Beyond Compare 4.1.2 (build 20720) I am seeing the following behavior:

* Beyond Compare begins parsing the files. As it does so, the scrollbar grows and lines are slowly inserted.
* Initially the UI is responsive, but about 10-20 seconds later the UI begins becoming unresponsive.
* If you keep the keyboard arrow key down, you will notice it "pauses" every second for a longer and longer period of time (every time lines are added?)
* About a minute later, the UI becomes so unresponsive (whether you move the caret or not) that the user cannot abort the DIFF (the UI stops responding) and Windows flags the process as "not responding".

I am guessing you have two problems here:
* Beyond Compare has a function whose complexity increases exponentially with the length of the diff window.
* The UI thread blocks on a regular basis to run this increasingly-complex function, until eventually it stops responding altogether.

This problem is reproducible 100% of the time on my end. Can you reproduce the problem on your end?

UPDATE: I should clarify, I am doing "rule comparison" with the following two rules defined:

Basic: "Ignore timestamps", regular expression = ^\d+:\d+:\d+\.\d+\s, Unimportant
Line: Ignore lines containing: "candidate":, and 0 lines after it.

The above rules slow down processing, making it easier to reproduce this problem.

Gili

[Aaron]

Hello,

The max file size supported in the 32bit version of the Text Compare is around 500 MB. We display and load the file simultaneously, so as it loads more resources are used. Windows will often show programs as Unresponsive if the main process is too busy to respond to a request; if we have not yet shown our crash dialog for running out of memory, then we are still attempting to load the file and may return control if given enough time.
http://www.scootersoftware.com/suppo...z=kb_maxfilev3

BC 4.1.2 is available as a 64bit application (you can check which is installed using the Help menu -> About dialog). We automatically detect and install the 64bit version if your OS is 64bit. If you give the 64bit several minutes to try and load the file, does it eventually load for you?

1GB files are fairly large. I've seen other applications appear to load files that large quickly, but are actually truncating the file, so I would be cautious when dealing with them in various text editors.

[cowwoc]

Hi Aaron,

My system has 32GB of ram so we're not going to run out of memory. In my initial bug report (below) I was already using version 4.1.2, 64-bit version. I understand that eventually the file will probably finish loading, but the question is why performance keeps on degrading in the meantime.

I don't understand though, why aren't you able to parse the file in one thread, and respond to UI events in a separate thread? Wouldn't that solve this problem regardless of how big the file is?

Gili

Hello,

The max file size supported in the 32bit version of the Text Compare is around 500 MB. We display and load the file simultaneously, so as it loads more resources are used. Windows will often show programs as Unresponsive if the main process is too busy to respond to a request; if we have not yet shown our crash dialog for running out of memory, then we are still attempting to load the file and may return control if given enough time.
http://www.scootersoftware.com/suppo...z=kb_maxfilev3

BC 4.1.2 is available as a 64bit application (you can check which is installed using the Help menu -> About dialog). We automatically detect and install the 64bit version if your OS is 64bit. If you give the 64bit several minutes to try and load the file, does it eventually load for you?

1GB files are fairly large. I've seen other applications appear to load files that large quickly, but are actually truncating the file, so I would be cautious when dealing with them in various text editors.

[Aaron]

Hello,

BC4 does use multiple threads, but some tasks are using the foreground thread for processing. For example, the navigation provided by the Thumbnail view. You can try using the View menu -> Thumbnail to disable this control to test how much this helps. Some processing and loading of large files in chunks still uses the foreground thread, so the loading process itself may be slow as more of the file is loaded. Once the load is complete, how responsive is the interface for you?

Sorry for the confusion about the 64bit build. I brought it up since the original post was in the BC3 forum and BC3 is only 32bit, and this would exceed our tested limits. As a 32bit application, BC3 would only have access to around 2gb of RAM. BC4.1+ is also available as 32bit or 64bit, and I wasn't sure yet which you were using or how much RAM your system had. Improving for large files is something on our wishlist; we only recently released the 64bit client capable of opening large files.

[cowwoc]

Once the file is loaded (26,539,152 lines long) every time I hit the arrow key, the caret moves around 2 seconds later. CPU usage is at 0% if I don't touch anything, but the UI is still incredibly slow.

Gili

Hello,

BC4 does use multiple threads, but some tasks are using the foreground thread for processing. For example, the navigation provided by the Thumbnail view. You can try using the View menu -> Thumbnail to disable this control to test how much this helps. Some processing and loading of large files in chunks still uses the foreground thread, so the loading process itself may be slow as more of the file is loaded. Once the load is complete, how responsive is the interface for you?

Sorry for the confusion about the 64bit build. I brought it up since the original post was in the BC3 forum and BC3 is only 32bit, and this would exceed our tested limits. As a 32bit application, BC3 would only have access to around 2gb of RAM. BC4.1+ is also available as 32bit or 64bit, and I wasn't sure yet which you were using or how much RAM your system had. Improving for large files is something on our wishlist; we only recently released the 64bit client capable of opening large files.

[Aaron]

Thanks for the report. Those files are quite large and have many lines over our initial test cases. We'll look into improving our performance when viewing files with that many lines.