Compare huge CSV files?

[krilbe]

I have two CSV files, each about 4 Gbyte and about 1 million rows. The first column is a unique integer key.

Most of the contents should be the same, but a few percent will differ. I could settle for comparing only the unique keys and get a report containing orphans for each side.

I tried a regular table compare, but it seems to take "forever". It has been "thinking" for about an hour now.

Any better way to do this?

[Aaron]

Hello,

4gb is well above our max test case for the BC3 Data Compare. BC3 is a 32bit application and will hit memory limits due to the limited amount of RAM we have access to. If you continue to let it load, it may be able to complete or it may crash when it runs out of memory
Our test case can be found here:
http://www.scootersoftware.com/suppo...z=kb_maxfilev3

BC4.1 recently added 64bit support. You can download the trial of BC4 to test if it works better with your files. You can install BC4's trial without removing or altering BC3, but you can also back up your current BC3 settings using the Tools menu -> Export, just in case you would like to uninstall and roll back your BC3 installation. Let us know how it works for you:
http://www.scootersoftware.com/download.php