FTP folder isn't loaded

[iklio]

I'm generally very happy with beyond compare but I have a problem with ftp comparisons. When I load/expand large directory hierarchies it sometimes stops working after listing some dirs, similar to this report:

https://forum.scootersoftware.com/fo...re-not-loading

This either causes or is caused by the ftp connection reconnecting to the server. After this I can't get beyond compare to load the contents of any ftp dir, but copying files from the ftp still works. I'm currently using a socks 5 proxy to access the ftp server but I seem to remember this happening with other servers without proxy too (I don't think the posts I linked mentioned proxies).

I can't send you the log because of privacy issues, but I can see the ftp commands working as expected, with cwd, pasv, list and then receiving a dir list, ftp parser, but then the message "Folder isn't loaded".

Tested and observed most recently using BC 4.4.5, build 27371 on debian.

[Aaron]

Hello,

Thanks. Similar to that report, do you think some security software might be injecting itself into the process. A key difference between BC and most other FTP clients is that BC is listing all subfolders, which is significantly more intensive than most other clients that only look at the current level.

You could disable automatic scanning in the Folder Compare's Handling tab (previous to launching the session or connecting to the FTP). This would allow more manual control over expanding and listing items as you visit them, but then loses the comparison status. Everything would appear yellow until all their child items could be visited and built (with an Open Subfolders command, for example).

It's a bit odd that transfer commands continue to work but listing commands fail. If you enable more of the debug logging options in the Tools menu -> Options dialog, Tweaks tab, scroll down to Log Panel, does anything turn up right before the error starts?

[iklio]

I enabled all the debug options but nothing in the log explains what exactly goes wrong. I can see it log successful dir lists, then a reconnect (a dir list seems to time out, I tried using longer and shorter timeout delays), and then I see the same ftp commands but "folder isn't loaded" gets logged inbetween the ftp commands. If I stop the current operation and wait a while it still can't load any dirs after this has happened.

I can see that the ftp server still replies with the correct dir listings, I tried switching the type of ftp command (list, msld) in the beyond compare settings but the same thing happens.

It's quite random, sometimes I can load a full dir tree with a few thousand subdirs and sometimes it fails at a random place (not the same dir).

I have a standard debian install without any software that I'd expect could cause the transfer to fail. I disabled the automatic background scanning, but if I manually expand the dirs then sooner or later the same thing happens.

[Aaron]

Hello,

I don't suppose you could randomize some of the private information in the log, so we could get the rough outline of what is happening? Knowing the connection information at the start might be useful. You can email us privately at [email protected] (including a link back to this forum thread for our reference).

As another shot in the dark: does your server support SFTP or FTPS connections? SFTP has less connection juggling, and might help if it's available server side. Client side, this would require the Pro edition, but if you are registered for Std, you can revert to trial mode temporarily to test this:
https://www.scootersoftware.com/supp...?zz=kb_evalpro

[iklio]

I have a pro license but I can't use SFTP on this server and I have the same problems with FTPS.

Here is a redacted log file of a session

03/03/2023 06:18:29 AM  Username: myusername
03/03/2023 06:18:29 AM  Sent>
03/03/2023 06:18:29 AM  Sent> socks login, user, pass,
03/03/2023 06:18:29 AM  Sent> ftpserver dns
03/03/2023 06:18:29 AM  Recv>
03/03/2023 06:18:29 AM  Stat> Connected.
03/03/2023 06:18:30 AM  Recv> 220 Ready set GO.........
03/03/2023 06:18:30 AM  Sent> HOST myserver
03/03/2023 06:18:30 AM  Recv> 500 'HOST': Command not understood
03/03/2023 06:18:30 AM  Sent> USER myusername
03/03/2023 06:18:31 AM  Recv> 331 Password required for myusername.
03/03/2023 06:18:31 AM  Sent> PASS ********
03/03/2023 06:18:31 AM  Recv> 230-                                
... server greeting received over multiple 230- response lines
03/03/2023 06:18:31 AM        230-              
03/03/2023 06:18:31 AM        230 User myusername logged in.
03/03/2023 06:18:31 AM  Sent> FEAT
03/03/2023 06:18:31 AM  Recv> 211-Extensions supported:
03/03/2023 06:18:31 AM         AUTH TLS
03/03/2023 06:18:31 AM         CLNT
03/03/2023 06:18:31 AM         CPSV
03/03/2023 06:18:31 AM         LIST -1aAdflLRsTU
03/03/2023 06:18:31 AM         MDTM
03/03/2023 06:18:31 AM         MDTM YYYYMMDDHHMMSS filename
03/03/2023 06:18:31 AM         PBSZ
03/03/2023 06:18:31 AM         PROT
03/03/2023 06:18:31 AM         REST STREAM
03/03/2023 06:18:31 AM         SIZE
03/03/2023 06:18:31 AM         SSCN
03/03/2023 06:18:31 AM         STAT -1aAdflLRsTU
03/03/2023 06:18:31 AM         TVFS
03/03/2023 06:18:31 AM         XCRC filename;start;end
03/03/2023 06:18:31 AM        211 END
03/03/2023 06:18:31 AM  Sent> CLNT Beyond Compare v4.4.5
03/03/2023 06:18:31 AM  Recv> 200 Noted.
03/03/2023 06:18:31 AM  Sent> TYPE A
03/03/2023 06:18:31 AM  Recv> 200 Type set to A.
03/03/2023 06:18:31 AM  Sent> SYST
03/03/2023 06:18:31 AM  Recv> 215 UNIX Type: L8
03/03/2023 06:18:31 AM  Sent> TYPE A
03/03/2023 06:18:31 AM  Recv> 200 Type set to A.
03/03/2023 06:18:31 AM  Sent> PWD
03/03/2023 06:18:31 AM  Recv> 257 "/" is current directory.
03/03/2023 06:18:31 AM  Sent> REST 1
03/03/2023 06:18:31 AM  Recv> 350 Restarting at 1. Send STORE or RETRIEVE to initiate transfer.
03/03/2023 06:18:31 AM  Sent> REST 0
03/03/2023 06:18:31 AM  Recv> 350 Restarting at 0. Send STORE or RETRIEVE to initiate transfer.
03/03/2023 06:18:31 AM  Sent> CWD /dirname
03/03/2023 06:18:32 AM  Recv> 250 CWD command successful.
03/03/2023 06:18:32 AM  Sent> PWD
03/03/2023 06:18:32 AM  Recv> 257 "/dirname" is current directory.
03/03/2023 06:18:32 AM  Sent> PASV
03/03/2023 06:18:32 AM  Recv> 227 Entering Passive Mode (ip,port...)
03/03/2023 06:18:32 AM  Sent> LIST
03/03/2023 06:18:32 AM  Recv> 150 Opening ASCII mode data connection for directory listing.
03/03/2023 06:18:32 AM  Recv> 226-
03/03/2023 06:18:32 AM        226
03/03/2023 06:18:32 AM  drwxrwxrwx  13 a       b                0 Nov 05  2021 dirname
03/03/2023 06:18:32 AM  FTP parser: Unix
03/03/2023 06:18:33 AM  Load comparison: ftp_path_with_password... <-> /localdir
03/03/2023 06:18:37 AM  Sent> CWD /subdir_path
03/03/2023 06:18:37 AM  Recv> 250 CWD command successful.
03/03/2023 06:18:37 AM  Sent> PASV
03/03/2023 06:18:37 AM  Recv> 227 Entering Passive Mode (ip)
03/03/2023 06:18:37 AM  Sent> LIST
03/03/2023 06:18:37 AM  Recv> 150 Opening ASCII mode data connection for directory listing.
03/03/2023 06:18:37 AM  Recv> 226-
03/03/2023 06:18:37 AM        226
03/03/2023 06:18:37 AM  drwxrwxrwx   3 a       b           225299 Nov 05  2021 dir_item
03/03/2023 06:18:37 AM  FTP parser: Unix

... repeat from CWD n times, successfully opening many dirs


03/03/2023 06:33:10 AM  Sent> CWD /yetanotherserverdir
03/03/2023 06:33:10 AM  Recv> 250 CWD command successful.
03/03/2023 06:33:10 AM  Sent> PASV
03/03/2023 06:33:10 AM  Recv> 227 Entering Passive Mode (ip)
03/03/2023 06:33:10 AM  Sent> LIST
03/03/2023 06:33:11 AM  Recv> 150 Opening ASCII mode data connection for directory listing.
03/03/2023 06:33:11 AM  Recv> 226-
03/03/2023 06:33:11 AM        226
03/03/2023 06:33:11 AM  -rw-r--r--   1 a       b              467 Dec 14  2019 filename_on_server
03/03/2023 06:33:11 AM  FTP parser: Unix
03/03/2023 06:33:11 AM  Sent> CWD /and_another_server_dir
03/03/2023 06:33:11 AM  Recv> 250 CWD command successful.
03/03/2023 06:33:11 AM  Sent> PASV
03/03/2023 06:33:11 AM  Recv> 227 Entering Passive Mode (ip)
03/03/2023 06:35:11 AM  Sent> QUIT
03/03/2023 06:35:11 AM  Recv> 221 server_quit_message
03/03/2023 06:35:11 AM  Stat> Disconnected.
03/03/2023 06:35:11 AM  Sent>
03/03/2023 06:35:11 AM  Sent> my login info
03/03/2023 06:35:12 AM  Recv>
03/03/2023 06:35:12 AM  Stat> Connected.
03/03/2023 06:35:13 AM  Recv> 220 Ready set GO.........
03/03/2023 06:35:13 AM  Sent> USER myusername
03/03/2023 06:35:13 AM  Recv> 331 Password required for myusername.
03/03/2023 06:35:13 AM  Sent> PASS ********
03/03/2023 06:35:13 AM  Recv> 230-                                 o
03/03/2023 06:35:13 AM        230 User myusername logged in.
03/03/2023 06:35:13 AM  Sent> FEAT
03/03/2023 06:35:13 AM  Recv> 211-Extensions supported:
03/03/2023 06:35:13 AM         AUTH TLS
03/03/2023 06:35:13 AM         CLNT
03/03/2023 06:35:13 AM         CPSV
03/03/2023 06:35:13 AM         LIST -1aAdflLRsTU
03/03/2023 06:35:13 AM         MDTM
03/03/2023 06:35:13 AM         MDTM YYYYMMDDHHMMSS filename
03/03/2023 06:35:13 AM         PBSZ
03/03/2023 06:35:13 AM         PROT
03/03/2023 06:35:13 AM         REST STREAM
03/03/2023 06:35:13 AM         SIZE
03/03/2023 06:35:13 AM         SSCN
03/03/2023 06:35:13 AM         STAT -1aAdflLRsTU
03/03/2023 06:35:13 AM         TVFS
03/03/2023 06:35:13 AM         XCRC filename;start;end
03/03/2023 06:35:13 AM        211 END
03/03/2023 06:35:13 AM  Sent> CLNT Beyond Compare v4.4.5
03/03/2023 06:35:13 AM  Recv> 200 Noted.
03/03/2023 06:35:13 AM  Sent> TYPE A
03/03/2023 06:35:13 AM  Recv> 200 Type set to A.
03/03/2023 06:35:13 AM  Sent> SYST
03/03/2023 06:35:13 AM  Recv> 215 UNIX Type: L8
03/03/2023 06:35:13 AM  Sent> TYPE A
03/03/2023 06:35:13 AM  Recv> 200 Type set to A.
03/03/2023 06:35:13 AM  Sent> PWD
03/03/2023 06:35:14 AM  Recv> 257 "/" is current directory.
03/03/2023 06:35:14 AM  Sent> CWD /nextdironserver
03/03/2023 06:35:14 AM  Recv> 250 CWD command successful.
03/03/2023 06:35:14 AM  Sent> PASV
03/03/2023 06:35:14 AM  Recv> 227 Entering Passive Mode (ip)
03/03/2023 06:35:14 AM  Sent> LIST
03/03/2023 06:35:14 AM  Recv> 150 Opening ASCII mode data connection for directory listing.
03/03/2023 06:35:14 AM  Recv> 226-
03/03/2023 06:35:14 AM        226
03/03/2023 06:35:14 AM  -rw-r--r--   1 a       b              401 Dec 14  2019 filename_on_server
03/03/2023 06:35:14 AM  FTP parser: Unix
03/03/2023 06:35:14 AM  Sent> CWD /next_dir_on_server
03/03/2023 06:35:14 AM  Folder isn't loaded
03/03/2023 06:35:14 AM  Recv> 250 CWD command successful.
03/03/2023 06:35:14 AM  Sent> PASV
03/03/2023 06:35:14 AM  Recv> 227 Entering Passive Mode (ip)
03/03/2023 06:35:15 AM  Sent> LIST
03/03/2023 06:35:15 AM  Recv> 150 Opening ASCII mode data connection for directory listing.
03/03/2023 06:35:15 AM  Recv> 226-
03/03/2023 06:35:15 AM        226
03/03/2023 06:35:15 AM  -rw-r--r--   1 a       b             6738 Dec 14  2019 some_file_on_server
03/03/2023 06:35:15 AM  FTP parser: Unix
03/03/2023 06:35:15 AM  Sent> CWD /another_dir
03/03/2023 06:35:15 AM  Folder isn't loaded
03/03/2023 06:35:15 AM  Recv> 250 CWD command successful.
03/03/2023 06:35:15 AM  Sent> PASV
03/03/2023 06:35:15 AM  Recv> 227 Entering Passive Mode (ip)
.. repeats for every remaining dir​

Something goes wrong around the time of the quit/reconnect. Not sure if it's timing out because of the server either, I haven't noticed timeouts with other ftp clients, but after the reconnect it reports folder isn't loaded for every dir.

[iklio]

Another observation that might help is that after this happens I can still run refresh on a previously loaded ftp dir and BC will pick up any changes on the server. The dirs that failed to load in the first operation seem to have permanently failed and won't respond to, e.g., open subfolders.

[iklio]

Even more confusing, if I use 2 connections to the ftp one could fail while the second keeps working. Even after both reconnect the one that could load dirs still works, while the one that failed still can't load any dirs. If I then for example repeatedly use "expand all" I can load half of the previously failed folders in the last iteration, until the second connection also fails.

[iklio]

I have found a workaround to keep going when this happens. I'll write it here in case someone else will find it useful:

While dirs are loading (well, failing to load), push "Stop". Abort and quit commands will be sent to the ftp before BC disconnects active connections. Right click one of the subdirs and open it n a new view. BC should reconnect, show a retry dialog, and load the dir successfully. Close the view, go back to the original comparison and use expand all to load the remaining dirs. Repeat as needed.

[Aaron]

This one has stumped me for a bit, but that's a great workaround! The new tab would act as a new session, so it would connect to the server. Meanwhile, tabs are part of the same process, so they share comparison data with each other, so you can update the original tab by updating in a new tab. Meanwhile, a refresh does not throw away the connection, which is why a refresh wasn't working to force a reconnect in the original tab. A full refresh would force the reconnection, but also throws away the current comparison, which would force you to start over and likely fail.

I've been working under the assumption this was an issue with a specific FTPS server or version, but re-reading the thread, do you see this behavior with multiple servers? Do you have any other servers you can try similar size/load testing with to see if my initial assumption was true?

[iklio]

I'm pretty sure I had the same problem on another ftp server a while back, but I can test it again in a few days.

[Aaron]

If it is multiple, different servers (running different server software), I'd suggest seeing if there is any firewall software in between that you can configure to allow BCompare.exe or Bcompare (process) through as a whitelist. Besides being intensive, you might have some security software that sees frequent or fast Listing of all directories and cutting that connection.

[iklio]

I've now tested with another computer and connecting to a different ftp server, and I have the same issues. One server is using ftps, one plain ftp. There is something that goes wrong with a per-connection state that makes it impossible to list any more directories on that connection until I can get it reset using the folder unavailable -> retry dialog. The main problem isn't that a single folder fails to load but that everything fails after that happens.

I can get it to keep working using the workaround, but then I need to monitor the process to catch when the problem occurs and since this can take hours it's a bit inconvenient.

[Aaron]

Were both test computers testing within the same physical network? I wonder if there's any part of the chain we could otherwise remove as another test, such as testing from home on your personal machine and connecting to either FTP/S? Or if you have access to one trying any other FTP/S server that isn't part of (what I assume) is a shared work environment.
We haven't had any other reports like this, so even if it's something we're able to try to improve, we'd need to know which part of your network chain is causing the issue so we can figure out what might be happening.
One other idea is if you have the ability to generate a test account, and that test account can reproduce the issue when used outside of your network, you could email us those details to [email protected] including a link back to this forum thread for our reference, and we could reproduce the issue that way.

[iklio]

So I set up a vsftpd server with ssl at home, and tested against that using beyond compare on a mac. While I still saw the same issue it seemed to happen a lot less often.

I also tried changing almost every ftp option in BC. It's difficult to say if anything helped, I couldn't find a configuration that made the problem disappear and since it happens at random times it's hard to say if anything reduced the frequency of the problem.

BC crashed in the middle of loading dirs on a few occasions. Not sure if it's related to the same problem.

Could you possibly make a build with more debug output added to the folder loading code, especially related to reconnecting, listing dirs and "Folder isn't loaded" ?

Edit: Ok, the crash is actually BC quitting if the ftp connection is closed unexpectedly by the server. I was testing some different server settings and restarting the ftpd.

[Aaron]

Hi,

Which version of Debian and vsftpd are you running at home? Can you reproduce the issue during BC4's listing/building by restarting vsftpd without altering any settings?