SFTP: Handling of recursive symlinks

[chrroe]

Hi!

One thing I would like to see in the final 3.0 release is a more robust handling of recursive symlinks when connecting to a SFTP linux server (maybe FTP is affected as well).

Example (a listing from my openSuSE 10.3 installation):

linux-g1jp:/boot # ll
insgesamt 9700
-rw------- 1 root root     512 13. Jan 14:45 backup_mbr
lrwxrwxrwx 1 root root       1 13. Jan 14:12 boot -> .
-rw-r--r-- 1 root root   80417 11. Feb 04:14 config-2.6.22.17-0.1-default
drwxr-xr-x 2 root root    4096 29. Feb 21:31 grub
lrwxrwxrwx 1 root root      28 29. Feb 21:31 initrd -> initrd-2.6.22.17-0.1-default
-rw-r--r-- 1 root root 4190802 29. Feb 21:31 initrd-2.6.22.17-0.1-default
-rw-r--r-- 1 root root  394752 19. Jan 12:40 message
-rw-r--r-- 1 root root  100315 11. Feb 04:15 symsets-2.6.22.17-0.1-default.tar.gz
-rw-r--r-- 1 root root  400576 11. Feb 04:16 symtypes-2.6.22.17-0.1-default.gz
-rw-r--r-- 1 root root  116297 11. Feb 04:15 symvers-2.6.22.17-0.1-default.gz
-rw-r--r-- 1 root root  843401 11. Feb 04:02 System.map-2.6.22.17-0.1-default
-rwxr-xr-x 1 root root 2147048 11. Feb 04:13 vmlinux-2.6.22.17-0.1-default.gz
lrwxrwxrwx 1 root root      29 29. Feb 21:30 vmlinuz -> vmlinuz-2.6.22.17-0.1-default
-rw-r--r-- 1 root root 1593936 11. Feb 04:02 vmlinuz-2.6.22.17-0.1-default

As you can see there is a symlinked directory called boot which points to the current directory. Using BC3's folder compare you can endlessly expand the boot folder resulting in a cascaded tree structure (screenshot1).
When you try to do some action with this folder (for example Compare Contents... or Copy To Right...) the background-scanning stops after a while with Unable to load boot: No such file in the log pane.
The calculated amount of files so far is 588 files, 43 folders, 201 MB.

Using the Calculate function of WinSCP's properties dialog for the same folder shows a different result (screenshot2). Somewhere in their forum or documentation I've read that WinSCP internally remembers every scanned directory in order to avoid endless loops caused by recursive symlinks.

It would be nice when BC3 could deliver the same results ...

Bye
Christoph

[Zoë]

WinSCP has trivial symlink behavior: It always shows the metadata for the symlink, and trying to open a symlinked directory changes the selected folder in its folder tree to the symlink's target. I think that works great for them, but I don't think it maps all that well to BC.

That said, symlink behavior has been an open question for a long time now, and we're open to improvements, especially since it's an even greater issue with the Linux build. To put it simply, what's your dream behavior for symlinks?

Do you like that BC opens them inline like it does now, or would you prefer it jump around the folder heirarchy like WinSCP does? Would your answer be different if a symlinked directory was aligned with a real directory?

Should copying a symlink copy it as the target or as a symlink? Copying as a symlink is really only possible in the Linux build, or possibly using SFTP, so the behavior might have to be different depending on the destination.

Should directory sizes include the size the symlinks or the targets? It sounds like you're suggesting they they only include the actual symlink size. If copying a folder follows symlinks that could make the final copy significantly larger than what's displayed on the source side.

Should symlinks show metadata (last modified, size, mode/owner/group) for the symlink or the target? WinSCP shows the data for the symlink; Konquerer shows the data for the target. In the Linux build we've had at least one complaint about showing it for the link, so the current behavior shows the target. On FTP it currently always shows the data for the link; I think SFTP could support showing the target with an addional round-trip per link to get the target's data. On BC2 this could be controlled by switching the FTP link resolution type to "Complete", but that was pretty complex code and could get tripped up by loops.

And a new one introduced with BC3: Trying to delete a directory when some of it's contents are filtered out will only delete the non-filtered children and leave the directory alone. If you selected a symlinked directory instead, should it just delete the link or follow the non-linked behavior?

[Mark L]

I typically like symlinks to copy as is (as a symlink, not the target) and would like the diff display to show the symlink size and other attributes rather than the target attributes. The icon could be similar to that shown in FTP mode. It makes sense that the target files get displayed when expanding the symlink directory.

Maybe a context menu or configuration option to switch between symlink or target view would satisfy others who prefer the other mode. I can see scenarios where you would copy the symlink's target contents rather than the symlink, so these options in the context menu could be very useful.

[magma596]

Just an Idea

what about an option to traverse sym links or not to? It is more complex but I feel that there is a benefit in both forms of functionality. Just my .02.

Patrick

[makholm]

FYI, I outlined my thoughts on symlink handling here:

http://www.scootersoftware.com/vbulletin/showthread.php?t=3581

[makholm]

You basically can't win all cases with one strategy only. As I've mentioned in my other thread about this, I think you need a "dereference symbolic links" option (from here on known as "-L" for brevity) like other Unix tools have... If this option is enabled, symlinks are followed and the data shown are that of the target file. If disabled the links data is shown. If you split everything into these two cases, decision become easier and behaviour more sane...

That said, symlink behavior has been an open question for a long time now, and we're open to improvements, especially since it's an even greater issue with the Linux build. To put it simply, what's your dream behavior for symlinks?

http://www.scootersoftware.com/vbulletin/showthread.php?t=3581

:-)

Do you like that BC opens them inline like it does now, or would you prefer it jump around the folder heirarchy like WinSCP does? Would your answer be different if a symlinked directory was aligned with a real directory?

For symlinks pointing to directories

If -L is enabled,

you can expand them inline in the tree as if they were real directories, but there should be a clear visual clue that the dir is in fact a symlink. In this case a right-click menu item for following the link (i.e. cd to the dir it points to) would be nice...

If one side is a symlink and the other is a dir, follow the link but with visual cues as above...

If -L is disabled,

the contents of the symlinks should be compared. In this case a right-click menu item for explicitly expanding a symlink inline in the tree would be useful. A refresh would undo this.

If one side is a symlink and the other is a dir, ask the user what to do...

Should copying a symlink copy it as the target or as a symlink? Copying as a symlink is really only possible in the Linux build, or possibly using SFTP, so the behavior might have to be different depending on the destination.

If -L is enabled. Copy the target.
If -L is disabled, copy links verbatim.

See the cp manpage.

Should directory sizes include the size the symlinks or the targets?

If -L is enabled. the size of targets. Follow symilnks recursively
If -L is disabled, the size of links. No following, ever.

See the du manpage.

It sounds like you're suggesting they they only include the actual symlink size. If copying a folder follows symlinks that could make the final copy significantly larger than what's displayed on the source side.

This should only be possible if you disable -L, calculate size, enable -L and then copy without recalculating size. You might want to always calculate size before copying directories (AFAIR, Total Commander for Windows does this). It can be slow for large trees though, so probably the size check should be breakable.

Should symlinks show metadata (last modified, size, mode/owner/group) for the symlink or the target?

If -L enabled, show target meta data, i.e. stat().
If -L disabled, show link meta data, i.e. lstat().

WinSCP shows the data for the symlink; Konquerer shows the data for the target. In the Linux build we've had at least one complaint about showing it for the link, so the current behavior shows the target. On FTP it currently always shows the data for the link; I think SFTP could support showing the target with an addional round-trip per link to get the target's data. On BC2 this could be controlled by switching the FTP link resolution type to "Complete", but that was pretty complex code and could get tripped up by loops.

Operating systems that implement support for symlinks generally apply a limit to how many links are followed in search of the ultimate target. In Linux this is:

/usr/include/sys/param.h:

#define MAXSYMLINKS 20

Which means that e.g. open() will simply give up with an error after following 20 links. The error in this case is ELOOP:

/usr/include/asm/errno.h:

#define  ELOOP 40 /* Too many symbolic links encountered */

...which strerror() will translate into "Too many levels of symbolic links".

If BC follows symlinks over FTP (as opposed to the server doing it), then a similar strategy would be reasonable. Recurse max N times and then give up.

And a new one introduced with BC3: Trying to delete a directory when some of it's contents are filtered out will only delete the non-filtered children and leave the directory alone. If you selected a symlinked directory instead, should it just delete the link or follow the non-linked behavior?

If -L enabled, delete target. If you decide to follow and delete all links recursively in this case, a big fat warning should probably be displayed, as that can be a potentially very desctructive operation.

If -L disabled, delete just the link.

Apart from that, if I select a dir and delete it, I will expect to disappear completely, regard of any filters. I will however appreciate a strong visual clue or confirmation dialog telling me that I might be about to delete something I can't currently see.

Generally speaking, I think you have to be consistent about symlinks. If -L is enabled, you always follow links for both FTP and local filesystems. If -L is disabled, you never follow links (unless explicitly action is taken to do so). IMO that's the only way you will get a sane and predictable behaviour...

For a tool like BC, I think it's reasonable to expect users to know what a symlink is and how it works. In any case experienced Unix users will be extremely frustrated with a tool that deviates significantly from the de-facto standard of using -L or not...

Come to think of it, "-L" might actually be a good icon for a tool bar buttong for enabling/disabling the "dereference symbolic links" option...

[oxygene]

Any news on this one? I would need an option to disable symlink dereferencing, maybe configurable per profile. Otherwise, it's nearly impossible to compare two Linux folders like /usr/

[Aaron]

We have an open tracker entry, and a developer will be investigating the issue when they can.

[mikej_w]

This is an issue I have been fighting for YEARS, but now it is becoming critical. Up to this point I have not run into self referencing links, but BC3 goes nuts in this case. Not knowing that the link is there, and then copying the (unseen) subdirectory with the link, wow. It just sat there copying till I stopped it. Yuk!

The behaviour right now is fine for one option, PLEASE just give me the option to treat a link as just another file. In this mode, it could have a link column, that shows where it points to, otherwise, it should act just like it is a "normal" file, nothing more/less.

Mike

[Aaron]

Hello Mike,

We've been investigating Makholm's suggestion above. If you have any additional thoughts on how it should be handled, please let us know in this thread.