Symlinks

[Marc Chamberlin]

I am a bit disgruntled with Beyond Compare Version 4.3.7 (build 25118) right now because it badly broke one of my systems (rendering it unbootable) and it took me many hours to find and resolve the problem. I used BC to upgrade and normalize many of my system configuration files so that where possible I have identical, or nearly identical configurations on all my systems. I compared files and copied changes or whole files from one system to another and thought everything was going OK. (BC never complained once, asked for help, or gave a warning about destination symlinks)

Unknown to me was the fact that BC is abysmal at handling symlinks! There is NO identification or warning that the destination file is a symlink AND WHAT'S WORSE is that the copy process DESTROYS the symlink and replaces it with the actual file copied across. Even though I have "follow symlinks" checked, BC does not obey that configuration when copying files. At the very least, when a symlink is the destination, the user ought to be asked what he/she wants the copy operation to do, copy the symlink itself across (if the source is also a symlink), replace the destination symlink, or replace the file that the symlink points to. To do something without clarification, blindly, is destructive programming and the cardinal rule of good software engineering practices is to NEVER destroy user data unless he/she specifically says to do so!

I did some Googling about BC and symlinks and found complaints about it dating all the way back to 2008, and it has been revisited in the meantime resulting in a partial but totally inadequate solution of supplying the Follow Symlinks checkbox that a user can check. Come on people, this is a serious and destructive bug that should have been given high priority, long ago, too getting it fixed! Especially after all these YEARS! Please help the user to find an adequate solution to his/her needs and problems, rather than blindly lead them into a trap. You are the experts after all and have received suggestions years ago about the the need to handle symlinks better, and given ideas and ways to solve this issue.

Marc....

[Aaron]

Hello,

I'm really sorry that happened. We've had a variety of issues or enhancement requests related to symlinks over the years; did you find an existing thread that has more information on your specific scenario? If you could share that link, that would help me cross reference what might have happened.

With Follow Symbolic Links enabled, the item becomes the target file, so any copies would create as a file in the destination and as a symlink. I could see that being an issue in reconstructing a system, although in some cases I'd expect some error when attempting to overwrite the destination side as they are often protected items.

BC4 does not have the capability to create symbolic links, and since the base folder syntax can differ it would have to create relative symbolic links. One related request (that we haven't been able to tackle yet) would be to filter on or exclude symbolic links. This would allow you to work on only the real files as a first pass, then narrow and review (and manually recreate) any symbolic links as a second pass. Would this have helped during your system reconstruction?

[wernerml]

I really love this software... But it really can't handle symlinks (it sould recreate the symlink in the other panel) !

I dream of this being solved !


[Edit]: I've just now read the solutions of Aaron, and it would be amazing to FILTER them... indeed !

then, just a quick List symlink command in cmd (save them) and recreate them later...
YES, a option to FILTER them would solve it !!!!

[Aaron]

Thanks. I'll add your notes to our wishlist entries on the subject.

[Marc Chamberlin]

My apologies for reviving this thread I started, I wasn't able to get back to it and now it is surfacing again on me. Aaron no I cannot find any threads that has particular relevance to my complaint about symlinks, just lots of generic complaints about how symlinks are not handled in an expected manner. A few things that need to be improved are -

1. If a file or a dir, shown in the comparison view, is a symlink then that clearly needs to be pointed out and the link pointer shown. BC doesn't do this all the time, but sometimes it does. I cannot understand why this is or how to correct it. But not knowing that a file or a dir is a symlink is really frustrating and causes all kinds of bad things to happen when such a file or dir is moved or copied from one side to the other.

2. When a symlink is edited, copied, or moved the user should be asked what part of the symlink does he/she want to work with, the pointer itself, or the target of the symlink. This needs to be recursive in nature since a symlink can point to another symlink.

3. When a comparison is made between two symlinks, both the pointers themselves and the targets of the symlink should be compared and results shown.

4. When a comparison between a symlink and an actual target is made then obviously only the targets are the objects of interest, but the user should still have the capability to modify the symlink pointer, if that is desired.

I know that it is possible to ask the file system whether a file or a directory is a symlink, other linux commands such as ll or ls -al show symlinks perfectly fine.

Finally, I don't think having the ability to exclude symlinks is a valid solution, it just covers up the problem at best.

[Marc Chamberlin]

BC4 does not have the capability to create symbolic links, and since the base folder syntax can differ it would have to create relative symbolic links

.

Aaron - I didn't understand this comment of yours, perhaps we have different models of how BC4 should interact with symlinks. I don't see why symlink's pointers and targets cannot be handled as two separate objects, both editable and managed by the user, not automagically by BC4. You are correct that the context of symlinks cannot be seamlessly managed by BC4, and I am not asking that. Just make sure the user is aware, at all times, when a symlink is in play, and let the user decide and manage which part of a symlink he/she wants to manage or work with. I would image that the GUI might be something like the following if the symlink pointer and target are the same -

]leftSideFileorDirLinkName                          ==                               rightSideFileorDirLinkName
    <left.Side.Symlink.Path>                          ==                                   <right.Side.Symlink.Path>
    leftSideTargetFileorDir                              ==                                   rightSideTargetFileorDir

If there are multi symlinks involve then just keep expanding them until the actual target is reached -

leftSideFileorDirLinkName                           ==                               rightSideFileorDirLinkName
    <left.Side.Symlink.Path>                         ==                                   <right.Side.Symlink.Path>
    nested-leftSideFileorDirName                 ==                                   nested-rightSideFileorDirName
         <nested-left.Side.Symlink.Path>         ==                                        <nested-right.Side.Symlink.Path>
         leftSideTargetFileorDir                        ==                                        rightSideTargetFileorDir

Alignment issues displayed in the above examples, derived from the usage of CODE tags and manual spacing, and a developer's indentation preferences aside; this should somewhat graphically show how I am suggesting that BC4 can be improved to handle symlinks in a more user friendly and manageable way. At least I hope you and the development team can get the jest of this idea. And regardless of whether absolute or relative symlink paths are used, simply show them as they were defined. Colors and fonts could also be used to further enhance the usage of symlinks. From this example the BC4 team should be able to work out how to show differences in each part of a symlink and highlight (in RED) broken or inconsistent symlink paths. Wish I had the time to grok the BC4 code and help, I don't, but I suspect this sort of addition to BC4 should not be hard to do, most of the components needed seem to be mostly there, this just requires a bit of tweaking to the BC4 model and some recursive code to follow and display symlinks.I focused on soft symlinks in these examples, hard symlinks are similar for comparison purposes, but the BC4 GUI should differentiate and show which kind of symlink is being used, perhaps using some sort of symbol to show whether a hard or soft symlink is involved. It matters to a user because hard links cannot cross mount points.

(As an aside, I suspect BC4 has troubles following soft symlinks across mount points also, though I don't grok why. I say this because it appears I am getting bogus permission error messages and not able to follow soft symlinks that cross mount point boundaries. That needs to be fixed also, I don't see any reason why BC4 cannot follow these soft symlinks when the Linux OS certainly allows them to be created, and handles the de-referencing of them, if following normal linux file and dir permissions allows such symlinks to be followed for a given user, of course.)

HTHs Marc..

[Aaron]

Hello,

Because of some of your comments in 1., I think you may have different kinds of links, which is adding to the complexity and confusion around your tasks. I'll focus on how we handle symbolic links for now for each of these questions, but if we can determine exactly what is on your system, it might help figure out what is happening when you attempt to sync. I suspect some of the issues might be when you have Follow disabled (and expect to get Links) but some types of links (like hardlinks) always follow so you'd end up with Targets; hence the "sometimes" behavior you are seeing.

1. BC4 does report if it is a symbolic link with an arrow icon overlay on top of the other icon. It also has the (path to target) in parenthesis after the "filename". This shows both if Follow Symbolic Links is enabled or disabled (but double clicking the item will either Follow and show followed content, or open as a file showing the target path if Not Followed).

You might have some other kind of link, like a hardlink or another type? The hardlink doesn't expose itself and is always the target (either in Explorer or BC4, it will look like a regular file), so it would always follow the target. I notice we're a little diverged in calling them soft and hard symbolic links or symbolic links and hardlinks; we're using the terminology derived from things like the mklink command, which references /H as "Creates a hard link instead of a symbolic link". "Symbolic Links" are always 'soft'.

2. Yes, that would be a great feature and the recursive nature, as well as other edge cases, make it a significant task to implement. It's on our radar and these are good ideas, it's just not something we'll be able to tackle short-term. If you'd like my help in troubleshooting what can be done with our current capabilities, please let me know and I'd be glad to help.

3. This one is a bit trickier. Somehow a file would have to be represented as two files, or as an overridable Comparison tab entry (when should one result override the other, and when should a difference flag in either mark the entire thing as different, would need to be customizable). Our current behavior can do this in two passes: this is how the comparison is performed based on if the Follow option in the Handling tab is enabled or disabled. When enabled, a content scan would scan the target content, and when disabled a content scan would scan the path to target. I see your follow up post has ideas around this, and I'll be sure to include those notes.

4. This is how the Follow option works, in that the user can enable it to perform that comparison, or disable it view the path inside. But BC4 does not allow modification of the symbolic links, only viewing the paths. Editing a symlink it outside of our support, but I can add this idea to our customer wishlist.

Beyond Compare has a lot of filtering tools, and is designed around focusing on specific sets of data or comparison. Strong filtering controls allow users to control exactly what they want to see and review or fix, and can work in a variety of workflows that prompts might be too limited.

[mightysprite]

Thanks for this thread, I just installed BC on Linux for the first time (after using it for almost 20 years now on Windows!) and the "follow links" was exactly what I was looking for.

BTW Marc there are no "hard" or "soft" symlinks. Symlinks are symbolic links, which is a file that just has the path to another file in it. While it "makes sense" to "just copy it," I think that could lead to a bunch of unexpected and undesirable behavior. E.g. I am copying a symlink that points to ~/bin/some_program to another place... Should the home directory be expanded, or should it use the home directory of whichever user is logged in? If you look at the 'alternatives' functionality in modern Linux, you have these chains of symlinks to choose the right program for 'editor' for instance, should it copy just the first symlink so that 'update-alternatives' changes propogate, or do you want any updates to be thrown away? I think we could come up with a pretty big matrix of behavior here, especially when we start getting into relative directories, and honestly I'm not confident that even the command line tools allow you to be "completely flexible" with how you handle symlinks.

Hard links, however, are a different beast entirely and they bring up an interesting question to me. One could argue that the target of the hard link should be opened and overwritten, so that all references to that file get changed, however I think it'd be much easier to argue that the file target should just be "unlinked" and then recreated, which would mean that the formerly shared hard links would point to the old file while the "synchronized" one is now sync'ed to the data you were copying but different.

"Follow links" does exactly what I want but what I use BC for, likely in excess of 99% of the time I open it, is to compare files in source control. I want to compare an old version of a file that's now in the /tmp directory to the version that's on-disk, but the file tree is recreated in /tmp as well using symlinks so that I can edit the "current" side as necessary. I understand and accept that if folks use it for synchronization that it may not work right; for syncing large datasets I tend to use rsync or robocopy depending on platform since I want to set up the sync process once in a script and then anyone can fire it off. (I realize that I can do this in BC but this is for environments where I can't just install and license this software however I want.)