Using BC3 To Find And Resolve Long Path Name Issues

[The Baker Family]

Please explain how to use any of BC3’s functionality to find long path names.

The remainder of this post is strictly FYI. Please do not be distracted by the additional content. Please focus on the above question in the first few lines of any reply to this post.

The purpose of the following material is to not only explain the reason for this post but also to hopefully present a compelling argument that will result in Scooter Software adding Path functionality to BC3’s file/folder name filter in a future BC3 version.

Scooter Software’s Forum post ‘Re: File Path length’ dated 13-Feb-2006, 06:31 PM, broached the Path issue but failed to answer the underlying Path question. This post explained how to search for individual files or folders using the filter functionality. However this reply was in response to the original question submitted by dburtis2 which focused on “path name lengths > 128 characters cause a fatal error.”

Provably the most common event that triggers Long Path Name issues is when a folder is copied to another folder or drive using the operating systems copy and past commands. The resulting file path name length for some of the file/folders may exceed the path name limit of the receiving folder. The most obvious issue is a violation of “Max_Path.” There are other events that produce path name issues such as the one dburtis2 encountered but they are outside this scope of this post. Unfortunately I have been bitten more times then I care to admit by the ‘Curse of Max_Path.’

Using BC3 to copy folders to another folder is obviously a better choice as BC3 will identify Max_Path violations in the source folder. The errant source files are presented as “cant access” error messages on the BC3 desktop and are subsequently listed in the BcLogTrace file. However, using BC3 is not a panacea when long path names are involved because BC3 does not notify the user when it encounters a source file or folder that will exceed acceptable file/folder naming limits if copied to the destination folder/drive. Apparently the ‘Curse of Max_Path’ is alive and well at Scooter Software as BC3 automatically and silently resolves the naming limit issue by renaming files and folders in accordance with 8.3 (8dot3) short naming conventions and places them in the destination folder.

The end result is the destination folder will contain all of the file/folder contents present in the source folder but the file/folder names will not be the same as they were in the Source folder. Therefore, a novice will lose confidence in the software if and when they become aware of these strangely named items in the destination folder. They will have no idea where these names came from and will provably assume they exist as a result of a glitch in the file system or BC software. Additionally, a SnapShot of the destination folder will be problematic when it is used to find future modifications made to the source folder.

To a knowledgeable user the previous paragraph is old news. A novice user, if they read this material will provably assume the contents are intended for deep space transmission. At a minimum, BC3 should notify the user when this situation exists otherwise the novice will assume that because no error messages were presented by BC3, the destination folder is an exact duplicate of the source folder.

What does all this conversation about copying folders have to do with the original request in this post? The answer is the only way to avoid the dreaded ‘Curse of Max_Path’ and encountering the automatic renaming issue is to manually rename (shorten) the files/folders before you copy them into a destination folder. My request is to find a way to identify these files/folders before executing a BC3 copy session. You have to know that these long paths exist if you are going to rename (shorten) them. As I am an advocate of using computers to resolve such issues, I would naturally want this process automated. This automation could reside in the requested future BC3 feature listed above. There are utility software products on the market that will do just what I am requesting in this post. However, my preference, and I assume Scooter Software’s preference, would be for me use BC3 instead of one of their competitor’s products for this purpose.

Well its time to change the focus and see if I can nudge Scooter Software into directly supporting the HP Media Smart Server & Windows Home Server. As an informational note, both of these products run on top of Microsoft’s Windows Server 2003 operating system.

Given the advent of the Home Server, the long path name issue now lives in the novice realm. For the novice to grasp the core concepts of files and paths, they encounter such things as the Windows API, MAX_PATH, prefixing the path with \\?\, 8.3 or (8dot3) equivalents, the shell (Explorer) 260 character limit, Dir /x, Dir /n, and NTFS’s 255 character limit. By definition, the novice will abandon any hope of understanding these core concepts. For this and other reasons, Scooter Software has a real opportunity to expand their market share by enhancing the BC3 product to support the Home Server user. By definition, the Home Server is targeted directly at and exists primarily to serve the novice user. Experts will adopt the technology but for the Home Server to survive in the main stream market it must be productively operable by a ‘novice’ user.

As an added incentive for Scooter Software to honor my request and get directly involved in supporting the Home Server, I will state the primary reason why I purchase and use Beyond Compares software. Having used BC for more years then I care to calculate, I know the value added by BC is its inherent ability to confirm and compare data. Many products do the same thing BC does but the main reason I use BC is because I trust this product. I have learned that I can trust BC to do what it says it does without consternation. In short, my last line of defense or if you will, my gold standard for data accuracy is Scooter Software’s Beyond Compare. A more objective and real world explanation is the fact that I have to be able to sleep at night without worrying about my data integrity assurance methodologies. You have to draw the line somewhere and my past experiences with BC allow me to put my professional reputation and my job on the line by relying on BC to do what is sais it will do reliably every time.

As I have moved into the Home Server realm I have learned one truth that is unavoidable, that truth being I can not trust the HP Media Smart Server (HP-MSS) or its underlying Windows Home Server operating system. These products were released into the market with known data corruption issues. Thereby moving the beta test bed into the general user population. The customer is apparently so trapped by the current market situation that major companies such as HP and Microsoft now appear to have become so brazen as to no longer fear their own customer’s wrath. I guess this must be one of those new world order things. It appears that the CEO’s of these firms have the same mindset that is costing us several trillion dollars to bail out the financial system and kick start the economy courtesy of Uncle Sam.

Enough of the pontificating, I need BC to confirm that the data stored on the HP-MSS has not been corrupted by the servers own software. Thankfully, I was able to determine that BC3 works fine running in the Windows Server 2003 environment. Loading and operating BC3 on the HP-MSS is well beyond the normal user capabilities. If Scooter were to come to market with an Add-On package for the HP-MSS, I am sure they would have no problem marketing it. Your marketing folks might name such a product ‘Glasnost for the HP-MSS’. Thanks to Ronald Reagan, many U.S. citizens confuse this Russian word for openness with the Russian expression Dovorey no provorey (Trust, but verify). Being that HP-MSS has shown that it can’t be trusted; the only other acceptable option if you want to use the new Home Server technology is to verify that the data entrusted to this system has not been corrupted. That’s where BC3 comes in. I hope you folks fly with the Add-on idea.

Oh well, back to the purpose of this post, the following is a collection of previous Forum Posts related to the Path Issue:

13-Feb-2006, 06:31 PM Re: File Path length
13-Apr-2006, 01:03 PM Re: Request: Tooltip for long filenames
17-Apr-2006, 07:08 PM Re: Long Filename are not treated correctly
29-Jan-2008, 06:46 PM Re: Long filenames, MS-DOS filenames
17-Feb-2009, 07:56 PM ClearCase versioned filenames longer than 256...
17-Feb-2009, 04:53 PM 8602 insufficient filepath references in error

[Zoë]

Beyond Compare isn't designed to make it easy to find files with paths longer than MAX_PATH, and I don't see that changing. The only way I can think of to do so is to load the directory you're interested in, expand everything, select everything, then use the "Copy Filenames" command to get a list of all of the filenames. You can then use a text editor to find just those lines that are longer than MAX_PATH. The regular expression to use would be "^.{260,}". The only editors I have that support that expression are BC's text compare and EditPad Pro.

If you're familiar with any programming language or scripting language, it would be much easier to bypass BC entirely, and just write something that enumerates all of the directories recursively and outputs a list of any files/directories with paths longer than MAX_PATH.

As to why BC doesn't show anything for files longer than MAX_PATH, the reason is simply that BC generally is not affected by it and can handle paths up to 32,768 characters long by using the \\?\ syntax automatically if it's necessary. In the cases where that doesn't work, it's a limitation in some third-party software you're using. I've successfully compared/copied/edited across computers using UNC paths with files over 1300 characters long. If your network is switching the files to their 8.3 equivalents, giving access denied errors, or silently failing in some other way you should complain to the vendor of your file server.

From your examples:

File Path Length: This is a limitation in the DVD structures, and is unrelated to MAX_PATH. There's no easy way for us to detect or correct for this.

Tooltip for long filenames: This was asking for tooltips for filenames that are truncated because the display is too narrow. It has nothing to do with MAX_PATH and is fixed in 3.0.

Long Filename are not treated correctly: The user never got back to us, but I suspect that they, like you, were running into a limitation in their file server. As I said, Vista, at least, supports much longer paths over UNC.

MS-DOS filenames: Like the first post, this is just a limitation in the CD structures. More specifcally, filenames on CDs can be limited to to 64 characters if they use the Joliet extensions, but the Rock Ridge extensions allow 255 characters. I'd like to handle comparing truncated filenames vs their original filenames better, but again, that's unrelated to MAX_PATH.

ClearCase: This is a MAX_PATH issue, but is a limitation in ClearCase, not BC. We can detect this case, so where possible we will work around it.

Insufficient filepath references: This is just asking that we include the fully qualified path in error messages instead of just the filename. It's unrelated to MAX_PATH and will be fixed in the next release.

I appreciate your suggestion to make an add-on product that runs on the server, but I don't see that happening soon. If we do make a server-side add-on it will be a general-purpose server designed for transferring differences between files, not something that just works around a specific vendor's MAX_PATH issues. There's almost no chance such a limited product would make enough money to pay for it's development and ongoing support.