2 hour discrepancy after sftp copy

[SSnow]

I've searched the forum messages and have found various references to problems that are similar to mine, but not quite exactly.

I have two notebook computers that I try to keep synchronized precisely because of the need to have a backup system for remote admin chores in case the primary system fails. This means I copy updated files every day between the systems, mostly from primary to backup, but also from backup to primary on occasion. (BC is installed on both systems, and I might initiate the comparison / copy from either system.)

I've been using BC for years on the Windows side and never saw this issue on that OS. But I just moved over to Ubuntu within the past couple of weeks. I just copied some files from primary to backup and checked that no differences showed up. I updated a file on the backup system and started to copy it to the primary system. Lo and behold I saw a LOT of files on primary listed as being two hours NEWER than the same files on backup.

These files are all identical! What's more, most of these files are several years old. I am using binary only copying, and I am using touch local files on FTP transfers.

Okay, I figure I'll outsmart it. I select all of the "older" local files on the backup system, right-click, and touch them using the timestamp on the primary system. All fixed.

Uh, no. A couple of hours later I open BC on primary to copy files over to backup, and I see all the same files listed as being -- drum roll please -- two hours NEWER (again) than the files on the local system.

I don't want to keep on having to select hundreds of files and touch them to make them match the files on the other system.

Can anyone help me?

Be gentle. I'm new to Linux, and I thought that I had kissed goodbye the slings and arrows of FILE TRANSFER PROTOCOL. Will the beast never die? Do I have an alternative, either in the way I use BC or the data protocol I use for synchronizing the two systems?

PS: I thought I should assure you that date/time and time zone are set identically on the two systems.

[SSnow]

It Gets Weirder

Just looked at Beyond Compare on the other system. It does NOT show the timestamp differences.



PS: Sorry for being a moron. Version is 3.07. Files were originally copied onto one of these systems from DVD, then across to the other system over the network SSH connection. Then I installed BC3 (3.04, I think) and saw some weird discrepancies in filesizes (html, text, etc). I changed sftp settings to do binary transfers and equilibrated. Went through version 3.06 with no problems. First saw this two hour thing in version 3.07.

[SSnow]

Additional Testing - Talking to Myself

I decided to experiment.

1. I burned all data from the primary machine to DVD.
2. I removed all data from the secondary machine.
3. I copied the data from the backup DVD onto the secondary machine.
4. I used BC 3.07 on the primary machine to compare data on the two machines. All date / time stamps matched perfectly.
5. I used BC3 on the secondary machine to compare data on the two machines. The date / time stamps for a large subset of the data on the primary system are exactly two hours newer than the date / time stamps for the same data on the secondary system.

Obviously, I can use a rules-based or other file content comparison to get around this. The downside is how much longer it takes. Because it has to deal with the apparent date / time disparity on the secondary machine, running a comparison from there takes over 10 minutes! The rules-based comparison from the primary takes much less time, of course. But I suspect that will change over time if it starts seeing date / time disparities in the file populations, too.

I've rechecked both systems and can't come up with any configuration difference that could account for this.

Does anyone have an idea as to what may be causing this?

[Michael Bulgrien]

I don't use Linux, but it sounds like your machines are configured with different timezones.

http://www.wikihow.com/Change-the-Timezone-in-Linux

[Michael Bulgrien]

Also Note:

On the Comparison tab in Session Settings you can check the option to "Ignore timezone differences".

On the Handling tab in Session Settings you can check the option to "Touch local files when copying to an FTP site".

If either of these options work for you, just be sure to "Also update session settings" so that it become the default behavior on your system.

[SSnow]

I don't use Linux, but it sounds like your machines are configured with different timezones.

http://www.wikihow.com/Change-the-Timezone-in-Linux

Yep, I know it looks like that. But, according to the machines, they're both configured with the same time and time zone settings. (I had that information in an earlier post, but I realize the danged post was long.)

Thank you for your reply. At least it's nice to have confirmation that steps I've taken so far are reasonable.

[SSnow]

Also Note:

On the Comparison tab in Session Settings you can check the option to "Ignore timezone differences".

On the Handling tab in Session Settings you can check the option to "Touch local files when copying to an FTP site".

If either of these options work for you, just be sure to "Also update session settings" so that it become the default behavior on your system.

I'm using the "touch local files" feature. I'm also using binary copying to prevent disparities in file sizes, and I'm using the preserve local date / time feature in the ftp profiles.

I looked at the ignore timezone differences setting, but haven't tried it because the systems are set to the same time zone and time. I did try using UTC in the ftp profiles, and that had no effect upon the issue.

I'm thinking that this may have something to do with the fact that these files resided previously on Windows systems, and that the Windows / ntfs way of storing date and time stamp data could have something to do with this issue.

I do appreciate your suggestions. I'm going to give the ignore timezone setting a try just for giggles when I get home this afternoon.

[Zoë]

Have you tried using FTP instead of SFTP? I agree with Michael that it sounds like a timezone issue, but I couldn't say whether it's a SFTP issue or a filesystem issue.

[Michael Bulgrien]

The date / time stamps for a large subset of the data on the primary system are exactly two hours newer than the date / time stamps for the same data on the secondary system.

When you say "exactly"... do you really mean "exactly" (to a fraction of a second)? What happens when you copy the file back to the original system. Does the date/time match what it was before it was backed up? Or does it reflect the 2 hour difference you saw on the other system?

[SSnow]

Have you tried using FTP instead of SFTP? I agree with Michael that it sounds like a timezone issue, but I couldn't say whether it's a SFTP issue or a filesystem issue.

I installed openssh-client and openssh-server (version 1:4.7p1-8ubuntu1.2) on both systems before I installed BC3. If I go into the BC3 ftp profiles dialog and try to change to one of the other transfer protocols (including plain ftp) I get an error when I try to connect indicating that the service cannot be located on the target system. I would have assumed as much. Would I have to reconfigure openssh-server or add an additional service? (I'm a beginner on this OS and with all of this software, except for BC.)

It's not a time zone issue, and it (probably) isn't a file system issue, per se.

First, both machines do have absolutely the same time zone and date/time settings. The only way I can see that this could be a time zone issue is if the ftp server on one (or both) of these systems are disputing the time zone with the operating system.

Second, not all of the files behave this way -- just some of them. There are plenty of files residing in the same folders with the same approximate creation dates and of the same type that are not affected this way. It's a roughly 50%-50% split, with absolutely no pattern as to which files have the behavior and which don't that I can discern.

I'm beginning to wonder if just creating / modifying the files again (rearchiving the archive type files and just opening and saving the other file types) would solve the issue in some way. It seems that the problem resides with specific files, and not with folders or systems as such.

If I'm missing something obvious, please smack me with the clue-by-four.

[SSnow]

When you say "exactly"... do you really mean "exactly" (to a fraction of a second)? What happens when you copy the file back to the original system. Does the date/time match what it was before it was backed up? Or does it reflect the 2 hour difference you saw on the other system?

I'm not sure how to see the time stamp to a fraction of a second (newcomer to this OS), but, yes, the time stamps are off from each other by exactly two hours, to the second.

I performed the following experiment on a single file in the hope that its results will be useful:

I start with a BC session on System A showing a comparison of System A and System B. No applications (other than BC) open on either system to avoid changes in status or locking of files in the file population. I use sftp as the transfer protocol, strictly set to use binary transfer. The session is set to for rules-based comparison with the skip and override features set on the comparison parameters. There are no files shown as being different.

On System B I start a BC session with the exact same settings, going the opposite way, of course. It also shows no files being different.

I change the comparison method on the System A and B sessions to use only the Quick Tests (standard features employed).

Now the fun begins with a list of the time stamp for the test file as listed in the BC sessions on both systems:

System A: local = 02/22/2002, 11:13:50 AM - remote = 02/22/2002, 11:13:50 AM
System B: local = 02/22/2002, 10:13:50 AM - remote = 02/22/2002, 12:13:50 PM

Now on System B I copy left, meaning I copy the file from System A (12:13:50 PM time stamp) to my local drive on System B. Refresh both systems, now record the time stamps again:

System A: local = 02/22/2002, 11:13:50 AM, remote = 02/22/2002, 01:13:50 PM
System B: local = 02/22/2002, 12:13:50 PM, remote = 02/22/2002, 12:13:50 PM

Isn't that cute? I made System B happy, but System A is sad.

Now I use touch in the session on System A to set the time stamp on its local copy of the file to use the time stamp for the file on System B. Results:

System A: local = 02/22/2002, 01:13:50 PM, remote = 02/22/2002, 01:13:50 PM
System B: local = 02/22/2002, 12:13:50 PM, remote = 02/22/2002, 02:13:50 PM

That's more or less what I would have expected after seeing the file copying behavior. Turning off the automatic touch feature in the File Handling dialog has no effect on the behavior.

The history of these files is that I tried migrating from Windows to Ubuntu a little over a year ago. I was working on files in the same Open Source applications on both platforms (one-at-a-time, mind you) and equilibrating the work between the systems at the end of each work station.

I was, if I remember correctly, using WinSCP on the Windows box and was seeing much the same sort of behavior. At that time I absolutely had to keep one of the systems on Windows, so I finally ditched Linux and went back to Windows.

Now I've left Windows behind, but I'm seeing the same problem again trying to synchronize this file population between system under Ubuntu. I saw absolutely no indication of this sort of trouble between systems (using BC) on the Windows side. Been doing this for years.

I'm all ears now, except that I really do not want to install more services for testing transfer protocols, particularly if they are less secure ones. I am able to work with the file population now using rules-based comparison. It's much more time-consuming, and it irks me that the time stamps are not synchronized, but I guess I can live with it.

Remember that only somewhere around half of the files are doing this. And there's no obvious difference in file sizes, types, or ages that are affected vs. those that aren't affected.

I originally restored these data files from DVD to System A, then copied them over to System B via network connection. I tried removing all of the files from System B and restoring the files from DVD to System B. Looked OK until I ran BC.

I don't remember seeing the time stamp difference until after I updated BC to 3.07, but I was very busy at the time. Who knows, I may have been copying the same unchanged files back-and-forth for a few days.

Does anyone know what's going on?

[Michael Bulgrien]

I'm going to give the ignore timezone setting a try just for giggles when I get home this afternoon.

Just curious... did you give "Ignore Timezone" a try? If so, what were the results?

[SSnow]

Just curious... did you give "Ignore Timezone" a try? If so, what were the results?

I saw no difference when I tried that. I've tried changing one little feature at a time, and nothing has made any difference. The whole thing finally lost its ability to amuse me. I opened BC on System B and just went down the list of affected files on System A, opening and saving them.

I found that I had to be careful about the way I handled this process to make it work. For instance, if I opened a bitmap in GIMP and saved it to the same folder with the same name, that didn't work. I had to change the graphics format (like to PNG). I changed the affected .zip files to .tar.gz. Text files and PDFs I dealt with by saving as some other name. In the end, this stupid workaround worked. At least so far. I switched the comparison basis for all sessions to the normal quick tests and did some file changing and some back-and-forth synchronization. I didn't see any more two hour disparities showing up in file -- EXCEPT in a very few current files down under the .mozilla and .mozilla-thunderbird locations. (I synchronize those to keep my Thunderbird e-mail, Sunbird calendar, and Firefox bookmarks the same on the two notebooks.)

I think there was something "spooky" about these files (headers?), but I think that's not all there is too it. The files themselves have never given a moment's trouble.

I'm just going to wait and watch closely to see if I start seeing the behavior again. If it does, I'm going to totally rethink my system-to-system synchronization strategy. This is just too weird for me to be comfortable trusting this procedure with my data.

[SSnow]

Additional Note

I mustered what little curiosity was left and looked at the four files under the .mozilla-thunderbird folder which were still showing the abberrant behavior. I deleted them, just for gigles. Three .js and one .rdf file. Most (all?) of them are associated with the Webmail extension for Thunderbird. They show the same exact two hour time difference.

And they get created again after they've been deleted and the Webmail extensions have been reinstalled! And they still have the same weird two hour thing going on! And -- get this -- their time stamps are from November, 2007. So, not new data files. I'm wondering if the originals in the software package were created on an ntfs file system.

I'm so confused. But I've got a Bloody Mary in one hand and a bad attitude in the other.

As I said before. I'm going to rest on this. I'm going to wait and watch. And I won't worry about it again unless "regular" files start showing the problem again. In my [alleged] mind, it's even money.

[SSnow]

This is Beyond Bizarre - PLEASE HELP ME

SYNOPSIS:

Why would Beyond Compare and Nautilus show diferrent Modified timestamps on one system, and not on another?

LONG VERSION:

I don't know whether to start a new thread, or to continue my old one on this matter. The 2 hour time differential has reared its ugly head again.

This new occurrence may help more precisely pin down what is happening.

On Saturday, 11/01/2008, I performed clean installations of Ubuntu 8.10 (just released on 10/30/2008) on both of my laptops.

The data had been saved to a separate physical hard drive (single ext3 partition) in the larger of the two laptops, and a second copy of the data was saved to another separate physical hard drive (usb-connected, single ext3 partition).

When the clean installations were finished, they were configured. Both systems were set to the proper, same time zone. I could see them keeping the same time to the second.

The data was brought back into the home directory on what I'll call "LaptopA" with Beyond Compare by copying it from the modular drive.. Data was copied back into the new home directory on "LaptopB" with Beyond Compare by copying it from the usb drive.

Last night (HOURS before the time change) I could run Beyond Compare 3.09 on LaptopA and show that the date/time stamps on the data in the home directory were identical to:
a. the date/time stamps on the data on the modular ext3 drive,
b. the date/time stamps on the data on the usb ext3 drive, and
c. the date/time stamps on the data in the home directory of LaptopB.

I ran Beyond Compare on LaptopB, and it showed the date/time stamps on the data in the two home directories to be identical.

Both laptops were shut down overnight.

This morning I started both laptops. I used LaptopA, getting e-mail, updating my calendar, etc. I went to synchronize my data between LaptopA's home directory and the modular drive. The normal few files were updated. I went to synchronize data between LaptopA's home directory and the usb drive. The normal few files were updated.

I went to syncrhonize data between LaptopA's home directory and LaptopB's home directory. Beyond Compare told me that it was going to update ALL of my files. ALL OF THEM.

I compared system times. Still the same, second to second. Both systems had successfully negotiated the change from "savings" to "standard" time.

I decided to do something drastic. I erased all of the data in LaptopB's home directory and copied ALL DATA from LaptopA to LaptopB over the network. I figured that would settle it.

When the synchronization was complete and there were no file differences showing between LaptopA's home directory and LaptopB's home directory in Beyond Compare, I started Beyond Compare on LaptopB and compared home directories.

It showed this:
a. On all files with modified dates of 11/01/2008 and earlier, the LaptopA version was precisely 2 hours "younger" (later time of day).
b. On all files with modified dates of 11/02/2008, the LaptopB version and LaptopA version had identical date/time stamps.

I disconnected the usb drive from LaptopA and connected it to LaptopB. Nautilus on LaptopB says that the date/time stamps of files from 11/01/2008 and earlier are ONE hour older than the same files in it's own home directory. And that date time stamp is identical to the one shown for the same file by both Nautilus AND Beyond Compare on LaptopA.

CUT TO THE CHASE:

LaptopA
-------------
Nautilus and Beyond Compare both show exactly the same Modified date/time for ALL files, locally and remotely.

LaptopB
-------------
-looking at its own home directory-
For all files with modified date of 11/01/2008 and older, Beyond Compare shows time/date 1 hour later (younger) than what Nautilus shows.
For all files with modified date of 11/02/2008, Beyond Compare shows SAME date and time.
-looking at home directory of LaptopA-
For all files with modified date of 11/01/2008 and older, Beyond Compare shows time/date 1 hour earlier (older) than what Nautilus shows.
For all files with modified date of 11/02/2008, Beyond compare shows SAME time/date as Nautilus

PLEASE HELP ME!

I am discontinuing any use of LaptopB for anything other than target repository until I get this solved.