Numeric tolerance

[Frank Phillips]

This has been asked before but the answers were pretty generic and didn't seem to apply.

I have two numeric columns that have rounding issues.

Right side value = .505
Left side value = .51

In this particular case, the file I'm comparing only contains 2 decimals, the new code output has three. Since it's obviously a rounding issue I should be able to use .05 and it should mark it unimportant but it's not.

I've tried several numeric tolerances to get this to work, and I know I've been able to have it work in the past, but for some reason even

Any help on what value i should use is greatly appreciated.

Thanks

Frank

[Aaron]

Hello,

Are you currently using the Text Compare or Data Compare session type for your data?

The Data Compare can ignore numeric differences if the column is detected as numeric and only contains the number. At this point, you can go to the Session menu -> Session Settings, Columns tab, edit that specific column, uncheck Use Default, and add a numeric tolerance.

If the column is not numeric, then it would need to be formatted to be numeric first.

If you are in the Text Compare, we do not support numeric tolerance. You can, however, define a grammar element that matches the text values that you are trying to compare, and mark them as unimportant. This does not translate well to a numeric tolerance, since that isn't easy to define as a regular expression or text, but if there's a specific number or string you'd like to ignore you could:
http://www.scootersoftware.com/suppo..._unimportantv3

A full screen screenshot could help if you still have any questions. Please post it here, or email us at [email protected] (and if you email, please include a link back to this forum thread for our reference). Thanks.