devxlogo

Compare Floating Point Values Using the Round Function

Compare Floating Point Values Using the Round Function

When you have to compare the results of floating point expressions, you can’t rely on the “=” operator due to the finite precision of Single or Double variables. To see this concept demonstrated, use this code:

 Dim i As Integer, d As DoubleFor i = 1 To 10	d = d + CDbl(0.1)NextMsgBox (d = 1)	' displays "False"

To more easily compare two floating numbers, use the new VB6 Round function, which rounds a number to the desired number of decimal digits. For example, you can rewrite the previous test like this:

 ' the difference is less than 1E-12MsgBox (Round(d, 12) = 1)  ' displays "True"

You can also use this method to check whether A and B variables contain values that match up to their 12th decimal digit:

 If Round(A - B, 12) = 0 Then Print "Equal"

devx-admin

Share the Post: