Login | Register   
RSS Feed
Download our iPhone app
Browse DevX
Sign up for e-mail newsletters from DevX

Tip of the Day
Language: Visual Basic
Expertise: Beginner
May 11, 1999

Compare Floating Point Values Using the Round Function

When you have to compare the results of floating point expressions, you can't rely on the "=" operator due to the finite precision of Single or Double variables. To see this concept demonstrated, use this code:
Dim i As Integer, d As Double
For i = 1 To 10
	d = d + CDbl(0.1)
MsgBox (d = 1)	' displays "False"
To more easily compare two floating numbers, use the new VB6 Round function, which rounds a number to the desired number of decimal digits. For example, you can rewrite the previous test like this:
' the difference is less than 1E-12
MsgBox (Round(d, 12) = 1)  ' displays "True"
You can also use this method to check whether A and B variables contain values that match up to their 12th decimal digit:
If Round(A - B, 12) = 0 Then Print "Equal"
Francesco Balena
Comment and Contribute






(Maximum characters: 1200). You have 1200 characters left.



Thanks for your registration, follow us on our social networks to keep up-to-date