AS3 core maths problem

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

hutn.jimbo - 30 Oct 2008 18:53 GMT

Hi. If you run the code at the bottom in AS3 you get this:
0.1
0.2
0.30000000000000004
0.4
0.5
0.6
0.7
0.7999999999999999
0.8999999999999999
0.9999999999999999

if you run it in AS2 you get

0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1

the as2 version is what you would expect. Anybody know why this is? :grin;

var myNum:Number = 0;

for (var i:Number = 0; i < 10; i++)
{
myNum += 0.1
trace(myNum)
}

Reply to this Message

Rothrock - 30 Oct 2008 22:59 GMT

The problem is the representation of decimal numbers in binary format. All
micro computer systems have this (or slight variations on this) problem.

In earlier versions of AS2 you would also have that same problem that you are
seeing in AS3. Somewhere along the way they made the player aware of the
problem and it would compensate for it.

Then (and my understanding is a bit fuzzy here and I might get this a bit
wrong, but the sentiment is right) when AS3 came along they were trying to be
closer to the ECMA standard. And that standard says that the player should
correct for that issue and the end user needs to incorporate their own code
fixes.

More or less something like that.

Reply to this Message

Rothrock - 30 Oct 2008 23:00 GMT

The last sentence of the penultimate paragraph should be:

And that standard says that the player [b]shouldn't[/b] correct for that issue and the end user needs to incorporate their own code fixes.

Reply to this Message

hutn.jimbo - 31 Oct 2008 10:27 GMT

Thanks for the reply Rothrock, but that does seem strange. If you add 0.1 to
something, it should go up by that. Or is my maths really bad?

I don't really know how to get the output I'm after, as sometimes you would
need to round down and other up. Does anybody know what the code should be to
fix this?

Thanks

Reply to this Message

David Stiller - 31 Oct 2008 14:38 GMT

hutn.jimbo,

> Thanks for the reply Rothrock, but that does
> seem strange. If you add 0.1 to something, it
> should go up by that.

In a theoretical sense, of course, you're right. The problem isn't with
your arithmetic ability, but rather with the basic problem of how computers
represent decimal values. I wish I could explain it better than that (it
definitely surprised me when I first saw it), but you do get used to it.

The same output occurs in JavaScript:

<script>
var counter = 0;
for (var i=0; i<10; i++) {
document.write((counter += 0.1) + "<br />");
}
</script>

> I don't really know how to get the output I'm after,
> as sometimes you would need to round down and
> other up.

I can think of two ways in this scenario. Either multiply your value by
10, round, then divide it by 10 (or whatever decimal place makes sense) ...

var myNum:Number = 0;
for (var i:Number = 0; i < 10; i++) {
myNum *= 10;
myNum += 1;
myNum /= 10;
trace(myNum);
}

... which is your only choice (I think) with ActionScript 2.0. In
JavaScript, you could use the Number.toFixed() method:

<script>
var counter = 0;
for (var i=0; i<10; i++) {
document.write((counter += 0.1).toFixed(1) + "<br />");
}
</script>

David Stiller
Co-author, Foundation Flash CS4 for Designers
http://tinyurl.com/5j55cv
"Luck is the residue of good design."

Reply to this Message

kglad - 31 Oct 2008 15:44 GMT

the easiest way, i've found, to explain this to people that don't necessarily
have a math background is to say, "the answers you're getting from flash are
correct to within flash'es limits of accuracy."

ie,

0.7999999999999999 is the same as 0.8 (to flash).

Reply to this Message

hutn.jimbo - 31 Oct 2008 16:32 GMT

Ah that works perfectly. Thanks you very much! Oh and Darren Richardson
says"Hi".

If anybody else has this problem, make sure you add the Math.round() else you
will keep getting funky numbers!

var myNum:Number = 0;
for (var i:Number = 0; i < 20; i++)
{
    myNum *= 100
    myNum = Math.round(myNum + 1);
    myNum /= 100;
    trace(myNum);
}

Reply to this Message

David Stiller - 31 Oct 2008 17:13 GMT

hutn.jimbo,

> Ah that works perfectly. Thanks you very much!
> Oh and Darren Richardson says"Hi".

Howdy, mate! Good to hear from Darren. :) (For fans of trivia, or
out-of-the-way factoids, check out the author names on this link:
http://tinyurl.com/2s28a5).

> If anybody else has this problem, make sure you
> add the Math.round() else you will keep getting
> funky numbers!

Ah, yes! I omitted that accidentally. Glad you caught it!

David Stiller
Contributor, How to Cheat in Adobe Flash CS3
http://tinyurl.com/2cp6na
"Luck is the residue of good design."

Reply to this Message

Webmaster Forum / Flash / General Flash Topics / October 2008

AS3 core maths problem