r/AskProgramming • u/Odd-Heron5704 • 3d ago
Can simple mathematical functions give different results in different CPU architectures?
I'm referring to modern CPUs. To make the question more specific, arm64 vs x86
And if so, how do you fix it?
I asked chatGPT and it gave me this example but I'd like to ask it here (since AI can make mistakes). Can you let me know? Thanks
It says this can give different results due to rounding errors. If so, how to you write code so that you don't have this issue?
#include <iostream>
int main() {
double a = 1.0 + 0x1p-27;
double b = 1.0 - 0x1p-27;
double c = -1.0;
std::cout << (a * b + c) << '\n';
}
15
u/tgm4mop 3d ago
Modern CPUs follow IEEE standards for basic floating point operations, so the same assembly code will give the same results. (With some caveat about rounding modes)
However C++ introduces some complexity, as it does not guarantee IEEE behavior itself, but you can usually control it with compiler flags. In the example you give, the compiler could choose to emit either an add and a multiply, or a single fused multiply-add. These could give different results. So if you want reproducible results across compilers, you will need to have the compiler flags carefully set to ensure the generated assembly is the same across builds.
1
u/Modi57 2d ago
Doesn't c++ guarantee order of operations? I thought you needed to explicitly enable -ffast-math for those optimizations
1
u/AutonomousOrganism 1d ago
C++ does not require exact reproducibility for floating point types.
Afaik fma is used even without -ffast-math, as it is actually more precise than doing mul and add.
13
u/daV1980 3d ago
Other posts here are correct for when CPUs and compilers are running in IEEE 754 strict mode. IEEE 754 strict mode requires the compiler and processor to run exactly the operations you've asked for and the resulting answer must be bitwise the same on any implementation (at the time of the result, any intermediate calculations are allowed to use implementation defined precision, but they are non-observable and so are not required to match).
However, many applications and games run with 'fast math' (on GCC or clang, this is the option -ffast-math). This can be beneficial for performance in many applications for a variety of reasons. One is that many CPUs have an operation to do a "fused multiply add (FMA)," which means that they can execute A * B + C as one instruction. But the result of using FMA is that the results will very frequently not be the same as if you actually multiplied A * B, then added C to the result.
Floating point math doesn't meet three of the four properties of ordinary arithmetic, primarily due to precision but also because of internal implementation details. That is that FP math is not commutative, associative, or distributive.
Changing the order of operations or the grouping of those operations absolutely can change the results.
--
I saw the post about AoE 2, and while I didn't work on AoE 2, I was a programmer on an extremely popular RTS in the early 2000s; if you are a fan of the genre and played in that era you almost definitely played the game I worked on.
RTS games of that era (and maybe now? but I don't work on them nowadays) used a networking architecture called peer to peer. This architecture relies on having clients send commands to each other and executing those commands at some point in the future (that point being decided based on the latency of the players from each other). Rather than there being a single, dedicated server who runs commands and sends back the results, this says "let's just send the commands to each other, and we'll both execute them at the same time and by the inductive property we will agree about the next state of the game (that is "state a + command 1 = state b").
Periodically these games need to verify that they are still in sync, and the fastest way to do this is to simply CRC all of the game state memory and exchange that CRC over the network. But doing a CRC in this way requires that you are bitwise accurate across all state in the game. For integers, this is pretty straightforward.
But because of the fuzziness of floats described above, this can be much harder to achieve across architectures, especially if the game uses fast math. For x86-based processors, even -fast-math is likely to yield identical results (it is in AMD's interests to match what Intel generates and vice versa, and both chips will be driven by the same compiled code). But when you are running on ARM (which is the processor used by modern Apple machines), you are either running in emulation or recompiling. In both cases, ensuring that you will get bit-accurate results in computations becomes more challenging. As a bonus problem, historically (though I haven't looked recently as I don't generally ship things for ARM these days), performance of fp strict mode on ARM was terrible. Like "30-50% slower" terrible, so not really feasible for a game.
Anyways, it's certainly possible, but it's difficult.
2
u/paulstelian97 2d ago
Not commutative? That’s interesting because I can’t think of any examples that don’t involve NaN that break this property…
3
u/Some-Key-6222 2d ago
It really is only because of NaN
2
u/paulstelian97 2d ago
And associativity and distributivity are only barely violated due to differences in rounding errors for different orders of operation right? (With some nasty edge cases, like doing 1.00000001 - 1 or some shit like that)
3
u/Some-Key-6222 2d ago
It's not so much about rounding errors as it is about rounding order and number of rounding operations.
The point is that arbitrarily rounding each part of the calculation doesn't guarantee the same results. Which makes sense, but when writing floating point math people aren't really aware that much more happens under the hood than what they've written.
0
u/paulstelian97 2d ago
Yeah. Thankfully in most cases the differences are minimal and if you know about them you can defend against them (considering equality when the difference is small enough)
1
u/SeriousPlankton2000 1d ago
small + big + smallmay bebig, butsmall + small + bigmay be2 * small + bigdue to rounding.2
u/paulstelian97 1d ago
That’s associativity though, because in the first one it’s small+big as the first operation and on the other one it’s small+small. So your example breaks associativity.
1
u/SeriousPlankton2000 1d ago
I commuted them, too.
1
u/paulstelian97 16h ago
Which didn’t itself contribute to the issue. The associativity is what’s broken, you just showed it in a poor fashion. Commutativity being broken means one operation receives two inputs and changes results when you swap the inputs. Your example shows the following four distinct operations: small+big, (small+big)+small, small+small, and (small+small)+big. None of these are another one with the inputs swapped.
1
u/SeriousPlankton2000 13h ago
Usually equations are evaluated from one side to the other in a fixed order by what the language dictates. On a sheet of paper you can swap them around all the way, but not on a computer doing floating point.
1
u/paulstelian97 13h ago
That’s informal enough to not mention which of the two properties is broken.
Commutativity is broken only if you can find an example of a+b=b+a being false. Someone mentioned it can happen with NaN, but otherwise it doesn’t happen.
Your example can be rephrased in an example of associativity being broken, as (small+small)+big can give a different result from small+(small+big).
1
u/AndrewBorg1126 3d ago
Often the commands can be replayed not only at some time in the future, but at some point in the past, subsequently rewinding and replaying as events are received with a time that has already passed.
10
u/AndrewBorg1126 3d ago
Not if they all correctly implement the same standard.
If there is an inplementation bug, or if they are not designed to fit the same standard, they could behave differently.
5
u/highlevelcomputing14 3d ago
The short answer is yes, floating point can bite you here, but the example you pasted is a bit of a red herring since x86 and ARM both use IEEE 754, the real gotcha is the x87 FPU's 80-bit internal precision on older compilers if you're not careful with compiler flags.
4
u/Mynameismikek 3d ago
Floating point values are usually disallowed in fully deterministic systems because differing order of operations can give rise to errors.
Equivalent hand written assembly should come to the same result, but any non-trivial compiled code can drift.
5
u/Chuu 2d ago
I'm just going to post this at the top level because a bunch of posts are giving an incorrect answer. Even following IEEE-754 strictly you can get different results on different architectures.
The problem is while the IEEE-754 standard is overwhelmingly popular for floating point representation, modern CPUs have expanded the scope of what floating point operators they support which might result in different results on different architectures.
For example, most modern FPUs can do a "fused multiply add" that which computes `a=a+(b*c)` as a single instruction much more efficiently than an add and multiply. This was standardized by IEEE-754 in 2008, and specifically only rounds once.
Which means if your CPU supports it, a=a+(b*c) might be computed with a FMA instruction and be strictly IEEE-754 compliant and produce one result. However for an architecture that does not support FMA, the compiler might compute it as (tmp=(b*c), a=a+tmp) which has *two* rounding instructions -- one after the multiply and one after the add. Which could produce a different result.
Both are IEEE-754 compliant.
3
u/real_kerim 3d ago
Is this related to the AoE2 cross-play post? The explanation in that post was... questionable.
3
u/Odd-Heron5704 3d ago
Yes, but in that post and others everyone starts talking in very generic terms without providing any actual examples, that's why I asked a more specific question. Those posts unfortunately end up in people giving opinions instead of technical answers.
1
u/LaughingIshikawa 2d ago edited 2d ago
u/daV1980 has the correct explanation, FWIW.
In general, programming is filled with these "leaky abstractions" where 99.99% of the time the result doesn't change based on the implementation details, but 0.01% of the time it does. Floating point math in particular is notorious for this, because by its nature it involves lots of rounding, and implementing the rounding differently (rounding at different times, or in different ways) will cause different implementations to "drift" away from each other.
Generally that's not a big deal, because floats store many digits, and you're likely to only use the first few digits, possibly rounding to an integer at some point. But it can be an issue if you're depending on floating point implementations to be exactly the same at all times, such as with AoE2.
This explaina the basics of why you get floating point rounding errors
To (probably inaccurately) simplify it and connect it back to what I was saying in the AOE2 thread) is that when you type something like:
float A = 1.5 float B = 2.0 float D = A + B * B...the underlying implementaion on different chips especially in "fast math" mode, or other kinds of compiler optimizations, might be either to add A + B, and then multiply by B, or multiply B * B, and A * B, then add the results.
Hypothetically those two results "should be" equivalent, because multiplication is communicative - but because floating point operations commonly introduce rounding errors, adding, subtracting, and multiplying stops being communicative, and order matters! ...at least when you care about being bit-wise identical across all implementations.
You might get 6.000000000000004 with one method, and 6.000000000000006 with a different method. (That's NOT accurate to the actual math because I'm not invested enough to find a floating point calculator to figure out what the actual values would be - but you get the idea.)
Generally is 99.99% of programs, that 0.000000000000002 difference doesn't matter because at some point you're going to convert back to integers and round to 6 in either case, or you're only displaying the first 2-3 digits at most anyway, or you're comparing values produced by CPUs that will in all cases compute those floating point operations in the same order, and thus produce the same value.
In the other 0.01% of programs though, it can matter a lot that the two values are not bit-wise identical: for example, this is likely to cause the CRC or "Cyclic Redundancy Check" check-sum values to not match, and if that's part of your system for detecting a desync, that could convince two computers that a desync has occured (which in a very technically sense it has...) and they should throw an error.
I'm probably getting some of the details of this explanation wrong, but is that enough of an "actual code example" for you? 😅😮💨
It isn't a difference in code, since in both cases the actual code is identical - it's a difference in underlying implementaion "leaking" through the abstraction layer that is "supposed to" abstract away all the messy details of how the CPU actually does math. The actual difference is in the CPU, and it will produce a different output, even when the code running on the CPU is exactly the same.
Other methods for calculating values do exist but the thing that's not always discussed (likely because most programmers are intuitively aware of it) is that this comes with tradeoffs - mostly notably in this case time tradeoffs, because maintaining exact precision will likely require you to do more calculations which can snowball into a big problem when you are trying to render game frames at 60 frames a second, and you only have 1/60th of a second to fully calculate the next game state. 👍
"But couldn't the developers reprogram AOE2 to just use fully deterministic math to enable cross-platform compatibility?!?". (after all modern CPUs are much faster now, ect ect ect...)
😮💨😮💨😮💨
Technically yes, but doing so would require them to systematically change every piece of code that currently does floating point math over to the new system, plus fixing the inevitable bugs that will be introduced by changing the code in that many places. (Even experienced programers aren't going to perfectly change the code everytime, plus it's possible that the legacy code was relying on the jankiness of floating point math in some cases, causing even bigger problems...)
...And that's an amount of work many time larger than is worthwhile, just to enable cross-platform compatibility for a game that's 27 years old. Even for a really popular game, there just aren't that many people still playing it in total, and the number of people who are significantly inconvenienced by the lack of cross platform compatibility is a fraction of a fraction of that...
So yeah. 🫤🤷
2
u/khedoros 3d ago
Taking your code above, I compiled it like this, forcing SSE floating-point in one compilation (64-bit internal calculations), and 387 floating-point (80-bit internal calculations) in another.
$ g++ -O0 -mfpmath=sse -msse2 float.cpp -o float-sse
$ g++ -O0 -mfpmath=387 float.cpp -o float-x87
$ ./float-sse
0
$ ./float-x87
-5.55112e-17
I could do a similar experiment on my Raspberry Pi (ARM64 architecture, currently running a 32-bit OS though, I think) if I wanted to, and I suspect it would match the SSE answer.
1
u/Odd-Heron5704 3d ago
Interesting. But why would you do one compilation or the other. Does it make sense in a real life case to be using one or the other and then cause these issues between arm64 and x86? Then your code behaves differently in e.g. Windows vs macOS.
2
u/khedoros 3d ago edited 3d ago
At one time (e.g. the original release of AoE2), it would've been because the game released when SSE technically existed, but it's likely that the development toolchain, developer's computers, and computers of most users of the software would be on older chips that didn't support it. And at the time, it would've been the difference between PCs using an x87 floating point unit and Macs using PowerPC's floating point instructions.
Building it for today, I don't think there's any reason to stick to the x87-based one. And I'd expect the X64 and ARM64 calculations to match, as far as floating point. (edit: At least mostly? Using fast-math compilation options might change things like rounding and order of operations and cause them to behave differently, but I haven't tried it).
1
u/engy1207 1d ago
Well, technically the 387 80bit-variant is "more correct". If you do that calculation with infinite precision that's (1+2-27)×(1-2-27)-1=(1-2-54)-1=-2-54=~-5.55×10-17
But then again it has more digits to calculate with... and the concrete values used are selected specifically for this.
The problem is FP and its representation - as well as order of operations and therefore number of rounding errors. FP is inherently messy, and even in IEEE mode there's multiple number of bits one can use (64bit being "double precision" for some reason). Don't know if this is still IEEE but in AI models they even use FP with only 4 bits (1 sign 2 exponent 1 mantissa).
2
u/Plus-Painter-2004 3d ago
On different architectures it shouldn’t assuming they follow the same standards eg ieee 754 floats. What might give different results is if different compilers are used which might optimise code differently, which in turn can cause certain operations to return different values since rounding errors are affected by order of operations (eg a(b+c) could give a slightly different result to ab + ac)
1
u/Whole-Chest90 3d ago
As others have said here, and what would make sense to me, is that the CPU itself shouldn't give different answers, but different layers (by that I mean compilers, IDEs, platforms) on those cpus might.
1
u/verdant_bloom_drift 3d ago
That specific expression evaluates identically on arm64 and x86 because it only uses add, subtract, multiply which IEEE 754 mandates as correctly rounded. You get divergence from compiler optimizations like FMA contraction or math library differences in functions such as sin and cos. Force strict FP conformance with /fp:strict on MSVC or -ffp-contract=off -fno-fast-math on GCC and Clang to guarantee bitwise reproducibility across architectures.
1
u/MyTinyHappyPlace 3d ago
IEEE floating point is very strictly standardized. You can have differences on C-level when it comes to numeric ranges of integers, though.
I'm referring to modern CPUs.
I know, but here is my favorite line: "Don't divide, Intel inside".
1
1
u/SpiritedInflation835 3d ago
I'm referring to modern CPUs
Well... I can only point out the chipsets in the TI-30 calculators with their famous logarithm bug: http://www.datamath.org/Story/LogarithmBug.htm
1
1
u/Torebbjorn 3d ago
Well yes, since C(++) do not give strict requirements for what an "int" or a "float" is, just that they must be at least of certain sizes and also adhere to some relational rules, you could have different results based on platform.
I would think that modern x86 and arm64 cpus all use the same standards, namely IEEE 754, but I don't know for certain if all do.
1
u/Affectionate-Slice70 3d ago
If you’re not doing anything funky they will be the same. At some level the hardware differs but that is intentionally abstract away from you and standardised as you go up the stack.
1
u/Traveling-Techie 2d ago
This may or may not be useful, but when I worked running benchmarks on parallel supercomputers we ran into the issue that, strictly speaking, floating point arithmetic is not associative. Order of operations matter, and parallel computations controlled by an optimizing compiler can give differing results. The errors usually occur in low order bits that are not reliable in the first place, but users were very attached to the idea of repeatability, and we had to educate them about why different runs under different loads gave slightly different results.
1
1
u/PvtRoom 3d ago
Different CPUs can have different optimisations.
1030 + 10-30 - 1030 = 10-30, but that's way smaller than rounding errors
CPUs do things like optimize code for speed, A+B-C, it could recognise as B (as A=C, fastest and most accurate), or it could do (A+B)-C = 0. Then you have what does the compiler do, does the compiler do fancy stuff to preserve accuracy.
35
u/Avereniect 3d ago edited 3d ago
So long as both C++ implementations adhere to IEEE-754 and follow strict math, that code should deliver identical results on both architectures because the rounding behavior is a well-defined part of spec. Although rounding may lead to inexact results, that is different than saying that the results will vary across compliant implementations.
For an actual example where results may differ, I'd suggest contrasting an x86 target with only x87 80-bit floats supported compared to a more recent x86 implementation that supports IEEE-754 via SSE or later.