How much cost can total store ordering add in cases where multiple CPUs each make repeated accesses to a different byte within the same cache line? Even if the costs of supporting total store ordering would usually be minimal, I would think the worst-case costs could be massive.
It's going to depend on the "coherency domain" that needs total store ordering, and the "cache coherency protocol" used by the CPU. For your question it would depend on the protocol being used (MOSI, MOESI, etc) and your core topology.
For example if your coherency domain only includes the CPU Cores on a given memory bus, then total store ordering is really just the cost of the cache coherency protocol operating (and general cache maintenance). RAM overhead isn't actually needed if the only thing touching RAM are the cpu cores - the coherency protocol will ensure that cores see the correct version of the data that's at the target RAM address (but cached on each cpu core). These have some overhead, but compared to blocking the world it's basically nothing.
Where things start becoming particularly expensive is when you need total store ordering on an uncacheable address because you now are forced to pay the cost of committing the store to main memory and blocking all cores that are interested in that address while the store is being performed. There you're going to be limited by the number memory channels present and the memory controller pressure from other cores.
Caches and their Coherency Protocols exist to alleviate much of these costs. As long as the address that's being contended over is cacheable, the contention cost is a property of the protocol being used. Also, some architectures have instructions to assist with this overhead - x86 has PAUSE which tells the core "hey, we're in a spin loop and if you keep trying to take ownership of that address I just read from you're going to steal it from the core trying to write to it." The "taking ownership" part is what the coherency protocol handles.
A bit off topic, but I like recommending people read (or at least skim a couple chapters of) memory-barriers.txt from the linux documentation if this type of stuff is interesting.
3
u/flatfinger 7d ago
How much cost can total store ordering add in cases where multiple CPUs each make repeated accesses to a different byte within the same cache line? Even if the costs of supporting total store ordering would usually be minimal, I would think the worst-case costs could be massive.