Float Divergence

IEEE 754 does not guarantee bit-identical results across platforms, and NVIDIA’s own CUDA documentation says so. The records below show what that looks like in practice.

Run the Divergence Script Download the script GPU harness →

$python3 prove_divergence.py

The Proof

The Record

Two findings, both reproducible. The GPU finding reproduces from the GPU harness repo; the script finding runs in under a minute from the float-divergence-script repo.

GPU FINDING: path-dependent and scheduling-dependent divergence

The same 1,000,000-element float32 corpus, run on a local NVIDIA RTX 3050 and two Google Colab Tesla T4 instances. Source code and compiler flags were identical, compiled independently on each machine.

Kernel	Compiler flags	Tesla T4 (sm_75)	RTX 3050 (sm_86)
fma_default	-O2 -DUSE_FMA	5b4f9db9…	5b4f9db9…
fma_no_fma	-O2 --fmad=false	80d7301c…	80d7301c…
fast_on (fast `__expf`)	-O2 -DUSE_FAST	743d2d5f…	743d2d5f…
fast_off (accurate `expf`)	-O2	cbe5cea0…	cbe5cea0…
reduction_tree	-DTREE_REDUCTION	37d5c6b3…	37d5c6b3…
reduction_atomic	-DATOMIC_REDUCTION	10 unique hashes in 10 runs	9 unique hashes in 10 runs

The deterministic kernels produce bit-identical output on both GPU architectures. Changing a compiler flag on the same GPU changes the output. Atomic reduction is non-deterministic from run to run at this size, while tree reduction was deterministic in every current run. The fast and accurate rows compare __expf against expf, so they are not an isolated flush-to-zero toggle. These are expected, documented behaviors of compiler flags and thread scheduling. The point is that output depends on the build and the schedule, not on a defect in either GPU.

Full manifests and hash logs are in the GPU harness repo. A CPU result previously published from this harness was retracted on November 2, 2026 after a fault in the harness was found. See the correction notice in the repository.

SCRIPT FINDING: one Python file, five environments, three answers

Environment	CPU	Python	Sorted-sum	Hex view	Result
Prime (WSL2)	Intel i5-12450HX	3.12.3	2190.0	00e00845	B
OnlineGDB	Intel Xeon @ 2.80GHz	3.13.7	2190.0	00e00845	B
Colab CPU	Intel Xeon @ 2.00GHz	3.13.16	2190.9140625	a0ee0845	A
Colab CPU	Intel Xeon @ 2.20GHz	3.13.16	2191.25390625	10f40845	C
Colab CPU	AMD EPYC 7B12	3.13.16	2191.25390625	10f40845	C

Same script, five environments, three sums. An Intel Xeon and an AMD EPYC agree; two Intel Xeons do not. The CPU vendor does not predict the result.

Spread · one sum, three landing pointsΔ 1.25390625 between lowest and highest

B · 2190.0Prime · OnlineGDB

A · 2190.9140625Colab Xeon 2.00

C · 2191.25390625Colab Xeon 2.20 · EPYC

Run it yourself

One Python file. Run it on whatever hardware you can get your hands on, then compare your hex against the record. Do not take the table above on trust.

prove_divergence.py

Result Aa0ee0845

Colab Xeon @ 2.00GHz. Sum 2190.9140625. Seen once.

Result B00e00845

i5-12450HX under WSL2, and OnlineGDB’s Xeon @ 2.80GHz. Sum 2190.0.

Result C10f40845

Colab Xeon @ 2.20GHz and AMD EPYC 7B12. Sum 2191.25390625.

Read the table the hard way. Result B appears on an Intel laptop part and on a hosted Xeon. Result C appears on an Intel Xeon and an AMD EPYC. Result A is an Intel Xeon that agrees with neither. Vendor does not predict the outcome. One script cannot be assumed to give one answer across your fleet.

Open the repo Download prove_divergence.py

The Divergence Baseline

Everything else on this page is measured against this one number.

The Test

prove_divergence.py prints the host CPU model and Python version, runs a sorting-then-summing experiment, and reports the sorted-sum with its hex view.

The Result

Five environments produced three different sums, spread across 1.25390625. An Intel Xeon and an AMD EPYC agree; two Intel Xeons disagree. The vendor on the chip does not predict the answer.

Not Claimed

The script does not isolate the cause. CPU instruction set, library build and interpreter version are all uncontrolled across these runs (Python 3.12.3, 3.13.7 and 3.13.16 appear). Results A and C share Python 3.13.16 and still differ, so the interpreter alone does not explain the spread. The script prints the Python version so you can compare like with like. Run it on your own fleet; that is the test that matters.

Reproduce it

prove_divergence.py · sample output

# 1. Get the script (or download it from this page)
git clone https://github.com/SwapnopamMitra/float-divergence-script.git
cd float-divergence-script

# 2. Run it
python3 prove_divergence.py

============================================================
HARDWARE & MEMORY TELEMETRY
============================================================
CPU Model : AMD EPYC 7B12
Python    : 3.13.16
============================================================

Running Architecture-Sensitive Sorting Divergence Experiment...
Sorted-Sum Result (Sum)   : 2191.25390625
Sorted-Sum Hex View       : 10f40845
============================================================

Test Reports

Primary source documents. Not marketing copy.

Divergence evidence

Source	Scope	Link
float-divergence-script	Sorted-sum experiment, 5 environments, 3 distinct results. Runs anywhere Python runs.	Repo · Script
prove-model-drift	Same PyTorch script, same seed, 4 environments: identical final loss, 4 different weights hashes	Repo · Script · Page
gpu-float32-divergence	GPU harness: FMA, fast-math and atomic-reduction divergence on Tesla T4 and RTX 3050, 1,000,000-element corpus	Repo

Ready to test it against your pipeline?

Request Pilot Access SPCMP Model Drift