LLM non-determinism experiment (run 2026-10-22, gpt-4o-mini, total cost < US$0.01)
Scripts: nondet.py (short answers, temp 1.0 vs 0, + first-token probabilities) and nondet2.py (120-word story, temp 0).
- “Pick a random number between 1 and 10. Reply with only the number.” temp 1.0: 19x “7”, 1x “4” (2 distinct/20). temp 0: 20x “7”.
- First-token probabilities: ‘7’ 98.3%, ‘4’ 1.6%, ‘6’ 0.1%.
- “Give me a name for a coffee shop.” temp 1.0: 13x Brewed Awakening, 3x Brewed Awakenings, 3x Brew Haven, 1x Brews & Bliss (4 distinct). temp 0: 20x Brewed Awakening.
- “Write a 120-word story about a robot learning to cook.” temp 0, 20 runs: 18 distinct texts. Divergences start mid-text (e.g. char 59, char 210) and then the rest differs. Takeaways: (1) “random” from an LLM is not random (98% ‘7’); (2) temperature 0 is near-deterministic for short outputs but NOT for long ones: one tiny early difference (ties/floating point/batching on the provider side) cascades.