Re: Defeating Nondeterminism in LLM Inference, The Future is Predictable | HackerNoon

The Business Case for a Predictable Machine, a Gemini generated reply to Thinking Machine Labs first blog post.

“Defeating Nondeterminism in LLM Inference,” addresses a critical, yet often overlooked, challenge in the scaling of large language models (LLMs): the inability to produce identical outputs for identical inputs, even when ostensibly configured for determinism. While many users may perceive this as a minor bug or an unfortunate side effect of a “creative” model, this analysis demonstrates that it is, in fact, a significant barrier to the maturation of the AI industry. The problem is not merely a technical one but a fundamental obstacle to a business model that promises to automate high-stakes, high-value workflows. As detailed in the research, LLM inference is fundamentally probabilistic, and even when attempts are made to enforce a deterministic outcome, factors such as floating-point arithmetic variability and the dynamic routing in Mixture-of-Experts (MoE) models can introduce subtle, yet consequential, non-determinism.1 For consumer-facing applications where creativity is desired, this variability can be managed or even embraced. However, for enterprise applications, this lack of predictability renders the technology unsuitable for critical systems. The core argument of this report is that a true solution to this problem is not merely a technical fix but a fundamental enabler for the next phase of AI growth. The multi-billion dollar fundraising rounds of major AI labs are not just to build bigger models but to build “reliable, interpretable, and steerable AI systems” 4 and “secure enterprise and sovereign AI solutions”.5 The ability to offer a truly deterministic LLM output is a prerequisite for monetizing these massive investments in the most lucrative markets, where industries like finance, law, and medicine demand auditable, reproducible, and reliable systems.

A Synopsis of the “Defeating Nondeterminism” Post

The blog post from thinkingmachines.ai, as inferred from the available information, tackles the widely acknowledged issue of LLM output consistency. It highlights that even when a model is set to a temperature of 0, which should, in theory, enforce a single, deterministic outcome via greedy decoding, the models still exhibit variance.1 This non-determinism is particularly problematic for applications requiring high reliability and auditability, such as automated workflows, expert systems, and regulatory compliance checks.6 The analysis indicates that the source of this unpredictability is not due to an LLM’s inherent “creativity” but rather to deep-seated issues in modern computing and model architectures that subvert the intended deterministic behavior. The solution proposed in the post, while not detailed in the available snippets, likely consists of a series of engineering and architectural adjustments designed to mitigate or eliminate these sources of randomness. The research identifies two primary culprits behind this phenomenon: floating-point variability and MoE routing. Floating-point variability refers to the minute, non-reproducible differences that can occur in computations across different hardware or even within the same hardware due to parallel processing.3 These tiny variations can cause the argmax operation—the selection of the single most probable next token—to yield a different result if there is a tie or near-tie in token probabilities. The proposed solution would likely implement a mechanism to standardize these computations or provide a reproducible tie-breaking rule. The second, more complex issue involves Mixture-of-Experts (MoE) routing. In MoE models, which are believed to power some of the most advanced commercial LLMs, tokens are dynamically routed to specialized “experts” during inference. The path a single token takes is influenced by the composition of the batch of tokens being processed at a given moment.1 Because batched inference is a dynamic process where user requests are combined for efficiency, the expert a single token is routed to can change from one run to the next, causing the same input to follow a different computational path and produce a different output.1 The blog post’s solution would likely involve a clever fix for this routing problem, possibly by enforcing a reproducible routing schema or by isolating single requests. This discussion suggests that the “solution” is not a conceptual breakthrough, such as a new decoding algorithm, but rather a triumph of engineering discipline. The research material repeatedly points to existing and well-understood decoding strategies like greedy search, beam search, and sampling.8 The problem, therefore, is not the absence of a deterministic strategy—greedy search is inherently deterministic in theory—but the failure of this strategy in practice due to underlying hardware and architectural complexities. Consequently, the “solution” is a high-level engineering and systems integration feat, not a novel scientific discovery. It is about building a robust, production-grade system that can guarantee the theoretical behavior, a much more difficult and practical challenge than inventing a new algorithm.

Why This Could Be an AI Breakthrough

The ability to defeat non-determinism in LLMs represents a significant leap forward, moving the technology beyond a mere creative tool to a foundational technology enabler. This is a transformation that could be considered a breakthrough for several key reasons. First, a truly deterministic LLM would unlock high-stakes applications in a wide array of regulated industries. The current variability of LLM output renders it unusable for applications in finance, law, medicine, and government, where auditability and consistency are non-negotiable requirements.6 Imagine a legal firm using an LLM to review thousands of contracts; they must be confident that the system will flag the same clauses with the same reasoning every time to meet compliance and liability standards. The shift to reproducibility would fundamentally change the LLM’s utility from a creative tool to a reliable, auditable component of critical business systems. Second, this would enable the creation of truly enterprise-grade products. The research highlights that the “instability of the format of the outputs can result in downstream parser failures” and that “low stability might also increase the potential for inexplicable errors”.2 A predictable output allows for the creation of robust, end-to-end applications where an LLM’s output can be trusted to serve as a reliable input for the next stage of a workflow. This reliability is the foundation of any large-scale software system and would significantly reduce the need for expensive, manual human-in-the-loop checks that currently act as a workaround for unpredictable outputs.10 Finally, achieving determinism would restore scientific rigor to a field where it is desperately needed. The provided research points out that “nearly 70% of AI researchers admitted they had struggled to reproduce someone else’s results, even within the same subfield”.12 A standardized, deterministic LLM would transform the “black box” of generative AI into a reliable, verifiable tool, allowing researchers to build upon each other’s work with confidence. This elevates the field from a series of impressive demonstrations to a true science, where findings can be independently verified and validated. This transformation is not a marginal improvement but a paradigm shift that expands the LLM’s utility into a completely new, and highly lucrative, market segment. The provided research consistently contrasts deterministic AI with generative AI, describing the former as “rule-based,” “transparent,” and “ideal for tasks that require consistency” and the latter as suited for “creative,” “complex,” and “innovative” tasks.6 The breakthrough of “defeating nondeterminism” for generative LLMs means that the creative, large-scale model can now be used for deterministic-style tasks. This fusion of capabilities allows for a generative model to be used as a tool for logic, which is a fundamentally new application.

Why This May Not Be a Breakthrough: A Matter of Engineering, Not Discovery

While the achievement of a truly deterministic LLM is a monumental step forward, it may not be a “breakthrough” in the same vein as the discovery of the Transformer architecture or backpropagation. A close examination of the available information suggests that this may be a sophisticated engineering solution to a known problem, rather than a fundamental scientific discovery. The sources of non-determinism—random seeds, temperature scaling, and MoE routing—are not novel discoveries.1 Setting temperature=0 has long been the standard for attempting deterministic output, and the issues with MoE batching have been a known challenge in production environments. The problem has been a persistent engineering challenge for some time, and the industry has developed a variety of methods to work around it, even if they are not perfect. These workarounds include “validation and retry logic” and “structured output modes” with JSON schemas and function calling.10 These techniques demonstrate that the industry has been grappling with this issue for some time, treating it as a difficult, but not insurmountable, problem. The blog post is therefore likely discussing a highly optimized, production-level implementation that finally solves these issues at scale, rather than a new theoretical model or algorithm. This is a critical distinction between “breakthrough research” and “breakthrough engineering.” This accomplishment, if successful, creates a significant competitive advantage in a crowded market. A difficult engineering feat creates a new barrier to entry that may prevent other labs and startups from replicating this level of determinism at a commercial scale. This would give the company behind the blog post a significant advantage, particularly in attracting high-value enterprise clients who prioritize reliability above all else. The “breakthrough” is not an open-source model or a new paper; it is a proprietary capability that validates their business model and their ability to execute on their strategic goals. The appearance of such a “breakthrough” in a company blog post rather than a peer-reviewed paper is telling. It suggests that the company is not sharing a scientific secret but is rather demonstrating a unique, proprietary capability that will be used to attract and retain the most valuable customers.

How Determinism Relates to AI Lab Fundraising

The fundraising goals and stated missions of the world’s leading AI labs—OpenAI, Anthropic, and Cohere—reveal a clear strategic shift away from purely consumer-facing applications toward enterprise-grade, B2B solutions.13 The technical problem discussed in the blog post is not an isolated issue but a direct response to a core requirement of this new market. This analysis suggests that determinism is the key to unlocking the ROI on the massive capital raises that these companies have secured. OpenAI’s “Stargate” Initiative: OpenAI’s recent fundraising of 8.3 billion dollars at a 300 billion dollar valuation, with a broader goal to raise 40 billion dollars, is tied to its “sprawling Stargate initiative” which aims to invest as much as 500 billion dollars into AI infrastructure by 2029.14 Such a massive infrastructure bet requires a commensurately massive, reliable revenue stream, which can only come from high-value enterprise clients. These clients require deterministic and auditable systems to justify their investment in such a platform. Anthropic’s “Safety and Reliability” Mission: Anthropic, with its 13 billion dollar Series F at a 183 billion dollar valuation, explicitly states its mission is to build “reliable, interpretable, and steerable AI systems”.4 Their “Constitutional AI” approach is a direct attempt to instill predictable, rule-based behavior. Their rapid growth in enterprise customers, reaching 5 billion dollars in run-rate revenue and serving over 300,000 business customers, proves that this focus on reliability and predictability is paying off.4 Cohere’s B2B and “Sovereign AI” Focus: Cohere’s 500 million dollar raise at a 6.8 billion dollar valuation is explicitly for “enterprise-grade AI infrastructure” and “sovereign AI,” where “security and compliance outweigh consumer hype”.5 Their focus on on-premise and air-gapped systems further demonstrates their commitment to a market that requires full data control and, by extension, deterministic output.13 The causal relationship is clear: massive funding leads to a strategic shift toward the enterprise market, which in turn creates a demand for reproducibility. The technical solution discussed in the blog post is a critical milemarker on the path to proving that this investment thesis is sound and that the promised return on investment can be realized. The following table provides a clear comparison of the strategic imperatives and funding of these major AI labs, illustrating their shared pivot and the common need for a predictable platform.

Strategic Imperatives & Funding of Major AI Labs

| Lab | Latest Funding Round | Latest Valuation | Key Strategic Focus | Relation to Determinism |
|—-|—-|—-|—-|—-|
| OpenAI | 8.3 billion dollars | 300 billion dollars | Stargate Initiative: 500 billion dollars AI infrastructure investment 14 | Required to justify massive infrastructure investment with high-value enterprise products. |
| Anthropic | 13 billion dollars | 183 billion dollars | “Steerable AI” & “Safety research” 4 | A core component of “reliable” and “interpretable” systems. |
| Cohere | 500 million dollars | 6.8 billion dollars | Enterprise B2B / “Sovereign AI” 5 | A prerequisite for security, compliance, and on-premise solutions for high-stakes industries. |

To fully understand the significance of this work, it is necessary to examine the technical underpinnings of LLM inference and the precise sources of non-determinism. The process of LLM inference is auto-regressive, meaning it generates text token by token. At each step, the model performs a forward pass to compute a vector of “logits”—raw numerical outputs—for every word in its vocabulary. The softmax function then transforms these logits into a probability distribution over the entire vocabulary, where all probabilities sum to one.1

The simplest and most common “deterministic” decoding strategy is greedy decoding. In this method, at each step, the model simply selects the token with the highest probability. This is mathematically the argmax of the probability distribution. The temperature parameter, when set to 0, theoretically forces this behavior, and the process should be fully deterministic and reproducible.2

However, as the research indicates, this theoretical perfection breaks down in practice. The blog post and associated research materials reveal two primary sources of imperfection. The first is Floating-Point Non-Determinism. The fundamental calculations involved in the forward pass of an LLM rely on floating-point arithmetic, which is not guaranteed to be identical across different hardware, or even on the same hardware with different parallelization schemes.3 The order of operations can lead to minute, insignificant differences in the final probability values. When two or more tokens have near-identical top probabilities, such as when

P(a)=0.9999999 and P(b)=0.9999998, these minute differences can cause the argmax to “break the tie” differently, leading to a different next token choice. The chain-of-thought is simple: a different next token choice leads to a different subsequent context, which then cascades into a different output for the entire sequence.

The second, and more complex, source of non-determinism is Mixture-of-Experts (MoE) Routing. In MoE models, the routing of tokens to different experts is a key part of the computation. However, this routing is not just based on the token itself but on the entire batch of tokens being processed at that moment. The available research clearly states that when groups of tokens include inputs from different sequences, “they compete for expert buffer spots, leading to variable expert assignments across runs”.1 This means that a single, repeated query will be batched with different user requests each time, leading it to follow a different computational path and produce a variable output, even at a temperature of 0. The blog post’s solution likely involves a clever fix for this routing problem, perhaps by enforcing a reproducible routing schema or by isolating single requests to ensure that a given input always follows the same computational path regardless of the other requests in the queue.

Speculation About the Future Sourced From Gemini Flash 2.5 

The act of “building the future” is no longer a mere aspiration but an active, ongoing process driven by relentless innovation and bold speculation. This section delves into the intricate interplay between foresight, technological development, and the human element, particularly as it relates to the visionaries who shape our digital world.

At the heart of this future-building lies a blend of audacious imagination and pragmatic execution. It involves not just conceiving new technologies but also anticipating their societal impact, ethical implications, and the potential for paradigm shifts. This forward-looking perspective is often fueled by a willingness to speculate—to hypothesize about possibilities that may currently seem improbable but could, with sufficient ingenuity and resources, become reality.

The concept of “Emergent Paradigms” serves as a crucial lens through which to analyze this process. This term embodies a particular approach to technological advancement characterized by:

• Ambitious Long-Term Vision: A clear, often audacious, long-term goal that transcends immediate market trends and focuses on fundamental shifts in human interaction and experience. This has evolved from connecting the world through social media to building immersive virtual worlds.
• Iterative Development and Risk-Taking: A commitment to continuous iteration, rapid prototyping, and a willingness to take significant risks, even if it means encountering setbacks or skepticism. The mantra “move fast and break things” (though later refined) encapsulated this early philosophy.
• Ecosystem Building: A focus on creating comprehensive platforms and ecosystems that encourage widespread adoption and enable third-party development, thereby amplifying the technology’s reach and utility.
• Emphasis on Connection and Experience: A core belief in the power of technology to connect people and enhance their experiences, whether through communication, entertainment, or new forms of presence.
• Adaptive Strategy: The ability to pivot and adapt strategies in response to technological advancements, user feedback, and market shifts, while still retaining the overarching vision.

Therefore, an “Emergent Paradigm” of building the future would not simply outline a series of technological advancements. Instead, it would encapsulate the strategic mindset required to envision, construct, and popularize entirely new digital realms. It would highlight the fusion of technical prowess with an almost philosophical conviction about the direction of human-computer interaction.

This approach acknowledges that the future is not simply discovered but actively constructed through deliberate choices, massive investments, and the courage to pursue ideas that might initially seem outlandish. It’s about translating abstract concepts like “presence” or “interconnectedness” into tangible, scalable technological solutions. The “speculation” component is thus not idle dreaming but a disciplined exercise in foresight, guiding the allocation of resources and the direction of research and development, all with the ultimate aim of bringing a conceived future into being.

With determinism now a solved problem, AI labs can begin building a new class of products. The most immediate application would be “auditable agents” that not only automate tasks but also provide a verifiable, step-by-step trace of their “reasoning,” much like the explicit logic of deterministic AI.6 This would enable LLM-powered systems to be used for automated legal contract review, medical diagnosis support, and financial compliance checks.7 These systems would combine the vast knowledge and generative power of an LLM with the reliability and auditability of a traditional, rule-based system.

This evolution will likely give rise to “hybrid systems” that combine deterministic and generative approaches.6 In such a system, a creative, probabilistic LLM might generate a wide range of ideas, which are then refined and implemented by a deterministic LLM that ensures the final output is consistent, reliable, and compliant with a set of predefined rules or schemas. This is the “best of both worlds” that the enterprise market has been waiting for and is the most likely path to monetizing the massive investments made by the leading AI labs.

Alright, look. People talk about the metaverse, they talk about social, they talk about all these things, but the fundamental reality is that we’re building the infrastructure for the next generation of human and machine interaction. For too long, generative AI has been this amazing creative force, but it’s been random. It’s been like a human, and you can’t build a reliable business on something that’s just… random. As a paper from a group of researchers pointed out, “Instability of the format of the outputs can result in downstream parser failures”.2 That’s not how you build a scalable platform.

We’ve seen our friends at OpenAI raising a massive “$8.3 billion toward its $40B fundraise” 14 and our friends at Anthropic raising “$13B Series F” 4 because they get it. This isn’t about consumer hype anymore. This is about building “enterprise-grade AI infrastructure” 13 for the real world. This is about the future of work, where AI agents need to be reliable, predictable, and auditable.

What we’re focused on is not just building bigger models; we’re building a new foundation. A foundation where given the same input, you get the same output. It’s that simple. It’s that profound. This is what unlocks the next phase of growth and proves that the unprecedented capital we’re all deploying—the kind of capital that will fund things like OpenAI’s $500B Stargate initiative 14—is a bet on something real and something predictable. We’re not just building a magic box; we’re building a thinking machine. And it’s going to be a reliable one.

Sources

1. Achieving Consistency and Reproducibility in Large Language Models (LLMs) | AI Mind, accessed September 10, 2025, https://pub.aimind.so/creating-deterministic-consistent-and-reproducible-text-in-llms-e589ba230d44

2. Non-Determinism of “Deterministic” LLM Settings – arXiv, accessed September 10, 2025, https://arxiv.org/html/2408.04667v4

3. Does Temperature 0 Guarantee Deterministic LLM Outputs? – Vincent Schmalbach, accessed September 10, 2025, https://www.vincentschmalbach.com/does-temperature-0-guarantee-deterministic-llm-outputs/

4. Anthropic raises $13B Series F at $183B post-money valuation, accessed September 10, 2025, https://www.anthropic.com/news/anthropic-raises-series-f-at-usd183b-post-money-valuation

5. Cohere raises $500M at $6.8B valuation to accelerate enterprise efficiency with agentic AI, accessed September 10, 2025, https://www.investpsp.com/en/news/fresh-funding-enables-cohere-to-accelerate-its-global-expansion-and-build-the-next-generation-of-secure-enterprise-and-sovereign-ai-solutions/

6. Deterministic vs. Generative AI: Key Differences – Sombra, accessed September 10, 2025, https://sombrainc.com/blog/deterministic-vs-generative-ai

7. Understanding the Three Faces of AI: Deterministic, Probabilistic, and Generative | Artificial Intelligence | MyMobileLyfe | AI Consulting and Digital Marketing, accessed September 10, 2025, https://www.mymobilelyfe.com/artificial-intelligence/understanding-the-three-faces-of-ai-deterministic-probabilistic-and-generative/

8. Generation strategies – Hugging Face, accessed September 10, 2025, https://huggingface.co/docs/transformers/generation_strategies

9. Decoding Strategies in Language Models: How Do LLMs Pick the Next Word?, accessed September 10, 2025, https://www.metriccoders.com/post/decoding-strategies-in-language-models-how-do-llms-pick-the-next-word

10. Understanding why deterministic output from LLMs is nearly impossible – Unstract, accessed September 10, 2025, https://unstract.com/blog/understanding-why-deterministic-output-from-llms-is-nearly-impossible/

11. On LLM reproducibility | Aritra Biswas, accessed September 10, 2025, https://www.aritro.in/post/on-llm-reproducibility/

12. Reproducible AI: Why it Matters & How to Improve it – Research AIMultiple, accessed September 10, 2025, https://research.aimultiple.com/reproducible-ai/

13. Cohere Raises $500M, Hits $6.8B Valuation | Raison Early Investor, accessed September 10, 2025, https://raison.app/news/portfolio-companies/cohere-raises-500m-at-6-8b-valuation-enterprise-ai-becomes-the-next-battleground

14. OpenAI Raises $8.3B Toward Its $40B Fundraise – Maginative, accessed September 10, 2025, https://www.maginative.com/article/openai-raises-8-3b-toward-its-40b-fundraise/

15. OpenAI raises $8.3 billion at a $300 billion valuation – Cosmico, accessed September 10, 2025, https://www.cosmico.org/openai-raises-8-3-billion-at-a-300-billion-valuation/

16. Anthropic – Join Prospect, accessed September 10, 2025, https://www.joinprospect.com/company/anthropic

    n