--- title: Untitled tags: - memory-as-a-service - agentic-ai - ai-nd-evals-framing - ai-adoption - code-agents created: '2026-05-24T11:11:10.402948Z' updated: '2026-05-24T11:11:18.595575Z' status: review type: note deprecated: false summary: Eleven vendors — three model providers (Anthropic, OpenAI, Google), seven evaluation tooling vendors (LangChain, Brai... --- # How Major AI Builders and Tooling Vendors Frame Non-Determinism, Reliability, and Production Readiness (2025–2026) ## Executive Summary Eleven vendors — three model providers (Anthropic, OpenAI, Google), seven evaluation tooling vendors (LangChain, Braintrust, Langfuse, Arize Phoenix, DeepEval, Weave, LangSmith), and one observability outlier (AgentOps) — have converged on the same operational answer to the question of how to ship LLM-based systems to production in 2025–2026. The answer is single-run LLM-as-judge scoring, deployed as an offline pre-deployment gate and an online production monitor, with a continuous score thresholded against a pass/fail bar. None of the eleven treats *bitwise output-level determinism* as the primary trust proxy, but at least three use *structural or parametric determinism* as a load-bearing component of their trust architecture: OpenAI ships the `seed` parameter and `system_fingerprint` response field as a "best effort" reproducibility primitive at the API surface; DeepEval ships the DAGMetric as a "deterministic decision tree" for per-output evaluation; Google prescribes "deterministic infrastructure around non-deterministic LLMs" as the surrounding system pattern. All eleven treat distributional quality — measured through systematic evaluation — as the de facto trust proxy at the output layer. The convergence is real; the gradient on determinism is finer than a universal denial. What is not real is the convergence on what that framework can deliver. Underneath the surface agreement lies a field-wide gap between what vendors know about their own methodology's limits and what their primary product documentation tells the practitioner. A stochastic judge evaluating a stochastic evaluand in a single run does not produce a confidence interval; it produces an unknown confidence interval that the field reports as a known fact. Production inference at temperature=0 still produces 80 unique completions per 1,000 identical requests because batch-size non-invariance drives endpoint non-determinism [[defeating-nondeterminism-in-llm-inference-thinking-machines-lab]]. Atil and colleagues measure "accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%" under "deterministic" settings [[non-determinism-of-deterministic-llm-settings-2]]. Schroeder and Wood-Doughty measure McDonald's omega (a reliability coefficient quantifying consistency of judge ratings across replications, where values above 0.8 indicate high consistency and below 0.6 are flagged "questionable") for Meta-Llama-3-8B-Instruct on MT-Bench at T=0.25 of 0.421 — below the 0.6 threshold and far below the 0.8 standard [[can-you-trust-llm-judgments-reliability-of-llm-as-a-judge-4]]. Two layers — evaluand variance and judge variance — compound in every single-shot verdict the industry reports. This report makes three claims. First, the field has access to the critique and to two sophisticated published solutions: Anthropic's Central Limit Theorem (CLT) methodology for benchmark-scale comparison [[adding-error-bars-to-evals-a-statistical-approach-to-language-model-evaluations]] and DeepEval's DAGMetric structural alternative for per-output CI gating [[dag-deep-acyclic-graph-deepeval-by-confident-ai-the-llm-evaluation-framework]]. Eighteen months after the CLT paper's publication, no competitor has adopted the methodology at the platform layer. Second, the regulatory frame practitioners commonly anticipate ("regulators will demand determinism we cannot deliver") is the wrong frame. EU AI Act Annex IV, FDA Software-as-a-Medical-Device guidance, and IMDRF Good Machine Learning Practice all use distributional and statistical framing, not bitwise determinism. The August 2026 EU enforcement date is fifteen months out from this writing; the gap is statistical rigor, not determinism. Third, even if the field repaired its single-run methodology tomorrow, it would still be measuring capability rather than reliability — Khanal and colleagues demonstrate across 23,392 episodes that "capability and reliability rankings diverge substantially, with multi-rank inversions at long horizons" [[260329231-beyond-pass1-a-reliability-science-framework-for-long-horizon-llm-agen]]. Pass@1 (the probability of success on a single attempt — the dominant capability metric) on short tasks is structurally blind to long-horizon production reliability. The taxonomy this report uses across every vendor section: *knows-and-tells* vendors (primary documentation matches the field's best evidence), *knows-and-hides* vendors (honest in secondary checklists and advanced cookbooks, optimistic in primary product surfaces), and *observability-only / scoped-out* vendors (opted out of the eval-gate discourse, deliberately scoped to forensic post-hoc trust). Anthropic, Arize Phoenix, Google Vertex AI (architecturally), and DeepEval (structurally) sit in the first category. OpenAI, LangChain, Braintrust, Langfuse on the eval side, Weave on enforcement, and LangSmith sit in the second. AgentOps occupies the third alone. The pattern is structural and commercial, not editorial — adoption-friendly surfaces win because they are the surfaces practitioners encounter first. | Vendor | Taxonomy bucket | ND framing | Reliability framing | Repeatable-behavior claim | |---|---|---|---|---| | Anthropic | knows-and-tells | "even with temperature of 0.0, the results will not be fully deterministic" (API ref) | CLT error bars, paired-difference inference, power analysis | Multiple trials for "more consistent results"; no output determinism claim | | OpenAI | knows-and-hides (evals) / parametric-determinism (API) | "non-deterministic by default"; `seed` for "(mostly) consistent" | Judge calibration via TPR/TNR (cookbook); dual graders (LLM + Python) | "Best effort to sample deterministically"; `system_fingerprint` to detect backend drift | | Google Vertex | knows-and-tells (architectural) | Non-determinism named as one of four reasons "vibe checks" fail | "Build deterministic infrastructure around non-deterministic LLMs"; trajectory + final-response evals | User Simulator multi-turn; METRIC_THRESHOLD 0.75 mean score | | LangChain | knows-and-hides | "temperature=0 produces more deterministic results" (primary); "non-determinism" requires multi-trial CIs (checklist) | Two-doc split: rubric-optimistic primary vs CI-rigorous readiness checklist | Reproducibility via test-case design; no output-determinism claim | | Braintrust | knows-and-hides (soft) | Implicit: "deterministic scenarios by stubbing external dependencies" | Pre-deployment eval as CI/CD gate; rubric design | Reproducibility *manufactured* by mocking, not by the LLM | | Langfuse | intra-vendor split | "AI is inherently non-deterministic" (observability) | "Scalable, Human-like, Repeatable" via rubric (judge docs) | "Rerun the same prompts to get consistent scores" (uncited 80–90% agreement claim) | | Arize Phoenix | knows-and-tells | "LLMs are probabilistic; the same prompt can yield different outputs" | Repetitions + function-calling output + provider-agnostic judge config | "A single execution cannot concretely tell you" — multi-run as primitive | | DeepEval | knows-and-tells (structural) | G-Eval "NOT as deterministic as you think" | Three-stack: G-Eval / DAGMetric / evaluation_steps | DAGMetric: "deterministic decision trees for evaluation" | | Weave | knows-and-tells (framing) / knows-and-hides (default) | "LLM outputs are non-deterministic, making debugging harder" | Observability + systematic eval; `trials` optional | Evaluation as reproducible blueprint; single-run default | | AgentOps | observability-only | Not engaged | SRE-style: uptime, traceability, replayability | Reproducible *observation*, not reproducible behavior | | LangSmith | knows-and-hides | "For production agents that require some determinism" | Offline + online eval; `repetitions` as optimization knob | Repetitions framed as performance/optimization, not statistical validation | --- ## 1. Anthropic (Claude, MCP, Evals) Anthropic is the only major AI builder whose primary API documentation, peer-reviewable research output, and enterprise compliance product all tell the same story about non-determinism — and that story is the right one. The framing is *knows-and-tells* in the strictest sense. The Model Context Protocol (MCP), Anthropic's open standard for connecting AI applications to external tools and data, sits orthogonal to the determinism debate: MCP standardizes the tool-call surface (the deterministic, structurally-asserted side of the agent loop) while leaving the model's output non-determinism untouched [[what-is-the-model-context-protocol-mcp-model-context-protocol]]. The architectural division of labor — deterministic plumbing, non-deterministic reasoning — mirrors Google's "deterministic infrastructure around non-deterministic LLMs" pattern at the protocol layer. The primary API reference states verbatim: "Note that even with temperature of 0.0, the results will not be fully deterministic" [[anthropic-api-temperature-websearch]]. The Anthropic Glossary entry pushes further: "identical inputs may produce different outputs across API calls. This applies both to Anthropic's first-party inference service and to inference through third-party cloud providers" [[anthropic-api-temperature-websearch]]. Temperature is framed not as a reliability lever but as "Amount of randomness injected into the response. Defaults to 1.0. Ranges from 0.0 to 1.0" [[anthropic-api-temperature-websearch]]. No other major-vendor primary doc places such a candid disclosure directly in the temperature parameter reference. A developer reading Anthropic's temperature docs cannot leave them believing temperature=0 produces reproducible behavior. Anthropic's reliability position is grounded in formal statistical methodology rather than vendor convention. "Adding Error Bars to Evals" (Evan Miller, arXiv:2411.00640, November 2024) opens with the claim that "Evaluations are critical for understanding the capabilities of large language models (LLMs). Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning" [[adding-error-bars-to-evals-a-statistical-approach-to-language-model-evaluations]]. The paper offers five concrete recommendations grounded in Central Limit Theorem reasoning: report standard errors of the mean via CLT; compute clustered standard errors when questions are drawn in related groups; reduce variance by resampling answers and analyzing next-token probabilities; conduct inference on question-level paired differences; use power analysis to determine eval adequacy. Anthropic's blog post quantifies what is at stake: "In practice, we have found that clustered standard errors on popular evals can be over three times as large as naive standard errors" [[a-statistical-approach-to-model-evaluations-anthropic]]. Resampling K from 1 to 10 reduces the Minimum Detectable Effect from 13.2% to 7.5% on a worked example. The single most consequential passage in the paper is the explicit rejection of temperature-zeroing as a variance-reduction technique: "It may be tempting to reduce the 'sampling temperature' of the model in order to reduce (or eliminate) the conditional variance. However, we advise against this practice ... adjusting the sampling temperature may simply shift the conditional variance (which can be mitigated using the two techniques above) into the variance of the conditional means (which cannot), or else reduce conditional variance by injecting bias into the estimator" [[adding-error-bars-to-evals-a-statistical-approach-to-language-model-evaluations]]. Temperature is not a reliability lever — it is a behavior modifier whose effect on variance is conserved rather than reduced. The paper closes with the framing Anthropic recommends for the field: "machine learning practitioners will think of their model evaluations as informative experiments rather than a series of contests to produce the largest number" [[adding-error-bars-to-evals-a-statistical-approach-to-language-model-evaluations]]. The engineering side reinforces this discipline. Anthropic's January 2026 "Demystifying evals for AI agents" post separates capability evals (can the agent do X?) from regression evals (does it still handle what it used to?), insisting that regression evals "should have a nearly 100% pass rate" and that "a decline in score signals that something is broken" [[demystifying-evals-for-ai-agents-anthropic]]. The post defines a trial structure explicitly because "model outputs vary between runs, we run multiple trials to produce more consistent results." This is determinism imposed at the measurement layer, not the model layer. Anthropic's March 2026 Compliance API completes the alignment. The product is "a control-plane audit, not a content audit" [[claudes-compliance-api-turns-ai-governance-into-audit-ready-evidence-resultsense]]. It records admin and system events, API key creation, file and skill activity — but explicitly excludes model interactions: "Direct model interactions, the prompts and completions themselves, remain outside the scope of this feed" [[claudes-compliance-api-turns-ai-governance-into-audit-ready-evidence-resultsense]]. The strategic framing: "Regulators do not care how capable your AI is. They care whether you can prove who configured it, who accessed it, and what changed" [[claudes-compliance-api-turns-ai-governance-into-audit-ready-evidence-resultsense]]. This is the most direct vendor-side statement in the corpus that determinism is not a regulatory trust proxy. The puzzle Anthropic poses is not how it frames non-determinism — the framing is correct, statistically rigorous, and consistent across the entire stack. The puzzle is why no competitor has adopted the CLT methodology in its tooling. The paper itself names the gap in passing: "We are not aware of an open-source evals framework which implements this technique," referring to next-token probability scoring [[adding-error-bars-to-evals-a-statistical-approach-to-language-model-evaluations]]. The only open-source system the paper cites for resampling-based standard errors is the Inspect framework's `epochs` parameter. Anthropic ships the science; Anthropic does not ship a hosted eval product that mandates the science as its default. The forcing function for adoption cannot, in other words, be Anthropic. The puzzle has a near-edge worth naming: Anthropic's own "Demystifying evals for AI agents" post recommends pass-rate threshold gates on regression evals ("should have a nearly 100% pass rate," "a decline in score signals that something is broken") without surfacing the CLT error-bar discipline its arXiv paper publishes [[demystifying-evals-for-ai-agents-anthropic]]. Anthropic ships the science in research and a softer pass-rate-threshold pattern in product guidance — which means even the *knows-and-tells* vendor practices a mild double-document pattern at the operational layer. The taxonomy position in §1 stands, but the disclosure is a useful corrective on its sharp edges. --- ## 2. OpenAI (Evals Cookbook and Product Docs) OpenAI is the canonical *knows-and-hides* case on the evaluation side and the corpus's clearest example of *parametric determinism as a trust primitive at the API surface* on the model-output side. The honest disclosures about evals live in cookbooks; the determinism primitives are foregrounded in the API reference. OpenAI's primary model-output framing of non-determinism is the most explicit of any model provider on the *problem-and-feature* gradient. The Cookbook entry "How to make your completions outputs consistent with the new seed parameter" opens: "The Chat Completions and Completions APIs are non-deterministic by default" and pitches the `seed` parameter as the mechanism to "receive (mostly) consistent outputs" [[how-to-make-your-completions-outputs-consistent-with-the-new-seed-parameter]]. The API reference is more carefully hedged: "If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed, and you should refer to the system_fingerprint response parameter to monitor changes in the backend" [[create-chat-completion-openai-api-reference]]. The `system_fingerprint` field "represents the backend configuration that the model runs with. Can be used in conjunction with the seed request parameter to understand when backend changes have been made that might impact determinism." This is the only primary-doc API surface in the corpus that ships an explicit determinism primitive with explicit hedge language — "best effort," "mostly consistent," "not guaranteed" — alongside a fingerprint field that lets the caller detect when the backend itself has shifted. The framing simultaneously treats non-determinism as the baseline ("non-deterministic by default") and reproducibility as a desirable feature ("receive (mostly) consistent outputs") — making OpenAI the strongest single case in the corpus where determinism functions as a trust proxy at the parameter layer, while remaining short of an output-determinism guarantee. The 2024 OpenAI Evals cookbook contains the earliest vendor-side acknowledgment of judge unreliability in the corpus: "Model grading will have an error rate, so it is important to validate the performance with human evaluation before running the evals at scale. For best results, it makes sense to use a different model to do grading from the one that did the completion, like using GPT-4 to grade GPT-3.5 answers" [[getting-started-with-openai-evals]]. An accompanying SQL eval YAML disclaimer reads: "Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa" [[getting-started-with-openai-evals]]. The recognition that judge-evaluand correlation is a confounding mechanism appears two years before most peers caught up. None of this language appears in the headline OpenAI hosted-evals product documentation. The October 2025 "Building resilient prompts using an evaluation flywheel" cookbook by Neel Kapse and Hamel Husain prescribes the most rigorous LLM-as-judge protocol any vendor in the corpus publishes [[building-resilient-prompts-using-an-evaluation-flywheel]]. The cookbook's load-bearing passage: "An automated LLM judge is only useful if its judgments are trustworthy. To ensure this, you must systematically measure its performance against a human subject-matter expert (SME) using a 'gold standard' dataset." It introduces True Positive Rate and True Negative Rate as the correct judge-calibration metrics: "True Positive Rate (TPR): How well does the judge correctly identify the failures? True Negative Rate (TNR): How well does the judge correctly identify the passes? The goal is to achieve high scores on both TPR and TNR" [[building-resilient-prompts-using-an-evaluation-flywheel]]. The data discipline is concrete: a 20%/40%/40% train/validation/test split for calibrating judges. What the flywheel cookbook does not do — and what no OpenAI primary doc does — is specify minimum TPR/TNR thresholds at which a judge is fit to gate production deployments. There is no minimum acceptable value. A practitioner adopting OpenAI's hosted evals via the primary product UI encounters no warnings about error rates and no prerequisites for TPR/TNR calibration. A practitioner who happens to find the cookbooks receives the better methodology. The asymmetry is not neutral: it is a commercial choice between adoption-friendly defaults and methodologically sound defaults, and OpenAI has reliably chosen the former in the products it sells and the latter in the materials it educates with. OpenAI's June 2025 "Evals API Use-case - MCP Evaluation" cookbook (Josiah Grace and Shikhar Kwatra) is the corpus's clearest single piece of OpenAI primary-source evidence that its published evals practice mixes stochastic LLM judgment with deterministic structural assertions. The cookbook ships a dual-grader pattern: an LLM-based pass/fail grader that scores semantic content, paired with a "Python MCP Grader: A Python function that checks whether the model actually used the MCP tool during its response (for auditing tool usage)" with `pass_threshold=1.0` [[evals-api-use-case-mcp-evaluation]]. The cookbook's framing: "Best Practice: Using both LLM-based and programmatic graders provides a more robust and transparent evaluation." This places OpenAI's evals discipline closer to DeepEval's structural escape pattern than the rest of this section suggests — when OpenAI's own engineers publish reference patterns, they pair LLM-as-judge with deterministic Python assertions on the tool-call surface, not LLM-as-judge alone. The headline product UI does not surface this dual-grader pattern as a default. The cookbook is co-authored by Hamel Husain, who in his independent writing critiques the entire prefab-eval category: "All you get from using these prefab evals is you don't know what they actually do and in the best case you're not learning anything" [[llm-evals-everything-you-need-to-know-hamels-blog-hamel-husain-gap-C]]. That critique applies to OpenAI's own hosted evals product as much as to anyone else's. The gap is structural — what happens when reliability warnings are treated as adoption friction to be channelled away from headline pages. --- ## 3. Google Vertex AI (Agent Evaluation) Google's framing is architecturally explicit — the underlying patterns (state machines, idempotent operations, undo stacks) are standard distributed-systems practice rather than Google-specific innovation; Google's contribution is naming them and applying them to LLM systems as a reliability discipline. Where most vendors meet non-determinism with measurement (eval more) or constraint (judge less), Google's developer guidance meets it with containment: build deterministic infrastructure around the LLM so the non-deterministic component does less of the load-bearing work. Google occupies a *knows-and-tells* position via this architectural register, with an internal gap on the evaluation side. Vertex AI Agent Builder's November 2025 update is the corpus's most direct primary-product-doc admission from a model provider on the non-determinism evaluation problem: "You told us that evaluating non-deterministic systems is a major challenge. We agree" [[more-ways-to-build-and-scale-ai-agents-with-vertex-ai-agent-builder-google-cloud]]. The product response is structural: a new User Simulator inside Agent Engine's Evaluation Layer, positioning multi-turn simulation as a primitive rather than an opt-in. The Google Cloud developer guidance for production agents frames the design pattern explicitly: "Build deterministic infrastructure around non-deterministic LLMs: transaction coordinators, idempotent operations, undo stacks" [[a-devs-guide-to-production-ready-ai-agents-google-cloud-blog]]. The framing concedes the LLM's non-determinism without trying to fight it. Architectural responsibility shifts to the surrounding code: state machines, idempotent tool calls, reversible action sequences. The unspoken proposition is that the LLM is the only stochastic component in a system that should otherwise be deterministic. Google Codelabs articulates non-determinism as one of four reasons "vibe checking" fails in production: "Non-Determinism: Agents can answer differently each time. You need statistically significant sample sizes" [[from-vibe-checks-to-data-driven-agent-evaluation-google-codelabs]]. The proposed mitigation is pass@k and pass^k with statistically significant samples, gated by a METRIC_THRESHOLD of 0.75 mean score. The threshold is explicit; bitwise reproducibility is not promised. The Vertex AI Gen AI Evaluation Service organizes agent quality as a multi-pillar problem. The "methodical approach to agent evaluation" post defines three pillars: "Pillar 1: Agent success and quality... the end result; Pillar 2: Analysis of process and trajectory... the agent's reasoning process and tool usage; Pillar 3: Trust and safety assessment... reliability under adverse conditions" [[a-methodical-approach-to-agent-evaluation-google-cloud-blog]]. The agent can produce "a correct output through an inefficient or incorrect process — what we call a 'silent failure'." Trajectory evaluation is positioned as the antidote to the result-only blindness of single-output scoring. The framework distinguishes "final response evaluation" from "trajectory evaluation" via six explicit metrics: trajectory_exact_match, trajectory_in_order_match, trajectory_any_order_match, trajectory_precision, trajectory_recall, trajectory_single_tool_use [[evaluate-your-ai-agents-with-vertex-gen-ai-evaluation-service-google-cloud-blog]]. Google also prescribes the upstream alignment step that several peers treat as optional: "Before deployment, you should align the LLM judge to the human evaluation by comparing the judge's output against the original manual human output, groundtruthing the results" [[a-methodical-approach-to-agent-evaluation-google-cloud-blog]]. This places Google in the second-most-honest position on judge calibration in the corpus, behind only OpenAI's flywheel cookbook in specificity. Where Vertex AI does default to LLM-as-judge — the "adaptive rubrics" mechanism explicitly compared to unit testing — the example pass rate Google highlights is 66.7%, informative rather than blocking [[gen-ai-evaluation-service-overview-generative-ai-on-vertex-ai-google-cloud-docum]]. Google exposes a structural alternative the rest of the corpus does not foreground. If the LLM is the only non-deterministic component in an otherwise deterministic system, the engineering question is not "how do we make the LLM deterministic?" but "how do we minimize the load-bearing weight of the non-deterministic component?" This is closer to a safety-engineering pattern (reduce the unsafe region) than to an evaluation-discipline pattern (measure the unsafe region). The framing is correct, and Google is the only vendor that operates in both registers — the Vertex Eval Service performs the measurement work while the developer guidance performs the containment work. Where Google's framing remains incomplete is the inner evaluation problem: the adaptive-rubrics mechanism is itself LLM-as-judge, with no statistical aggregation across repetitions disclosed in primary docs. Containment moves the goalposts on the outer system without solving the inner evaluation methodology. Google has the most regulated-domain-ready evaluation foundation in the corpus that has not yet been framed as such. --- ## 4. LangChain Agent Evals LangChain is the cleanest exhibit of the double-document pattern in the entire survey, and the canonical *knows-and-hides* case. The contradiction is stark enough to read in one paragraph. The same vendor publishes the field's clearest propagation of the temperature=0-as-reproducibility myth in its primary evaluation guide and the field's most candid LLM-as-judge unreliability disclosure in its March 2026 readiness checklist. The primary "What Is LLM Evaluation" practitioner guide states: "Temperature settings control output randomness, and setting temperature to zero produces more deterministic results for test cases that need reproducibility" [[what-is-llm-evaluation-a-practical-guide-to-evals-metrics-and-regression-testing-2]]. This is the single most explicit propagation of the temperature=0 myth in the surveyed vendor corpus, published by an evaluation-tooling vendor for the express audience of practitioners building test cases that need reproducibility. The claim is empirically false in production environments — refuted by two orthogonal analytical paths covered in §12 — and Anthropic's own API documentation directly contradicts it. Against this primary-doc framing sits Victor Moreira's March 2026 "Agent Evaluation Readiness Checklist," published under the LangChain banner. The checklist contains the corpus's most direct vendor-side acknowledgment of LLM-as-judge unreliability: "LLM-as-judge grading for objective tasks can be unreliable, inconsistent judgments can mask real regressions. Switching to deterministic comparison can often eliminate inconsistency and provide better signal. Reserve LLM-as-judge for genuinely subjective assessments" [[agent-evaluation-readiness-checklist]]. On non-determinism: "Run multiple trials per task to account for non-determinism. When running multiple trials, compute confidence intervals before declaring improvement — single-run benchmarks are noisy" [[agent-evaluation-readiness-checklist]]. On judge breadth: "no single judge is uniformly reliable across all benchmarks." On calibration: "Calibrate LLM-as-a-Judge graders to human preferences. Start with 20+ labeled examples using LangSmith's Align Evaluator feature, then grow toward ~100 for production-grade confidence. Include reasoning in the judge's output; this improves accuracy and lets you audit why it scored something" [[agent-evaluation-readiness-checklist]]. On the capability/regression separation: "Capability evals push your agent forward by measuring progress on hard tasks, while regression evals protect what already works" [[agent-evaluation-readiness-checklist]]. The two texts are not in conversation; they are in different rooms. The interpretive question is not which document is correct — the readiness checklist is — but how a vendor can simultaneously publish both. Feature documentation rewards adoption; readiness checklists are written by specialist deployed engineers (Moreira's title in the byline). The practitioners who most need the readiness guidance are the ones least likely to read it. The LangChain case is the field's clearest single demonstration that vendor primary docs cannot be treated as authoritative on reliability methodology. A practitioner's reliability depends on reading order — find the checklist first and you adopt the right methodology; find the primary guide first and you deploy a fundamentally broken eval. The checklist's existence does not absolve the primary guide of its empirically falsified claim. Some of this pattern likely reflects documentation-update inertia rather than active commercial framing — vendors publish corrections in new documents rather than rewriting old ones for SEO and reference-stability reasons — but the inertia itself becomes a methodological problem when the old documents remain the top search result. --- ## 5. Braintrust Braintrust is *knows-and-hides* in a softer register than LangChain. The honesty is not absent so much as displaced — it appears in the product architecture (the mocking recommendation, the two-scorer typology) rather than in honest documentation about judge stochasticity. The commercial pitch is the most aggressive pre-deployment trust model in the survey: evaluation is positioned as infrastructure for shipping, with non-determinism treated as a problem solved by good scorer design. The defining framing of trust is pre-deployment measurement: "The broken feedback loop in AI development turns engineering into guesswork. This creates a painful reality: teams ship AI on vibes. They deploy changes hoping for improvement, only to discover regressions when users complain" [[how-to-evaluate-llms-and-ai-agents-in-production-the-braintrust-way-articles-bra]]. The Braintrust pitch is operational: "Evaluation isn't overhead — it's infrastructure that enables fast iteration without breaking things." Offline agent evals "function like unit tests or integration tests, emphasizing reproducibility." The CI/CD integration is mature: "GitHub Actions integration brings production-grade CI/CD to AI development. Every pull request automatically runs evals and posts detailed results as comments showing exactly which test cases improved, which regressed, and by how much. Quality gates prevent regressions from reaching production" [[how-to-evaluate-llms-and-ai-agents-in-production-the-braintrust-way-articles-bra]]. The Notion case study quantifies the value: 3 issues triaged per day to 30, a 10x improvement, after introducing Braintrust eval infrastructure [[how-notion-evaluates-ai-at-scale-across-70-engineers-customers-braintrust]]. The non-determinism position is bifurcated structurally. Braintrust offers two scorer types: "Code-based for deterministic checks, LLM-as-judge for nuanced qualities" [[how-to-evaluate-llms-and-ai-agents-in-production-the-braintrust-way-articles-bra]]. The framing is symmetric — pick the right tool for the job — and the LLM-as-judge tool comes with no reliability caveat in the primary docs. Recommended judge design is rubric-quality-conditioned: "Design clear rubrics with explicit instructions and examples of good vs. bad outputs. Use chain of thought to understand scoring decisions. Choose the right model for judging, which may differ from your task model" [[how-to-evaluate-llms-and-ai-agents-in-production-the-braintrust-way-articles-bra]]. Critically absent: any acknowledgment of judge stochasticity, any TPR/TNR requirement, any minimum sample count for multi-run aggregation. The most revealing framing is the reproducibility framing. The explicit recommendation: "Create deterministic scenarios by stubbing external dependencies with production snapshots" [[how-to-evaluate-llms-and-ai-agents-in-production-the-braintrust-way-articles-bra]]. Reproducibility is a property *manufactured* by mocking — not a property of the LLM, but a property of the test harness around it. This is a defensible engineering position. It is also a tacit admission that the LLM cannot be expected to behave deterministically: trust is built by isolating the deterministic substrate (mocked dependencies) from the non-deterministic core (the model). Braintrust's product architecture knows that LLM determinism is unattainable. Braintrust's documentation does not surface this knowledge as a methodological warning. The Notion 10x velocity improvement is real and worth taking seriously as evidence that systematic evaluation improves engineering throughput. The strongest counter-argument to this section's methodological critique deserves explicit engagement: a fast-and-noisy single-run LLM-as-judge pipeline that runs ten eval cycles per week may catch more real regressions than a slow-and-rigorous multi-run CLT pipeline that runs two — even if any individual eval result is statistically weaker. Methodological soundness and engineering throughput trade off; the field's adoption of single-run LLM-as-judge as the default may reflect a defensible engineering judgment that fast-and-noisy beats slow-and-rigorous in unregulated product contexts. The Notion evidence sits inside this defense rather than outside it. What changes the equation is the regulatory frame: an audit under EU AI Act Annex IV or IMDRF GMLP does not score on regressions-caught-per-week, it scores on the statistical soundness of any given test plan. In regulated contexts the trade-off inverts; in unregulated ones it does not. Braintrust's framing is defensible for the latter and inadequate for the former — and the vendor does not currently differentiate. A team adopting Braintrust as its CI/CD gate, following only the primary docs, will deploy LLM-as-judge evaluators with unknown TPR and TNR — and the GitHub Actions pass/fail signal will be treated as load-bearing despite that fact. A team can ship faster and still ship regressions invisible to its eval suite if the eval suite uses single-shot LLM-as-judge scorers whose own variance is not measured. Braintrust's framing prevents the practitioner from asking the methodological question. --- ## 6. Langfuse Langfuse displays an intra-vendor split that is itself diagnostic of the field's commercial framing pattern. The observability primary docs contain the corpus's clearest single-sentence non-determinism framing. The eval primary docs contain the corpus's most aggressive uncited reliability claim. The same product holds both positions because practitioners read them in different sittings. The Langfuse tracing overview opens: "Because AI is inherently non-deterministic, debugging your application without any observability tool is more like guesswork" [[llm-observability-application-tracing-open-source-langfuse-2]]. This is unambiguous and correct. Non-determinism is named as the baseline condition, not the exception. Langfuse's value proposition in tracing is making non-deterministic behavior observable, not eliminating it. Trust is operationalized as trace completeness — "captures the exact prompt sent, the model's response, token usage, latency, and any tools or retrieval steps in between." This is a methodologically clean reliability framing in the observability register. The Langfuse LLM-as-a-Judge documentation falls into the genre default. It positions LLM-as-judge as combining "the nuance of human judgment with the scalability of automated evaluation," lists three properties — "Scalable: Judge thousands of outputs quickly versus human annotators. Human-like: Captures nuance (e.g. helpfulness, toxicity, relevance) better than simple metrics, especially when rubric-guided. Repeatable: With a fixed rubric, you can rerun the same prompts to get consistent scores" — and conditions repeatability on rubric quality alone [[llm-as-a-judge-langfuse]]. The "Repeatable" claim is the corpus's cleanest single example of repeatability framed as a feature of the eval protocol rather than as a function of judge stochasticity. The framing is directly falsified by Schroeder and Wood-Doughty's omega measurements [[can-you-trust-llm-judgments-reliability-of-llm-as-a-judge-4]]. The FAQ contains the corpus's single most aggressive uncited reliability claim: "strong LLM judges (such as GPT-5 class models) achieve 80-90% agreement with human evaluators on many quality dimensions, which is comparable to inter-annotator agreement between humans" [[llm-as-a-judge-langfuse]]. No citation is given. No specification of "which dimensions." No methodology. No acknowledgment that the academic literature on LLM-as-judge inter-run reliability is far more pessimistic than this figure implies. The closest empirical anchor in the literature is Zheng et al.'s MT-Bench paper, which reported "over 80% agreement" between GPT-4 and humans on multi-turn benchmark questions [[230605685-judging-llm-as-a-judge-with-mt-bench-and-chatbot-arena]] — but that result is benchmark-specific and was directly challenged by Schroeder and Wood-Doughty's omega analysis, which found judge reliability at MT-Bench T=0.25 of 0.421 and inter-rater reliability ranging from 0.167 to 1.000 across seed replications. Langfuse's framing does not engage either side of this debate. The 80–90% claim, asserted without citation, will function in adoption as ground truth because no Langfuse reader has access to the per-task omega values that contradict it. The pattern is sharpest at Langfuse: the framing is honest about whatever the product *cannot* solve and softer about whatever the product is *sold to* solve. Tracing makes non-determinism visible; the LLM-as-judge feature is sold as the remedy, so the remedy cannot itself be stochastic. --- ## 7. Arize Phoenix Arize Phoenix is the corpus exception. Its primary documentation is the most explicit acknowledgment of structural non-determinism of any vendor surveyed, and it is the only vendor whose primary docs name single-run inadequacy in unqualified terms. Phoenix breaks the double-document pattern from the primary-doc side: the honest version is the headline version. From Phoenix's primary repetitions documentation: "Since LLMs are probabilistic, their synthesis can differ even when the supplied prompts are exactly the same. This can make it challenging to determine if a particular change is warranted as a single execution cannot concretely tell you whether a given change improves or degrades your task. So what can you do when an execution can change from one run to the next? That's where repetitions come in. Repetitions help you reduce uncertainty in systems prone to variability, notably more 'agentic' systems, where the model orchestrates multi-step reasoning" [[repetitionsmd]]. This passage is operationally load-bearing. It names probabilistic generation as the cause, names single-execution insufficiency as the practical consequence, and positions multi-run aggregation as a primary primitive — not an optional optimization. The Phoenix repetition cookbook for customer-review evals extends the framing into methodology: "Large Language Models (LLMs) are probabilistic; the same prompt can yield different outputs across runs. This variability makes it hard to tell if a change truly improves performance or is just random noise. Repetitions help address this by running the same input multiple times, reducing uncertainty and revealing patterns that single runs can miss" [[analyzing-customer-review-evals-with-repetition-experimentsmd]]. The single sentence — "a single execution cannot concretely tell you whether a given change improves or degrades your task" — is the most load-bearing single statement from any vendor primary doc in the corpus. It is not in a footnote, not in an advanced checklist, not in an academic preprint. It is in the headline definition of what Phoenix Repetitions are for. Where other vendors frame multi-run as an optional optimization (Weave's `trials=N`, LangSmith's `repetitions`), Phoenix frames it as a structural requirement for valid measurement. Phoenix's evaluation overview reinforces the framing: "Evaluations measure the quality of your AI application's outputs — whether responses are accurate, grounded, safe, or relevant to the user's intent. Unlike traditional software, LLM outputs can't be tested with simple assertions. Evaluations give you a way to score outputs systematically, moving from observation to measurement" [[llm-evalsmd-2]]. Phoenix supports "both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM evaluators for more nuanced assessments." The structural mitigation for judge stochasticity is function-calling for output: "LLM evaluators use function calling (tool use) to extract structured judgments rather than parsing free-form text. This approach produces more consistent and reliable outputs than asking the model to format its response in a specific way" [[llm-evalsmd-2]]. Function-calling-based judgment extraction is one of two structural mitigations in the corpus — the other is DeepEval's DAGMetric — that attacks judge stochasticity at the protocol level rather than the aggregation level. Phoenix also recursively instruments its own evaluators: "Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model's full reasoning, the final scores, and execution timing" [[llm-evalsmd-2]]. Phoenix's third structural primitive is the provider-agnostic judge abstraction: "Invocation behavior (temperature, token limits, or other generation controls) is configured separately on the evaluator" [[run-evals-with-built-in-eval-templates-phoenix]], decoupling judge-model identity from judge-call settings so the same judge can be reused with audited per-evaluation parameters. A gap remains. Phoenix exposes repetitions; it does not mandate them. There is no default minimum K, no power-analysis guidance, no required confidence-interval reporting on eval outputs. The framing is honest about the problem and provides the tool; the practitioner still has to know to use it. But the framing itself is the deviation from field norm. Phoenix is the existence proof that primary-doc honesty is compatible with product adoption. A reliability-honest commercial eval platform is not commercially fatal. Other vendors could match Phoenix's framing and have chosen not to. --- ## 8. DeepEval DeepEval is the corpus's most sophisticated vendor on the subjective/objective distinction in evaluation, and the only vendor that ships an explicit structural alternative to statistical accommodation of LLM-as-judge non-determinism. DeepEval's position is *knows-and-tells* in a structural register the rest of the field has ignored. DeepEval's G-Eval documentation acknowledges the stochasticity of the dominant LLM-as-judge protocol head-on. The DAGMetric guide states: "this will NOT give you the exact score according to your criteria, and is NOT as deterministic as you think" [[dag-deep-acyclic-graph-deepeval-by-confident-ai-the-llm-evaluation-framework]]. This is the corpus's most direct vendor admission that LLM-as-judge — in the form most other vendors ship — is not, in fact, deterministic. The admission is positioned as the motivating problem for DAGMetric: if G-Eval is not as deterministic as you think, the path forward is structural. Simultaneously, G-Eval's marketing claims "human-like accuracy" [[g-eval-deepeval-by-confident-ai-the-llm-evaluation-framework]]. The reconciliation comes through architecture. G-Eval's residual stochasticity is partially mitigated by token-probability weighting — "G-Eval take[s] the probabilities of the output tokens from the LLM to normalize the score and take their weighted summation as the final result" [[g-eval-deepeval-by-confident-ai-the-llm-evaluation-framework]]. This is a more principled aggregation than single-text scoring, though DeepEval does not publish the residual variance reduction it produces. The DAGMetric is the corpus's only deterministic structural escape from LLM-as-judge stochasticity at the per-output CI/CD gate. The framing relative to G-Eval is explicit: "The DAGMetric gives you more deterministic control over GEval" [[dag-deep-acyclic-graph-deepeval-by-confident-ai-the-llm-evaluation-framework]]. The product positioning is direct: "The deep acyclic graph (DAG) metric in deepeval is currently the most versatile custom metric for you to easily build deterministic decision trees for evaluation with the help of using LLM-as-a-judge" [[dag-deep-acyclic-graph-deepeval-by-confident-ai-the-llm-evaluation-framework]]. The architecture: four node types (TaskNode, BinaryJudgementNode, NonBinaryJudgementNode, VerdictNode) composed into a directed acyclic graph traversed in topological order. The LLM is never asked to produce a score; it is asked only to make local routing decisions whose verdict integers are pre-defined. The final score emerges deterministically from the path through the DAG. The LLM's stochasticity is quarantined to discrete classification decisions at each node. The product line splits the eval problem on the subjective/objective axis: "DAG: Directed-acyclic-graph metrics for objective, multi-step conditional scoring ... G-Eval: Criteria-based, chain-of-thought scoring via form-filling for reliable subjective evals" [[dag-deep-acyclic-graph-deepeval-by-confident-ai-the-llm-evaluation-framework]]. Where subjective criteria are unavoidable, accept stochasticity and mitigate via token-probability aggregation. Where objective criteria apply, eliminate stochasticity through structural constraint. This produces a three-architecture stack — G-Eval (token-probability aggregation for residual subjective variance), DAGMetric (structural elimination via routed decision trees for objective criteria), and `evaluation_steps` (manual rubric specification that "provides more reliability in how the metric is scored") — that is the most complete reliability architecture in the corpus, even though DeepEval does not publish quantified residual variance for the probability-weighted G-Eval path. DAGMetric is the per-output CI gate analogue to Anthropic's benchmark-scale CLT methodology. The two are complementary, not competing. CLT-style error bars are the right tool for benchmark-scale comparative evaluation (is Model A meaningfully better than Model B across N hundreds of questions?); DAGMetric is the right tool for per-case CI/CD gating (did this single output pass the defined criterion?). Phoenix's function-calling output constraint is in the same family as DAG routing, but it is per-evaluator rather than multi-step, and Phoenix does not publish inter-run variance for its function-calling judges either. The field has effectively published both alternatives — one statistical, one structural — and adopted neither at scale. The under-discussed weakness of DeepEval's position is that it does not publish residual variance measurements for DAGMetric, so the claim of "deterministic scores" is structural rather than quantitative. A practitioner cannot point to an omega measurement and say "DAGMetric achieves omega = 0.9X." The structural argument is nonetheless strong: if the LLM's role is reduced to discrete routing on explicit criteria, the residual variance is bounded by the variance of those routing decisions, which is empirically much lower than free-form continuous scoring. DeepEval has the most defensible split-positioning available to a vendor under regulatory pressure: it can claim DAGMetric produces "deterministic scores" without overstating, because the structural design genuinely makes that claim true. --- ## 9. Weave (Weights & Biases) Weave occupies the widest gap in the corpus between the strength of its philosophical framing and the weakness of its default product behavior. It is *knows-and-tells* on framing and *knows-and-hides* on enforcement. Weave's headline copy is more candid about non-determinism than any other vendor in the survey; Weave's default workflow runs single-shot evals. The Weave overview is the strongest single-paragraph framing of why traditional software methodology fails for LLM systems: "W&B Weave is an observability and evaluation platform for building reliable LLM applications. Weave helps you understand what your AI application is doing, measure how well it performs, and systematically improve it over time. Building LLM applications is fundamentally different from traditional software development. LLM outputs are non-deterministic, making debugging harder. Quality is subjective and context-dependent. Small prompt changes can cause unexpected behavior changes. Traditional testing approaches fall short" [[what-is-weave-weights-biases-documentation]]. Three propositions in two sentences: non-determinism, subjectivity, and brittleness to prompt change. The reliability architecture is dual-track: "Visibility into every LLM call, input, and output" (observability) plus "Systematic evaluation to measure performance against curated test cases" (eval). The Evaluation object is positioned as a reproducible blueprint — each Evaluation captures the test cases, scorers, and configuration, and "each run as a measurement of how your application performs under that setup" [[evaluations-overview-weights-biases-documentation]]. The multi-run mechanism is the `trials=N` parameter: "You can set the `trials` parameter on the `Evaluation` object to run each example multiple times" [[evaluations-overview-weights-biases-documentation]]. The framing in the docs labels it explicitly as "(Optional)" — an advanced feature. The default workflow shown in primary docs is a single `.evaluate()` call. The gap between framing and default is the widest of any vendor in the corpus. Weave's documentation says LLM outputs are non-deterministic, that quality is subjective, that traditional testing falls short. Then the default eval runs once, with a single score per test case, with no guidance on how many trials are needed, how to aggregate scores across trials, or how to compute confidence intervals from trial results. The structural problem Weave's own opening paragraph names is not addressed by Weave's default behavior. A team adopting Weave will accept the single-run default because that is what the primary tutorial shows. The empirical question is whether opt-in multi-sample tooling is sufficient given the default. Bjarnason and colleagues answer it: across 60,000 trajectories on SWE-Bench-Verified, "standard deviations exceeding 1.5 percentage points even at temperature 0," with the diagnosis that "Despite the name suggesting a statistical estimator, in practice, most researchers run the agent exactly once per task and report the fraction that succeeded. This single-run approach has become standard practice across research papers, model releases, and community leaderboards" [[260207150-on-randomness-in-agentic-evals]]. If practitioners default to single-run when given the choice — and the empirical evidence is that they do — exposing `trials` without defaulting to a statistically meaningful K is functionally equivalent to not exposing it at all. The population transfer here is incomplete: Bjarnason et al. studied research papers, model releases, and community leaderboards — academic publication norms — not paid platform users with explicit `trials` parameters whose overview documentation names non-determinism as the baseline condition. Whether Weave platform users follow academic-paper defaults is empirically unverified. Subject to that caveat, the path-of-least-resistance argument applies symmetrically: Weave has the right product architecture and a default that the available evidence suggests is misaligned with its own framing. --- ## 10. AgentOps AgentOps is the corpus's clearest *observability-only* case — and not as a criticism, but as a market-segmentation data point. The eleven vendors split across a three-way temporal taxonomy of trust models: *pre-deployment* (eval gates that block ship — LangSmith, Braintrust, DeepEval, Phoenix, Weave, OpenAI, Google), *runtime* (live monitoring on production traffic — Langfuse, LangSmith's online evaluation, Phoenix), and *forensic* (post-hoc reconstruction after a failure — AgentOps's pure position). AgentOps has opted out of the eval-gate and live-monitoring discourse and built the corpus's purest forensic stack. The silence on eval methodology is itself the finding. The AgentOps documentation contains no mention of non-determinism, stochasticity, evaluation methodology, scoring frameworks, judges, or quality measurement. The platform is positioned as "Observability and monitoring for your AI agents and LLM apps. And we do it all in just two lines of code" [[introduction-agentops-2]]. The headline framing is "Powering thousands of engineers building reliable agents" — where "testing" in this vocabulary means "session inspection," not statistical evaluation. The features named are session waterfalls, time-travel debugging ("Rewind and replay agent runs with point in time precision"), and the audit-log function ("Debug and Audit: Keep a full data trail of logs, errors, and prompt injection attacks from prototype to production"). The architecture is OpenTelemetry-based: "AgentOps is built on OpenTelemetry, a widely-adopted standard for observability instrumentation" [[core-conceptsmd]]. Reliability in AgentOps is framed as a function of session visibility and forensic capability — the SRE conception of reliability (uptime, traceability, replayability) applied to agents, rather than the eval conception of reliability (distributional quality, measurement, regression detection). AgentOps does not claim repeatable behavior. It claims repeatable *observation*: the user can replay the agent's behavior precisely because the trace is durable, even if the underlying agent is not. This is a conceptual distinction the corpus does not name explicitly elsewhere — the difference between reproducible behavior (the system does the same thing each time) and reproducible observation (the system's behavior, however non-deterministic, is captured durably enough to be examined later). AgentOps is not making a claim about output quality; it is making a claim about audit-trail completeness. The two are sometimes confused in agent-reliability discourse, and AgentOps is the corpus's clearest example of a vendor whose offering occupies the audit-trail position without claiming the quality position. The trust model is forensic, not pre-deployment. A practitioner reading AgentOps's marketing learns they need visibility, replayability, and an OpenTelemetry-compatible trace store. A practitioner reading Arize Phoenix's marketing learns they need repetitions, function-calling-based judges, and statistical reasoning about variance. Both vendors claim to make agents production-ready. They are not solving the same problem. The field has converged on neither a definition of "production-ready" nor a methodology for asserting it. AgentOps's scoped-out position is a coherent product-design choice rather than a deficiency: the vendor judges output-quality evaluation to be a different product category and ships a deliberately narrow stack accordingly. The corpus's own "reproducible behavior vs reproducible observation" distinction is the strongest single argument for taking the AgentOps stance as a legitimate market segment rather than as opting out of methodology. AgentOps cannot be held responsible for the field's methodology failures because it has not joined the methodology conversation. Under regulatory pressure, this posture will not be sufficient on its own — a regulated buyer cannot satisfy EU AI Act Annex IV's statistical-soundness requirement with trace-replay alone. Under product pressure, it has clearly been sufficient enough to power "thousands of engineers." That says more about the field's current bar for "reliable agents" than about AgentOps's product. --- ## 11. LangSmith LangSmith is the same vendor ecosystem as LangChain Agent Evals but deserves separate treatment because the `repetitions` parameter framing is uniquely revealing and the "require some determinism" phrase is the most honest uncalibrated piece of vendor product copy in the corpus. The double-document pattern is at its most concentrated here. LangSmith's primary evaluation documentation divides the trust architecture into two temporal buckets: "Offline Evaluation: Test before you ship — Run evaluations on curated datasets during development to compare versions, benchmark performance, and catch regressions. Online Evaluation: Monitor in production — Evaluate real user interactions in real-time to detect issues and measure quality on live traffic" [[langsmith-evaluation-docs-by-langchain]]. The feedback loop is explicit: "Add failing production traces to your dataset, create targeted evaluators, validate fixes with offline experiments, and redeploy" [[langsmith-evaluation-docs-by-langchain]]. Four evaluator types are listed without reliability stratification: "Human review, Code rules, LLM-as-judge, Pairwise comparison" [[langsmith-evaluation-docs-by-langchain]]. The architecture is methodologically reasonable; the population of evaluators within it inherits all the reliability problems documented elsewhere. The non-determinism framing reduces to one phrase: "For production agents that require some determinism" [[langchain-observe-evaluate-and-deploy-reliable-ai-agents]]. The phrase deserves close reading. "Require some determinism" implicitly acknowledges non-determinism as the default ("require" implies the typical agent does not have it); it positions determinism as an engineering target rather than a system property; it offers no threshold for what "some" means. A production-readiness checklist that uses "some" as a quantifier is operating at a level of imprecision that no software-engineering discipline would accept on its primary metric. These are simultaneously the most honest three words about non-determinism in any vendor primary doc surveyed and the least operationalized. The `repetitions` parameter framing is the second piece of revealing copy. From the primary docs: "Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs" [[langsmith-evaluation-docs-by-langchain]]. Repetitions are listed alongside concurrency and caching — performance-tuning knobs, not statistical requirements. The verb is "optimize," not "validate." A practitioner reading this guidance encounters multi-run as a way to speed up or stabilize experiments, not as a precondition for the eval result being meaningful. There is no companion documentation in primary docs requiring repetitions for small-effect detection, no confidence-interval calculation surfaced in the eval UI by default, no warning that single-run pass@1 has structural variance. The contrast with the LangChain Agent Evaluation Readiness Checklist is the double-document pattern at its most concentrated. The same vendor ecosystem publishes feature documentation that treats `repetitions` as an optimization knob and a secondary readiness checklist that warns "single-run benchmarks are noisy" and prescribes confidence-interval computation [[agent-evaluation-readiness-checklist]]. A practitioner who builds against LangSmith primary docs alone will deploy a single-run eval pipeline by default, with multi-run aggregation available but not surfaced as methodologically necessary. The vendor publishes both; the vendor does not reconcile them; the vendor's product surface defaults to the less-rigorous configuration. LangSmith's October 2025 Insights Agent and Multi-turn Evals announcement extends the platform to multi-turn agent trajectory evaluation: "Multi-turn evals are online evaluations that let you measure things like: Semantic intent... Semantic outcomes... Agent trajectory" [[improve-agent-quality-with-insights-agent-and-multi-turn-evals-now-in-langsmith]]. The multi-turn framing addresses the silent-failure problem that Google Vertex's Pillar 2 names, but does not address the underlying judge stochasticity. --- ## 12. Field-Level Synthesis: Determinism as a Proxy for Trust The eleven vendors above converge on a single dominant framing: non-determinism is a structural property of LLMs to be managed via measurement, not eliminated via engineering. Determinism is *not* used as a proxy for trust in any vendor's primary documentation. This is the right framing in the abstract. What is wrong is the methodology that has filled the vacuum: single-run LLM-as-judge scoring, deployed by default in every platform surveyed except DeepEval (DAGMetric) and Arize Phoenix (repetitions as primary mechanism), and even there only as opt-in. The synthesis splits into six load-bearing findings. ### A. Compounding Unreliability — the Judge Is as Stochastic as the Evaluand The strongest single argument against current eval practice is that the judge fails for the same structural reason as the system it judges. Both are LLMs. Both run on production inference infrastructure with batch-size non-invariance. Both produce probabilistic distributions of outputs, not point estimates. A single-shot judge invocation against a single-shot system output produces a verdict that compounds the stochastic uncertainty of both layers — and the field reports this verdict as a binary pass/fail with no confidence interval. Quantitatively. Atil and colleagues, across 5 LLMs × 8 tasks at temperature=0, top-p=1, fixed seed, 10 runs each — maximally deterministic settings — report a 70 percentage points best-worst accuracy gap across "deterministic" runs on MMLU and BBH subtasks, with the conclusion that "an LLM rarely produces the same response ten times given the same input" and "non-determinism perhaps essential to the efficient use of compute resources via co-mingled data in input buffers so this issue is not going away anytime soon" [[non-determinism-of-deterministic-llm-settings-2]]. Bjarnason and colleagues: standard deviations exceeding 1.5 percentage points even at temperature 0 on SWE-Bench-Verified across 60,000 trajectories, with observed ranges of 2.2–6.0 percentage points and the explicit conclusion that "reported improvements of 2–3 percentage points may reflect evaluation noise rather than genuine algorithmic progress" [[260207150-on-randomness-in-agentic-evals]]. The causal mechanism Bjarnason names connects token-level non-determinism to trajectory-level variance: "trajectories diverge early, often within the first few percent of tokens, and that these small differences cascade into different solution strategies" — yielding gaps of up to 24.9 percentage points between optimistic (pass@k — at least one success across k attempts) and pessimistic (pass^k — k consecutive successes, a stricter reliability variant) bounds on the same model and task. Schroeder and Wood-Doughty are unambiguous on the determinism shortcut and coin the field's sharpest single term for the epistemic confusion underneath it — *fixed randomness*: "Many research works aim to circumvent this issue by setting a fixed seed and using deterministic settings. However, this results in 'fixed randomness' — a single, reproducible but potentially misleading sample from a distribution" [[can-you-trust-llm-judgments-reliability-of-llm-as-a-judge-4]]. *Fixed randomness* names what temperature=0 + seed actually deliver — a reproducible draw, not a reliable measurement — and the term is load-bearing for everything in §12.B that follows. And on inter-run variance: "Table 1 details the drastic variation in inter-rater reliability, ranging from 0.167 to 1.00" [[can-you-trust-llm-judgments-reliability-of-llm-as-a-judge-4]]. Mustahsan and colleagues (December 2025) sharpen the framing further. Their Intraclass Correlation Coefficient (ICC) decomposition partitions variance into between-query (task difficulty) and within-query (agent inconsistency), allowing practitioners to ask whether reported improvements reflect true capability or measurement noise [[251206710-stochasticity-in-agentic-evaluations-quantifying-inconsistency-with-in]]. Their finding across GAIA and FRAMES: ICC varies 0.304–0.774, and "ICC converges by n=8–16 trials for structured tasks and n≥32 for complex reasoning." The recommendation: report accuracy alongside ICC and within-query variance as standard practice. The compounding argument can now be stated precisely. If the evaluand's within-query variance is non-trivial (up to 70pp gap; 1.5pp+ SD on SWE-Bench), and the judge's omega is non-trivially below 1.0 on subjective tasks (0.421 on MT-Bench T=0.25), then the variance of a single-shot judge verdict against a single-shot system output is the convolution of two distributions whose marginals are both wide. The field reports the resulting verdict as a binary score and aggregates across test cases to produce a percentage pass rate. The confidence interval around that pass rate is unknown — not "wide," not "moderate," *unknown* — because no vendor publishes the joint variance of their evaluand × judge pipeline. This is the methodological crisis in one sentence: the industry's primary automated quality gate produces verdicts whose confidence intervals it has not measured and cannot report. ### B. The Temperature=0 Myth Is Refuted from Two Convergent Paths The corpus contains the field's clearest empirically falsified shared belief: temperature=0 produces deterministic outputs. Two independent analytical paths refute it through non-overlapping mechanisms, arriving at the same practical conclusion. The infrastructure path is Thinking Machines Lab's September 2025 analysis. The team ran 1,000 completions at temperature=0 with Qwen3-235B-A22B-Instruct-2507 on the prompt "Tell me about Richard Feynman" and "Surprisingly, we generate 80 unique completions, with the most common of these occurring 78 times" [[defeating-nondeterminism-in-llm-inference-thinking-machines-lab]]. The divergence is user-visible, not just token-level wobble: 992 of the 1,000 completions produced "Queens, New York" and 8 produced "New York City" on identical inputs at temperature=0 — same prefix, divergence at a single token, observable content-level difference. The root-cause analysis identifies the mechanism: "the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies." The forward pass of an LLM inference server is technically deterministic given identical inputs, but batch-size non-invariance — driven by continuous batching in vLLM, TensorRT-LLM, and similar systems — means each user's result depends on concurrent load from other users. The mechanism is structural and persistent. Thinking Machines' normative stance is the corpus's sharpest single rhetorical move: "After all, our systems are already 'probabilistic', so what's wrong with a little more nondeterminism? We reject this defeatism" [[defeating-nondeterminism-in-llm-inference-thinking-machines-lab]]. The team ships batch-invariant kernels on top of vLLM with quantified latency cost: an unoptimized deterministic vLLM runs 2.1x slower than baseline, the improved kernel 1.6x slower. The cost of architectural determinism is bounded and measurable; the field's non-adoption of batch-invariant kernels — connected to §12.D's regulatory argument — is a vendor velocity choice, not a technical impossibility. The statistical path is the Anthropic CLT paper. The paper explicitly rejects temperature-zeroing as a variance-reduction technique on theoretical grounds: "adjusting the sampling temperature may simply shift the conditional variance (which can be mitigated using the two techniques above) into the variance of the conditional means (which cannot)" [[adding-error-bars-to-evals-a-statistical-approach-to-language-model-evaluations]]. The decomposition is rigorous: a question's score is the sum of a conditional mean (true skill) and a zero-mean random component. Resampling and next-token probability scoring target the random component; temperature manipulation moves it elsewhere where it cannot be mitigated, and additionally injects estimator bias by altering the model's behavior. Two analytical paths, one falsified belief, two independent derivations. The infrastructure path says temperature=0 fails because the forward pass is batched non-deterministically. The statistical path says temperature=0 fails because variance shifts rather than reduces. Convergent evidence from orthogonal analytical angles is the strongest epistemic warrant the field offers on any question of practice. The practitioner norm that conflates temperature=0 with reproducibility is empirically wrong and theoretically wrong simultaneously. The norm persists because temperature=0 reduces sampling variance — it narrows the distribution without collapsing it to a point — and practitioners conflate variance reduction with determinism. The cleanest correction lives in LangChain's own readiness checklist; the practitioner norm persists in LangChain's own evaluation guide. The field is documenting the refutation in its secondary documents and propagating the myth in its primary ones. ### C. The Double-Document Pattern Is a Field-Wide Commercial Framing Decision Across the eleven vendors, the same structural gap recurs: the most honest reliability framing lives in optional secondary documents (readiness checklists, advanced cookbooks, methodology papers) while the primary feature documentation presents reliability as conditioned on rubric quality, scorer choice, or trace coverage — never on multi-sample aggregation or judge calibration. LangChain primary docs versus the March 2026 readiness checklist is the cleanest case. OpenAI hosted evals docs versus the October 2025 flywheel cookbook is the second. Langfuse observability docs (honest) versus Langfuse LLM-as-judge docs (rubric-optimistic) is the intra-vendor inversion. Braintrust marketing copy versus Braintrust's mocking recommendation (manufacture reproducibility because the LLM cannot provide it) is the implicit version. LangSmith's `repetitions`-as-optimization framing versus the same vendor's readiness checklist is the most concentrated case. The exceptions, partial and full, are Arize Phoenix (primary repetitions docs are honest), Anthropic (primary API docs deny temperature-0 determinism), and DeepEval (DAGMetric docs explicitly state G-Eval is "NOT as deterministic as you think"). The pattern is therefore not universal — three vendors deviate — but it is dominant enough to constitute a field-wide framing failure. The mechanism is identifiable. The practitioners most likely to uncritically deploy LLM-as-judge in automated CI gates are the same practitioners least likely to have read secondary checklists. The vendor's commercial interest (driving adoption of evaluation features) is in tension with the vendor's epistemic interest (warning users about reliability limits). When the two tensions are resolved by publishing both documents on different surfaces, the adoption-friendly surface wins by default because it is the surface practitioners encounter first. The double-document pattern is therefore not a documentation oversight but a commercial framing decision. The corrective is not "publish better documentation"; it is "publish the secondary-document standard as the primary-document default." No major vendor has yet done so. Phoenix's repetitions docs come closest because the primary doc opens with the structural framing, but even Phoenix does not mandate repetitions. ### D. The Regulatory Frame Requires Statistical Rigor, Not Determinism The most consequential misreading in the field is the assumption that the EU AI Act, the FDA's Good Machine Learning Practice (GMLP), and similar regulatory frameworks will require deterministic reproducibility of AI outputs. The corpus evidence is decisive against this reading. EU AI Act Annex IV — the technical documentation required under Article 11(1) for high-risk systems — does not mandate output determinism. It requires "the validation and testing procedures used, including information about the validation and testing data used and their main characteristics; metrics used to measure accuracy, robustness and compliance with other relevant requirements set out in Chapter III, Section 2, as well as potentially discriminatory impacts; test logs and all test reports dated and signed by the responsible persons" [[annex-iv-technical-documentation-referred-to-in-article-111-eu-artificial-intell]]. The term "reproducibility" appears only in the context of system-state version control: "M11.2 Version control ensures full traceability and reproducibility of AI system states over time" [[assessing-high-risk-ai-systems-eu-ai-act]]. Buscemi and colleagues conclude that "conformity emerges from structured combinations of procedural controls and empirical testing, rather than from isolated checks or single metrics" [[251213907-assessing-high-risk-ai-systems-under-the-eu-ai-act-from-legal-requirem]]. The FDA's evolving position on Software as a Medical Device (SaMD) requires training reproducibility (version control, datasets, seed values, random splits, hyperparameter choices) but not output-level determinism, and the locked-vs-adaptive distinction explicitly permits adaptive models so long as they ship a Predetermined Change Control Plan (PCCP) [[fda-expectations-samd-2025-guide]]. The FDA's own admission: "The FDA's traditional paradigm of medical device regulation was not designed for adaptive artificial intelligence and machine learning technologies" [[fda-samd-ai-ml-overview]]. IMDRF's Good Machine Learning Practice requires "methodologically and statistically sound test plans" [[imdrf-gmlp-10-principles]] — a statistical mandate, not a determinism mandate. This produces an inversion that matters. The regulatory framing is *better aligned with Anthropic's CLT paper than with the dominant industry practice*. A vendor that ships single-run LLM-as-judge with no confidence intervals as its sole quality gate will struggle to satisfy "methodologically and statistically sound test plans" in any defensible reading — though Buscemi et al. concede that procedural controls (version control, human review, change-management documentation) can compensate for statistical-testing gaps, so vendors that pair single-run evals with rigorous procedural controls may pass audit while vendors that rely on single-run evals alone will not. A vendor that ships Anthropic-style CLT error bars, paired-differences tests, and power analysis does. A vendor that ships DeepEval-style DAGMetric routing does, too, on per-case CI/CD criteria. The regulatory mandate, in other words, is asking for what the field already has tools to produce — and is not asking for what the field cannot produce (output-level determinism in hosted inference). The non-adoption is therefore not a regulatory blocker but a vendor choice. The EU AI Act's enforcement timeline for high-risk systems begins August 2, 2026 — fifteen months out from this writing. Article 16's provider obligations and Article 17's quality management system requirements apply on that date. Annex IV documentation must exist. Vendors selling eval tooling into high-risk deployments will need to be able to defend their methodology under audit. The non-adoption of CLT and DAG patterns is therefore a regulatory liability that the field has not priced in. The common practitioner concern — "regulators will demand determinism we cannot provide" — is the wrong concern. The accurate concern is: regulators demand statistical rigor in test plans, and the single-run LLM-as-judge methodology that vendors default to does not provide statistical rigor. The deeper finding sits behind this gap. No vendor in the surveyed corpus draws the regulated-vs-product-context boundary explicitly. Anthropic's "building effective agents" examples are customer support and coding agents; Google's production guides do not distinguish healthcare from consumer apps; LangChain's checklist has no regulated-industry section; Braintrust's Notion case study is a productivity SaaS deployment; OpenAI's hosted evals product UI does not surface regulated-deployment options. Every vendor surveyed recommends the same distributional-quality methodology for customer support chatbots and (implicitly) for higher-stakes regulated deployments. The single-tier trust framework — distributional quality is sufficient everywhere — is itself a structural vendor choice. The field has not differentiated its own production-readiness standards by deployment risk context, which means a buyer in a regulated industry has to construct the regulated tier themselves from primitives the vendor did not segment. ### E. The Non-Adoption Paradox — Sophistication Exists, Adoption Does Not Both sophisticated solutions to the methodological problem are already published, documented, and shipping. Anthropic's CLT framework has been available since November 2024 — eighteen months at the time of writing. DeepEval's DAGMetric documentation has been live throughout 2025. The Inspect framework's `epochs` parameter implements CLT-correct resampling open source. Arize Phoenix's `repetitions` parameter has existed since v11.37.0. Eighteen months after the publication of Miller's CLT paper, the situation has not materially changed across the platform vendors. The most damning self-admission in the corpus is Anthropic's own concession in the CLT paper itself: "We are not aware of an open-source evals framework which implements this technique" — referring to next-token probability scoring, the paper's most powerful variance-reduction technique [[adding-error-bars-to-evals-a-statistical-approach-to-language-model-evaluations]]. The originator of the methodology acknowledges that even the open-source ecosystem has not implemented its strongest single component. The non-adoption paradox is not "others haven't caught up"; it is "even the originator has not shipped the full methodology." LangSmith's `repetitions` parameter exists but is framed as performance optimization. Weave's `trials` parameter exists but is framed as optional. Braintrust offers no multi-run aggregation in primary docs. Langfuse's LLM-as-judge docs do not mention judge inter-run variance. AgentOps does not engage the methodological question at all. Google Vertex AI's adaptive rubrics use LLM-as-judge with no published variance characterization. OpenAI's hosted evals product (distinct from the cookbooks) ships without TPR/TNR calibration as a default discipline. Across the field, the methodologically defensible patterns exist as opt-in features behind UX defaults that produce methodologically indefensible single-run scores. This is not a knowledge gap. Schroeder and Wood-Doughty published the omega quantification in December 2024. Miller published the CLT framework in November 2024. Atil and colleagues published the 70pp gap finding in August 2024. Bjarnason and colleagues published the 60,000-trajectory SWE-Bench study in February 2026. Thinking Machines Lab published the batch-invariance root cause analysis in September 2025. Khanal and colleagues published the long-horizon reliability framework in March 2026. Mustahsan and colleagues published the ICC variance decomposition in December 2025. The ICLR 2025 Blogposts analysis of GSM-Symbolic went further: papers with millions of social-media views made degradation claims that did not survive Wilson-score confidence interval analysis on 100-question datasets, demonstrating that the methodology gap is not solely a vendor problem — high-visibility academic work also fails basic CI scrutiny. The empirical record is dense, consistent, and accessible. The vendors have access to it. They have chosen not to redesign their defaults around it. A parallel enterprise-procurement definition of production-readiness has emerged in 2025 that runs on different rails from the methodology axes this report has surveyed: full-lifecycle coverage (Experiment → Simulate & Evaluate → Observe), OpenTelemetry-based tracing for agent and tool actions, human-in-the-loop review queues, and enterprise controls (RBAC, SSO, in-VPC deployment, SOC2/HIPAA posture) — used to compare platforms like Maxim AI, LangSmith, Langfuse, Arize Phoenix, Comet, Confident AI, and RAGAS [[top-agent-evaluation-tools-in-2025-best-platforms-for-reliable-enterprise-evals]]. These procurement axes do not overlap with the methodology axes: a vendor can score well on procurement (OpenTelemetry tracing, in-VPC, SOC2) and still ship single-run LLM-as-judge as its default quality gate. The two definitions of "production-ready" coexist without engaging each other. A second explanation worth naming for the methodological non-adoption: Miller's CLT framework assumes benchmark-scale paired-question designs that do not cleanly fit the custom-dataset / custom-scorer workflow most vendors expose, and Miller himself notes the most powerful technique (next-token probability scoring) has no open-source implementation. The non-adoption is therefore partially a tooling gap, not purely a commercial one — though the tooling gap has itself persisted eighteen months, which returns the question to incentives. The most parsimonious remaining explanation is competitive velocity pressure. Vendors compete on adoption metrics (developers signed up, integrations shipped, GitHub stars). Multi-run eval defaults increase compute cost per eval run by 5–10x. Confidence-interval reporting is a UX complication. TPR/TNR judge-calibration prerequisites slow time-to-first-eval. The vendor that ships honest methodology by default takes a velocity hit relative to the vendor that ships single-run by default. In a market where adoption is the dominant signal, the methodology gap is rational — for the vendor. It is not rational for the practitioner deploying production AI under regulatory exposure, and it will not be rational for the field once the EU AI Act's high-risk-system enforcement begins. ### F. Capability and Reliability Are Distinct Dimensions; the Field Measures Only One Even if the field repaired its single-run methodology — adopted CLT error bars, multi-sample aggregation, proper TPR/TNR judge calibration — it would still be measuring capability, not reliability, in long-horizon agent deployments. Khanal, Tao and Zhou (arXiv 2603.29231, March 2026) state the load-bearing claim: "Existing benchmarks measure capability — whether a model succeeds on a single attempt — but production deployments require reliability — consistent success across repeated attempts on tasks of varying duration. We show these properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to this divergence" [[260329231-beyond-pass1-a-reliability-science-framework-for-long-horizon-llm-agen]]. The empirical scope is decisive: 10 models, 23,392 episodes, 396-task benchmark spanning four duration buckets and three domains. The four reliability metrics — Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP) — are the first published reliability-science framework for long-horizon agents. The key findings: "(1) reliability decay is domain-stratified — SE GDS drops from 0.90 to 0.44 while document processing is nearly flat (0.74 to 0.71); (2) VAF bifurcates by capability tier — high VAF is a capability signature, not an instability signal; (3) capability and reliability rankings diverge substantially, with multi-rank inversions at long horizons; (4) frontier models have the highest meltdown rates (up to 19%) because they attempt ambitious multi-step strategies that sometimes spiral; and (5) memory scaffolds universally hurt long-horizon performance across all 10 models" [[260329231-beyond-pass1-a-reliability-science-framework-for-long-horizon-llm-agen]]. ReliabilityBench (Gupta, January 2026) confirms the production-stress framing empirically: success drops from 96.9% at ε=0 perturbation to 88.1% at ε=0.2, and "rate limiting is the most damaging fault in ablations" [[260106112-reliabilitybench-evaluating-llm-agent-reliability-under-production-lik]]. A replication caveat sits underneath the load-bearing claim: the Khanal et al. framework is a single arXiv preprint introducing four new metrics on a 396-task benchmark, has not been independently replicated in the corpus, and the multi-rank inversion finding may be partly an artifact of the benchmark's domain mix or task-correlation structure. The claim that pass@1 is "structurally blind" to long-horizon reliability is a strong reading of one study, not yet a field-validated paradigm. ReliabilityBench (Gupta, arXiv 260106112, January 2026) provides convergent directional evidence on production-stress degradation — which strengthens the reliability-is-a-distinct-axis claim — but the specific capability-vs-reliability ranking-inversion result needs replication before it becomes settled science. The implications for the vendor corpus are uncomfortable. The entire evaluation industry operates within the capability paradigm. Even Anthropic's CLT methodology, the field's most sophisticated statistical framing, is a refinement of capability measurement: it produces better confidence intervals around pass@1 estimates, but the estimand is still the probability of success on a single attempt. A model that scores well on short-task capability benchmarks may be deeply unreliable at long task horizons for structural reasons — meltdown cascades, memory scaffold interference — that pass@1 never surfaces. The "multi-rank inversions at long horizons" finding means that the leaderboard ranking under capability metrics can be reversed under reliability metrics on the same models. The capability/reliability blind spot is not a fixable bug in the current evaluation paradigm; it is a missing paradigm. The field has not yet developed reliability-first evaluation methodology. ### G. Forward-Looking Analysis: What Changes Next Three forcing functions could break the inertia in 2026–2027. First, the EU AI Act's August 2026 high-risk-system deadline creates a procurement-side requirement that vendors will have to satisfy or accept disqualification from those segments. A buyer subject to Annex IV cannot defensibly source an eval platform that ships single-run LLM-as-judge as its sole quality gate. The vendor that most credibly closes the gap between secondary-document honesty and primary-document default in the next twelve months will capture disproportionate regulated-domain market share. Phoenix's repetitions framing is the closest existing template; the vendor that adopts it as a primary-doc default — with prescribed minimum K, surfaced confidence intervals, and integrated judge calibration via TPR/TNR — wins the high-stakes deployment segment. Second, a high-profile public failure — a regression shipped to production that single-run evals missed and multi-run evals would have caught, attributed publicly to single-run methodology — would shift practitioner sentiment in a way that academic critique has not. A regulated-industry incident, such as an FDA-cleared medical AI deployment that fails a post-market surveillance audit because its single-run evaluation methodology did not detect a regression, would provide the public case study that the academic evidence has not yet provided. The probability of such an incident before August 2027 is non-trivial. Third, the capability-versus-reliability distinction will become operationally consequential as agentic deployments grow longer task horizons. The first vendor to ship a Reliability Decay Curve as a first-class evaluation output, alongside pass@1 metrics, will set the methodological template for long-horizon agent evaluation. PromptEvals (Vir, Shankar, Chase, Fu-Hinthorn, Parameswaran, April 2025) is an early signal of the open-source angle: a dataset of 2,087 LLM pipeline prompts with 12,623 corresponding assertion criteria, explicitly framed as "creating assertions or guardrails for LLM outputs to run alongside the pipelines is essential" for production reliability [[250414738-promptevals-a-dataset-of-assertions-and-guardrails-for-custom-producti]]. The structural pattern — pre-specified assertions verified per output rather than LLM judges scoring on continuous scales — points in the same direction as DAGMetric: structural elimination of judge stochasticity at the per-case gate by reducing the judge's task to discrete binary decisions. If this pattern matures into a default rather than a research artifact, the methodological floor of the field will rise. ### H. Where Determinism Is and Is Not a Trust Proxy Among the eleven vendors surveyed, *none* uses bitwise output-level determinism as the primary trust proxy in 2025–2026 — but three use weaker forms of determinism as load-bearing trust components, and the gradient matters. Anthropic, Arize Phoenix, and Weave actively disclaim output-level determinism. OpenAI ships parametric determinism via the `seed` and `system_fingerprint` API surface as a hedged ("best effort," "mostly consistent") reproducibility feature — the only major model provider that exposes a determinism primitive as a first-class API parameter. Google reduces non-deterministic surface area architecturally without claiming output determinism. DeepEval uses structural determinism for the DAGMetric specifically — applied to the *evaluator's* output via routed-decision paths, not the *evaluand's*. The rest of the field uses distributional quality (pass rates, mean scores, threshold gates) as the operative trust signal at the output layer. Where determinism *is* used as a trust proxy, in the broader 2025–2026 AI ecosystem outside platform vendors: in academic research using temperature=0 as an experimental isolation technique (with hedged language about residual variance), in RL training pipelines where on-policy correctness requires deterministic inference (Thinking Machines' motivating case), and in the practitioner subculture that propagates the temperature=0 myth via secondary guides like LangChain's primary evaluation doc. None of these uses survives close scrutiny. The academic uses acknowledge residual variance; the RL training case is narrow and tools-bound; the practitioner subculture is empirically refuted. The strongest single statement the field can make in 2025–2026 is that determinism is not the trust frame and was never going to be. The trust frame is statistical rigor under explicit acknowledgment of non-determinism — and the field has the tools for that framing and has not yet shipped them as defaults. The methodological crisis is therefore not "the wrong frame has won." The frame is right. The methodology has not caught up to the frame. ### Closing A field that has the critique (Schroeder and Wood-Doughty; Atil; Bjarnason; Mustahsan; Miller's CLT paper), the structural alternative (DAGMetric), the regulatory mandate (EU AI Act Annex IV; IMDRF GMLP), and convergent empirical evidence (Thinking Machines on batch non-invariance; Anthropic on variance shifts) — and chooses the convenient default anyway — is not suffering from ignorance. It is suffering from misaligned incentives between vendor velocity and methodological soundness. The August 2026 EU AI Act enforcement deadline is the first external forcing function that will price methodological rigor into vendor adoption. The vendors that adopt CLT error bars, DAG routing, ICC reporting, or equivalent structural mitigations before that deadline will own the regulated-AI segment. The vendors that ship single-run LLM-as-judge as their CI/CD gate will not. The current trajectory — eighteen months after Miller's CLT paper, with no platform adoption — predicts late, expensive, and grudging migration rather than considered methodology. The practical recommendation that follows: treat any single vendor's primary documentation as a partial account of reliability methodology. Cross-reference primary docs against secondary checklists from the same vendor. Treat the Anthropic CLT methodology and DeepEval DAGMetric as published alternatives the rest of the field has not yet caught up to. Treat the Phoenix repetitions framing as the standard the other vendors will eventually have to match. And treat the gap between vendor marketing and academic literature as a permanent feature of the field, not a temporary documentation lag.