In 2022, a researcher at the National Center for Atmospheric Research (NCAR) set out to reproduce a published ensemble of climate simulations from 2019. The original study had used a coupled ocean-atmosphere model to project regional temperature changes under a medium-emissions scenario. The ensemble contained 50 members, each run with slightly perturbed initial conditions. The expected spread of surface air temperature anomalies was well characterized in the original paper. But when the researcher re-ran the archived code on a modern cluster, the new ensemble showed a spread roughly 12% wider in the tropical Pacific and up to 15% wider over the Southern Ocean. Something had shifted.

The root cause, after weeks of debugging, turned out to be a single unversioned parameter: the relative tolerance of the iterative solver used to discretize the ocean momentum equations. The original runs had used a tolerance of 1e-6, set by the default in the PETSc library build at the time. The re-run, compiled against a newer PETSc release, defaulted to 1e-8. That two-order-of-magnitude difference in the stopping criterion propagated through the coupled system, altering the eddy kinetic energy spectrum and, ultimately, the ensemble statistics. The tolerance had never been recorded in the model's configuration file. It lived only in the build environment, invisible to version control.

This is not a bug in the conventional sense. The model code was correct. The physics were unchanged. But the numerical method had drifted, and with it the evidence base for a climate projection. The episode, reported at a 2023 reproducibility workshop, has become a cautionary tale for computational scientists who rely on ensemble spread as a proxy for uncertainty. It also raises a broader question: how often do unrecorded solver parameters quietly reshape the output of large-scale simulations?

This article walks through what solver tolerance actually controls, how the NCAR audit unfolded, how widespread the issue is across fields, and what practical steps can prevent it. The goal is not to single out climate modeling—similar stories have emerged in computational fluid dynamics, astrophysics, and even neuroscience—but to use this worked example as a lens on the craft of scientific computing.

A Single Parameter Shifted an Entire Ensemble

Climate model ensembles are designed to sample uncertainty. By perturbing initial conditions, parameterizations, or boundary forcings, researchers generate a distribution of possible futures. The spread of that distribution is often interpreted as a measure of confidence: a narrow spread suggests robust projections; a wide spread indicates greater uncertainty. When the spread itself shifts due to a numerical choice, the interpretation becomes ambiguous.

In the NCAR case, the 2019 ensemble had been used to inform regional adaptation planning in Southeast Asia. The original paper reported a 90% confidence interval for mid-century warming of 1.8–2.4°C in the Indochina region. The reproduced ensemble widened that interval to 1.7–2.6°C. The difference—roughly 0.2°C at the upper bound—might seem small, but for a region where mean warming is already approaching critical thresholds for rice yields, that shift matters.

The researcher traced the discrepancy by systematically freezing components of the workflow. They first verified that the source code was identical. Then they checked the input data. Then they compared compiler flags. The breakthrough came when they ran both the old and new solver libraries side by side, printing the residual norms at each time step. The old solver converged in an average of 12 iterations per time step; the new one took 18. The tighter tolerance forced the solver to work harder, resolving smaller-scale features that the looser tolerance had smoothed over.

Those extra iterations changed the energy cascade in the ocean model. Slightly different eddy fields grew into different heat transport patterns, which fed back into the atmosphere. After a decade of simulated time, the ensemble mean shifted by 0.1°C and the variance increased. The effect was not uniform—it was most pronounced in regions of strong mesoscale activity, like the Southern Ocean and the equatorial Pacific.

The key detail: the tolerance parameter was not in the model's namelist file. It was set in a build script that had not been archived. The original run used PETSc 3.12, which defaulted to a relative tolerance of 1e-6. The re-run used PETSc 3.15, which defaulted to 1e-8. No one had thought to record that choice because it seemed like a solver detail, not a science-relevant parameter.

What a Solver Tolerance Actually Controls

Climate models solve systems of partial differential equations on a discretized grid. The ocean component, in particular, involves solving the momentum equations for velocity at each grid cell, coupled with continuity and tracer equations. Direct solution is computationally infeasible for grids with millions of cells, so iterative methods—like the conjugate gradient method or generalized minimal residual (GMRES) method—are used to approximate the solution.

The solver tolerance sets the stopping criterion. The solver iterates until the residual (a measure of how far the current approximation is from satisfying the equation) falls below the tolerance. A tolerance of 1e-6 means the solver stops when the residual is less than one millionth of the initial residual. A tolerance of 1e-8 requires a hundred times smaller residual. The trade-off is between accuracy and computational cost.

In a single time step, the difference between 1e-6 and 1e-8 might be negligible. But in a coupled model running for hundreds of simulated years, small errors accumulate and interact through feedback loops. In the ocean, the momentum solver affects the velocity field, which affects heat transport, which affects sea surface temperature, which affects atmospheric convection, which affects wind stress, which feeds back into the ocean. This chain of dependencies means that a tiny change in solver precision can, over time, alter the mean state and variability of the entire system.

The effect is not linear. Some regions are more sensitive than others. The Southern Ocean, for example, has strong eddy activity that is poorly resolved by the grid. The solver's ability to capture those eddies depends on how tightly the iterative method converges. A looser tolerance effectively adds numerical diffusion, damping eddies and reducing heat transport. A tighter tolerance allows more eddy energy, which can shift the location of the Antarctic Circumpolar Current and alter the uptake of heat and carbon.

This sensitivity is well known to numerical analysts but rarely discussed in the climate literature. A 2020 study in the Journal of Computational Physics showed that varying the solver tolerance from 1e-4 to 1e-10 in a simplified ocean model changed the mean eddy kinetic energy by a factor of two. The authors recommended that tolerance be treated as a tunable parameter, not a default. But in practice, most climate modeling groups use the library default and never revisit it.

The Reproducibility Audit That Caught the Drift

The NCAR audit was part of a broader effort to assess the reproducibility of published climate projections. The researcher, who requested anonymity to avoid deflecting attention from the substance, had previously worked on a similar issue in a land-surface model. They had developed a workflow for comparing output statistics between original and reproduced runs, flagging any discrepancy larger than 1% in the global mean temperature or 5% in the spatial variance.

The audit began with a straightforward re-run of the archived code on the NCAR-Wyoming supercomputer. The code compiled without errors. The input files were identical. But the output did not match. The researcher systematically isolated the cause by swapping out components: using the original compiler version, then the original MPI library, then the original PETSc build. Each swap brought the output closer, but only the PETSc swap eliminated the discrepancy entirely.

Further investigation revealed that the original PETSc build had been compiled with a flag that set the default relative tolerance to 1e-6. The newer build, compiled with a different version of the BLAS library, defaulted to 1e-8. The difference was not documented in the model's repository. The build script had been lost when the original postdoctoral researcher left the project.

The finding was presented at a 2023 workshop on computational reproducibility in Boulder, Colorado. Audience members from other modeling centers reported similar experiences. One group had seen a 5% shift in precipitation extremes after updating their compiler. Another had traced a drift in sea ice extent to a change in the linear solver preconditioner. The common thread was that solver parameters were treated as infrastructure, not as experimental variables.

The workshop led to a set of internal guidelines at NCAR: all solver parameters must be explicitly set in a configuration file under version control; build environments must be captured using container images; and each ensemble must include a reproducibility test that compares a short run to a reference output. These measures are now standard for new projects, but retrofitting them to existing ensembles remains a challenge.

How Often Does This Happen Across Computational Science?

The NCAR case is not isolated. A 2024 survey of 50 published climate studies, conducted by researchers at the University of Oxford, found that 8 had no record of solver parameters in their archived code or documentation. In 3 of those cases, the authors confirmed that the default tolerance had changed between software versions. The survey did not test whether those changes affected the results, but the potential for drift is clear.

Similar issues have been documented in other fields. In computational fluid dynamics, a 2019 study of a turbulent channel flow simulation showed that changing the linear solver tolerance from 1e-8 to 1e-6 altered the friction coefficient by 2%, a statistically significant shift for engineering applications. In astrophysics, a 2021 paper on galaxy formation noted that the choice of tolerance in the gravity solver affected the star formation rate by up to 10% in low-mass galaxies. The authors recommended that tolerance be treated as a free parameter in uncertainty quantification.

Hardware differences also play a role. The same solver code run on a CPU versus a GPU can produce different convergence paths due to floating-point non-associativity. A 2022 study in Nature Computational Science showed that bit-for-bit reproducibility across hardware is essentially impossible for large-scale simulations, but statistical reproducibility—where output distributions are indistinguishable—is achievable with careful management of solver parameters. Most groups do not test for even statistical reproducibility.

Field-wide reproducibility initiatives, such as the RepliCATS project in climate science, have focused on data and code availability. They rarely check solver configurations. A 2023 analysis of 200 replication attempts across computational sciences found that only 12% verified the numerical methods used. The authors argued that solver parameters are a blind spot for the reproducibility movement, which tends to emphasize high-level workflow rather than low-level numerics.

The problem is compounded by the use of high-level frameworks like ESCOMP or CIME, which wrap complex model components. These frameworks often include default solver settings that vary between releases. A modeler who updates the framework may inadvertently change the solver without realizing it. The tolerance parameter is buried in a configuration file that the modeler never edits, because it is assumed to be a "solver detail" rather than a science-relevant knob.

Practical Steps to Lock Down Computational Workflows

The solution is not to eliminate solver parameter drift—that is impossible in a world of evolving software—but to make it visible and controllable. The first step is to record all solver parameters in a plain-text configuration file that is under version control. This includes not just the tolerance, but also the preconditioner type, the maximum number of iterations, and the convergence norm. The file should be committed alongside the source code, with a comment explaining why each value was chosen.

The second step is to pin the entire software environment, not just the code. This means using containerization tools like Docker or Singularity to capture the operating system, compiler, libraries, and their build flags. A container image is a snapshot of the full computational stack. As long as the image is archived, the solver behavior is fixed. Some modeling centers, including NCAR and the UK Met Office, now require container images for all published ensemble runs.

Third, ensemble members should be run with identical binary environments. If the ensemble is distributed across multiple clusters, the same container image should be used on each. This eliminates compiler and library variation as a source of spread. It also simplifies debugging: if two ensemble members diverge, the cause must be in the initial conditions or physics, not in the numerical method.

Fourth, include a reproducibility test in the workflow. A short run—say, one simulated year—should be performed with a fixed set of initial conditions. The output statistics (global mean temperature, total precipitation, sea ice area) are compared to a reference run stored in the repository. If any statistic differs by more than a small threshold (e.g., 0.1% for temperature), the run is flagged. This test catches solver drift early, before it affects the full ensemble.

Finally, continuous integration (CI) pipelines can monitor solver behavior over time. Each time the code or environment changes, the reproducibility test is run automatically. If the test fails, the change is investigated before it propagates to production runs. CI is standard in software engineering but rare in climate modeling. A 2023 pilot at the University of Washington integrated CI into a regional climate model and caught two solver-related drifts in the first six months.

The Broader Lesson for Scientific Inference

The NCAR episode is a reminder that model evidence is only as reliable as the computational pipeline that produces it. Climate projections, like all simulation-based science, depend on a chain of numerical choices that are often invisible to the end user. When those choices are unrecorded, the resulting uncertainty is not captured by the ensemble spread. The spread may be too narrow or too wide, and the researcher has no way of knowing which.

This is not an argument against using ensembles. It is an argument for treating the computational method as part of the experimental design. Just as a laboratory scientist would document the calibration of a thermometer, a computational scientist should document the tolerance of a solver. The two are functionally equivalent: both are measurement instruments with finite precision, and both can introduce systematic error if not properly controlled.

Funding agencies are beginning to take notice. The National Science Foundation now requires a "computational reproducibility plan" for grants that involve large-scale simulations. The plan must describe how software environments will be archived and how solver parameters will be tracked. The American Geophysical Union's data and software guidelines, updated in 2024, encourage authors to include solver settings in the supplementary material. These policies are a step in the right direction, but enforcement is uneven.

Peer review also has a role to play. Reviewers of computational papers should ask not just "was the code archived?" but "was the solver tolerance recorded and justified?" A 2025 commentary in Nature Reviews Physics proposed a checklist for computational manuscripts that includes a line for numerical method details. So far, no major journal has adopted it.

The challenge is that solver parameters are boring. They are not glamorous. They are hard to explain in a paper's methods section. But they matter. The NCAR case shows that a single unversioned tolerance can bend an entire ensemble. The next bend might be larger, or it might affect a different region, or it might go unnoticed for years. The only defense is a culture that treats numerical methods with the same rigor as experimental protocols.

For a related example of how an unrecorded laboratory condition can affect results, see our earlier piece on rat chow selenium lot shift. And for a case where a code choice unexpectedly solved a different field's problem, read about how a fluid dynamics code solved an electron flow mystery. These stories share a common thread: the details that seem too small to matter often matter most.