A Decade of Software Reproducibility in the Nix Package Ecosystem
Abstract
We report a large-scale empirical study of two aspects of software reproducibility — rebuildability and bitwise reproducibility — in the Nix package ecosystem. Using 29 evenly spaced historical snapshots over a decade of history of the nixpkgs repository (2015-2024) we attempted to rebuild and bitwise-compare the build outputs of tens of thousands of packages per snapshot. Our experiment produced a dataset of build metadata and logs for 1 321 000 package builds and preserved 166 523 diffoscopes for unreproducible outputs. We find that functional package management enables extremely high rebuildability over time (near-universal ability to reconstitute historical build environments and rebuild software packages), while bitwise reproducibility has steadily improved and reaches a high point in recent years (up to 93% in 2024). Early years show substantially lower bitwise reproducibility, indicating that functional package management alone does not guarantee bitwise-identical outputs, and that the observed high level of bitwise reproducibility is not solely due to the package management approach. Common causes of unreproducibility, both in the rebuildability and bitwise reproducibility dimensions, include management of dates in build and test processes; we quantify their prevalence and other common causes using manual analysis of logs of rebuild failures and automated analysis of diffoscopes.