Reproducible Builds and Insights from an Independent Verifier for Arch Linux

Esther H\"anggi; Iy\'an M\'endez Veiga; Joshua Drexel

REVIEW 3 major objections 5 minor 2 references

Reproducible Builds and Insights from an Independent Verifier for Arch Linux

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An independent Arch Linux rebuild verifier uncovered a security-relevant packaging bug that makes 16 Certbot packages unreproducible, and traced fwupd's unreproducibility to a gzip timestamp that an upstream patch now fixes.

desk verdict A modest but real contribution: one accepted upstream fix and a plausible packaging-bug report, undercut by thin artifacts and an overclaiming abstract. read the letter →

arxiv 2505.21642 v1 pith:7JP5HY2L submitted 2025-05-27 cs.CR

Joshua Drexel , Esther Hänggi , Iyán Méndez Veiga This is my paper

classification cs.CR

keywords reproduciblebuildssupplychainsecurityArchLinuxCertbotfwupdrebuilderdindependentverificationSBOM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that independent verifiers for reproducible builds are the missing link in supply-chain security, and demonstrates the argument by running one. The authors set up their own rebuilderd instance that rebuilds official Arch Linux packages from source and compares them bit-for-bit with the distributed binaries. Using its BAD results, they found a packaging flaw affecting 16 Certbot packages, making them unreproducible on Arch even though Debian's tests show the same software is reproducible there. They also traced unreproducibility in fwupd to a timestamp written by Python's gzip module and submitted an upstream patch that was accepted.

What carries the argument

The load-bearing object is the independent rebuilderd verifier: a sync daemon that polls Arch's official repositories every five minutes, queues package builds to workers, and labels results GOOD (bit-identical), BAD (unreproducible or failed), or UNKWN (untested). The two concrete bugs are found with diffoscope on BAD results: the Certbot issue is a version-race between pkgver() and .BUILDINFO, and the fwupd issue is a timestamp in the gzip header. The PKGBUILD build script and .BUILDINFO SBOM files are the precise artifacts that carry the mismatch.

What would settle it

Run the exact PKGBUILD scripts for the 16 Certbot packages in a clean Arch Linux build chroot using the official toolchain and archive, and check whether the produced artifact matches the distributed one and whether the .BUILDINFO python-acme version matches the actually resolved dependency. If the artifact matches despite the reported mismatch, or if the mismatch cannot be reproduced, the paper's central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that an independent rebuilder can expose real, security-relevant packaging defects that distribution-level testing missed. In the Certbot case the flaw is in the Arch Linux packaging workflow: the pkgver() function, which updates the package version from a git commit, was run before the version of a dependency (python-acme) was resolved, and the SBOM (.BUILDINFO) recorded the version from that earlier step. The resulting package depends on a python-acme version that does not match what the build actually used, making the 16 packages unreliable to reproduce and opening a window for malicious substitution. In the fwupd case, a gzip header timestamp written by a Python script rather than GNU gzip made the package non-reproducible; the paper's accepted upstream patch removes that timestamp, making fwupd reproducible in Arch and future releases elsewhere.

Load-bearing premise

The whole finding rests on the verifier's rebuild environment faithfully matching the official Arch Linux build environment; if the official build uses a different toolchain, timestamp, or archive, then a BAD result may not reflect the official artifact.

Editorial extensions

If this is right

If independent rebuilders become common, users can check whether a distributed binary provably matches audited source before running it.
The Certbot packaging flaw can be fixed either by patching the build toolchain to abort when PKGBUILD is modified after dependencies are installed, or by manual packaging changes; similar latent bugs will surface in other packages using the same pattern.
The fwupd upstream patch, once released, makes fwupd reproducible in Arch Linux and every other distribution that packages new releases.
With only a modest machine (e.g., an older i7 with 16GB RAM, plus workers) a verifier can cover the official Arch repository, lowering the barrier for more independent instances.
SBOMs (.BUILDINFO) are only trustworthy if they reflect what the build actually used; this case shows an SBOM can be inconsistent with the build inputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same pkgver()-after-dependency-resolution pattern likely affects other Arch packages where the version is derived from a git tag, so a toolchain check that forbids mutating PKGBUILD after dependency resolution would be a broadly valuable fix.
The failure of distribution CI to notice the Certbot bug despite having similar test infrastructure suggests that independent verifiers should be treated as complementary, not redundant, and that test coverage across environments matters.
A testable extension: run a rebuilderd-like verifier over archived old package versions to identify historical unreproducibility that may have already shipped to users; that would measure the actual exposure window.
Since the fwupd timestamp was written by Python's gzip rather than GNU gzip, upstream projects should test reproducibility of their Python-based release tooling; the class of bug is not specific to fwupd.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a solid workshop-level contribution to the reproducible-builds effort, not a breakthrough. The authors set up an independent rebuilderd verifier for Arch, found and reported a Certbot packaging defect affecting 16 packages, and fixed a real upstream unreproducibility in fwupd. That patch was accepted, which is the strongest piece of evidence in the paper.

What's genuinely new is the packaging failure mode: pkgver() mutating the package version after dependencies have been resolved, so the SBOM in .BUILDINFO disagrees with what the PKGBUILD would install on a later rebuild. That's a concrete, non-obvious way a package can become unreproducible, and the proposed toolchain guard (abort when a function modifies the build description) is reasonable. The fwupd root cause—a gzip timestamp embedded via Python's gzip module rather than GNU gzip—is exactly the kind of thing that slips through, and the accepted PR shows the fix is real.

Soft spots: the Certbot claim is under-documented. The paper gives no list of the 16 packages, no diffoscope logs, no timestamps, and no comparison with Arch's official reproducible status page. The stress-test's worry is fair: it is not clear whether the verifier reconstructs the official environment from .BUILDINFO or resolves dependencies from a current mirror. If it does the latter, the BAD verdicts could be dependency-drift false positives. The paper's own root-cause narrative leans that way, but it never states which behavior rebuilderd exhibits. That is a testable, should-be-answered question. The abstract overreaches: "effectively eradicate compromises" is not supported by anything in the paper; the conclusion itself says "mitigate." The survey chapters are competent but add little for anyone already familiar with the field.

Bottom line: the fwupd result alone justifies reading the paper. The Certbot finding is plausible and worth reporting, but it needs the missing artifacts to be checked. I would not cite this as a rigorous empirical study, but I would cite it as an example of an independent verifier surfacing real packaging defects. It deserves peer review—a serious referee could push the authors to ship logs and clarify the verifier's dependency resolution. Bring it to reading group if you care about supply-chain tooling.

Referee Report

3 major / 5 minor

Summary. The paper introduces reproducible builds (R-B) and bootstrappable builds (B-B), surveys their history and current state, and describes the authors' independent rebuilderd-based verifier for Arch Linux. The empirical contribution reports two findings: (1) 16 Certbot-related packages are declared unreproducible due to a packaging bug involving pkgver() and the .BUILDINFO metadata, and (2) the root cause of fwupd's unreproducibility is a timestamp introduced by Python's gzip module, with an accepted upstream patch. The paper argues that independent verifiers are valuable for supply-chain security and encourages broader adoption.

Significance. If the empirical claims are substantiated, the paper demonstrates a valuable use case: a modest independent verifier can surface real packaging defects missed by distribution-level testing. The fwupd finding is externally corroborated by an accepted upstream pull request, and the public verifier instance is a useful community resource. The Certbot finding, however, currently rests on an unvalidated rebuilderd configuration; the paper provides no build logs, diffoscope outputs, or package list, and the causal mechanism is under-specified. The paper's introductory and survey sections are competent but largely derivative. Overall, the contribution is potentially significant for the reproducible-builds community, but the central empirical claim needs stronger supporting evidence.

major comments (3)

[Section 6.1] The description of the rebuilderd setup does not include any validation that the BAD verdicts produced by the instance are true positives. The paper omits the list of the 16 Certbot packages, their diffoscope logs, the .BUILDINFO files, and any comparison against the official Arch Linux rebuilderd status (https://reproducible.archlinux.org). Without these artifacts, configuration drift (e.g., mirror lag, differing makepkg flags, or a nonstandard toolchain) cannot be excluded as the cause of the BAD verdicts. This evidence is load-bearing for the paper's central claim that the official Certbot packages are unreproducible.
[Section 6.2] The causal explanation of the Certbot issue is under-specified. The text states that pkgver() updates pkgver after dependencies have been installed and that .BUILDINFO records the older python-acme version, but it does not state how rebuilderd resolves build dependencies. If dependencies are resolved from the .BUILDINFO of the official package, the official environment is reconstructed and one would expect a reproducible artifact; if dependencies are resolved from the mutated PKGBUILD against a current mirror, the BAD verdict could be a false positive caused by dependency drift. The paper must clarify which behavior its instance exhibits and provide evidence from an affected build (e.g., a diffoscope excerpt or the relevant .BUILDINFO content) to confirm that the described mismatch actually underlies the BAD verdict.
[Abstract and Sections 2, 7] The abstract claims that R-B, B-B, and periodic audits 'can effectively eradicate compromises in the building process.' This statement is not supported by the paper itself, which in Section 2 notes that R-B does not make insecure software secure and in Section 3 explains that the compiler/toolchain remains a trust anchor (e.g., Thompson's trusting-trust attack). The claim should be qualified, for example to 'can substantially reduce the risk of compromises,' so that it does not overstate the paper's own analysis.

minor comments (5)

[Abstract] There is a typo: 'unreproduciblity' should be 'unreproducibility'.
[Section 6.2] The phrase 'a fixed and manually defined git commit is used by this function' is vague; please name the affected Certbot package and explain the exact version-determination workflow more concretely.
[Section 6.2] 'bash-alike script' should be 'bash-like script'.
[Figure 2] The diagram would be clearer if it distinguished the sync daemon from the rebuilderd daemon and defined the arrows (e.g., data flow versus control flow).
[Section 6.1] The URL https://reproducible.crypto-lab.ch appears in the body text but not in the reference list; consider adding it as a permanent artifact reference so that the verifier's outputs can be consulted by readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the report is an empirical systems study whose claims are measured against external artifacts and accepted upstream fixes, not derived from its own definitions.

full rationale

The paper does not contain a derivation chain of the kind the circularity check targets. Section 6.1 describes a rebuilderd instance that rebuilds official Arch Linux packages and compares them bit-for-bit with distributed binaries; a BAD verdict is an empirical measurement against the external Arch repository, not a quantity fitted from the paper's own outputs. The Certbot finding (Section 6.2) is an inference about a PKGBUILD/.BUILDINFO mismatch that is checked against Arch bugtracker context and Debian's reproducible-build status for python-certbot, and the fwupd finding (Section 6.3) is independently corroborated by an accepted upstream pull request. There are no fitted parameters, no predicted quantities defined in terms of their inputs, and no load-bearing self-citations: the authors cite external tools (rebuilderd, PKGBUILD(5), BUILDINFO(5), Debian test infrastructure, fwupd PR) rather than their own prior results. The reviewer-flagged weakness, that the setup may not faithfully reconstruct the official build environment, is a threat to external validity and to the strength of the Certbot claim, but it is not circularity: the paper does not define "unreproducible" as "BAD in my instance"; it uses that instance as a measurement instrument. Under the hard rule that circularity is only claimed when a specific reduction can be exhibited, no step qualifies.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its empirical findings rest on two domain assumptions: that the verifier environment matches the official build environment, and that the combination of R-B, B-B, and audits can effectively eradicate build-process compromises, the latter being an unsupported aspirational claim.

assumptions (2)

domain assumption A BAD result from the rebuilderd instance accurately indicates that the official Arch Linux package is unreproducible.
Section 6.1 describes the verifier setup; the entire contribution depends on the verifier's build environment matching the official build conditions closely enough that a failed rebuild implies a real defect in the official package.
ad hoc to paper Reproducible builds combined with bootstrappable builds and audits can effectively eradicate compromises in the building process.
Stated in the abstract and Section 7, this strong claim is not established by the paper and is in tension with Section 2's note that reproducible builds do not make insecure software secure.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Reproducible Builds and Insights from an Independent Verifier for Arch Linux." pith.science (2026). https://pith.science/paper/7JP5HY2L

@misc{pith2026250521642,
  author       = {Pith},
  title        = {Pith review of: Reproducible Builds and Insights from an Independent Verifier for Arch Linux},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7JP5HY2L}},
  note         = {Machine review of arXiv:2505.21642}
}

read the original abstract

Supply chain attacks have emerged as a prominent cybersecurity threat in recent years. Reproducible and bootstrappable builds have the potential to reduce such attacks significantly. In combination with independent, exhaustive and periodic source code audits, these measures can effectively eradicate compromises in the building process. In this paper we introduce both concepts, we analyze the achievements over the last ten years and explain the remaining challenges. We contribute to the reproducible builds effort by setting up a rebuilder and verifier instance to test the reproducibility of Arch Linux packages. Using the results from this instance, we uncover an unnoticed and security-relevant packaging issue affecting 16 packages related to Certbot, the recommended software to install TLS certificates from Let's Encrypt, making them unreproducible. Additionally, we find the root cause of unreproduciblity in the source code of fwupd, a critical software used to update device firmware on Linux devices, and submit an upstream patch to fix it.

Figures

Figures reproduced from arXiv: 2505.21642 by the authors.

**Figure 2.** The rebuilderd daemon requires few resources and can be run on a small cloud instance (e.g., 1 vCPU and 1 GB of RAM). Our independent verifier (sync daemon and a small worker) runs on a modest refurbish PC with an Intel i7-4790K CPU and 16 GB of RAM.Workers need at least 32 GB of RAM to be ableto buildallthe official packages.A significant speedup onlarge builds (e.g.linux kernel,web browsers, etc.) can be achieved … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

2 extracted references

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...