What exactly are "Truly reproducible builds"?

Question

What exactly are they? Why are they important, in the domain of Continuous Delivery?

Context: I have seen in one of (I guess reddit)'s comments that Truly Reproducible builds are still an under-research technology, and is very difficult to create.

So, I wanted to know why are they so difficult to create?

Answer 1

What exactly are they?

Here is a quote from reproducible-builds.org:

Reproducible builds are a set of software development practices that create a verifiable path from human readable source code to the binary code used by computers.

Why are they important?

IMO the easiest way to explain their importance is to consider them as a variation of a backup procedure.

As an example:

• Assume a business which uses (depends) on some software package licensed from some software vendor. Whereas the business only gets the executables, not the sources, etc that were used to create those executables.

• Everything goes well, but at some point something goes wrong with the software vendor, e.g. they go out of business (eg bankrupcy).

• This may expose a risk to the business (in the long run). I.e. if there is no procedure/agreement in place for the business to get (legal) access to all required sources, documentation, build procedures, etc related to anything from the software vendor used (back in the days) when the executables (used by the business) were created (and shipped to the business).

• That's were "Software Escrow" comes to the rescue: if there is such agreement in place, one would think that via a 3rd party, it would still be possible for the business to get access to "whatever was used" to be able to reproduce the executables, so that from there on, the business might have a chance to continue to use that software, and where apropriate start maintaining it themselves (for only running their own business).

• However, the "whatever was used" in the previous bullet is the toughest part to make this work. It requires that the 3rd party performs appropriate validations upfront. And trust me, it takes a while before you can recreate an executable for which you can prove that, apart from (eg) the link date, it is a perfect match with what the software vendor delivers to the software agent.

And why are they so difficult to create?

If the above sample is still not clear enough, imagine you're my software escrow agent, tell me what you need as input to recreate a copy of the software licensed by my customer. Get it? You didn't forget to check about which version of my compiler, maybe my OS, compile/link options, versions of reusable components (includes), libraries, etc?

Answer 2

To provide a practical example of an attempt at creating a truly repeatable build consider the following -

A build pipeline which starts with a git repository for which no user can ever rewrite history or delete unmerged branches.

The first "build" step after checking out the source code is to spin up a container which contains all the build time dependencies.

The output of the build time container running is a container which contains the compiled binary.

More important to the repeatability of the build the following tags are added to the final container:

• The exact hash of the source code in the original repository and the url of both the git repo and a tar ball snapshot of the code that is uploaded to an artifact repository.
• The exact version of the build container that was used to run the build.
• The exact version of the original base image that the binary was loaded into.
• The values of all build-time variables used to create the binary.
• The version of docker that all three containers where built with as well as the version they where running under when they built.

By adding all of this meta data we can ensure that at any point in the future we can pull out the exact set of build dependencies (via the build container), compile the binary with an exact known set of steps (enshrined on the build container) and package this into another known base image with all run-time dependencies (using the base image tag) and this can all be based on the exact correct version of the source code based on the tag on the container.

Theoretically this should give us he ability to exactly reproduce a build version.

The importance of this is that it allows us to look at what's running in production and, even if everything has progressed versions significantly, go back and pull out the version of code, base image and build container originally used so that we can, for example, apply a hot fix to that version before rebuilding exactly as before so that we can redeploy knowing that it is exactly the same artifact with the only delta being the hot fix.