Skip to content

Commit

Permalink
Clarifying offline build process using gbuild --url and noting it is not
Browse files Browse the repository at this point in the history 
done automatically.

At some point along the line, fully offline builds were no longer happening
when strictly following the release-process.md instructions.

We should ensure that users who might want to torify or build offline need
to take extra steps to remain offline.

Also, corrections to build process: including gverify examples for new builders.
  • Loading branch information
@midnightmagic
midnightmagic committed Sep 17, 2015
1 parent 83f0e22 commit c73a8ea
Show file tree
Hide file tree
Showing 2 changed files with 106 additions and 21 deletions.
60 changes: 56 additions & 4 deletions doc/gitian-building.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -330,10 +330,11 @@ There will be a lot of warnings printed during the build of the image. These can
Getting and building the inputs
--------------------------------

Follow the instructions in [doc/release-process.md](release-process.md#fetch-and-build-inputs-first-time-or-when-dependency-versions-change)
in the bitcoin repository to install sources which require manual intervention. Also follow
the next step: 'Seed the Gitian sources cache', which will fetch all the necessary source
files to allow gitian to work offline.
Follow the instructions in [doc/release-process.md](release-process.md#fetch-and-build-inputs-first-time-or-when-dependency-versions-change)
in the bitcoin repository under 'Fetch and build inputs' to install sources which require
manual intervention. Also optionally follow the next step: 'Seed the Gitian sources cache
and offline git repositories' which will fetch the remaining files required for building
offline.

Building Bitcoin
----------------
Expand Down Expand Up @@ -391,6 +392,57 @@ COMMIT=2014_03_windows_unicode_path
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
```

Building fully offline
-----------------------

For building fully offline including attaching signatures to unsigned builds, the detached-sigs repository
and the bitcoin git repository with the desired tag must both be available locally, and then gbuild must be
told where to find them. It also requires an apt-cacher-ng which is fully-populated but set to offline mode, or
manually disabling gitian-builder's use of apt-get to update the VM build environment.

To configure apt-cacher-ng as an offline cacher, you will need to first populate its cache with the relevant
files. You must additionally patch target-bin/bootstrap-fixup to set its apt sources to something other than
plain archive.ubuntu.com: us.archive.ubuntu.com works.

So, if you use LXC:

```bash
export PATH="$PATH":/path/to/gitian-builder/libexec
export USE_LXC=1
cd /path/to/gitian-builder
./libexec/make-clean-vm --suite precise --arch amd64

LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root apt-get update
LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root \
-e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install \
$( sed -ne '/^packages:/,/[^-] .*/ {/^- .*/{s/"//g;s/- //;p}}' ../bitcoin/contrib/gitian-descriptors/*|sort|uniq )
LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root apt-get -q -y purge grub
LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
```

And then set offline mode for apt-cacher-ng:

```
/etc/apt-cacher-ng/acng.conf
[...]
Offlinemode: 1
[...]
service apt-cacher-ng restart
```

Then when building, override the remote URLs that gbuild would otherwise pull from the gitian descriptors::
```bash

cd /some/root/path/
git clone https://github.com/bitcoin/bitcoin-detached-sigs.git

BTCPATH=/some/root/path/bitcoin.git
SIGPATH=/some/root/path/bitcoin-detached-sigs.git

./bin/gbuild --url bitcoin=${BTCPATH},signature=${SIGPATH} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
```

Signing externally
-------------------

Expand Down
67 changes: 50 additions & 17 deletions doc/release-process.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,39 +6,54 @@ Release Process

* * *

###update (commit) version in sources
###first time only or for new builders, check out the source in the following directory hierarchy

cd /path/to/your/toplevel/build
git clone https://github.com/bitcoin/gitian.sigs.git
git clone https://github.com/devrandom/gitian-builder.git
git clone https://github.com/bitcoin/bitcoin.git

###for bitcoin maintainers/release engineers, update (commit) version in sources

pushd ./bitcoin
contrib/verifysfbinaries/verify.sh
doc/README*
share/setup.nsi
src/clientversion.h (change CLIENT_VERSION_IS_RELEASE to true)

###tag version in git
###for bitcoin maintainers/release engineers, tag version in git

git tag -s v(new version, e.g. 0.8.0)

###write release notes. git shortlog helps a lot, for example:
###for bitcoin maintainers/release engineers, write release notes. git shortlog helps a lot, for example:

git shortlog --no-merges v(current version, e.g. 0.7.2)..v(new version, e.g. 0.8.0)
popd

* * *

###update gitian

In order to take advantage of the new caching features in gitian, be sure to update to a recent version (`e9741525c` or later is recommended)
###update gitian, gitian.sigs, checkout bitcoin version, and perform gitian builds

###perform gitian builds

From a directory containing the bitcoin source, gitian-builder and gitian.sigs
To ensure your gitian descriptors are accurate for direct reference for gbuild, below, run the following from a directory containing the bitcoin source:

pushd ./bitcoin
export SIGNER=(your gitian key, ie bluematt, sipa, etc)
export VERSION=(new version, e.g. 0.8.0)
pushd ./bitcoin
git checkout v${VERSION}
popd

Ensure your gitian.sigs are up-to-date if you wish to gverify your builds against other gitian signatures:

pushd ./gitian.sigs
git pull
popd

Ensure your gitian-builder sources are up-to-date to take advantage of the new caching features of gitian (`e9741525c` or later is recommended)

pushd ./gitian-builder
git pull

###fetch and build inputs: (first time, or when dependency versions change)
###fetch and create inputs: (first time, or when dependency versions change)

mkdir -p inputs
wget -P inputs https://bitcoincore.org/cfields/osslsigncode-Backports-to-1.7.1.patch
Expand All @@ -52,28 +67,44 @@ Release Process

tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.9.sdk.tar.gz MacOSX10.9.sdk

###Optional: Seed the Gitian sources cache
###Optional: Seed the Gitian sources cache and offline git repositories

By default, gitian will fetch source files as needed. For offline builds, they can be fetched ahead of time:
By default, gitian will fetch source files as needed. To cache them ahead of time:

make -C ../bitcoin/depends download SOURCES_PATH=`pwd`/cache/common

Only missing files will be fetched, so this is safe to re-run for each build.
Only missing files will be fetched, so this is safe to re-run for each build.

Clone the detached-sigs repository:

popd
git clone https://github.com/bitcoin/bitcoin-detached-sigs.git
pushd ./bitcoin-builder

NOTE: Offline builds must use the --url flag to ensure gitian fetches only from local URLs.
For example: ./bin/bguild --url bitcoin=/path/to/bitcoin,signature=/path/to/sigs {rest of arguments}
The following gbuild invocations DO NOT DO THIS by default.

###Build Bitcoin Core for Linux, Windows, and OS X:
###Build (and optionally verify) Bitcoin Core for Linux, Windows, and OS X:

./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-linux --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-linux ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
mv build/out/bitcoin-*.tar.gz build/out/src/bitcoin-*.tar.gz ../

./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-win-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
mv build/out/bitcoin-*-win-unsigned.tar.gz inputs/bitcoin-win-unsigned.tar.gz
mv build/out/bitcoin-*.zip build/out/bitcoin-*.exe ../

./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-osx-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
mv build/out/bitcoin-*-osx-unsigned.tar.gz inputs/bitcoin-osx-unsigned.tar.gz
mv build/out/bitcoin-*.tar.gz build/out/bitcoin-*.dmg ../
popd

Build output expected:

1. source tarball (bitcoin-${VERSION}.tar.gz)
Expand All @@ -98,19 +129,21 @@ Commit your signature to gitian.sigs:
Once the Windows/OSX builds each have 3 matching signatures, they will be signed with their respective release keys.
Detached signatures will then be committed to the bitcoin-detached-sigs repository, which can be combined with the unsigned apps to create signed binaries.

Create the signed OSX binary:
Create (and optionally verify) the signed OSX binary:

pushd ./gitian-builder
./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
mv build/out/bitcoin-osx-signed.dmg ../bitcoin-${VERSION}-osx.dmg
popd

Create the signed Windows binaries:
Create (and optionally verify) the signed Windows binaries:

pushd ./gitian-builder
./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-win-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-signed ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
mv build/out/bitcoin-*win64-setup.exe ../bitcoin-${VERSION}-win64-setup.exe
mv build/out/bitcoin-*win32-setup.exe ../bitcoin-${VERSION}-win32-setup.exe
popd
Expand Down

0 comments on commit c73a8ea

Please sign in to comment.