Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: Add Rust to reprobuilds so we can bundle the cln-grpc plugin with it #5421

Merged
merged 6 commits into from
Jul 21, 2022
+1,689 −404
Merged

build: Add Rust to reprobuilds so we can bundle the cln-grpc plugin with it #5421

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b2bb02b
reprobuild: Add Rust compiler to repro build docker images
cdecker Jul 13, 2022
4b07bac
repro: Update ubuntu jammy reprobuild
cdecker Jul 13, 2022
354d8c3
py: Update poetry.lock using poetry update
cdecker Jul 13, 2022
25e39f1
rs: Strip binaries when compiling them for release
cdecker Jul 13, 2022
1c04a1f
rs: Add Cargo.lock for reproducible builds
cdecker Jul 21, 2022
6725d02
doc: Spell out the reprobuild instructions for each distro
cdecker Jul 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
@@ -72,5 +72,4 @@ tests/primitives_pb2_grpc.py

# Rust targets
target
Cargo.lock
plugins/cln-grpc
1,440 changes: 1,440 additions & 0 deletions Cargo.lock
View file

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
[profile.release]
strip = "debuginfo"

[workspace]
members = [
"cln-rpc",
8 changes: 7 additions & 1 deletion contrib/reprobuild/Dockerfile.bionic
View file
Original file line number Diff line number Diff line change
@@ -2,6 +2,8 @@ FROM bionic

ENV TZ=UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
ENV RUST_PROFILE=release
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mh! by following the what there is written in https://lightning.readthedocs.io/REPRODUCIBLE.html#builder-image-setup during the command exectution

sudo docker build -t cl-repro-bionic - < Dockerfile.bionic

i receive the following message

gitpod /workspace/tmp/lightning/contrib/reprobuild ((f9bcb2510...)) $ sudo docker build -t cl-repro-bionic - < Dockerfile.bionic
Sending build context to Docker daemon  3.584kB
Step 1/13 : FROM bionic
pull access denied for bionic, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

assuming that we need to update the build system? and you are not see this because you have the image already cloned?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to also build the base image, a couple of lines above: https://lightning.readthedocs.io/REPRODUCIBLE.html#base-image-creation

That will build the ubuntu bionic image for example using solely the installation medium contents (fetched from the web), and then we build on top of that known good state.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To get the tags you might have to git fetch --tags, but I thought it should work out of the box. If that doesn't work you can always git fetch cdecker refs/tags/v0.99.1:v0.99.1

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To get the tags you might have to git fetch --tags, but I thought it should work out of the box. If that doesn't work you can always git fetch cdecker refs/tags/v0.99.1:v0.99.1

Mh! I do not find any tag with this name on your repository https://github.com/cdecker/lightning/tags

Copy link
Member Author

@cdecker cdecker Jul 14, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, my bad:

git push origin --tags
Enumerating objects: 18, done.
Counting objects: 100% (18/18), done.
Delta compression using up to 8 threads
Compressing objects: 100% (18/18), done.
Writing objects: 100% (18/18), 13.00 KiB | 1.18 MiB/s, done.
Total 18 (delta 0), reused 17 (delta 0)
To github.com:cdecker/lightning.git
 * [new tag]             v0.99.1 -> v0.99.1

Please ignore my comments above, I forgot to push the tag 🤦

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, no problem :)

In addition, when I ran the following command
➜ lightning git:(repobuild/v0.99.1) ✗ sudo docker run --rm -v $(pwd):/repo -ti cl-repro-bionic

I receive the following error at the end
cp: target '/repo/release/' is not a directory

and I also noted that the docs contain a path different from what you posted in the PR

In particular, you posted

release/clightning-v0.99.1-Ubuntu-18.04.tar.xz

The doc contains

/repo/release/clightning-v0.9.0rc1-Ubuntu-18.04.tar.xz

ENV PATH=/root/.cargo/bin:/root/.pyenv/shims:/root/.pyenv/bin:$PATH

RUN sed -i '/updates/d' /etc/apt/sources.list && \
sed -i '/security/d' /etc/apt/sources.list
@@ -25,7 +27,6 @@ RUN apt-get update \
# Need to fetch a python version that is >= 3.7 since that's the
# lowest version supported by pyln. This is just temporary until we
# drop support for ubuntu 18.04
ENV PATH=/root/.pyenv/shims:/root/.pyenv/bin:$PATH
RUN git clone https://github.com/pyenv/pyenv.git /root/.pyenv \
&& apt-get install -y --no-install-recommends \
libbz2-dev \
@@ -41,6 +42,11 @@ RUN wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py && python3 /tmp
&& rm /tmp/get-pip.py \
&& pip install poetry

RUN wget https://sh.rustup.rs -O rustup-install.sh && \
bash rustup-install.sh --default-toolchain none --quiet -y && \
rm rustup-install.sh && \
/root/.cargo/bin/rustup install 1.62

RUN mkdir /build
WORKDIR /build

7 changes: 7 additions & 0 deletions contrib/reprobuild/Dockerfile.focal
View file
Original file line number Diff line number Diff line change
@@ -2,6 +2,8 @@ FROM focal

ENV TZ=UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
ENV RUST_PROFILE=release
ENV PATH=/root/.cargo/bin:$PATH

RUN sed -i '/updates/d' /etc/apt/sources.list && \
sed -i '/security/d' /etc/apt/sources.list
@@ -29,6 +31,11 @@ RUN wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py && python3 /tmp
&& rm /tmp/get-pip.py \
&& pip install poetry

RUN wget https://sh.rustup.rs -O rustup-install.sh && \
bash rustup-install.sh --default-toolchain none --quiet -y && \
rm rustup-install.sh && \
/root/.cargo/bin/rustup install 1.62

RUN mkdir /build
WORKDIR /build

10 changes: 9 additions & 1 deletion contrib/reprobuild/Dockerfile.jammy
View file
Original file line number Diff line number Diff line change
@@ -2,6 +2,8 @@ FROM jammy

ENV TZ=UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
ENV RUST_PROFILE=release
ENV PATH=/root/.cargo/bin:$PATH

RUN sed -i '/updates/d' /etc/apt/sources.list && \
sed -i '/security/d' /etc/apt/sources.list
@@ -14,7 +16,8 @@ RUN apt-get update \
file \
gettext \
git \
libgmp-dev \
libgmp-dev \
libsqlite3-dev \
libpq-dev \
libsodium23 \
libtool \
@@ -29,6 +32,11 @@ RUN wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py && python3 /tmp
&& rm /tmp/get-pip.py \
&& pip install poetry

RUN wget https://sh.rustup.rs -O rustup-install.sh && \
bash rustup-install.sh --default-toolchain none --quiet -y && \
rm rustup-install.sh && \
/root/.cargo/bin/rustup install 1.62

RUN mkdir /build
WORKDIR /build

8 changes: 6 additions & 2 deletions doc/REPRODUCIBLE.md
View file
Original file line number Diff line number Diff line change
@@ -70,7 +70,7 @@ The following table lists the codenames of distributions that we
currently support:

| Distribution Version | Codename |
|----------------------|----------|
|:---------------------|:---------|
| Ubuntu 18.04 | bionic |
| Ubuntu 20.04 | focal |
| Ubuntu 22.04 | jammy |
@@ -116,7 +116,9 @@ We can then build the builder image by calling `docker build` and passing it
the `Dockerfile`:

```bash
sudo docker build -t cl-repro-bionic - < Dockerfile.bionic
sudo docker build -t cl-repro-bionic - < contrib/reprobuild/Dockerfile.bionic
sudo docker build -t cl-repro-focal - < contrib/reprobuild/Dockerfile.focal
sudo docker build -t cl-repro-jammy - < contrib/reprobuild/Dockerfile.jammy
```

Since we pass the `Dockerfile` through `stdin` the build command will not
@@ -146,6 +148,8 @@ repository (remember to checkout the tag you are trying to build):

```bash
sudo docker run --rm -v $(pwd):/repo -ti cl-repro-bionic
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe this is a change that need to be done inside the docker container, but during the the process that start in this line I need to ran mkdir release and rerun the process again.

Maybe can be done inside the docker container or at least documented it?

sudo docker run --rm -v $(pwd):/repo -ti cl-repro-focal
sudo docker run --rm -v $(pwd):/repo -ti cl-repro-jammy
```

The last few lines of output also contain the `sha256sum` hashes of all
613 changes: 214 additions & 399 deletions poetry.lock
View file

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions tools/repro-build.sh
View file
Original file line number Diff line number Diff line change
@@ -161,6 +161,9 @@ d8b8653388e676a3ae2fcf565c2b1a42a01a1104062317f641e8d24f0eaff9c3 /var/cache/apt
572a544d2c18bf49d25c465720c570cd8e6e38731386ac9c0a7f29bed2486f3e /var/cache/apt/archives/m4_1.4.18-5ubuntu2_amd64.deb
080b79a1a1623a2e6c6eead37d62b15fdf2c3dbfeafe8ecf5e31c54eb09eadcc /var/cache/apt/archives/make_4.3-4.1build1_amd64.deb
52449467942cc943d651fd16867014e9339f3657935fc09b75b3347aa5a78066 /var/cache/apt/archives/zlib1g_1%3a1.2.11.dfsg-2ubuntu9_amd64.deb
5722d6ef8435a9dc3736e474040b4c7e6512b889ad9f74b6d52cdf11eec7e219 /var/cache/apt/archives/libsqlite3-dev_3.37.2-2_amd64.deb
ddbadadcbfe2669de79eabac36a990f0f1666bb86a87d1a9cd56fd72620ca2db /var/cache/apt/archives/zlib1g-dev_1%3a1.2.11.dfsg-2ubuntu9_amd64.deb
59e3890fc8407bcf8ccc9f709d6513156346d5c942e8c624dc90435e58f6f978 /var/cache/apt/archives/automake_1%3a1.16.5-1.3_all.deb
EOF
;;
*)