Skip to content

Commit

Permalink
Debian packaging via dh_virtualenv (matrix-org#4285)
Browse files Browse the repository at this point in the history
  • Loading branch information
richvdh committed Dec 20, 2018
1 parent fd4070a commit 1a6d5bf
Show file tree
Hide file tree
Showing 36 changed files with 310 additions and 509 deletions.
2 changes: 2 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,5 @@ demo/etc
tox.ini
.git/*
.tox/*
debian/matrix-synapse/
debian/matrix-synapse-*/
8 changes: 7 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,16 @@ homeserver*.db
homeserver*.log
homeserver*.log.*
homeserver*.pid
homeserver*.yaml
/homeserver*.yaml

*.signing.key
*.tls.crt
*.tls.dh
*.tls.key

.coverage
.coverage.*
!.coverage.rc
htmlcov

demo/*/*.db
Expand Down Expand Up @@ -57,3 +59,7 @@ env/

.vscode/
.ropeproject/

*.deb

/debs
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ prune demo/etc
prune docker
prune .circleci
prune .coveragerc
prune debian

exclude jenkins*
recursive-exclude jenkins *.sh
1 change: 1 addition & 0 deletions changelog.d/4212.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Debian packages utilising a virtualenv with bundled dependencies can now be built.
7 changes: 7 additions & 0 deletions debian/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
/matrix-synapse-py3.*.debhelper
/matrix-synapse-py3.debhelper.log
/matrix-synapse-py3.substvars
/matrix-synapse-*/
/files
/debhelper-build-stamp
/.debhelper
22 changes: 22 additions & 0 deletions debian/NEWS
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
matrix-synapse-py3 (0.34.0) stable; urgency=medium

matrix-synapse-py3 is intended as a drop-in replacement for the existing
matrix-synapse package. The replacement should be relatively seamless,
however, please note the following important differences to matrix-synapse:

* Most importantly, the matrix-synapse service now runs under Python 3 rather
than Python 2.7.

* Synapse is installed into its own virtualenv (in /opt/venvs/matrix-synapse)
instead of using the system python libraries. (This may mean that you can
remove a number of old dependencies with `apt-get autoremove`).

matrix-synapse-py3 will take over responsibility for the existing
configuration files, including the matrix-synapse systemd service.

Beware, however, that `apt-get purge matrix-synapse` will *disable* the
matrix-synapse service (so that it will not be started on reboot), even
though that service is no longer being provided by the matrix-synapse
package. It can be re-enabled with `systemctl enable matrix-synapse`.

-- Richard van der Hoff <[email protected]> Wed, 19 Dec 2018 14:00:00 +0000
48 changes: 48 additions & 0 deletions debian/build_virtualenv
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/bin/bash
#
# runs dh_virtualenv to build the virtualenv in the build directory,
# and then runs the trial tests against the installed synapse.

set -e

export DH_VIRTUALENV_INSTALL_ROOT=/opt/venvs
SNAKE=/usr/bin/python3

# try to set the CFLAGS so any compiled C extensions are compiled with the most
# generic as possible x64 instructions, so that compiling it on a new Intel chip
# doesn't enable features not available on older ones or AMD.
#
# TODO: add similar things for non-amd64, or figure out a more generic way to
# do this.

case `dpkg-architecture -q DEB_HOST_ARCH` in
amd64)
export CFLAGS=-march=x86-64
;;
esac

# Use --builtin-venv to use the better `venv` module from CPython 3.4+ rather
# than the 2/3 compatible `virtualenv`.

dh_virtualenv \
--install-suffix "matrix-synapse" \
--builtin-venv \
--setuptools \
--python "$SNAKE" \
--upgrade-pip \
--preinstall="lxml" \
--preinstall="mock" \
--extra-pip-arg="--no-cache-dir" \
--extra-pip-arg="--compile"

# we copy the tests to a temporary directory so that we can put them on the
# PYTHONPATH without putting the uninstalled synapse on the pythonpath.
tmpdir=`mktemp -d`
trap "rm -r $tmpdir" EXIT

cp -r tests "$tmpdir"
cd debian/matrix-synapse-py3

PYTHONPATH="$tmpdir" \
./opt/venvs/matrix-synapse/bin/python \
-B -m twisted.trial --reporter=text -j2 tests
11 changes: 11 additions & 0 deletions debian/changelog
Original file line number Diff line number Diff line change
@@ -1,3 +1,14 @@
matrix-synapse-py3 (0.34.0) stable; urgency=medium

* New synapse release 0.34.0.
* Synapse is now installed into a Python 3 virtual environment with
up-to-date dependencies.
* The matrix-synapse service will now be restarted when the package is
upgraded.
(Fixes https://github.com/matrix-org/package-synapse-debian/issues/18)

-- Synapse packaging team <[email protected]> Wed, 19 Dec 2018 14:00:00 +0000

matrix-synapse (0.33.9-1matrix1) stretch; urgency=medium

[ Erik Johnston ]
Expand Down
92 changes: 26 additions & 66 deletions debian/control
Original file line number Diff line number Diff line change
@@ -1,77 +1,37 @@
Source: matrix-synapse
Maintainer: Erik Johnston <[email protected]>
Section: python
Priority: optional
Source: matrix-synapse-py3
Section: contrib/python
Priority: extra
Maintainer: Synapse Packaging team <[email protected]>
Build-Depends:
debhelper (>= 9),
dh-python,
dh-systemd (>= 1.5),
po-debconf,
python (>= 2.6.6-3),
python-bcrypt,
python-blist,
python-canonicaljson (>=1.1.3),
python-daemonize,
python-frozendict (>= 0.4),
python-lxml,
python-mock,
python-msgpack (>=0.3.0),
python-nacl (>= 0.3.0),
python-netaddr,
python-openssl (>= 0.14),
python-pil,
python-psutil,
python-pyasn1,
python-pydenticon,
python-pymacaroons-pynacl,
python-pysaml2,
python-service-identity (>= 1.0.0),
python-setuptools (>= 0.6b3),
python-signedjson (>= 1.0.0),
python-sortedcontainers,
python-syutil (>= 0.0.7),
python-treq (>= 15.1.0),
python-twisted (>= 17.1.0),
python-unpaddedbase64 (>= 1.0.1),
python-yaml,
python-phonenumbers (>= 8.2.0),
python-jsonschema (>=2.5.1),
python-prometheus-client,
python-attr
Standards-Version: 3.9.8
X-Python-Version: >= 2.7
dh-systemd,
dh-virtualenv (>= 1.0),
lsb-release,
python3-dev,
python3,
python3-setuptools,
python3-pip,
python3-venv,
tar,
Standards-Version: 3.9.5
Homepage: https://github.com/matrix-org/synapse

Package: matrix-synapse
Architecture: all
Package: matrix-synapse-py3
Architecture: amd64
Conflicts: matrix-synapse
Pre-Depends: dpkg (>= 1.16.1)
Depends:
${misc:Depends},
${python:Depends},
adduser,
debconf,
lsb-base (>= 3.0-6),
python-attr (>= 16.0.0),
python-twisted (>= 17.1.0),
python-canonicaljson (>=1.1.3),
python-prometheus-client (>=0.0.14),
python3-distutils|libpython3-stdlib (<< 3.6),
python3,
${misc:Depends},
# some of our scripts use perl, but none of them are important,
# so we put perl:Depends in Suggests rather than Depends.
Suggests:
python-bleach (>= 1.4.2),
python-jinja2 (>= 2.8),
Recommends:
python-psycopg2,
python-lxml,
sqlite3,
${perl:Depends},
Description: Open federated Instant Messaging and VoIP server
Matrix is an ambitious new ecosystem for open federated Instant
Messaging and VoIP. Synapse is a reference Matrix server
implementation.
.
Everything in Matrix happens in a room. Rooms are distributed and do
not exist on any single server. Rooms can be located using
convenience aliases like #matrix:matrix.org or #test:localhost:8448.
.
Matrix user IDs look like @matthew:matrix.org (although in the future
you will normally refer to yourself and others using a 3PID: email
address, phone number, etc rather than manipulating Matrix user IDs)
.
The overall architecture is:
client <------> homeserver <=============> homeserver <------> client
https://a.org/_matrix https://b.net/_matrix
2 changes: 1 addition & 1 deletion debian/copyright
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ Upstream-Name: synapse
Source: https://github.com/matrix-org/synapse

Files: *
Copyright: 2014-2017, OpenMarket Ltd
Copyright: 2014-2017, OpenMarket Ltd, 2017-2018 New Vector Ltd
License: Apache-2.0

Files: synapse/config/saml2.py
Expand Down
5 changes: 0 additions & 5 deletions debian/gbp.conf

This file was deleted.

10 changes: 3 additions & 7 deletions debian/homeserver.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -70,13 +70,9 @@ pid_file: "/var/run/matrix-synapse.pid"
#
# cpu_affinity: 0xFFFFFFFF

# Whether to serve a web client from the HTTP/HTTPS root resource.
web_client: False

# The root directory to server for the above web client.
# If left undefined, synapse will serve the matrix-angular-sdk web client.
# Make sure matrix-angular-sdk is installed with pip if web_client is True
# and web_client_location is undefined
# The path to the web client which will be served at /_matrix/client/
# if 'webclient' is configured under the 'listeners' configuration.
#
# web_client_location: "/path/to/web/root"

# The public-facing base URL for the client API (not including _matrix/...)
Expand Down
4 changes: 4 additions & 0 deletions debian/matrix-synapse-py3.links
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
opt/venvs/matrix-synapse/bin/hash_password usr/bin/hash_password
opt/venvs/matrix-synapse/bin/register_new_matrix_user usr/bin/register_new_matrix_user
opt/venvs/matrix-synapse/bin/synapse_port_db usr/bin/synapse_port_db
opt/venvs/matrix-synapse/bin/synctl usr/bin/synctl
File renamed without changes.
31 changes: 31 additions & 0 deletions debian/matrix-synapse-py3.preinst
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/bin/sh -e

# Attempt to undo some of the braindamage caused by
# https://github.com/matrix-org/package-synapse-debian/issues/18.
#
# Due to reasons [1], the old python2 matrix-synapse package will not stop the
# service when the package is uninstalled. Our maintainer scripts will do the
# right thing in terms of ensuring the service is enabled and unmasked, but
# then do a `systemctl start matrix-synapse`, which of course does nothing -
# leaving the old (py2) service running.
#
# There should normally be no reason for the service to be running during our
# preinst, so we assume that if it *is* running, it's due to that situation,
# and stop it.
#
# [1] dh_systemd_start doesn't do anything because it sees that there is an
# init.d script with the same name, so leaves it to dh_installinit.
#
# dh_installinit doesn't do anything because somebody gave it a --no-start
# for unknown reasons.

if [ -x /bin/systemctl ]; then
if /bin/systemctl --quiet is-active -- matrix-synapse; then
echo >&2 "stopping existing matrix-synapse service"
/bin/systemctl stop matrix-synapse || true
fi
fi

#DEBHELPER#

exit 0
9 changes: 9 additions & 0 deletions debian/matrix-synapse-py3.triggers
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Register interest in Python interpreter changes and
# don't make the Python package dependent on the virtualenv package
# processing (noawait)
interest-noawait /usr/bin/python3.5
interest-noawait /usr/bin/python3.6
interest-noawait /usr/bin/python3.7

# Also provide a symbolic trigger for all dh-virtualenv packages
interest dh-virtualenv-interpreter-update
Loading

0 comments on commit 1a6d5bf

Please sign in to comment.