**Warning**: This release removes the example email notification temp…

…lates from

`res/templates` (they are now internal to the python package). This should only
affect you if you (a) deploy your Synapse instance from a git checkout or a
github snapshot URL, and (b) have email notifications enabled.

If you have email notifications enabled, you should ensure that
`email.template_dir` is either configured to point at a directory where you
have installed customised templates, or leave it unset to use the default
templates.

The configuration parser will try to detect the situation where
`email.template_dir` is incorrectly set to `res/templates` and do the right
thing, but will warn about this.

Features
--------

- Ship the example email templates as part of the package ([\matrix-org#4052](matrix-org#4052))
- Add support for end-to-end key backup (MSC1687) ([\matrix-org#4019](matrix-org#4019))

Bugfixes
--------

- Fix bug which made get_missing_events return too few events ([\matrix-org#4045](matrix-org#4045))
- Fix bug in event persistence logic which caused 'NoneType is not iterable' ([\matrix-org#3995](matrix-org#3995))
- Fix exception in background metrics collection ([\matrix-org#3996](matrix-org#3996))
- Fix exception handling in fetching remote profiles ([\matrix-org#3997](matrix-org#3997))
- Fix handling of rejected threepid invites ([\matrix-org#3999](matrix-org#3999))
- Workers now start on Python 3. ([\matrix-org#4027](matrix-org#4027))
- Synapse now starts on Python 3.7. ([\matrix-org#4033](matrix-org#4033))

Internal Changes
----------------

- Log exceptions in looping calls ([\matrix-org#4008](matrix-org#4008))
- Optimisation for serving federation requests ([\matrix-org#4017](matrix-org#4017))
- Add metric to count number of non-empty sync responses ([\matrix-org#4022](matrix-org#4022))

Features

--------

- Ship the example email templates as part of the package ([\matrix-org#4052](matrix-org#4052))

Bugfixes
--------

- Fix bug which made get_missing_events return too few events ([\matrix-org#4045](matrix-org#4045))

Features

--------

- Add support for end-to-end key backup (MSC1687) ([\matrix-org#4019](matrix-org#4019))

Bugfixes
--------

- Fix bug in event persistence logic which caused 'NoneType is not iterable' ([\matrix-org#3995](matrix-org#3995))
- Fix exception in background metrics collection ([\matrix-org#3996](matrix-org#3996))
- Fix exception handling in fetching remote profiles ([\matrix-org#3997](matrix-org#3997))
- Fix handling of rejected threepid invites ([\matrix-org#3999](matrix-org#3999))
- Workers now start on Python 3. ([\matrix-org#4027](matrix-org#4027))
- Synapse now starts on Python 3.7. ([\matrix-org#4033](matrix-org#4033))

Internal Changes
----------------

- Log exceptions in looping calls ([\matrix-org#4008](matrix-org#4008))
- Optimisation for serving federation requests ([\matrix-org#4017](matrix-org#4017))
- Add metric to count number of non-empty sync responses ([\matrix-org#4022](matrix-org#4022))

Features

--------

- Adding the ability to change MAX_UPLOAD_SIZE for the docker container
variables. ([\matrix-org#3883](matrix-org#3883))
- Report "python_version" in the phone home stats
([\matrix-org#3894](matrix-org#3894))
- Always LL ourselves if we're in a room
([\matrix-org#3916](matrix-org#3916))
- Include eventid in log lines when processing incoming federation
transactions
([\matrix-org#3959](matrix-org#3959))
- Remove spurious check which made 'localhost' servers not work
([\matrix-org#3964](matrix-org#3964))

Bugfixes
--------

- Fix problem when playing media from Chrome using direct URL (thanks
@remjey!) ([\matrix-org#3578](matrix-org#3578))
- support registering regular users non-interactively with
register_new_matrix_user script
([\matrix-org#3836](matrix-org#3836))
- Fix broken invite email links for self hosted riots
([\matrix-org#3868](matrix-org#3868))
- Don't ratelimit autojoins
([\matrix-org#3879](matrix-org#3879))
- Fix 500 error when deleting unknown room alias
([\matrix-org#3889](matrix-org#3889))
- Fix some b'abcd' noise in logs and metrics
([\matrix-org#3892](matrix-org#3892),
[\matrix-org#3895](matrix-org#3895))
- When we join a room, always try the server we used for the alias
lookup first, to avoid unresponsive and out-of-date servers.
([\matrix-org#3899](matrix-org#3899))
- Fix incorrect server-name indication for outgoing federation requests
([\matrix-org#3907](matrix-org#3907))
- Fix adding client IPs to the database failing on Python 3.
([\matrix-org#3908](matrix-org#3908))
- Fix bug where things occaisonally were not being timed out correctly.
([\matrix-org#3910](matrix-org#3910))
- Fix bug where outbound federation would stop talking to some servers
when using workers
([\matrix-org#3914](matrix-org#3914))
- Fix some instances of ExpiringCache not expiring cache items
([\matrix-org#3932](matrix-org#3932),
[\matrix-org#3980](matrix-org#3980))
- Fix out-of-bounds error when LLing yourself
([\matrix-org#3936](matrix-org#3936))
- Sending server notices regarding user consent now works on Python 3.
([\matrix-org#3938](matrix-org#3938))
- Fix exceptions from metrics handler
([\matrix-org#3956](matrix-org#3956))
- Fix error message for events with m.room.create missing from
auth_events
([\matrix-org#3960](matrix-org#3960))
- Fix errors due to concurrent monthly_active_user upserts
([\matrix-org#3961](matrix-org#3961))
- Fix exceptions when processing incoming events over federation
([\matrix-org#3968](matrix-org#3968))
- Replaced all occurences of e.message with str(e). Contributed by
Schnuffle ([\matrix-org#3970](matrix-org#3970))
- Fix lazy loaded sync in the presence of rejected state events
([\matrix-org#3986](matrix-org#3986))
- Fix error when logging incomplete HTTP requests
([\matrix-org#3990](matrix-org#3990))

Internal Changes
----------------

- Unit tests can now be run under PostgreSQL in Docker using
``test_postgresql.sh``.
([\matrix-org#3699](matrix-org#3699))
- Speed up calculation of typing updates for replication
([\matrix-org#3794](matrix-org#3794))
- Remove documentation regarding installation on Cygwin, the use of WSL
is recommended instead.
([\matrix-org#3873](matrix-org#3873))
- Fix typo in README, synaspse -> synapse
([\matrix-org#3897](matrix-org#3897))
- Increase the timeout when filling missing events in federation
requests ([\matrix-org#3903](matrix-org#3903))
- Improve the logging when handling a federation transaction
([\matrix-org#3904](matrix-org#3904),
[\matrix-org#3966](matrix-org#3966))
- Improve logging of outbound federation requests
([\matrix-org#3906](matrix-org#3906),
[\matrix-org#3909](matrix-org#3909))
- Fix the docker image building on python 3
([\matrix-org#3911](matrix-org#3911))
- Add a regression test for logging failed HTTP requests on Python 3.
([\matrix-org#3912](matrix-org#3912))
- Comments and interface cleanup for on_receive_pdu
([\matrix-org#3924](matrix-org#3924))
- Fix spurious exceptions when remote http client closes conncetion
([\matrix-org#3925](matrix-org#3925))
- Log exceptions thrown by background tasks
([\matrix-org#3927](matrix-org#3927))
- Add a cache to get_destination_retry_timings
([\matrix-org#3933](matrix-org#3933),
[\matrix-org#3991](matrix-org#3991))
- Automate pushes to docker hub
([\matrix-org#3946](matrix-org#3946))
- Require attrs 16.0.0 or later
([\matrix-org#3947](matrix-org#3947))
- Fix incompatibility with python3 on alpine
([\matrix-org#3948](matrix-org#3948))
- Run the test suite on the oldest supported versions of our
dependencies in CI.
([\matrix-org#3952](matrix-org#3952))
- CircleCI now only runs merged jobs on PRs, and commit jobs on develop,
master, and release branches.
([\matrix-org#3957](matrix-org#3957))
- Fix docstrings and add tests for state store methods
([\matrix-org#3958](matrix-org#3958))
- fix docstring for FederationClient.get_state_for_room
([\matrix-org#3963](matrix-org#3963))
- Run notify_app_services as a bg process
([\matrix-org#3965](matrix-org#3965))
- Clarifications in FederationHandler
([\matrix-org#3967](matrix-org#3967))
- Further reduce the docker image size
([\matrix-org#3972](matrix-org#3972))
- Build py3 docker images for docker hub too
([\matrix-org#3976](matrix-org#3976))
- Updated the installation instructions to point to the matrix-synapse
package on PyPI.
([\matrix-org#3985](matrix-org#3985))
- Disable USE_FROZEN_DICTS for unittests by default.
([\matrix-org#3987](matrix-org#3987))
- Remove unused Jenkins and development related files from the repo.
([\matrix-org#3988](matrix-org#3988))
- Improve stacktraces in certain exceptions in the logs
([\matrix-org#3989](matrix-org#3989))
- Pin to prometheus_client<0.4 to avoid renaming all of our metrics
([\matrix-org#4002](matrix-org#4002))

Features

--------

- Adding the ability to change MAX_UPLOAD_SIZE for the docker container variables. ([\matrix-org#3883](matrix-org#3883))
- Report "python_version" in the phone home stats ([\matrix-org#3894](matrix-org#3894))
- Always LL ourselves if we're in a room ([\matrix-org#3916](matrix-org#3916))
- Include eventid in log lines when processing incoming federation transactions ([\matrix-org#3959](matrix-org#3959))
- Remove spurious check which made 'localhost' servers not work ([\matrix-org#3964](matrix-org#3964))

Bugfixes
--------

- Fix problem when playing media from Chrome using direct URL (thanks @remjey!) ([\matrix-org#3578](matrix-org#3578))
- support registering regular users non-interactively with register_new_matrix_user script ([\matrix-org#3836](matrix-org#3836))
- Fix broken invite email links for self hosted riots ([\matrix-org#3868](matrix-org#3868))
- Don't ratelimit autojoins ([\matrix-org#3879](matrix-org#3879))
- Fix 500 error when deleting unknown room alias ([\matrix-org#3889](matrix-org#3889))
- Fix some b'abcd' noise in logs and metrics ([\matrix-org#3892](matrix-org#3892), [\matrix-org#3895](matrix-org#3895))
- When we join a room, always try the server we used for the alias lookup first, to avoid unresponsive and out-of-date servers. ([\matrix-org#3899](matrix-org#3899))
- Fix incorrect server-name indication for outgoing federation requests ([\matrix-org#3907](matrix-org#3907))
- Fix adding client IPs to the database failing on Python 3. ([\matrix-org#3908](matrix-org#3908))
- Fix bug where things occaisonally were not being timed out correctly. ([\matrix-org#3910](matrix-org#3910))
- Fix bug where outbound federation would stop talking to some servers when using workers ([\matrix-org#3914](matrix-org#3914))
- Fix some instances of ExpiringCache not expiring cache items ([\matrix-org#3932](matrix-org#3932), [\matrix-org#3980](matrix-org#3980))
- Fix out-of-bounds error when LLing yourself ([\matrix-org#3936](matrix-org#3936))
- Sending server notices regarding user consent now works on Python 3. ([\matrix-org#3938](matrix-org#3938))
- Fix exceptions from metrics handler ([\matrix-org#3956](matrix-org#3956))
- Fix error message for events with m.room.create missing from auth_events ([\matrix-org#3960](matrix-org#3960))
- Fix errors due to concurrent monthly_active_user upserts ([\matrix-org#3961](matrix-org#3961))
- Fix exceptions when processing incoming events over federation ([\matrix-org#3968](matrix-org#3968))
- Replaced all occurences of e.message with str(e). Contributed by Schnuffle ([\matrix-org#3970](matrix-org#3970))
- Fix lazy loaded sync in the presence of rejected state events ([\matrix-org#3986](matrix-org#3986))
- Fix error when logging incomplete HTTP requests ([\matrix-org#3990](matrix-org#3990))
- Add a cache to get_destination_retry_timings ([\matrix-org#3991](matrix-org#3991))

Internal Changes
----------------

- Unit tests can now be run under PostgreSQL in Docker using
  ``test_postgresql.sh``. ([\matrix-org#3699](matrix-org#3699))
- Speed up calculation of typing updates for replication ([\matrix-org#3794](matrix-org#3794))
- Remove documentation regarding installation on Cygwin, the use of WSL is
  recommended instead. ([\matrix-org#3873](matrix-org#3873))
- Fix typo in README, synaspse -> synapse ([\matrix-org#3897](matrix-org#3897))
- Increase the timeout when filling missing events in federation requests ([\matrix-org#3903](matrix-org#3903))
- Improve the logging when handling a federation transaction ([\matrix-org#3904](matrix-org#3904), [\matrix-org#3966](matrix-org#3966))
- Improve logging of outbound federation requests ([\matrix-org#3906](matrix-org#3906), [\matrix-org#3909](matrix-org#3909))
- Fix the docker image building on python 3 ([\matrix-org#3911](matrix-org#3911))
- Add a regression test for logging failed HTTP requests on Python 3. ([\matrix-org#3912](matrix-org#3912))
- Comments and interface cleanup for on_receive_pdu ([\matrix-org#3924](matrix-org#3924))
- Fix spurious exceptions when remote http client closes conncetion ([\matrix-org#3925](matrix-org#3925))
- Log exceptions thrown by background tasks ([\matrix-org#3927](matrix-org#3927))
- Add a cache to get_destination_retry_timings ([\matrix-org#3933](matrix-org#3933))
- Automate pushes to docker hub ([\matrix-org#3946](matrix-org#3946))
- Require attrs 16.0.0 or later ([\matrix-org#3947](matrix-org#3947))
- Fix incompatibility with python3 on alpine ([\matrix-org#3948](matrix-org#3948))
- Run the test suite on the oldest supported versions of our dependencies in CI. ([\matrix-org#3952](matrix-org#3952))
- CircleCI now only runs merged jobs on PRs, and commit jobs on develop, master, and release branches. ([\matrix-org#3957](matrix-org#3957))
- Fix docstrings and add tests for state store methods ([\matrix-org#3958](matrix-org#3958))
- fix docstring for FederationClient.get_state_for_room ([\matrix-org#3963](matrix-org#3963))
- Run notify_app_services as a bg process ([\matrix-org#3965](matrix-org#3965))
- Clarifications in FederationHandler ([\matrix-org#3967](matrix-org#3967))
- Further reduce the docker image size ([\matrix-org#3972](matrix-org#3972))
- Build py3 docker images for docker hub too ([\matrix-org#3976](matrix-org#3976))
- Updated the installation instructions to point to the matrix-synapse package on PyPI. ([\matrix-org#3985](matrix-org#3985))
- Disable USE_FROZEN_DICTS for unittests by default. ([\matrix-org#3987](matrix-org#3987))
- Remove unused Jenkins and development related files from the repo. ([\matrix-org#3988](matrix-org#3988))
- Improve stacktraces in certain exceptions in the logs ([\matrix-org#3989](matrix-org#3989))

Internal Changes

----------------

- Fix incompatibility with older Twisted version in tests. Thanks
  @OlegGirko!
([\matrix-org#3940](matrix-org#3940))

Features

--------

- Python 3.5 and 3.6 support is now in beta.
([\matrix-org#3576](matrix-org#3576))
- Implement `event_format` filter param in `/sync`
([\matrix-org#3790](matrix-org#3790))
- Add synapse_admin_mau:registered_reserved_users metric to expose
number of real reaserved users
([\matrix-org#3846](matrix-org#3846))

Bugfixes
--------

- Remove connection ID for replication prometheus metrics, as it creates
a large number of new series.
([\matrix-org#3788](matrix-org#3788))
- guest users should not be part of mau total
([\matrix-org#3800](matrix-org#3800))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with
recent Twisted.
([\matrix-org#3804](matrix-org#3804))
- Fix existing room tags not coming down sync when joining a room
([\matrix-org#3810](matrix-org#3810))
- Fix jwt import check
([\matrix-org#3824](matrix-org#3824))
- fix VOIP crashes under Python 3 (matrix-org#3821)
([\matrix-org#3835](matrix-org#3835))
- Fix manhole so that it works with latest openssh clients
([\matrix-org#3841](matrix-org#3841))
- Fix outbound requests occasionally wedging, which can result in
federation breaking between servers.
([\matrix-org#3845](matrix-org#3845))
- Show heroes if room name/canonical alias has been deleted
([\matrix-org#3851](matrix-org#3851))
- Fix handling of redacted events from federation
([\matrix-org#3859](matrix-org#3859))
-  ([\matrix-org#3874](matrix-org#3874))
- Mitigate outbound federation randomly becoming wedged
([\matrix-org#3875](matrix-org#3875))

Internal Changes
----------------

- CircleCI tests now run on the potential merge of a PR.
([\matrix-org#3704](matrix-org#3704))
- http/ is now ported to Python 3.
([\matrix-org#3771](matrix-org#3771))
- Improve human readable error messages for threepid
registration/account update
([\matrix-org#3789](matrix-org#3789))
- Make /sync slightly faster by avoiding needless copies
([\matrix-org#3795](matrix-org#3795))
- handlers/ is now ported to Python 3.
([\matrix-org#3803](matrix-org#3803))
- Limit the number of PDUs/EDUs per federation transaction
([\matrix-org#3805](matrix-org#3805))
- Only start postgres instance for postgres tests on Travis CI
([\matrix-org#3806](matrix-org#3806))
- tests/ is now ported to Python 3.
([\matrix-org#3808](matrix-org#3808))
- crypto/ is now ported to Python 3.
([\matrix-org#3822](matrix-org#3822))
- rest/ is now ported to Python 3.
([\matrix-org#3823](matrix-org#3823))
- add some logging for the keyring queue
([\matrix-org#3826](matrix-org#3826))
- speed up lazy loading by 2-3x
([\matrix-org#3827](matrix-org#3827))
- Improved Dockerfile to remove build requirements after building
reducing the image size.
([\matrix-org#3834](matrix-org#3834))
- Disable lazy loading for incremental syncs for now
([\matrix-org#3840](matrix-org#3840))
- federation/ is now ported to Python 3.
([\matrix-org#3847](matrix-org#3847))
- Log when we retry outbound requests
([\matrix-org#3853](matrix-org#3853))
- Removed some excess logging messages.
([\matrix-org#3855](matrix-org#3855))
- Speed up purge history for rooms that have been previously purged
([\matrix-org#3856](matrix-org#3856))
- Refactor some HTTP timeout code.
([\matrix-org#3857](matrix-org#3857))
- Fix running merged builds on CircleCI
([\matrix-org#3858](matrix-org#3858))
- Fix typo in replication stream exception.
([\matrix-org#3860](matrix-org#3860))
- Add in flight real time metrics for Measure blocks
([\matrix-org#3871](matrix-org#3871))
- Disable buffering and automatic retrying in treq requests to prevent
timeouts. ([\matrix-org#3872](matrix-org#3872))
- mention jemalloc in the README
([\matrix-org#3877](matrix-org#3877))
- Remove unmaintained "nuke-room-from-db.sh" script
([\matrix-org#3888](matrix-org#3888))

Features

--------

- Python 3.5 and 3.6 support is now in beta.
([\matrix-org#3576](matrix-org#3576))
- Implement `event_format` filter param in `/sync`
([\matrix-org#3790](matrix-org#3790))
- Add synapse_admin_mau:registered_reserved_users metric to expose
number of real reaserved users
([\matrix-org#3846](matrix-org#3846))

Bugfixes
--------

- Remove connection ID for replication prometheus metrics, as it creates
a large number of new series.
([\matrix-org#3788](matrix-org#3788))
- guest users should not be part of mau total
([\matrix-org#3800](matrix-org#3800))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with
recent Twisted.
([\matrix-org#3804](matrix-org#3804))
- Fix existing room tags not coming down sync when joining a room
([\matrix-org#3810](matrix-org#3810))
- Fix jwt import check
([\matrix-org#3824](matrix-org#3824))
- fix VOIP crashes under Python 3 (matrix-org#3821)
([\matrix-org#3835](matrix-org#3835))
- Fix manhole so that it works with latest openssh clients
([\matrix-org#3841](matrix-org#3841))
- Fix outbound requests occasionally wedging, which can result in
federation breaking between servers.
([\matrix-org#3845](matrix-org#3845))
- Show heroes if room name/canonical alias has been deleted
([\matrix-org#3851](matrix-org#3851))
- Fix handling of redacted events from federation
([\matrix-org#3859](matrix-org#3859))
-  ([\matrix-org#3874](matrix-org#3874))
- Mitigate outbound federation randomly becoming wedged
([\matrix-org#3875](matrix-org#3875))

Internal Changes
----------------

- CircleCI tests now run on the potential merge of a PR.
([\matrix-org#3704](matrix-org#3704))
- http/ is now ported to Python 3.
([\matrix-org#3771](matrix-org#3771))
- Improve human readable error messages for threepid
registration/account update
([\matrix-org#3789](matrix-org#3789))
- Make /sync slightly faster by avoiding needless copies
([\matrix-org#3795](matrix-org#3795))
- handlers/ is now ported to Python 3.
([\matrix-org#3803](matrix-org#3803))
- Limit the number of PDUs/EDUs per federation transaction
([\matrix-org#3805](matrix-org#3805))
- Only start postgres instance for postgres tests on Travis CI
([\matrix-org#3806](matrix-org#3806))
- tests/ is now ported to Python 3.
([\matrix-org#3808](matrix-org#3808))
- crypto/ is now ported to Python 3.
([\matrix-org#3822](matrix-org#3822))
- rest/ is now ported to Python 3.
([\matrix-org#3823](matrix-org#3823))
- add some logging for the keyring queue
([\matrix-org#3826](matrix-org#3826))
- speed up lazy loading by 2-3x
([\matrix-org#3827](matrix-org#3827))
- Improved Dockerfile to remove build requirements after building
reducing the image size.
([\matrix-org#3834](matrix-org#3834))
- Disable lazy loading for incremental syncs for now
([\matrix-org#3840](matrix-org#3840))
- federation/ is now ported to Python 3.
([\matrix-org#3847](matrix-org#3847))
- Log when we retry outbound requests
([\matrix-org#3853](matrix-org#3853))
- Removed some excess logging messages.
([\matrix-org#3855](matrix-org#3855))
- Speed up purge history for rooms that have been previously purged
([\matrix-org#3856](matrix-org#3856))
- Refactor some HTTP timeout code.
([\matrix-org#3857](matrix-org#3857))
- Fix running merged builds on CircleCI
([\matrix-org#3858](matrix-org#3858))
- Fix typo in replication stream exception.
([\matrix-org#3860](matrix-org#3860))
- Add in flight real time metrics for Measure blocks
([\matrix-org#3871](matrix-org#3871))
- Disable buffering and automatic retrying in treq requests to prevent
timeouts. ([\matrix-org#3872](matrix-org#3872))
- mention jemalloc in the README
([\matrix-org#3877](matrix-org#3877))
- Remove unmaintained "nuke-room-from-db.sh" script
([\matrix-org#3888](matrix-org#3888))

Merge pull request matrix-org#3856 from matrix-org/erikj/speed_up_purge

Make purge history slightly faster

Merge pull request matrix-org#3846 from matrix-org/neilj/expose-regis…

…tered-users

expose number of real reserved users