Synapse 1.4.0rc1 (2019-09-26)

=============================

Note that this release includes significant changes around 3pid
verification. Administrators are reminded to review the [upgrade notes](UPGRADE.rst##upgrading-to-v140).

Features
--------

- Changes to 3pid verification:
  - Add the ability to send registration emails from the homeserver rather than delegating to an identity server. ([\matrix-org#5835](matrix-org#5835), [\matrix-org#5940](matrix-org#5940), [\matrix-org#5993](matrix-org#5993), [\matrix-org#5994](matrix-org#5994), [\matrix-org#5868](matrix-org#5868))
  - Replace `trust_identity_server_for_password_resets` config option with `account_threepid_delegates`, and make the `id_server` parameteter optional on `*/requestToken` endpoints, as per [MSC2263](matrix-org/matrix-spec-proposals#2263). ([\matrix-org#5876](matrix-org#5876), [\matrix-org#5969](matrix-org#5969), [\matrix-org#6028](matrix-org#6028))
  - Switch to using the v2 Identity Service `/lookup` API where available, with fallback to v1. (Implements [MSC2134](matrix-org/matrix-spec-proposals#2134) plus `id_access_token authentication` for v2 Identity Service APIs from [MSC2140](matrix-org/matrix-spec-proposals#2140)). ([\matrix-org#5897](matrix-org#5897))
  - Remove `bind_email` and `bind_msisdn` parameters from `/register` ala [MSC2140](matrix-org/matrix-spec-proposals#2140). ([\matrix-org#5964](matrix-org#5964))
  - Add `m.id_access_token` to `unstable_features` in `/versions` as per [MSC2264](matrix-org/matrix-spec-proposals#2264). ([\matrix-org#5974](matrix-org#5974))
  - Use the v2 Identity Service API for 3PID invites. ([\matrix-org#5979](matrix-org#5979))
  - Add `POST /_matrix/client/unstable/account/3pid/unbind` endpoint from [MSC2140](matrix-org/matrix-spec-proposals#2140) for unbinding a 3PID from an identity server without removing it from the homeserver user account. ([\matrix-org#5980](matrix-org#5980), [\matrix-org#6062](matrix-org#6062))
  - Use `account_threepid_delegate.email` and `account_threepid_delegate.msisdn` for validating threepid sessions. ([\matrix-org#6011](matrix-org#6011))
  - Allow homeserver to handle or delegate email validation when adding an email to a user's account. ([\matrix-org#6042](matrix-org#6042))
  - Implement new Client Server API endpoints `/account/3pid/add` and `/account/3pid/bind` as per [MSC2290](matrix-org/matrix-spec-proposals#2290). ([\matrix-org#6043](matrix-org#6043))
  - Add an unstable feature flag for separate add/bind 3pid APIs. ([\matrix-org#6044](matrix-org#6044))
  - Remove `bind` parameter from Client Server POST `/account` endpoint as per [MSC2290](matrix-org/matrix-spec-proposals#2290). ([\matrix-org#6067](matrix-org#6067))
  - Add `POST /add_threepid/msisdn/submit_token` endpoint for proxying submitToken on an `account_threepid_handler`. ([\matrix-org#6078](matrix-org#6078))
  - Add `submit_url` response parameter to `*/msisdn/requestToken` endpoints. ([\matrix-org#6079](matrix-org#6079))
  - Add `m.require_identity_server` flag to /version's unstable_features. ([\matrix-org#5972](matrix-org#5972))
- Enhancements to OpenTracing support:
  - Make OpenTracing work in worker mode. ([\matrix-org#5771](matrix-org#5771))
  - Pass OpenTracing contexts between servers when transmitting EDUs. ([\matrix-org#5852](matrix-org#5852))
  - OpenTracing for device list updates. ([\matrix-org#5853](matrix-org#5853))
  - Add a tag recording a request's authenticated entity and corresponding servlet in OpenTracing. ([\matrix-org#5856](matrix-org#5856))
  - Add minimum OpenTracing for client servlets. ([\matrix-org#5983](matrix-org#5983))
  - Check at setup that OpenTracing is installed if it's enabled in the config. ([\matrix-org#5985](matrix-org#5985))
  - Trace replication send times. ([\matrix-org#5986](matrix-org#5986))
  - Include missing OpenTracing contexts in outbout replication requests. ([\matrix-org#5982](matrix-org#5982))
  - Fix sending of EDUs when OpenTracing is enabled with an empty whitelist. ([\matrix-org#5984](matrix-org#5984))
  - Fix invalid references to None while OpenTracing if the log context slips. ([\matrix-org#5988](matrix-org#5988), [\matrix-org#5991](matrix-org#5991))
  - OpenTracing for room and e2e keys. ([\matrix-org#5855](matrix-org#5855))
  - Add OpenTracing span over HTTP push processing. ([\matrix-org#6003](matrix-org#6003))
- Add an admin API to purge old rooms from the database. ([\matrix-org#5845](matrix-org#5845))
- Retry well-known lookups if we have recently seen a valid well-known record for the server. ([\matrix-org#5850](matrix-org#5850))
- Add support for filtered room-directory search requests over federation ([MSC2197](matrix-org/matrix-spec-proposals#2197), in order to allow upcoming room directory query performance improvements. ([\matrix-org#5859](matrix-org#5859))
- Correctly retry all hosts returned from SRV when we fail to connect. ([\matrix-org#5864](matrix-org#5864))
- Add admin API endpoint for setting whether or not a user is a server administrator. ([\matrix-org#5878](matrix-org#5878))
- Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. ([\matrix-org#5884](matrix-org#5884))
- Add config option to sign remote key query responses with a separate key. ([\matrix-org#5895](matrix-org#5895))
- Add support for config templating. ([\matrix-org#5900](matrix-org#5900))
- Users with the type of "support" or "bot" are no longer required to consent. ([\matrix-org#5902](matrix-org#5902))
- Let synctl accept a directory of config files. ([\matrix-org#5904](matrix-org#5904))
- Increase max display name size to 256. ([\matrix-org#5906](matrix-org#5906))
- Add admin API endpoint for getting whether or not a user is a server administrator. ([\matrix-org#5914](matrix-org#5914))
- Redact events in the database that have been redacted for a month. ([\matrix-org#5934](matrix-org#5934))
- New prometheus metrics:
  - `synapse_federation_known_servers`: represents the total number of servers your server knows about (i.e. is in rooms with), including itself. Enable by setting `metrics_flags.known_servers` to True in the configuration.([\matrix-org#5981](matrix-org#5981))
  - `synapse_build_info`: exposes the Python version, OS version, and Synapse version of the running server. ([\matrix-org#6005](matrix-org#6005))
- Give appropriate exit codes when synctl fails. ([\matrix-org#5992](matrix-org#5992))
- Apply the federation blacklist to requests to identity servers. ([\matrix-org#6000](matrix-org#6000))
- Add `report_stats_endpoint` option to configure where stats are reported to, if enabled. Contributed by @Sorunome. ([\matrix-org#6012](matrix-org#6012))
- Add config option to increase ratelimits for room admins redacting messages. ([\matrix-org#6015](matrix-org#6015))
- Stop sending federation transactions to servers which have been down for a long time. ([\matrix-org#6026](matrix-org#6026))
- Make the process for mapping SAML2 users to matrix IDs more flexible. ([\matrix-org#6037](matrix-org#6037))
- Return a clearer error message when a timeout occurs when attempting to contact an identity server. ([\matrix-org#6073](matrix-org#6073))
- Prevent password reset's submit_token endpoint from accepting trailing slashes. ([\matrix-org#6074](matrix-org#6074))
- Return 403 on `/register/available` if registration has been disabled. ([\matrix-org#6082](matrix-org#6082))
- Explicitly log when a homeserver does not have the `trusted_key_servers` config field configured. ([\matrix-org#6090](matrix-org#6090))
- Add support for pruning old rows in `user_ips` table. ([\matrix-org#6098](matrix-org#6098))

Bugfixes
--------

- Don't create broken room when `power_level_content_override.users` does not contain `creator_id`. ([\matrix-org#5633](matrix-org#5633))
- Fix database index so that different backup versions can have the same sessions. ([\matrix-org#5857](matrix-org#5857))
- Fix Synapse looking for config options `password_reset_failure_template` and `password_reset_success_template`, when they are actually `password_reset_template_failure_html`, `password_reset_template_success_html`. ([\matrix-org#5863](matrix-org#5863))
- Fix stack overflow when recovering an appservice which had an outage. ([\matrix-org#5885](matrix-org#5885))
- Fix error message which referred to `public_base_url` instead of `public_baseurl`. Thanks to @aaronraimist for the fix! ([\matrix-org#5909](matrix-org#5909))
- Fix 404 for thumbnail download when `dynamic_thumbnails` is `false` and the thumbnail was dynamically generated. Fix reported by rkfg. ([\matrix-org#5915](matrix-org#5915))
- Fix a cache-invalidation bug for worker-based deployments. ([\matrix-org#5920](matrix-org#5920))
- Fix admin API for listing media in a room not being available with an external media repo. ([\matrix-org#5966](matrix-org#5966))
- Fix list media admin API always returning an error. ([\matrix-org#5967](matrix-org#5967))
- Fix room and user stats tracking. ([\matrix-org#5971](matrix-org#5971), [\matrix-org#5998](matrix-org#5998), [\matrix-org#6029](matrix-org#6029))
- Return a `M_MISSING_PARAM` if `sid` is not provided to `/account/3pid`. ([\matrix-org#5995](matrix-org#5995))
- `federation_certificate_verification_whitelist` now will not cause `TypeErrors` to be raised (a regression in 1.3). Additionally, it now supports internationalised domain names in their non-canonical representation. ([\matrix-org#5996](matrix-org#5996))
- Only count real users when checking for auto-creation of auto-join room. ([\matrix-org#6004](matrix-org#6004))
- Ensure support users can be registered even if MAU limit is reached. ([\matrix-org#6020](matrix-org#6020))
- Fix bug where login error was shown incorrectly on SSO fallback login. ([\matrix-org#6024](matrix-org#6024))
- Fix bug in calculating the federation retry backoff period. ([\matrix-org#6025](matrix-org#6025))
- Prevent exceptions being logged when extremity-cleanup events fail due to lack of user consent to the terms of service. ([\matrix-org#6053](matrix-org#6053))
- Remove POST method from password-reset `submit_token` endpoint until we implement `submit_url` functionality. ([\matrix-org#6056](matrix-org#6056))
- Fix logcontext spam on non-Linux platforms. ([\matrix-org#6059](matrix-org#6059))
- Ensure query parameters in email validation links are URL-encoded. ([\matrix-org#6063](matrix-org#6063))
- Fix a bug which caused SAML attribute maps to be overridden by defaults. ([\matrix-org#6069](matrix-org#6069))
- Fix the logged number of updated items for the `users_set_deactivated_flag` background update. ([\matrix-org#6092](matrix-org#6092))
- Add `sid` to `next_link` for email validation. ([\matrix-org#6097](matrix-org#6097))
- Threepid validity checks on msisdns should not be dependent on `threepid_behaviour_email`. ([\matrix-org#6104](matrix-org#6104))
- Ensure that servers which are not configured to support email address verification do not offer it in the registration flows. ([\matrix-org#6107](matrix-org#6107))

Updates to the Docker image
---------------------------

- Avoid changing `UID/GID` if they are already correct. ([\matrix-org#5970](matrix-org#5970))
- Provide `SYNAPSE_WORKER` envvar to specify python module. ([\matrix-org#6058](matrix-org#6058))

Improved Documentation
----------------------

- Convert documentation to markdown (from rst) ([\matrix-org#5849](matrix-org#5849))
- Update `INSTALL.md` to say that Python 2 is no longer supported. ([\matrix-org#5953](matrix-org#5953))
- Add developer documentation for using SAML2. ([\matrix-org#6032](matrix-org#6032))
- Add some notes on rolling back to v1.3.1. ([\matrix-org#6049](matrix-org#6049))
- Update the upgrade notes. ([\matrix-org#6050](matrix-org#6050))

Deprecations and Removals
-------------------------

- Remove shared-secret registration from `/_matrix/client/r0/register` endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\matrix-org#5877](matrix-org#5877))
- Deprecate the `trusted_third_party_id_servers` option. ([\matrix-org#5875](matrix-org#5875))

Internal Changes
----------------

- Lay the groundwork for structured logging output. ([\matrix-org#5680](matrix-org#5680))
- Retry well-known lookup before the cache expires, giving a grace period where the remote well-known can be down but we still use the old result. ([\matrix-org#5844](matrix-org#5844))
- Remove log line for debugging issue matrix-org#5407. ([\matrix-org#5860](matrix-org#5860))
- Refactor the Appservice scheduler code. ([\matrix-org#5886](matrix-org#5886))
- Compatibility with v2 Identity Service APIs other than /lookup. ([\matrix-org#5892](matrix-org#5892), [\matrix-org#6013](matrix-org#6013))
- Stop populating some unused tables. ([\matrix-org#5893](matrix-org#5893), [\matrix-org#6047](matrix-org#6047))
- Add missing index on `users_in_public_rooms` to improve the performance of directory queries. ([\matrix-org#5894](matrix-org#5894))
- Improve the logging when we have an error when fetching signing keys. ([\matrix-org#5896](matrix-org#5896))
- Add support for database engine-specific schema deltas, based on file extension. ([\matrix-org#5911](matrix-org#5911))
- Update Buildkite pipeline to use plugins instead of buildkite-agent commands. ([\matrix-org#5922](matrix-org#5922))
- Add link in sample config to the logging config schema. ([\matrix-org#5926](matrix-org#5926))
- Remove unnecessary parentheses in return statements. ([\matrix-org#5931](matrix-org#5931))
- Remove unused `jenkins/prepare_sytest.sh` file. ([\matrix-org#5938](matrix-org#5938))
- Move Buildkite pipeline config to the pipelines repo. ([\matrix-org#5943](matrix-org#5943))
- Remove unnecessary return statements in the codebase which were the result of a regex run. ([\matrix-org#5962](matrix-org#5962))
- Remove left-over methods from v1 registration API. ([\matrix-org#5963](matrix-org#5963))
- Cleanup event auth type initialisation. ([\matrix-org#5975](matrix-org#5975))
- Clean up dependency checking at setup. ([\matrix-org#5989](matrix-org#5989))
- Update OpenTracing docs to use the unified `trace` method. ([\matrix-org#5776](matrix-org#5776))
- Small refactor of function arguments and docstrings in` RoomMemberHandler`. ([\matrix-org#6009](matrix-org#6009))
- Remove unused `origin` argument on `FederationHandler.add_display_name_to_third_party_invite`. ([\matrix-org#6010](matrix-org#6010))
- Add a `failure_ts` column to the `destinations` database table. ([\matrix-org#6016](matrix-org#6016), [\matrix-org#6072](matrix-org#6072))
- Clean up some code in the retry logic. ([\matrix-org#6017](matrix-org#6017))
- Fix the structured logging tests stomping on the global log configuration for subsequent tests. ([\matrix-org#6023](matrix-org#6023))
- Clean up the sample config for SAML authentication. ([\matrix-org#6064](matrix-org#6064))
- Change mailer logging to reflect Synapse doesn't just do chat notifications by email now. ([\matrix-org#6075](matrix-org#6075))
- Move last-seen info into devices table. ([\matrix-org#6089](matrix-org#6089))
- Remove unused parameter to `get_user_id_by_threepid`. ([\matrix-org#6099](matrix-org#6099))
- Refactor the user-interactive auth handling. ([\matrix-org#6105](matrix-org#6105))
- Refactor code for calculating registration flows. ([\matrix-org#6106](matrix-org#6106))

Bump version

v1.3.2-alpha.4+modular

Bump version

Merge pull request matrix-org#5932 from matrix-org/babolivier/account…

…_validity_template_encode

Fix encoding for account validity HTML files on Python 2

Merge pull request matrix-org#5815 from matrix-org/babolivier/dinsic-…

…message-retention

Message retention policies at the room and server levels

Synapse 1.3.1 (2019-08-17)

==========================

Features
--------

- Drop hard dependency on `sdnotify` python package. ([\matrix-org#5871](matrix-org#5871))

Bugfixes
--------

- Fix startup issue (hang on ACME provisioning) due to ordering of Twisted reactor startup. Thanks to @chrismoos for supplying the fix. ([\matrix-org#5867](matrix-org#5867))

Synapse 1.3.0 (2019-08-15)

==========================

Bugfixes
--------

- Fix 500 Internal Server Error on `publicRooms` when the public room list was
  cached. ([\matrix-org#5851](matrix-org#5851))

Synapse 1.3.0rc1 (2019-08-13)
==========================

Features
--------

- Use `M_USER_DEACTIVATED` instead of `M_UNKNOWN` for errcode when a deactivated user attempts to login. ([\matrix-org#5686](matrix-org#5686))
- Add sd_notify hooks to ease systemd integration and allows usage of Type=Notify. ([\matrix-org#5732](matrix-org#5732))
- Synapse will no longer serve any media repo admin endpoints when `enable_media_repo` is set to False in the configuration. If a media repo worker is used, the admin APIs relating to the media repo will be served from it instead. ([\matrix-org#5754](matrix-org#5754), [\matrix-org#5848](matrix-org#5848))
- Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events) over federation. This option can be used to prevent adverse performance on resource-constrained homeservers. ([\matrix-org#5783](matrix-org#5783))
- Allow defining HTML templates to serve the user on account renewal attempt when using the account validity feature. ([\matrix-org#5807](matrix-org#5807))

Bugfixes
--------

- Fix UISIs during homeserver outage. ([\matrix-org#5693](matrix-org#5693), [\matrix-org#5789](matrix-org#5789))
- Fix stack overflow in server key lookup code. ([\matrix-org#5724](matrix-org#5724))
- start.sh no longer uses deprecated cli option. ([\matrix-org#5725](matrix-org#5725))
- Log when we receive an event receipt from an unexpected origin. ([\matrix-org#5743](matrix-org#5743))
- Fix debian packaging scripts to correctly build sid packages. ([\matrix-org#5775](matrix-org#5775))
- Correctly handle redactions of redactions. ([\matrix-org#5788](matrix-org#5788))
- Return 404 instead of 403 when accessing /rooms/{roomId}/event/{eventId} for an event without the appropriate permissions. ([\matrix-org#5798](matrix-org#5798))
- Fix check that tombstone is a state event in push rules. ([\matrix-org#5804](matrix-org#5804))
- Fix error when trying to login as a deactivated user when using a worker to handle login. ([\matrix-org#5806](matrix-org#5806))
- Fix bug where user `/sync` stream could get wedged in rare circumstances. ([\matrix-org#5825](matrix-org#5825))
- The purge_remote_media.sh script was fixed. ([\matrix-org#5839](matrix-org#5839))

Deprecations and Removals
-------------------------

- Synapse now no longer accepts the `-v`/`--verbose`, `-f`/`--log-file`, or `--log-config` command line flags, and removes the deprecated `verbose` and `log_file` configuration file options. Users of these options should migrate their options into the dedicated log configuration. ([\matrix-org#5678](matrix-org#5678), [\matrix-org#5729](matrix-org#5729))
- Remove non-functional 'expire_access_token' setting. ([\matrix-org#5782](matrix-org#5782))

Internal Changes
----------------

- Make Jaeger fully configurable. ([\matrix-org#5694](matrix-org#5694))
- Add precautionary measures to prevent future abuse of `window.opener` in default welcome page. ([\matrix-org#5695](matrix-org#5695))
- Reduce database IO usage by optimising queries for current membership. ([\matrix-org#5706](matrix-org#5706), [\matrix-org#5738](matrix-org#5738), [\matrix-org#5746](matrix-org#5746), [\matrix-org#5752](matrix-org#5752), [\matrix-org#5770](matrix-org#5770), [\matrix-org#5774](matrix-org#5774), [\matrix-org#5792](matrix-org#5792), [\matrix-org#5793](matrix-org#5793))
- Improve caching when fetching `get_filtered_current_state_ids`. ([\matrix-org#5713](matrix-org#5713))
- Don't accept opentracing data from clients. ([\matrix-org#5715](matrix-org#5715))
- Speed up PostgreSQL unit tests in CI. ([\matrix-org#5717](matrix-org#5717))
- Update the coding style document. ([\matrix-org#5719](matrix-org#5719))
- Improve database query performance when recording retry intervals for remote hosts. ([\matrix-org#5720](matrix-org#5720))
- Add a set of opentracing utils. ([\matrix-org#5722](matrix-org#5722))
- Cache result of get_version_string to reduce overhead of `/version` federation requests. ([\matrix-org#5730](matrix-org#5730))
- Return 'user_type' in admin API user endpoints results. ([\matrix-org#5731](matrix-org#5731))
- Don't package the sytest test blacklist file. ([\matrix-org#5733](matrix-org#5733))
- Replace uses of returnValue with plain return, as returnValue is not needed on Python 3. ([\matrix-org#5736](matrix-org#5736))
- Blacklist some flakey tests in worker mode. ([\matrix-org#5740](matrix-org#5740))
- Fix some error cases in the caching layer. ([\matrix-org#5749](matrix-org#5749))
- Add a prometheus metric for pending cache lookups. ([\matrix-org#5750](matrix-org#5750))
- Stop trying to fetch events with event_id=None. ([\matrix-org#5753](matrix-org#5753))
- Convert RedactionTestCase to modern test style. ([\matrix-org#5768](matrix-org#5768))
- Allow looping calls to be given arguments. ([\matrix-org#5780](matrix-org#5780))
- Set the logs emitted when checking typing and presence timeouts to DEBUG level, not INFO. ([\matrix-org#5785](matrix-org#5785))
- Remove DelayedCall debugging from the test suite, as it is no longer required in the vast majority of Synapse's tests. ([\matrix-org#5787](matrix-org#5787))
- Remove some spurious exceptions from the logs where we failed to talk to a remote server. ([\matrix-org#5790](matrix-org#5790))
- Improve performance when making `.well-known` requests by sharing the SSL options between requests. ([\matrix-org#5794](matrix-org#5794))
- Disable codecov GitHub comments on PRs. ([\matrix-org#5796](matrix-org#5796))
- Don't allow clients to send tombstone events that reference the room it's sent in. ([\matrix-org#5801](matrix-org#5801))
- Deny redactions of events sent in a different room. ([\matrix-org#5802](matrix-org#5802))
- Deny sending well known state types as non-state events. ([\matrix-org#5805](matrix-org#5805))
- Handle incorrectly encoded query params correctly by returning a 400. ([\matrix-org#5808](matrix-org#5808))
- Handle pusher being deleted during processing rather than logging an exception. ([\matrix-org#5809](matrix-org#5809))
- Return 502 not 500 when failing to reach any remote server. ([\matrix-org#5810](matrix-org#5810))
- Reduce global pauses in the events stream caused by expensive state resolution during persistence. ([\matrix-org#5826](matrix-org#5826))
- Add a lower bound to well-known lookup cache time to avoid repeated lookups. ([\matrix-org#5836](matrix-org#5836))
- Whitelist history visbility sytests in worker mode tests. ([\matrix-org#5843](matrix-org#5843))