Merge pull request matrix-org#5115 from matrix-org/babolivier/lookup_…

…path

[DINSIC] Move 3PID lookup endpoint to CS API and proxy bulk lookups

Synapse 0.99.3.2 (2019-05-03)

=============================

Internal Changes
----------------

- Ensure that we have `urllib3` <1.25, to resolve incompatibility with `requests`. ([\matrix-org#5135](matrix-org#5135))

Synapse 0.99.3.1 (2019-05-03)

=============================

Security update
---------------

This release includes two security fixes:

- Switch to using a cryptographically-secure random number generator for token strings, ensuring they cannot be predicted by an attacker. Thanks to @opnsec for identifying and responsibly disclosing this issue! ([\matrix-org#5133](matrix-org#5133))
- Blacklist 0.0.0.0 and :: by default for URL previews. Thanks to @opnsec for identifying and responsibly disclosing this issue too! ([\matrix-org#5134](matrix-org#5134))

Add config option to block users from looking up 3PIDs (matrix-org#5010)

Prevent exception when hs or shadow_hs missing (matrix-org#5012)

* Prevent exception when hs or shadow_hs missing

Fix registration with register_mxid_from_3pid enabled (matrix-org#5011)

Rewrite more internal IS urls. (matrix-org#5006)

* Rewrite more internal IS urls.

Synapse 0.99.3 (2019-04-01)

No significant changes.

Synapse 0.99.3rc1 (2019-03-27)
Features
The user directory has been rewritten to make it faster, with less chance of falling behind on a large server. (matrix-org#4537, matrix-org#4846, matrix-org#4864, matrix-org#4887, matrix-org#4900, matrix-org#4944)
Add configurable rate limiting to the /register endpoint. (matrix-org#4735, matrix-org#4804)
Move server key queries to federation reader. (matrix-org#4757)
Add support for /account/3pid REST endpoint to client_reader worker. (matrix-org#4759)
Add an endpoint to the admin API for querying the server version. Contributed by Joseph Weston. (matrix-org#4772)
Include a default configuration file in the 'docs' directory. (matrix-org#4791, matrix-org#4801)
Synapse is now permissive about trailing slashes on some of its federation endpoints, allowing zero or more to be present. (matrix-org#4793)
Add support for /keys/query and /keys/changes REST endpoints to client_reader worker. (matrix-org#4796)
Add checks to incoming events over federation for events evading auth (aka "soft fail"). (matrix-org#4814)
Add configurable rate limiting to the /login endpoint. (matrix-org#4821, matrix-org#4865)
Remove trailing slashes from certain outbound federation requests. Retry if receiving a 404. Context: matrix-org#3622. (matrix-org#4840)
Allow passing --daemonize flags to workers in the same way as with master. (matrix-org#4853)
Batch up outgoing read-receipts to reduce federation traffic. (matrix-org#4890, matrix-org#4927)
Add option to disable searching the user directory. (matrix-org#4895)
Add option to disable searching of local and remote public room lists. (matrix-org#4896)
Add ability for password providers to login/register a user via 3PID (email, phone). (matrix-org#4931)
Bugfixes
Fix a bug where media with spaces in the name would get a corrupted name. (matrix-org#2090)
Fix attempting to paginate in rooms where server cannot see any events, to avoid unnecessarily pulling in lots of redacted events. (matrix-org#4699)
'event_id' is now a required parameter in federated state requests, as per the matrix spec. (matrix-org#4740)
Fix tightloop over connecting to replication server. (matrix-org#4749)
Fix parsing of Content-Disposition headers on remote media requests and URL previews. (matrix-org#4763)
Fix incorrect log about not persisting duplicate state event. (matrix-org#4776)
Fix v4v6 option in HAProxy example config. Contributed by Flakebi. (matrix-org#4790)
Handle batch updates in worker replication protocol. (matrix-org#4792)
Fix bug where we didn't correctly throttle sending of USER_IP commands over replication. (matrix-org#4818)
Fix potential race in handling missing updates in device list updates. (matrix-org#4829)
Fix bug where synapse expected an un-specced prev_state field on state events. (matrix-org#4837)
Transfer a user's notification settings (push rules) on room upgrade. (matrix-org#4838)
fix test_auto_create_auto_join_where_no_consent. (matrix-org#4886)
Fix a bug where hs_disabled_message was sometimes not correctly enforced. (matrix-org#4888)
Fix bug in shutdown room admin API where it would fail if a user in the room hadn't consented to the privacy policy. (matrix-org#4904)
Fix bug where blocked world-readable rooms were still peekable. (matrix-org#4908)
Internal Changes
Add a systemd setup that supports synapse workers. Contributed by Luca Corbatto. (matrix-org#4662)
Change from TravisCI to Buildkite for CI. (matrix-org#4752)
When presence is disabled don't send over replication. (matrix-org#4757)
Minor docstring fixes for MatrixFederationAgent. (matrix-org#4765)
Optimise EDU transmission for the federation_sender worker. (matrix-org#4770)
Update test_typing to use HomeserverTestCase. (matrix-org#4771)
Update URLs for riot.im icons and logos in the default notification templates. (matrix-org#4779)
Removed unnecessary $ from some federation endpoint path regexes. (matrix-org#4794)
Remove link to deleted title in README. (matrix-org#4795)
Clean up read-receipt handling. (matrix-org#4797)
Add some debug about processing read receipts. (matrix-org#4798)
Clean up some replication code. (matrix-org#4799)
Add some docstrings. (matrix-org#4815)
Add debug logger to try and track down matrix-org#4422. (matrix-org#4816)
Make shutdown API send explanation message to room after users have been forced joined. (matrix-org#4817)
Update example_log_config.yaml. (matrix-org#4820)
Document the generate option for the docker image. (matrix-org#4824)
Fix check-newsfragment for debian-only changes. (matrix-org#4825)
Add some debug logging for device list updates to help with matrix-org#4828. (matrix-org#4828)
Improve federation documentation, specifically .well-known support. Many thanks to @vaab. (matrix-org#4832)
Disable captcha registration by default in unit tests. (matrix-org#4839)
Add stuff back to the .gitignore. (matrix-org#4843)
Clarify what registration_shared_secret allows for. (matrix-org#4844)
Correctly log expected errors when fetching server keys. (matrix-org#4847)
Update install docs to explicitly state a full-chain (not just the top-level) TLS certificate must be provided to Synapse. This caused some people's Synapse ports to appear correct in a browser but still (rightfully so) upset the federation tester. (matrix-org#4849)
Move client read-receipt processing to federation sender worker. (matrix-org#4852)
Refactor federation TransactionQueue. (matrix-org#4855)
Comment out most options in the generated config. (matrix-org#4863)
Fix yaml library warnings by using safe_load. (matrix-org#4869)
Update Apache setup to remove location syntax. Thanks to @cwmke! (matrix-org#4870)
Reinstate test case that runs unit tests against oldest supported dependencies. (matrix-org#4879)
Update link to federation docs. (matrix-org#4881)
fix test_auto_create_auto_join_where_no_consent. (matrix-org#4886)
Use a regular HomeServerConfig object for unit tests rater than a Mock. (matrix-org#4889)
Add some notes about tuning postgres for larger deployments. (matrix-org#4895)
Add a config option for torture-testing worker replication. (matrix-org#4902)
Log requests which are simulated by the unit tests. (matrix-org#4905)
Allow newsfragments to end with exclamation marks. Exciting! (matrix-org#4912)
Refactor some more tests to use HomeserverTestCase. (matrix-org#4913)
Refactor out the state deltas portion of the user directory store and handler. (matrix-org#4917)
Fix nginx example in ACME doc. (matrix-org#4923)
Use an explicit dbname for postgres connections in the tests. (matrix-org#4928)
Fix ClientReplicationStreamProtocol.__str__(). (matrix-org#4929)

Synapse 0.99.3rc1 (2019-03-27)

-----------------------------

Features
--------

- The user directory has been rewritten to make it faster, with less chance of falling behind on a large server. ([\matrix-org#4537](matrix-org#4537), [\matrix-org#4846](matrix-org#4846), [\matrix-org#4864](matrix-org#4864), [\matrix-org#4887](matrix-org#4887), [\matrix-org#4900](matrix-org#4900), [\matrix-org#4944](matrix-org#4944))
- Add configurable rate limiting to the /register endpoint. ([\matrix-org#4735](matrix-org#4735), [\matrix-org#4804](matrix-org#4804))
- Move server key queries to federation reader. ([\matrix-org#4757](matrix-org#4757))
- Add support for /account/3pid REST endpoint to client_reader worker. ([\matrix-org#4759](matrix-org#4759))
- Add an endpoint to the admin API for querying the server version. Contributed by Joseph Weston. ([\matrix-org#4772](matrix-org#4772))
- Include a default configuration file in the 'docs' directory. ([\matrix-org#4791](matrix-org#4791), [\matrix-org#4801](matrix-org#4801))
- Synapse is now permissive about trailing slashes on some of its federation endpoints, allowing zero or more to be present. ([\matrix-org#4793](matrix-org#4793))
- Add support for /keys/query and /keys/changes REST endpoints to client_reader worker. ([\matrix-org#4796](matrix-org#4796))
- Add checks to incoming events over federation for events evading auth (aka "soft fail"). ([\matrix-org#4814](matrix-org#4814))
- Add configurable rate limiting to the /login endpoint. ([\matrix-org#4821](matrix-org#4821), [\matrix-org#4865](matrix-org#4865))
- Remove trailing slashes from certain outbound federation requests. Retry if receiving a 404. Context: matrix-org#3622. ([\matrix-org#4840](matrix-org#4840))
- Allow passing --daemonize flags to workers in the same way as with master. ([\matrix-org#4853](matrix-org#4853))
- Batch up outgoing read-receipts to reduce federation traffic. ([\matrix-org#4890](matrix-org#4890), [\matrix-org#4927](matrix-org#4927))
- Add option to disable searching the user directory. ([\matrix-org#4895](matrix-org#4895))
- Add option to disable searching of local and remote public room lists. ([\matrix-org#4896](matrix-org#4896))
- Add ability for password providers to login/register a user via 3PID (email, phone). ([\matrix-org#4931](matrix-org#4931))

Bugfixes
--------

- Fix a bug where media with spaces in the name would get a corrupted name. ([\matrix-org#2090](matrix-org#2090))
- Fix attempting to paginate in rooms where server cannot see any events, to avoid unnecessarily pulling in lots of redacted events. ([\matrix-org#4699](matrix-org#4699))
- 'event_id' is now a required parameter in federated state requests, as per the matrix spec. ([\matrix-org#4740](matrix-org#4740))
- Fix tightloop over connecting to replication server. ([\matrix-org#4749](matrix-org#4749))
- Fix parsing of Content-Disposition headers on remote media requests and URL previews. ([\matrix-org#4763](matrix-org#4763))
- Fix incorrect log about not persisting duplicate state event. ([\matrix-org#4776](matrix-org#4776))
- Fix v4v6 option in HAProxy example config. Contributed by Flakebi. ([\matrix-org#4790](matrix-org#4790))
- Handle batch updates in worker replication protocol. ([\matrix-org#4792](matrix-org#4792))
- Fix bug where we didn't correctly throttle sending of USER_IP commands over replication. ([\matrix-org#4818](matrix-org#4818))
- Fix potential race in handling missing updates in device list updates. ([\matrix-org#4829](matrix-org#4829))
- Fix bug where synapse expected an un-specced `prev_state` field on state events. ([\matrix-org#4837](matrix-org#4837))
- Transfer a user's notification settings (push rules) on room upgrade. ([\matrix-org#4838](matrix-org#4838))
- fix test_auto_create_auto_join_where_no_consent. ([\matrix-org#4886](matrix-org#4886))
- Fix a bug where hs_disabled_message was sometimes not correctly enforced. ([\matrix-org#4888](matrix-org#4888))
- Fix bug in shutdown room admin API where it would fail if a user in the room hadn't consented to the privacy policy. ([\matrix-org#4904](matrix-org#4904))
- Fix bug where blocked world-readable rooms were still peekable. ([\matrix-org#4908](matrix-org#4908))

Internal Changes
----------------

- Add a systemd setup that supports synapse workers. Contributed by Luca Corbatto. ([\matrix-org#4662](matrix-org#4662))
- Change from TravisCI to Buildkite for CI. ([\matrix-org#4752](matrix-org#4752))
- When presence is disabled don't send over replication. ([\matrix-org#4757](matrix-org#4757))
- Minor docstring fixes for MatrixFederationAgent. ([\matrix-org#4765](matrix-org#4765))
- Optimise EDU transmission for the federation_sender worker. ([\matrix-org#4770](matrix-org#4770))
- Update test_typing to use HomeserverTestCase. ([\matrix-org#4771](matrix-org#4771))
- Update URLs for riot.im icons and logos in the default notification templates. ([\matrix-org#4779](matrix-org#4779))
- Removed unnecessary $ from some federation endpoint path regexes. ([\matrix-org#4794](matrix-org#4794))
- Remove link to deleted title in README. ([\matrix-org#4795](matrix-org#4795))
- Clean up read-receipt handling. ([\matrix-org#4797](matrix-org#4797))
- Add some debug about processing read receipts. ([\matrix-org#4798](matrix-org#4798))
- Clean up some replication code. ([\matrix-org#4799](matrix-org#4799))
- Add some docstrings. ([\matrix-org#4815](matrix-org#4815))
- Add debug logger to try and track down matrix-org#4422. ([\matrix-org#4816](matrix-org#4816))
- Make shutdown API send explanation message to room after users have been forced joined. ([\matrix-org#4817](matrix-org#4817))
- Update example_log_config.yaml. ([\matrix-org#4820](matrix-org#4820))
- Document the `generate` option for the docker image. ([\matrix-org#4824](matrix-org#4824))
- Fix check-newsfragment for debian-only changes. ([\matrix-org#4825](matrix-org#4825))
- Add some debug logging for device list updates to help with matrix-org#4828. ([\matrix-org#4828](matrix-org#4828))
- Improve federation documentation, specifically .well-known support. Many thanks to @vaab. ([\matrix-org#4832](matrix-org#4832))
- Disable captcha registration by default in unit tests. ([\matrix-org#4839](matrix-org#4839))
- Add stuff back to the .gitignore. ([\matrix-org#4843](matrix-org#4843))
- Clarify what registration_shared_secret allows for. ([\matrix-org#4844](matrix-org#4844))
- Correctly log expected errors when fetching server keys. ([\matrix-org#4847](matrix-org#4847))
- Update install docs to explicitly state a full-chain (not just the top-level) TLS certificate must be provided to Synapse. This caused some people's Synapse ports to appear correct in a browser but still (rightfully so) upset the federation tester. ([\matrix-org#4849](matrix-org#4849))
- Move client read-receipt processing to federation sender worker. ([\matrix-org#4852](matrix-org#4852))
- Refactor federation TransactionQueue. ([\matrix-org#4855](matrix-org#4855))
- Comment out most options in the generated config. ([\matrix-org#4863](matrix-org#4863))
- Fix yaml library warnings by using safe_load. ([\matrix-org#4869](matrix-org#4869))
- Update Apache setup to remove location syntax. Thanks to @cwmke! ([\matrix-org#4870](matrix-org#4870))
- Reinstate test case that runs unit tests against oldest supported dependencies. ([\matrix-org#4879](matrix-org#4879))
- Update link to federation docs. ([\matrix-org#4881](matrix-org#4881))
- fix test_auto_create_auto_join_where_no_consent. ([\matrix-org#4886](matrix-org#4886))
- Use a regular HomeServerConfig object for unit tests rater than a Mock. ([\matrix-org#4889](matrix-org#4889))
- Add some notes about tuning postgres for larger deployments. ([\matrix-org#4895](matrix-org#4895))
- Add a config option for torture-testing worker replication. ([\matrix-org#4902](matrix-org#4902))
- Log requests which are simulated by the unit tests. ([\matrix-org#4905](matrix-org#4905))
- Allow newsfragments to end with exclamation marks. Exciting! ([\matrix-org#4912](matrix-org#4912))
- Refactor some more tests to use HomeserverTestCase. ([\matrix-org#4913](matrix-org#4913))
- Refactor out the state deltas portion of the user directory store and handler. ([\matrix-org#4917](matrix-org#4917))
- Fix nginx example in ACME doc. ([\matrix-org#4923](matrix-org#4923))
- Use an explicit dbname for postgres connections in the tests. ([\matrix-org#4928](matrix-org#4928))
- Fix `ClientReplicationStreamProtocol.__str__()`. ([\matrix-org#4929](matrix-org#4929))

Merge pull request matrix-org#4910 from matrix-org/erikj/third_party_…

…invite_create_spam

Add third party invite support to spam checker