Skip to content

Commit

Permalink
Merge branch 'release-v0.16.0' of github.com:matrix-org/synapse
Browse files Browse the repository at this point in the history
  • Loading branch information
erikjohnston committed Jun 9, 2016
2 parents 42a9ea3 + 8327d5d commit ba0406d
Show file tree
Hide file tree
Showing 218 changed files with 13,408 additions and 5,466 deletions.
3 changes: 3 additions & 0 deletions AUTHORS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,6 @@ Florent Violleau <floviolleau at gmail dot com>

Niklas Riekenbrauck <nikriek at gmail dot.com>
* Add JWT support for registration and login

Christoph Witzany <christoph at web.crofting.com>
* Add LDAP support for authentication
131 changes: 127 additions & 4 deletions CHANGES.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,126 @@
Changes in synapse v0.16.0 (2016-06-09)
=======================================

NB: As of v0.14 all AS config files must have an ID field.


Bug fixes:

* Don't make rooms published by default (PR #857)

Changes in synapse v0.16.0-rc2 (2016-06-08)
===========================================

Features:

* Add configuration option for tuning GC via ``gc.set_threshold`` (PR #849)

Changes:

* Record metrics about GC (PR #771, #847, #852)
* Add metric counter for number of persisted events (PR #841)

Bug fixes:

* Fix 'From' header in email notifications (PR #843)
* Fix presence where timeouts were not being fired for the first 8h after
restarts (PR #842)
* Fix bug where synapse sent malformed transactions to AS's when retrying
transactions (Commits 310197b, 8437906)

Performance Improvements:

* Remove event fetching from DB threads (PR #835)
* Change the way we cache events (PR #836)
* Add events to cache when we persist them (PR #840)


Changes in synapse v0.16.0-rc1 (2016-06-03)
===========================================

Version 0.15 was not released. See v0.15.0-rc1 below for additional changes.

Features:

* Add email notifications for missed messages (PR #759, #786, #799, #810, #815,
#821)
* Add a ``url_preview_ip_range_whitelist`` config param (PR #760)
* Add /report endpoint (PR #762)
* Add basic ignore user API (PR #763)
* Add an openidish mechanism for proving that you own a given user_id (PR #765)
* Allow clients to specify a server_name to avoid 'No known servers' (PR #794)
* Add secondary_directory_servers option to fetch room list from other servers
(PR #808, #813)

Changes:

* Report per request metrics for all of the things using request_handler (PR
#756)
* Correctly handle ``NULL`` password hashes from the database (PR #775)
* Allow receipts for events we haven't seen in the db (PR #784)
* Make synctl read a cache factor from config file (PR #785)
* Increment badge count per missed convo, not per msg (PR #793)
* Special case m.room.third_party_invite event auth to match invites (PR #814)


Bug fixes:

* Fix typo in event_auth servlet path (PR #757)
* Fix password reset (PR #758)


Performance improvements:

* Reduce database inserts when sending transactions (PR #767)
* Queue events by room for persistence (PR #768)
* Add cache to ``get_user_by_id`` (PR #772)
* Add and use ``get_domain_from_id`` (PR #773)
* Use tree cache for ``get_linearized_receipts_for_room`` (PR #779)
* Remove unused indices (PR #782)
* Add caches to ``bulk_get_push_rules*`` (PR #804)
* Cache ``get_event_reference_hashes`` (PR #806)
* Add ``get_users_with_read_receipts_in_room`` cache (PR #809)
* Use state to calculate ``get_users_in_room`` (PR #811)
* Load push rules in storage layer so that they get cached (PR #825)
* Make ``get_joined_hosts_for_room`` use get_users_in_room (PR #828)
* Poke notifier on next reactor tick (PR #829)
* Change CacheMetrics to be quicker (PR #830)


Changes in synapse v0.15.0-rc1 (2016-04-26)
===========================================

Features:

* Add login support for Javascript Web Tokens, thanks to Niklas Riekenbrauck
(PR #671,#687)
* Add URL previewing support (PR #688)
* Add login support for LDAP, thanks to Christoph Witzany (PR #701)
* Add GET endpoint for pushers (PR #716)

Changes:

* Never notify for member events (PR #667)
* Deduplicate identical ``/sync`` requests (PR #668)
* Require user to have left room to forget room (PR #673)
* Use DNS cache if within TTL (PR #677)
* Let users see their own leave events (PR #699)
* Deduplicate membership changes (PR #700)
* Increase performance of pusher code (PR #705)
* Respond with error status 504 if failed to talk to remote server (PR #731)
* Increase search performance on postgres (PR #745)

Bug fixes:

* Fix bug where disabling all notifications still resulted in push (PR #678)
* Fix bug where users couldn't reject remote invites if remote refused (PR #691)
* Fix bug where synapse attempted to backfill from itself (PR #693)
* Fix bug where profile information was not correctly added when joining remote
rooms (PR #703)
* Fix bug where register API required incorrect key name for AS registration
(PR #727)


Changes in synapse v0.14.0 (2016-03-30)
=======================================

Expand Down Expand Up @@ -511,7 +634,7 @@ Configuration:

* Add support for changing the bind host of the metrics listener via the
``metrics_bind_host`` option.


Changes in synapse v0.9.0-r5 (2015-05-21)
=========================================
Expand Down Expand Up @@ -853,7 +976,7 @@ See UPGRADE for information about changes to the client server API, including
breaking backwards compatibility with VoIP calls and registration API.

Homeserver:
* When a user changes their displayname or avatar the server will now update
* When a user changes their displayname or avatar the server will now update
all their join states to reflect this.
* The server now adds "age" key to events to indicate how old they are. This
is clock independent, so at no point does any server or webclient have to
Expand Down Expand Up @@ -911,7 +1034,7 @@ Changes in synapse 0.2.2 (2014-09-06)
=====================================

Homeserver:
* When the server returns state events it now also includes the previous
* When the server returns state events it now also includes the previous
content.
* Add support for inviting people when creating a new room.
* Make the homeserver inform the room via `m.room.aliases` when a new alias
Expand All @@ -923,7 +1046,7 @@ Webclient:
* Handle `m.room.aliases` events.
* Asynchronously send messages and show a local echo.
* Inform the UI when a message failed to send.
* Only autoscroll on receiving a new message if the user was already at the
* Only autoscroll on receiving a new message if the user was already at the
bottom of the screen.
* Add support for ban/kick reasons.

Expand Down
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ recursive-include synapse/storage/schema *.sql
recursive-include synapse/storage/schema *.py

recursive-include docs *
recursive-include res *
recursive-include scripts *
recursive-include scripts-dev *
recursive-include tests *.py
Expand Down
48 changes: 39 additions & 9 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ Installing prerequisites on Ubuntu or Debian::

sudo apt-get install build-essential python2.7-dev libffi-dev \
python-pip python-setuptools sqlite3 \
libssl-dev python-virtualenv libjpeg-dev
libssl-dev python-virtualenv libjpeg-dev libxslt1-dev

Installing prerequisites on ArchLinux::

Expand All @@ -119,7 +119,6 @@ Installing prerequisites on CentOS 7::
python-virtualenv libffi-devel openssl-devel
sudo yum groupinstall "Development Tools"


Installing prerequisites on Mac OS X::

xcode-select --install
Expand Down Expand Up @@ -151,12 +150,7 @@ In case of problems, please see the _Troubleshooting section below.
Alternatively, Silvio Fricke has contributed a Dockerfile to automate the
above in Docker at https://registry.hub.docker.com/u/silviof/docker-matrix/.

Another alternative is to install via apt from http://matrix.org/packages/debian/.
Note that these packages do not include a client - choose one from
https://matrix.org/blog/try-matrix-now/ (or build your own with
https://github.com/matrix-org/matrix-js-sdk/).

Finally, Martin Giess has created an auto-deployment process with vagrant/ansible,
Also, Martin Giess has created an auto-deployment process with vagrant/ansible,
tested with VirtualBox/AWS/DigitalOcean - see https://github.com/EMnify/matrix-synapse-auto-deploy
for details.

Expand Down Expand Up @@ -230,6 +224,19 @@ For information on how to install and use PostgreSQL, please see
Platform Specific Instructions
==============================

Debian
------

Matrix provides official Debian packages via apt from http://matrix.org/packages/debian/.
Note that these packages do not include a client - choose one from
https://matrix.org/blog/try-matrix-now/ (or build your own with one of our SDKs :)

Fedora
------

Oleg Girko provides Fedora RPMs at
https://obs.infoserver.lv/project/monitor/matrix-synapse

ArchLinux
---------

Expand Down Expand Up @@ -271,11 +278,17 @@ During setup of Synapse you need to call python2.7 directly again::
FreeBSD
-------

Synapse can be installed via FreeBSD Ports or Packages:
Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:

- Ports: ``cd /usr/ports/net/py-matrix-synapse && make install clean``
- Packages: ``pkg install py27-matrix-synapse``

NixOS
-----

Robin Lambertz has packaged Synapse for NixOS at:
https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/misc/matrix-synapse.nix

Windows Install
---------------
Synapse can be installed on Cygwin. It requires the following Cygwin packages:
Expand Down Expand Up @@ -545,6 +558,23 @@ as the primary means of identity and E2E encryption is not complete. As such,
we are running a single identity server (https://matrix.org) at the current
time.


URL Previews
============

Synapse 0.15.0 introduces an experimental new API for previewing URLs at
/_matrix/media/r0/preview_url. This is disabled by default. To turn it on
you must enable the `url_preview_enabled: True` config parameter and explicitly
specify the IP ranges that Synapse is not allowed to spider for previewing in
the `url_preview_ip_range_blacklist` configuration parameter. This is critical
from a security perspective to stop arbitrary Matrix users spidering 'internal'
URLs on your network. At the very least we recommend that your loopback and
RFC1918 IP addresses are blacklisted.

This also requires the optional lxml and netaddr python dependencies to be
installed.


Password reset
==============

Expand Down
8 changes: 8 additions & 0 deletions UPGRADE.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,14 @@ running:
python synapse/python_dependencies.py | xargs -n1 pip install
Upgrading to v0.15.0
====================

If you want to use the new URL previewing API (/_matrix/media/r0/preview_url)
then you have to explicitly enable it in the config and update your dependencies
dependencies. See README.rst for details.


Upgrading to v0.11.0
====================

Expand Down
3 changes: 1 addition & 2 deletions docs/application_services.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,5 +32,4 @@ The format of the AS configuration file is as follows:
See the spec_ for further details on how application services work.

.. _spec: https://github.com/matrix-org/matrix-doc/blob/master/specification/25_application_service_api.rst#application-service-api

.. _spec: https://matrix.org/docs/spec/application_service/unstable.html
10 changes: 10 additions & 0 deletions docs/log_contexts.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
What do I do about "Unexpected logging context" debug log-lines everywhere?

<Mjark> The logging context lives in thread local storage
<Mjark> Sometimes it gets out of sync with what it should actually be, usually because something scheduled something to run on the reactor without preserving the logging context.
<Matthew> what is the impact of it getting out of sync? and how and when should we preserve log context?
<Mjark> The impact is that some of the CPU and database metrics will be under-reported, and some log lines will be mis-attributed.
<Mjark> It should happen auto-magically in all the APIs that do IO or otherwise defer to the reactor.
<Erik> Mjark: the other place is if we branch, e.g. using defer.gatherResults

Unanswered: how and when should we preserve log context?
58 changes: 58 additions & 0 deletions docs/replication.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
Replication Architecture
========================

Motivation
----------

We'd like to be able to split some of the work that synapse does into multiple
python processes. In theory multiple synapse processes could share a single
postgresql database and we'd scale up by running more synapse processes.
However much of synapse assumes that only one process is interacting with the
database, both for assigning unique identifiers when inserting into tables,
notifying components about new updates, and for invalidating its caches.

So running multiple copies of the current code isn't an option. One way to
run multiple processes would be to have a single writer process and multiple
reader processes connected to the same database. In order to do this we'd need
a way for the reader process to invalidate its in-memory caches when an update
happens on the writer. One way to do this is for the writer to present an
append-only log of updates which the readers can consume to invalidate their
caches and to push updates to listening clients or pushers.

Synapse already stores much of its data as an append-only log so that it can
correctly respond to /sync requests so the amount of code changes needed to
expose the append-only log to the readers should be fairly minimal.

Architecture
------------

The Replication API
~~~~~~~~~~~~~~~~~~~

Synapse will optionally expose a long poll HTTP API for extracting updates. The
API will have a similar shape to /sync in that clients provide tokens
indicating where in the log they have reached and a timeout. The synapse server
then either responds with updates immediately if it already has updates or it
waits until the timeout for more updates. If the timeout expires and nothing
happened then the server returns an empty response.

However unlike the /sync API this replication API is returning synapse specific
data rather than trying to implement a matrix specification. The replication
results are returned as arrays of rows where the rows are mostly lifted
directly from the database. This avoids unnecessary JSON parsing on the server
and hopefully avoids an impedance mismatch between the data returned and the
required updates to the datastore.

This does not replicate all the database tables as many of the database tables
are indexes that can be recovered from the contents of other tables.

The format and parameters for the api are documented in
``synapse/replication/resource.py``.


The Slaved DataStore
~~~~~~~~~~~~~~~~~~~~

There are read-only version of the synapse storage layer in
``synapse/replication/slave/storage`` that use the response of the replication
API to invalidate their caches.
Loading

0 comments on commit ba0406d

Please sign in to comment.