Commit c7128061a0fe2a78d344c3f6f9700bc687ef003d - matrix-synapse

+93

-0

CHANGES.rst less more

	0	Changes in synapse v0.21.0 (2017-05-18)
	1	=======================================
	2
	3	No changes since v0.21.0-rc3
	4
	5
	6	Changes in synapse v0.21.0-rc3 (2017-05-17)
	7	===========================================
	8
	9	Features:
	10
	11	* Add per user rate-limiting overrides (PR #2208)
	12	* Add config option to limit maximum number of events requested by ``/sync``
	13	and ``/messages`` (PR #2221) Thanks to @psaavedra!
	14
	15
	16	Changes:
	17
	18	* Various small performance fixes (PR #2201, #2202, #2224, #2226, #2227, #2228,
	19	#2229)
	20	* Update username availability checker API (PR #2209, #2213)
	21	* When purging, don't de-delta state groups we're about to delete (PR #2214)
	22	* Documentation to check synapse version (PR #2215) Thanks to @hamber-dick!
	23	* Add an index to event_search to speed up purge history API (PR #2218)
	24
	25
	26	Bug fixes:
	27
	28	* Fix API to allow clients to upload one-time-keys with new sigs (PR #2206)
	29
	30
	31	Changes in synapse v0.21.0-rc2 (2017-05-08)
	32	===========================================
	33
	34	Changes:
	35
	36	* Always mark remotes as up if we receive a signed request from them (PR #2190)
	37
	38
	39	Bug fixes:
	40
	41	* Fix bug where users got pushed for rooms they had muted (PR #2200)
	42
	43
	44	Changes in synapse v0.21.0-rc1 (2017-05-08)
	45	===========================================
	46
	47	Features:
	48
	49	* Add username availability checker API (PR #2183)
	50	* Add read marker API (PR #2120)
	51
	52
	53	Changes:
	54
	55	* Enable guest access for the 3pl/3pid APIs (PR #1986)
	56	* Add setting to support TURN for guests (PR #2011)
	57	* Various performance improvements (PR #2075, #2076, #2080, #2083, #2108,
	58	#2158, #2176, #2185)
	59	* Make synctl a bit more user friendly (PR #2078, #2127) Thanks @APwhitehat!
	60	* Replace HTTP replication with TCP replication (PR #2082, #2097, #2098,
	61	#2099, #2103, #2014, #2016, #2115, #2116, #2117)
	62	* Support authenticated SMTP (PR #2102) Thanks @DanielDent!
	63	* Add a counter metric for successfully-sent transactions (PR #2121)
	64	* Propagate errors sensibly from proxied IS requests (PR #2147)
	65	* Add more granular event send metrics (PR #2178)
	66
	67
	68
	69	Bug fixes:
	70
	71	* Fix nuke-room script to work with current schema (PR #1927) Thanks
	72	@zuckschwerdt!
	73	* Fix db port script to not assume postgres tables are in the public schema
	74	(PR #2024) Thanks @jerrykan!
	75	* Fix getting latest device IP for user with no devices (PR #2118)
	76	* Fix rejection of invites to unreachable servers (PR #2145)
	77	* Fix code for reporting old verify keys in synapse (PR #2156)
	78	* Fix invite state to always include all events (PR #2163)
	79	* Fix bug where synapse would always fetch state for any missing event (PR #2170)
	80	* Fix a leak with timed out HTTP connections (PR #2180)
	81	* Fix bug where we didn't time out HTTP requests to ASes (PR #2192)
	82
	83
	84	Docs:
	85
	86	* Clarify doc for SQLite to PostgreSQL port (PR #1961) Thanks @benhylau!
	87	* Fix typo in synctl help (PR #2107) Thanks @HarHarLinks!
	88	* ``web_client_location`` documentation fix (PR #2131) Thanks @matthewjwolff!
	89	* Update README.rst with FreeBSD changes (PR #2132) Thanks @feld!
	90	* Clarify setting up metrics (PR #2149) Thanks @encks!
	91
	92
0	93	Changes in synapse v0.20.0 (2017-04-11)
1	94	=======================================
2	95

+28

-5

README.rst less more

83	83	Synapse is the reference python/twisted Matrix homeserver implementation.
84	84
85	85	System requirements:
	86
86	87	- POSIX-compliant system (tested on Linux & OS X)
87	88	- Python 2.7
88	89	- At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org

107	108	sudo pacman -S base-devel python2 python-pip \
108	109	python-setuptools python-virtualenv sqlite3
109	110
110		Installing prerequisites on CentOS 7::
	111	Installing prerequisites on CentOS 7 or Fedora 25::
111	112
112	113	sudo yum install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
113		lcms2-devel libwebp-devel tcl-devel tk-devel \
	114	lcms2-devel libwebp-devel tcl-devel tk-devel redhat-rpm-config \
114	115	python-virtualenv libffi-devel openssl-devel
115	116	sudo yum groupinstall "Development Tools"
116	117

244	245	For reliable VoIP calls to be routed via this homeserver, you MUST configure
245	246	a TURN server. See `<docs/turn-howto.rst>`_ for details.
246	247
	248	IPv6
	249	----
	250
	251	As of Synapse 0.19 we finally support IPv6, many thanks to @kyrias and @glyph
	252	for providing PR #1696.
	253
	254	However, for federation to work on hosts with IPv6 DNS servers you must
	255	be running Twisted 17.1.0 or later - see https://github.com/matrix-org/synapse/issues/1002
	256	for details. We can't make Synapse depend on Twisted 17.1 by default
	257	yet as it will break most older distributions (see https://github.com/matrix-org/synapse/pull/1909)
	258	so if you are using operating system dependencies you'll have to install your
	259	own Twisted 17.1 package via pip or backports etc.
	260
	261	If you're running in a virtualenv then pip should have installed the newest
	262	Twisted automatically, but if your virtualenv is old you will need to manually
	263	upgrade to a newer Twisted dependency via:
	264
	265	pip install Twisted>=17.1.0
	266
247	267
248	268	Running Synapse
249	269	===============

334	354	---------
335	355
336	356	The quickest way to get up and running with ArchLinux is probably with the community package
337		https://www.archlinux.org/packages/community/any/matrix-synapse/, which should pull in all
338		the necessary dependencies.
	357	https://www.archlinux.org/packages/community/any/matrix-synapse/, which should pull in most of
	358	the necessary dependencies. If the default web client is to be served (enabled by default in
	359	the generated config),
	360	https://www.archlinux.org/packages/community/any/python2-matrix-angular-sdk/ will also need to
	361	be installed.
339	362
340	363	Alternatively, to install using pip a few changes may be needed as ArchLinux
341	364	defaults to python 3, but synapse currently assumes python 2.7 by default:

372	395
373	396	Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:
374	397
375		- Ports: ``cd /usr/ports/net/py-matrix-synapse && make install clean``
	398	- Ports: ``cd /usr/ports/net-im/py-matrix-synapse && make install clean``
376	399	- Packages: ``pkg install py27-matrix-synapse``
377	400
378	401

+9

-0

UPGRADE.rst less more

27	27	git pull
28	28	# Update the versions of synapse's python dependencies.
29	29	python synapse/python_dependencies.py \| xargs -n1 pip install --upgrade
	30
	31	To check whether your update was sucessfull, run:
	32
	33	.. code:: bash
	34
	35	# replace your.server.domain with ther domain of your synaspe homeserver
	36	curl https://<your.server.domain>/_matrix/federation/v1/version
	37
	38	So for the Matrix.org HS server the URL would be: https://matrix.org/_matrix/federation/v1/version.
30	39
31	40
32	41	Upgrading to v0.15.0

+6

-10

contrib/cmdclient/http.py less more

35	35	the request body. This will be encoded as JSON.
36	36
37	37	Returns:
38		Deferred: Succeeds when we get any HTTP response.
39
40		The result of the deferred is a tuple of `(code, response)`,
41		where `response` is a dict representing the decoded JSON body.
	38	Deferred: Succeeds when we get a 2xx HTTP response. The result
	39	will be the decoded JSON body.
42	40	"""
43	41	pass
44	42
45	43	def get_json(self, url, args=None):
46		""" Get's some json from the given host homeserver and path
	44	""" Gets some json from the given host homeserver and path
47	45
48	46	Args:
49	47	url (str): The URL to GET data from.

53	51	and not a string.
54	52
55	53	Returns:
56		Deferred: Succeeds when we get any HTTP response.
57
58		The result of the deferred is a tuple of `(code, response)`,
59		where `response` is a dict representing the decoded JSON body.
	54	Deferred: Succeeds when we get a 2xx HTTP response. The result
	55	will be the decoded JSON body.
60	56	"""
61	57	pass
62	58

213	209	pass
214	210
215	211	def stopProducing(self):
216		pass⏎
	212	pass

+73

-0

docs/admin_api/user_admin_api.rst less more

	0	Query Account
	1	=============
	2
	3	This API returns information about a specific user account.
	4
	5	The api is::
	6
	7	GET /_matrix/client/r0/admin/whois/<user_id>
	8
	9	including an ``access_token`` of a server admin.
	10
	11	It returns a JSON body like the following:
	12
	13	.. code:: json
	14
	15	{
	16	"user_id": "<user_id>",
	17	"devices": {
	18	"": {
	19	"sessions": [
	20	{
	21	"connections": [
	22	{
	23	"ip": "1.2.3.4",
	24	"last_seen": 1417222374433,
	25	"user_agent": "Mozilla/5.0 ..."
	26	},
	27	{
	28	"ip": "1.2.3.10",
	29	"last_seen": 1417222374500,
	30	"user_agent": "Dalvik/2.1.0 ..."
	31	}
	32	]
	33	}
	34	]
	35	}
	36	}
	37	}
	38
	39	``last_seen`` is measured in milliseconds since the Unix epoch.
	40
	41	Deactivate Account
	42	==================
	43
	44	This API deactivates an account. It removes active access tokens, resets the
	45	password, and deletes third-party IDs (to prevent the user requesting a
	46	password reset).
	47
	48	The api is::
	49
	50	POST /_matrix/client/r0/admin/deactivate/<user_id>
	51
	52	including an ``access_token`` of a server admin, and an empty request body.
	53
	54
	55	Reset password
	56	==============
	57
	58	Changes the password of another user.
	59
	60	The api is::
	61
	62	POST /_matrix/client/r0/admin/reset_password/<user_id>
	63
	64	with a body of:
	65
	66	.. code:: json
	67
	68	{
	69	"new_password": "<secret>"
	70	}
	71
	72	including an ``access_token`` of a server admin.

+2

-3

docs/metrics-howto.rst less more

20	20
21	21	3. Add a prometheus target for synapse.
22	22
23		It needs to set the ``metrics_path`` to a non-default value::
	23	It needs to set the ``metrics_path`` to a non-default value (under ``scrape_configs``)::
24	24
25	25	- job_name: "synapse"
26	26	metrics_path: "/_synapse/metrics"
27	27	static_configs:
28		- targets:
29		"my.server.here:9092"
	28	- targets: ["my.server.here:9092"]
30	29
31	30	If your prometheus is older than 1.5.2, you will need to replace
32	31	``static_configs`` in the above with ``target_groups``.

+3

-3

docs/postgres.rst less more

111	111	run::
112	112
113	113	synapse_port_db --sqlite-database homeserver.db \
114		--postgres-config database_config.yaml
	114	--postgres-config homeserver-postgres.yaml
115	115
116	116	Once that has completed, change the synapse config to point at the PostgreSQL
117		database configuration file using the ``database_config`` parameter (see
118		`Synapse Config`_) and restart synapse. Synapse should now be running against
	117	database configuration file ``homeserver-postgres.yaml`` (i.e. rename it to
	118	``homeserver.yaml``) and restart synapse. Synapse should now be running against
119	119	PostgreSQL.

+3

-21

docs/replication.rst less more

25	25	Architecture
26	26	------------
27	27
28		The Replication API
29		~~~~~~~~~~~~~~~~~~~
	28	The Replication Protocol
	29	~~~~~~~~~~~~~~~~~~~~~~~~
30	30
31		Synapse will optionally expose a long poll HTTP API for extracting updates. The
32		API will have a similar shape to /sync in that clients provide tokens
33		indicating where in the log they have reached and a timeout. The synapse server
34		then either responds with updates immediately if it already has updates or it
35		waits until the timeout for more updates. If the timeout expires and nothing
36		happened then the server returns an empty response.
37
38		However unlike the /sync API this replication API is returning synapse specific
39		data rather than trying to implement a matrix specification. The replication
40		results are returned as arrays of rows where the rows are mostly lifted
41		directly from the database. This avoids unnecessary JSON parsing on the server
42		and hopefully avoids an impedance mismatch between the data returned and the
43		required updates to the datastore.
44
45		This does not replicate all the database tables as many of the database tables
46		are indexes that can be recovered from the contents of other tables.
47
48		The format and parameters for the api are documented in
49		``synapse/replication/resource.py``.
	31	See ``tcp_replication.rst``
50	32
51	33
52	34	The Slaved DataStore

+223

-0

docs/tcp_replication.rst less more

	0	TCP Replication
	1	===============
	2
	3	Motivation
	4	----------
	5
	6	Previously the workers used an HTTP long poll mechanism to get updates from the
	7	master, which had the problem of causing a lot of duplicate work on the server.
	8	This TCP protocol replaces those APIs with the aim of increased efficiency.
	9
	10
	11
	12	Overview
	13	--------
	14
	15	The protocol is based on fire and forget, line based commands. An example flow
	16	would be (where '>' indicates master to worker and '<' worker to master flows)::
	17
	18	> SERVER example.com
	19	< REPLICATE events 53
	20	> RDATA events 54 ["$foo1:bar.com", ...]
	21	> RDATA events 55 ["$foo4:bar.com", ...]
	22
	23	The example shows the server accepting a new connection and sending its identity
	24	with the ``SERVER`` command, followed by the client asking to subscribe to the
	25	``events`` stream from the token ``53``. The server then periodically sends ``RDATA``
	26	commands which have the format ``RDATA <stream_name> <token> <row>``, where the
	27	format of ``<row>`` is defined by the individual streams.
	28
	29	Error reporting happens by either the client or server sending an `ERROR`
	30	command, and usually the connection will be closed.
	31
	32
	33	Since the protocol is a simple line based, its possible to manually connect to
	34	the server using a tool like netcat. A few things should be noted when manually
	35	using the protocol:
	36
	37	* When subscribing to a stream using ``REPLICATE``, the special token ``NOW`` can
	38	be used to get all future updates. The special stream name ``ALL`` can be used
	39	with ``NOW`` to subscribe to all available streams.
	40	* The federation stream is only available if federation sending has been
	41	disabled on the main process.
	42	* The server will only time connections out that have sent a ``PING`` command.
	43	If a ping is sent then the connection will be closed if no further commands
	44	are receieved within 15s. Both the client and server protocol implementations
	45	will send an initial PING on connection and ensure at least one command every
	46	5s is sent (not necessarily ``PING``).
	47	* ``RDATA`` commands usually include a numeric token, however if the stream
	48	has multiple rows to replicate per token the server will send multiple
	49	``RDATA`` commands, with all but the last having a token of ``batch``. See
	50	the documentation on ``commands.RdataCommand`` for further details.
	51
	52
	53	Architecture
	54	------------
	55
	56	The basic structure of the protocol is line based, where the initial word of
	57	each line specifies the command. The rest of the line is parsed based on the
	58	command. For example, the `RDATA` command is defined as::
	59
	60	RDATA <stream_name> <token> <row_json>
	61
	62	(Note that `<row_json>` may contains spaces, but cannot contain newlines.)
	63
	64	Blank lines are ignored.
	65
	66
	67	Keep alives
	68	~~~~~~~~~~~
	69
	70	Both sides are expected to send at least one command every 5s or so, and
	71	should send a ``PING`` command if necessary. If either side do not receive a
	72	command within e.g. 15s then the connection should be closed.
	73
	74	Because the server may be connected to manually using e.g. netcat, the timeouts
	75	aren't enabled until an initial ``PING`` command is seen. Both the client and
	76	server implementations below send a ``PING`` command immediately on connection to
	77	ensure the timeouts are enabled.
	78
	79	This ensures that both sides can quickly realize if the tcp connection has gone
	80	and handle the situation appropriately.
	81
	82
	83	Start up
	84	~~~~~~~~
	85
	86	When a new connection is made, the server:
	87
	88	* Sends a ``SERVER`` command, which includes the identity of the server, allowing
	89	the client to detect if its connected to the expected server
	90	* Sends a ``PING`` command as above, to enable the client to time out connections
	91	promptly.
	92
	93	The client:
	94
	95	* Sends a ``NAME`` command, allowing the server to associate a human friendly
	96	name with the connection. This is optional.
	97	* Sends a ``PING`` as above
	98	* For each stream the client wishes to subscribe to it sends a ``REPLICATE``
	99	with the stream_name and token it wants to subscribe from.
	100	* On receipt of a ``SERVER`` command, checks that the server name matches the
	101	expected server name.
	102
	103
	104	Error handling
	105	~~~~~~~~~~~~~~
	106
	107	If either side detects an error it can send an ``ERROR`` command and close the
	108	connection.
	109
	110	If the client side loses the connection to the server it should reconnect,
	111	following the steps above.
	112
	113
	114	Congestion
	115	~~~~~~~~~~
	116
	117	If the server sends messages faster than the client can consume them the server
	118	will first buffer a (fairly large) number of commands and then disconnect the
	119	client. This ensures that we don't queue up an unbounded number of commands in
	120	memory and gives us a potential oppurtunity to squawk loudly. When/if the client
	121	recovers it can reconnect to the server and ask for missed messages.
	122
	123
	124	Reliability
	125	~~~~~~~~~~~
	126
	127	In general the replication stream should be considered an unreliable transport
	128	since e.g. commands are not resent if the connection disappears.
	129
	130	The exception to that are the replication streams, i.e. RDATA commands, since
	131	these include tokens which can be used to restart the stream on connection
	132	errors.
	133
	134	The client should keep track of the token in the last RDATA command received
	135	for each stream so that on reconneciton it can start streaming from the correct
	136	place. Note: not all RDATA have valid tokens due to batching. See
	137	``RdataCommand`` for more details.
	138
	139
	140	Example
	141	~~~~~~~
	142
	143	An example iteraction is shown below. Each line is prefixed with '>' or '<' to
	144	indicate which side is sending, these are not included on the wire::
	145
	146	* connection established *
	147	> SERVER localhost:8823
	148	> PING 1490197665618
	149	< NAME synapse.app.appservice
	150	< PING 1490197665618
	151	< REPLICATE events 1
	152	< REPLICATE backfill 1
	153	< REPLICATE caches 1
	154	> POSITION events 1
	155	> POSITION backfill 1
	156	> POSITION caches 1
	157	> RDATA caches 2 ["get_user_by_id",["@01register-user:localhost:8823"],1490197670513]
	158	> RDATA events 14 ["$149019767112vOHxz:localhost:8823",
	159	"!AFDCvgApUmpdfVjIXm:localhost:8823","m.room.guest_access","",null]
	160	< PING 1490197675618
	161	> ERROR server stopping
	162	* connection closed by server *
	163
	164	The ``POSITION`` command sent by the server is used to set the clients position
	165	without needing to send data with the ``RDATA`` command.
	166
	167
	168	An example of a batched set of ``RDATA`` is::
	169
	170	> RDATA caches batch ["get_user_by_id",["@test:localhost:8823"],1490197670513]
	171	> RDATA caches batch ["get_user_by_id",["@test2:localhost:8823"],1490197670513]
	172	> RDATA caches batch ["get_user_by_id",["@test3:localhost:8823"],1490197670513]
	173	> RDATA caches 54 ["get_user_by_id",["@test4:localhost:8823"],1490197670513]
	174
	175	In this case the client shouldn't advance their caches token until it sees the
	176	the last ``RDATA``.
	177
	178
	179	List of commands
	180	~~~~~~~~~~~~~~~~
	181
	182	The list of valid commands, with which side can send it: server (S) or client (C):
	183
	184	SERVER (S)
	185	Sent at the start to identify which server the client is talking to
	186
	187	RDATA (S)
	188	A single update in a stream
	189
	190	POSITION (S)
	191	The position of the stream has been updated
	192
	193	ERROR (S, C)
	194	There was an error
	195
	196	PING (S, C)
	197	Sent periodically to ensure the connection is still alive
	198
	199	NAME (C)
	200	Sent at the start by client to inform the server who they are
	201
	202	REPLICATE (C)
	203	Asks the server to replicate a given stream
	204
	205	USER_SYNC (C)
	206	A user has started or stopped syncing
	207
	208	FEDERATION_ACK (C)
	209	Acknowledge receipt of some federation data
	210
	211	REMOVE_PUSHER (C)
	212	Inform the server a pusher should be removed
	213
	214	INVALIDATE_CACHE (C)
	215	Inform the server a cache should be invalidated
	216
	217	SYNC (S, C)
	218	Used exclusively in tests
	219
	220
	221	See ``synapse/replication/tcp/commands.py`` for a detailed description and the
	222	format of each command.

+34

-4

docs/turn-howto.rst less more

49	49
50	50	pwgen -s 64 1
51	51
52		5. Ensure youe firewall allows traffic into the TURN server on
	52	5. Consider your security settings. TURN lets users request a relay
	53	which will connect to arbitrary IP addresses and ports. At the least
	54	we recommend:
	55
	56	# VoIP traffic is all UDP. There is no reason to let users connect to arbitrary TCP endpoints via the relay.
	57	no-tcp-relay
	58
	59	# don't let the relay ever try to connect to private IP address ranges within your network (if any)
	60	# given the turn server is likely behind your firewall, remember to include any privileged public IPs too.
	61	denied-peer-ip=10.0.0.0-10.255.255.255
	62	denied-peer-ip=192.168.0.0-192.168.255.255
	63	denied-peer-ip=172.16.0.0-172.31.255.255
	64
	65	# special case the turn server itself so that client->TURN->TURN->client flows work
	66	allowed-peer-ip=10.0.0.1
	67
	68	# consider whether you want to limit the quota of relayed streams per user (or total) to avoid risk of DoS.
	69	user-quota=12 # 4 streams per video call, so 12 streams = 3 simultaneous relayed calls per user.
	70	total-quota=1200
	71
	72	Ideally coturn should refuse to relay traffic which isn't SRTP;
	73	see https://github.com/matrix-org/synapse/issues/2009
	74
	75	6. Ensure your firewall allows traffic into the TURN server on
53	76	the ports you've configured it to listen on (remember to allow
54		both TCP and UDP if you've enabled both).
	77	both TCP and UDP TURN traffic)
55	78
56		6. If you've configured coturn to support TLS/DTLS, generate or
	79	7. If you've configured coturn to support TLS/DTLS, generate or
57	80	import your private key and certificate.
58	81
59		7. Start the turn server::
	82	8. Start the turn server::
60	83
61	84	bin/turnserver -o
62	85

82	105	to refresh credentials. The TURN REST API specification recommends
83	106	one day (86400000).
84	107
	108	4. "turn_allow_guests": Whether to allow guest users to use the TURN
	109	server. This is enabled by default, as otherwise VoIP will not
	110	work reliably for guests. However, it does introduce a security risk
	111	as it lets guests connect to arbitrary endpoints without having gone
	112	through a CAPTCHA or similar to register a real account.
	113
85	114	As an example, here is the relevant section of the config file for
86	115	matrix.org::
87	116
88	117	turn_uris: [ "turn:turn.matrix.org:3478?transport=udp", "turn:turn.matrix.org:3478?transport=tcp" ]
89	118	turn_shared_secret: n0t4ctuAllymatr1Xd0TorgSshar3d5ecret4obvIousreAsons
90	119	turn_user_lifetime: 86400000
	120	turn_allow_guests: True
91	121
92	122	Now, restart synapse::
93	123

+7

-11

docs/workers.rst less more

11	11	postgres anyway if you care about scalability).
12	12
13	13	The workers communicate with the master synapse process via a synapse-specific
14		HTTP protocol called 'replication' - analogous to MySQL or Postgres style
	14	TCP protocol called 'replication' - analogous to MySQL or Postgres style
15	15	database replication; feeding a stream of relevant data to the workers so they
16	16	can be kept in sync with the main synapse process and database state.
17	17

20	20	listeners:
21	21	- port: 9092
22	22	bind_address: '127.0.0.1'
23		type: http
24		tls: false
25		x_forwarded: false
26		resources:
27		- names: [replication]
28		compress: false
	23	type: replication
29	24
30	25	Under no circumstances should this replication API listener be exposed to the
31	26	public internet; it currently implements no authentication whatsoever and is
32		unencrypted HTTP.
	27	unencrypted.
33	28
34	29	You then create a set of configs for the various worker processes. These should be
35	30	worker configuration files should be stored in a dedicated subdirectory, to allow

49	44	You should minimise the number of overrides though to maintain a usable config.
50	45
51	46	You must specify the type of worker application (worker_app) and the replication
52		endpoint that it's talking to on the main synapse process (worker_replication_url).
	47	endpoint that it's talking to on the main synapse process (worker_replication_host
	48	and worker_replication_port).
53	49
54	50	For instance::
55	51
56	52	worker_app: synapse.app.synchrotron
57	53
58	54	# The replication listener on the synapse to talk to.
59		worker_replication_url: http://127.0.0.1:9092/_synapse/replication
	55	worker_replication_host: 127.0.0.1
	56	worker_replication_port: 9092
60	57
61	58	worker_listeners:
62	59	- type: http

94	91	All of the above is highly experimental and subject to change as Synapse evolves,
95	92	but documenting it here to help folks needing highly scalable Synapses similar
96	93	to the one running matrix.org!
97

+1

-3

scripts/synapse_port_db less more

446	446
447	447	postgres_tables = yield self.postgres_store._simple_select_onecol(
448	448	table="information_schema.tables",
449		keyvalues={
450		"table_schema": "public",
451		},
	449	keyvalues={},
452	450	retcol="distinct table_name",
453	451	)
454	452

+33

-10

scripts-dev/nuke-room-from-db.sh less more

8	8	ROOMID="$1"
9	9
10	10	sqlite3 homeserver.db <<EOF
11		DELETE FROM context_depth WHERE context = '$ROOMID';
12		DELETE FROM current_state WHERE context = '$ROOMID';
	11	DELETE FROM event_forward_extremities WHERE room_id = '$ROOMID';
	12	DELETE FROM event_backward_extremities WHERE room_id = '$ROOMID';
	13	DELETE FROM event_edges WHERE room_id = '$ROOMID';
	14	DELETE FROM room_depth WHERE room_id = '$ROOMID';
	15	DELETE FROM state_forward_extremities WHERE room_id = '$ROOMID';
	16	DELETE FROM events WHERE room_id = '$ROOMID';
	17	DELETE FROM event_json WHERE room_id = '$ROOMID';
	18	DELETE FROM state_events WHERE room_id = '$ROOMID';
	19	DELETE FROM current_state_events WHERE room_id = '$ROOMID';
	20	DELETE FROM room_memberships WHERE room_id = '$ROOMID';
13	21	DELETE FROM feedback WHERE room_id = '$ROOMID';
14		DELETE FROM messages WHERE room_id = '$ROOMID';
15		DELETE FROM pdu_backward_extremities WHERE context = '$ROOMID';
16		DELETE FROM pdu_edges WHERE context = '$ROOMID';
17		DELETE FROM pdu_forward_extremities WHERE context = '$ROOMID';
18		DELETE FROM pdus WHERE context = '$ROOMID';
19		DELETE FROM room_data WHERE room_id = '$ROOMID';
20		DELETE FROM room_memberships WHERE room_id = '$ROOMID';
	22	DELETE FROM topics WHERE room_id = '$ROOMID';
	23	DELETE FROM room_names WHERE room_id = '$ROOMID';
21	24	DELETE FROM rooms WHERE room_id = '$ROOMID';
22		DELETE FROM state_pdus WHERE context = '$ROOMID';
	25	DELETE FROM room_hosts WHERE room_id = '$ROOMID';
	26	DELETE FROM room_aliases WHERE room_id = '$ROOMID';
	27	DELETE FROM state_groups WHERE room_id = '$ROOMID';
	28	DELETE FROM state_groups_state WHERE room_id = '$ROOMID';
	29	DELETE FROM receipts_graph WHERE room_id = '$ROOMID';
	30	DELETE FROM receipts_linearized WHERE room_id = '$ROOMID';
	31	DELETE FROM event_search_content WHERE c1room_id = '$ROOMID';
	32	DELETE FROM guest_access WHERE room_id = '$ROOMID';
	33	DELETE FROM history_visibility WHERE room_id = '$ROOMID';
	34	DELETE FROM room_tags WHERE room_id = '$ROOMID';
	35	DELETE FROM room_tags_revisions WHERE room_id = '$ROOMID';
	36	DELETE FROM room_account_data WHERE room_id = '$ROOMID';
	37	DELETE FROM event_push_actions WHERE room_id = '$ROOMID';
	38	DELETE FROM local_invites WHERE room_id = '$ROOMID';
	39	DELETE FROM pusher_throttle WHERE room_id = '$ROOMID';
	40	DELETE FROM event_reports WHERE room_id = '$ROOMID';
	41	DELETE FROM public_room_list_stream WHERE room_id = '$ROOMID';
	42	DELETE FROM stream_ordering_to_exterm WHERE room_id = '$ROOMID';
	43	DELETE FROM event_auth WHERE room_id = '$ROOMID';
	44	DELETE FROM appservice_room_list WHERE room_id = '$ROOMID';
	45	VACUUM;
23	46	EOF

+1

-1

synapse/__init__.py less more

15	15	""" This is a reference implementation of a Matrix home server.
16	16	"""
17	17
18		__version__ = "0.20.0"
	18	__version__ = "0.21.0"

+11

-0

synapse/api/errors.py less more

65	65	return cs_error(self.msg)
66	66
67	67
	68	class MatrixCodeMessageException(CodeMessageException):
	69	"""An error from a general matrix endpoint, eg. from a proxied Matrix API call.
	70
	71	Attributes:
	72	errcode (str): Matrix error code e.g 'M_FORBIDDEN'
	73	"""
	74	def __init__(self, code, msg, errcode=Codes.UNKNOWN):
	75	super(MatrixCodeMessageException, self).__init__(code, msg)
	76	self.errcode = errcode
	77
	78
68	79	class SynapseError(CodeMessageException):
69	80	"""A base exception type for matrix errors which have an errcode and error
70	81	message (as well as an HTTP status code).

+22

-28

synapse/app/appservice.py less more

25	25	from synapse.replication.slave.storage.events import SlavedEventStore
26	26	from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
27	27	from synapse.replication.slave.storage.registration import SlavedRegistrationStore
	28	from synapse.replication.tcp.client import ReplicationClientHandler
28	29	from synapse.storage.engines import create_engine
29		from synapse.util.async import sleep
30	30	from synapse.util.httpresourcetree import create_resource_tree
31		from synapse.util.logcontext import LoggingContext, PreserveLoggingContext
	31	from synapse.util.logcontext import LoggingContext, PreserveLoggingContext, preserve_fn
32	32	from synapse.util.manhole import manhole
33	33	from synapse.util.rlimit import change_resource_limit
34	34	from synapse.util.versionstring import get_version_string
35	35
36	36	from synapse import events
37	37
38		from twisted.internet import reactor, defer
	38	from twisted.internet import reactor
39	39	from twisted.web.resource import Resource
40	40
41	41	from daemonize import Daemonize

119	119	else:
120	120	logger.warn("Unrecognized listener type: %s", listener["type"])
121	121
122		@defer.inlineCallbacks
123		def replicate(self):
124		http_client = self.get_simple_http_client()
125		store = self.get_datastore()
126		replication_url = self.config.worker_replication_url
127		appservice_handler = self.get_application_service_handler()
128
129		@defer.inlineCallbacks
130		def replicate(results):
131		stream = results.get("events")
132		if stream:
133		max_stream_id = stream["position"]
134		yield appservice_handler.notify_interested_services(max_stream_id)
135
136		while True:
137		try:
138		args = store.stream_positions()
139		args["timeout"] = 30000
140		result = yield http_client.get_json(replication_url, args=args)
141		yield store.process_replication(result)
142		replicate(result)
143		except:
144		logger.exception("Error replicating from %r", replication_url)
145		yield sleep(30)
	122	self.get_tcp_replication().start_replication(self)
	123
	124	def build_tcp_replication(self):
	125	return ASReplicationHandler(self)
	126
	127
	128	class ASReplicationHandler(ReplicationClientHandler):
	129	def __init__(self, hs):
	130	super(ASReplicationHandler, self).__init__(hs.get_datastore())
	131	self.appservice_handler = hs.get_application_service_handler()
	132
	133	def on_rdata(self, stream_name, token, rows):
	134	super(ASReplicationHandler, self).on_rdata(stream_name, token, rows)
	135
	136	if stream_name == "events":
	137	max_stream_id = self.store.get_room_max_stream_ordering()
	138	preserve_fn(
	139	self.appservice_handler.notify_interested_services
	140	)(max_stream_id)
146	141
147	142
148	143	def start(config_options):

198	193	reactor.run()
199	194
200	195	def start():
201		ps.replicate()
202	196	ps.get_datastore().start_profiling()
203	197	ps.get_state_handler().start_caching()
204	198

+6

-18

synapse/app/client_reader.py less more

29	29	from synapse.replication.slave.storage.directory import DirectoryStore
30	30	from synapse.replication.slave.storage.registration import SlavedRegistrationStore
31	31	from synapse.replication.slave.storage.transactions import TransactionStore
	32	from synapse.replication.tcp.client import ReplicationClientHandler
32	33	from synapse.rest.client.v1.room import PublicRoomListRestServlet
33	34	from synapse.server import HomeServer
34	35	from synapse.storage.client_ips import ClientIpStore
35	36	from synapse.storage.engines import create_engine
36		from synapse.util.async import sleep
37	37	from synapse.util.httpresourcetree import create_resource_tree
38	38	from synapse.util.logcontext import LoggingContext, PreserveLoggingContext
39	39	from synapse.util.manhole import manhole

44	44	from synapse import events
45	45
46	46
47		from twisted.internet import reactor, defer
	47	from twisted.internet import reactor
48	48	from twisted.web.resource import Resource
49	49
50	50	from daemonize import Daemonize

144	144	else:
145	145	logger.warn("Unrecognized listener type: %s", listener["type"])
146	146
147		@defer.inlineCallbacks
148		def replicate(self):
149		http_client = self.get_simple_http_client()
150		store = self.get_datastore()
151		replication_url = self.config.worker_replication_url
152
153		while True:
154		try:
155		args = store.stream_positions()
156		args["timeout"] = 30000
157		result = yield http_client.get_json(replication_url, args=args)
158		yield store.process_replication(result)
159		except:
160		logger.exception("Error replicating from %r", replication_url)
161		yield sleep(5)
	147	self.get_tcp_replication().start_replication(self)
	148
	149	def build_tcp_replication(self):
	150	return ReplicationClientHandler(self.get_datastore())
162	151
163	152
164	153	def start(config_options):

208	197	def start():
209	198	ss.get_state_handler().start_caching()
210	199	ss.get_datastore().start_profiling()
211		ss.replicate()
212	200
213	201	reactor.callWhenRunning(start)
214	202

+6

-18

synapse/app/federation_reader.py less more

26	26	from synapse.replication.slave.storage.room import RoomStore
27	27	from synapse.replication.slave.storage.transactions import TransactionStore
28	28	from synapse.replication.slave.storage.directory import DirectoryStore
	29	from synapse.replication.tcp.client import ReplicationClientHandler
29	30	from synapse.server import HomeServer
30	31	from synapse.storage.engines import create_engine
31		from synapse.util.async import sleep
32	32	from synapse.util.httpresourcetree import create_resource_tree
33	33	from synapse.util.logcontext import LoggingContext, PreserveLoggingContext
34	34	from synapse.util.manhole import manhole

41	41	from synapse import events
42	42
43	43
44		from twisted.internet import reactor, defer
	44	from twisted.internet import reactor
45	45	from twisted.web.resource import Resource
46	46
47	47	from daemonize import Daemonize

133	133	else:
134	134	logger.warn("Unrecognized listener type: %s", listener["type"])
135	135
136		@defer.inlineCallbacks
137		def replicate(self):
138		http_client = self.get_simple_http_client()
139		store = self.get_datastore()
140		replication_url = self.config.worker_replication_url
141
142		while True:
143		try:
144		args = store.stream_positions()
145		args["timeout"] = 30000
146		result = yield http_client.get_json(replication_url, args=args)
147		yield store.process_replication(result)
148		except:
149		logger.exception("Error replicating from %r", replication_url)
150		yield sleep(5)
	136	self.get_tcp_replication().start_replication(self)
	137
	138	def build_tcp_replication(self):
	139	return ReplicationClientHandler(self.get_datastore())
151	140
152	141
153	142	def start(config_options):

197	186	def start():
198	187	ss.get_state_handler().start_caching()
199	188	ss.get_datastore().start_profiling()
200		ss.replicate()
201	189
202	190	reactor.callWhenRunning(start)
203	191

+79

-116

synapse/app/federation_sender.py less more

22	22	from synapse.crypto import context_factory
23	23	from synapse.http.site import SynapseSite
24	24	from synapse.federation import send_queue
25		from synapse.federation.units import Edu
26	25	from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
27	26	from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
28	27	from synapse.replication.slave.storage.events import SlavedEventStore
29	28	from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
30	29	from synapse.replication.slave.storage.registration import SlavedRegistrationStore
	30	from synapse.replication.slave.storage.presence import SlavedPresenceStore
31	31	from synapse.replication.slave.storage.transactions import TransactionStore
32	32	from synapse.replication.slave.storage.devices import SlavedDeviceStore
	33	from synapse.replication.tcp.client import ReplicationClientHandler
33	34	from synapse.storage.engines import create_engine
34		from synapse.storage.presence import UserPresenceState
35		from synapse.util.async import sleep
	35	from synapse.util.async import Linearizer
36	36	from synapse.util.httpresourcetree import create_resource_tree
37		from synapse.util.logcontext import LoggingContext, PreserveLoggingContext
	37	from synapse.util.logcontext import LoggingContext, PreserveLoggingContext, preserve_fn
38	38	from synapse.util.manhole import manhole
39	39	from synapse.util.rlimit import change_resource_limit
40	40	from synapse.util.versionstring import get_version_string

49	49	import sys
50	50	import logging
51	51	import gc
52		import ujson as json
53	52
54	53	logger = logging.getLogger("synapse.app.appservice")
55	54
56	55
57	56	class FederationSenderSlaveStore(
58	57	SlavedDeviceInboxStore, TransactionStore, SlavedReceiptsStore, SlavedEventStore,
59		SlavedRegistrationStore, SlavedDeviceStore,
	58	SlavedRegistrationStore, SlavedDeviceStore, SlavedPresenceStore,
60	59	):
61		pass
	60	def __init__(self, db_conn, hs):
	61	super(FederationSenderSlaveStore, self).__init__(db_conn, hs)
	62
	63	# We pull out the current federation stream position now so that we
	64	# always have a known value for the federation position in memory so
	65	# that we don't have to bounce via a deferred once when we start the
	66	# replication streams.
	67	self.federation_out_pos_startup = self._get_federation_out_pos(db_conn)
	68
	69	def _get_federation_out_pos(self, db_conn):
	70	sql = (
	71	"SELECT stream_id FROM federation_stream_position"
	72	" WHERE type = ?"
	73	)
	74	sql = self.database_engine.convert_param_style(sql)
	75
	76	txn = db_conn.cursor()
	77	txn.execute(sql, ("federation",))
	78	rows = txn.fetchall()
	79	txn.close()
	80
	81	return rows[0][0] if rows else -1
62	82
63	83
64	84	class FederationSenderServer(HomeServer):

126	146	else:
127	147	logger.warn("Unrecognized listener type: %s", listener["type"])
128	148
129		@defer.inlineCallbacks
130		def replicate(self):
131		http_client = self.get_simple_http_client()
132		store = self.get_datastore()
133		replication_url = self.config.worker_replication_url
134		send_handler = FederationSenderHandler(self)
135
136		send_handler.on_start()
137
138		while True:
139		try:
140		args = store.stream_positions()
141		args.update((yield send_handler.stream_positions()))
142		args["timeout"] = 30000
143		result = yield http_client.get_json(replication_url, args=args)
144		yield store.process_replication(result)
145		yield send_handler.process_replication(result)
146		except:
147		logger.exception("Error replicating from %r", replication_url)
148		yield sleep(30)
	149	self.get_tcp_replication().start_replication(self)
	150
	151	def build_tcp_replication(self):
	152	return FederationSenderReplicationHandler(self)
	153
	154
	155	class FederationSenderReplicationHandler(ReplicationClientHandler):
	156	def __init__(self, hs):
	157	super(FederationSenderReplicationHandler, self).__init__(hs.get_datastore())
	158	self.send_handler = FederationSenderHandler(hs, self)
	159
	160	def on_rdata(self, stream_name, token, rows):
	161	super(FederationSenderReplicationHandler, self).on_rdata(
	162	stream_name, token, rows
	163	)
	164	self.send_handler.process_replication_rows(stream_name, token, rows)
	165
	166	def get_streams_to_replicate(self):
	167	args = super(FederationSenderReplicationHandler, self).get_streams_to_replicate()
	168	args.update(self.send_handler.stream_positions())
	169	return args
149	170
150	171
151	172	def start(config_options):

204	225	reactor.run()
205	226
206	227	def start():
207		ps.replicate()
208	228	ps.get_datastore().start_profiling()
209	229	ps.get_state_handler().start_caching()
210	230

228	248	"""Processes the replication stream and forwards the appropriate entries
229	249	to the federation sender.
230	250	"""
231		def __init__(self, hs):
	251	def __init__(self, hs, replication_client):
232	252	self.store = hs.get_datastore()
233	253	self.federation_sender = hs.get_federation_sender()
	254	self.replication_client = replication_client
	255
	256	self.federation_position = self.store.federation_out_pos_startup
	257	self._fed_position_linearizer = Linearizer(name="_fed_position_linearizer")
	258
	259	self._last_ack = self.federation_position
234	260
235	261	self._room_serials = {}
236	262	self._room_typing = {}

242	268	self.store.get_room_max_stream_ordering()
243	269	)
244	270
245		@defer.inlineCallbacks
246	271	def stream_positions(self):
247		stream_id = yield self.store.get_federation_out_pos("federation")
248		defer.returnValue({
249		"federation": stream_id,
250
251		# Ack stuff we've "processed", this should only be called from
252		# one process.
253		"federation_ack": stream_id,
254		})
255
256		@defer.inlineCallbacks
257		def process_replication(self, result):
	272	return {"federation": self.federation_position}
	273
	274	def process_replication_rows(self, stream_name, token, rows):
258	275	# The federation stream contains things that we want to send out, e.g.
259	276	# presence, typing, etc.
260		fed_stream = result.get("federation")
261		if fed_stream:
262		latest_id = int(fed_stream["position"])
263
264		# The federation stream containis a bunch of different types of
265		# rows that need to be handled differently. We parse the rows, put
266		# them into the appropriate collection and then send them off.
267		presence_to_send = {}
268		keyed_edus = {}
269		edus = {}
270		failures = {}
271		device_destinations = set()
272
273		# Parse the rows in the stream
274		for row in fed_stream["rows"]:
275		position, typ, content_js = row
276		content = json.loads(content_js)
277
278		if typ == send_queue.PRESENCE_TYPE:
279		destination = content["destination"]
280		state = UserPresenceState.from_dict(content["state"])
281
282		presence_to_send.setdefault(destination, []).append(state)
283		elif typ == send_queue.KEYED_EDU_TYPE:
284		key = content["key"]
285		edu = Edu(**content["edu"])
286
287		keyed_edus.setdefault(
288		edu.destination, {}
289		)[(edu.destination, tuple(key))] = edu
290		elif typ == send_queue.EDU_TYPE:
291		edu = Edu(**content)
292
293		edus.setdefault(edu.destination, []).append(edu)
294		elif typ == send_queue.FAILURE_TYPE:
295		destination = content["destination"]
296		failure = content["failure"]
297
298		failures.setdefault(destination, []).append(failure)
299		elif typ == send_queue.DEVICE_MESSAGE_TYPE:
300		device_destinations.add(content["destination"])
301		else:
302		raise Exception("Unrecognised federation type: %r", typ)
303
304		# We've finished collecting, send everything off
305		for destination, states in presence_to_send.items():
306		self.federation_sender.send_presence(destination, states)
307
308		for destination, edu_map in keyed_edus.items():
309		for key, edu in edu_map.items():
310		self.federation_sender.send_edu(
311		edu.destination, edu.edu_type, edu.content, key=key,
312		)
313
314		for destination, edu_list in edus.items():
315		for edu in edu_list:
316		self.federation_sender.send_edu(
317		edu.destination, edu.edu_type, edu.content, key=None,
318		)
319
320		for destination, failure_list in failures.items():
321		for failure in failure_list:
322		self.federation_sender.send_failure(destination, failure)
323
324		for destination in device_destinations:
325		self.federation_sender.send_device_messages(destination)
326
327		# Record where we are in the stream.
328		yield self.store.update_federation_out_pos(
329		"federation", latest_id
330		)
	277	if stream_name == "federation":
	278	send_queue.process_rows_for_federation(self.federation_sender, rows)
	279	preserve_fn(self.update_token)(token)
331	280
332	281	# We also need to poke the federation sender when new events happen
333		event_stream = result.get("events")
334		if event_stream:
335		latest_pos = event_stream["position"]
336		self.federation_sender.notify_new_events(latest_pos)
	282	elif stream_name == "events":
	283	self.federation_sender.notify_new_events(token)
	284
	285	@defer.inlineCallbacks
	286	def update_token(self, token):
	287	self.federation_position = token
	288
	289	# We linearize here to ensure we don't have races updating the token
	290	with (yield self._fed_position_linearizer.queue(None)):
	291	if self._last_ack < self.federation_position:
	292	yield self.store.update_federation_out_pos(
	293	"federation", self.federation_position
	294	)
	295
	296	# We ACK this token over replication so that the master can drop
	297	# its in memory queues
	298	self.replication_client.send_federation_ack(self.federation_position)
	299	self._last_ack = self.federation_position
337	300
338	301
339	302	if __name__ == '__main__':

+13

-6

synapse/app/homeserver.py less more

24	24	from synapse.config._base import ConfigError
25	25
26	26	from synapse.python_dependencies import (
27		check_requirements, DEPENDENCY_LINKS
	27	check_requirements, CONDITIONAL_REQUIREMENTS
28	28	)
29	29
30	30	from synapse.rest import ClientRestResource

54	54	from synapse.util.logcontext import LoggingContext, PreserveLoggingContext
55	55	from synapse.metrics import register_memory_metrics, get_metrics_for
56	56	from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
57		from synapse.replication.resource import ReplicationResource, REPLICATION_PREFIX
	57	from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
58	58	from synapse.federation.transport.server import TransportLayerServer
59	59
60	60	from synapse.util.rlimit import change_resource_limit

91	91	"\n"
92	92	"You can also disable hosting of the webclient via the\n"
93	93	"configuration option `web_client`\n"
94		% {"dep": DEPENDENCY_LINKS["matrix-angular-sdk"]}
	94	% {"dep": CONDITIONAL_REQUIREMENTS["web_client"].keys()[0]}
95	95	)
96	96	syweb_path = os.path.dirname(syweb.__file__)
97	97	webclient_path = os.path.join(syweb_path, "webclient")

164	164
165	165	if name == "metrics" and self.get_config().enable_metrics:
166	166	resources[METRICS_PREFIX] = MetricsResource(self)
167
168		if name == "replication":
169		resources[REPLICATION_PREFIX] = ReplicationResource(self)
170	167
171	168	if WEB_CLIENT_PREFIX in resources:
172	169	root_resource = RootRedirect(WEB_CLIENT_PREFIX)

221	218	),
222	219	interface=address
223	220	)
	221	elif listener["type"] == "replication":
	222	bind_addresses = listener["bind_addresses"]
	223	for address in bind_addresses:
	224	factory = ReplicationStreamProtocolFactory(self)
	225	server_listener = reactor.listenTCP(
	226	listener["port"], factory, interface=address
	227	)
	228	reactor.addSystemEventTrigger(
	229	"before", "shutdown", server_listener.stopListening,
	230	)
224	231	else:
225	232	logger.warn("Unrecognized listener type: %s", listener["type"])
226	233

+6

-18

synapse/app/media_repository.py less more

24	24	from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
25	25	from synapse.replication.slave.storage.registration import SlavedRegistrationStore
26	26	from synapse.replication.slave.storage.transactions import TransactionStore
	27	from synapse.replication.tcp.client import ReplicationClientHandler
27	28	from synapse.rest.media.v0.content_repository import ContentRepoResource
28	29	from synapse.rest.media.v1.media_repository import MediaRepositoryResource
29	30	from synapse.server import HomeServer
30	31	from synapse.storage.client_ips import ClientIpStore
31	32	from synapse.storage.engines import create_engine
32	33	from synapse.storage.media_repository import MediaRepositoryStore
33		from synapse.util.async import sleep
34	34	from synapse.util.httpresourcetree import create_resource_tree
35	35	from synapse.util.logcontext import LoggingContext, PreserveLoggingContext
36	36	from synapse.util.manhole import manhole

44	44	from synapse import events
45	45
46	46
47		from twisted.internet import reactor, defer
	47	from twisted.internet import reactor
48	48	from twisted.web.resource import Resource
49	49
50	50	from daemonize import Daemonize

141	141	else:
142	142	logger.warn("Unrecognized listener type: %s", listener["type"])
143	143
144		@defer.inlineCallbacks
145		def replicate(self):
146		http_client = self.get_simple_http_client()
147		store = self.get_datastore()
148		replication_url = self.config.worker_replication_url
149
150		while True:
151		try:
152		args = store.stream_positions()
153		args["timeout"] = 30000
154		result = yield http_client.get_json(replication_url, args=args)
155		yield store.process_replication(result)
156		except:
157		logger.exception("Error replicating from %r", replication_url)
158		yield sleep(5)
	144	self.get_tcp_replication().start_replication(self)
	145
	146	def build_tcp_replication(self):
	147	return ReplicationClientHandler(self.get_datastore())
159	148
160	149
161	150	def start(config_options):

205	194	def start():
206	195	ss.get_state_handler().start_caching()
207	196	ss.get_datastore().start_profiling()
208		ss.replicate()
209	197
210	198	reactor.callWhenRunning(start)
211	199

+45

-77

synapse/app/pusher.py less more

26	26	from synapse.replication.slave.storage.pushers import SlavedPusherStore
27	27	from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
28	28	from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
	29	from synapse.replication.tcp.client import ReplicationClientHandler
29	30	from synapse.storage.engines import create_engine
30	31	from synapse.storage import DataStore
31		from synapse.util.async import sleep
32	32	from synapse.util.httpresourcetree import create_resource_tree
33	33	from synapse.util.logcontext import LoggingContext, preserve_fn, \
34	34	PreserveLoggingContext

88	88
89	89
90	90	class PusherServer(HomeServer):
91
92	91	def get_db_conn(self, run_new_connection=True):
93	92	# Any param beginning with cp_ is a parameter for adbapi, and should
94	93	# not be passed to the database engine.

108	107	logger.info("Finished setting up.")
109	108
110	109	def remove_pusher(self, app_id, push_key, user_id):
111		http_client = self.get_simple_http_client()
112		replication_url = self.config.worker_replication_url
113		url = replication_url + "/remove_pushers"
114		return http_client.post_json_get_json(url, {
115		"remove": [{
116		"app_id": app_id,
117		"push_key": push_key,
118		"user_id": user_id,
119		}]
120		})
	110	self.get_tcp_replication().send_remove_pusher(app_id, push_key, user_id)
121	111
122	112	def _listen_http(self, listener_config):
123	113	port = listener_config["port"]

165	155	else:
166	156	logger.warn("Unrecognized listener type: %s", listener["type"])
167	157
	158	self.get_tcp_replication().start_replication(self)
	159
	160	def build_tcp_replication(self):
	161	return PusherReplicationHandler(self)
	162
	163
	164	class PusherReplicationHandler(ReplicationClientHandler):
	165	def __init__(self, hs):
	166	super(PusherReplicationHandler, self).__init__(hs.get_datastore())
	167
	168	self.pusher_pool = hs.get_pusherpool()
	169
	170	def on_rdata(self, stream_name, token, rows):
	171	super(PusherReplicationHandler, self).on_rdata(stream_name, token, rows)
	172	preserve_fn(self.poke_pushers)(stream_name, token, rows)
	173
168	174	@defer.inlineCallbacks
169		def replicate(self):
170		http_client = self.get_simple_http_client()
171		store = self.get_datastore()
172		replication_url = self.config.worker_replication_url
173		pusher_pool = self.get_pusherpool()
174
175		def stop_pusher(user_id, app_id, pushkey):
176		key = "%s:%s" % (app_id, pushkey)
177		pushers_for_user = pusher_pool.pushers.get(user_id, {})
178		pusher = pushers_for_user.pop(key, None)
179		if pusher is None:
180		return
181		logger.info("Stopping pusher %r / %r", user_id, key)
182		pusher.on_stop()
183
184		def start_pusher(user_id, app_id, pushkey):
185		key = "%s:%s" % (app_id, pushkey)
186		logger.info("Starting pusher %r / %r", user_id, key)
187		return pusher_pool._refresh_pusher(app_id, pushkey, user_id)
188
189		@defer.inlineCallbacks
190		def poke_pushers(results):
191		pushers_rows = set(
192		map(tuple, results.get("pushers", {}).get("rows", []))
	175	def poke_pushers(self, stream_name, token, rows):
	176	if stream_name == "pushers":
	177	for row in rows:
	178	if row.deleted:
	179	yield self.stop_pusher(row.user_id, row.app_id, row.pushkey)
	180	else:
	181	yield self.start_pusher(row.user_id, row.app_id, row.pushkey)
	182	elif stream_name == "events":
	183	yield self.pusher_pool.on_new_notifications(
	184	token, token,
193	185	)
194		deleted_pushers_rows = set(
195		map(tuple, results.get("deleted_pushers", {}).get("rows", []))
	186	elif stream_name == "receipts":
	187	yield self.pusher_pool.on_new_receipts(
	188	token, token, set(row.room_id for row in rows)
196	189	)
197		for row in sorted(pushers_rows \| deleted_pushers_rows):
198		if row in deleted_pushers_rows:
199		user_id, app_id, pushkey = row[1:4]
200		stop_pusher(user_id, app_id, pushkey)
201		elif row in pushers_rows:
202		user_id = row[1]
203		app_id = row[5]
204		pushkey = row[8]
205		yield start_pusher(user_id, app_id, pushkey)
206
207		stream = results.get("events")
208		if stream and stream["rows"]:
209		min_stream_id = stream["rows"][0][0]
210		max_stream_id = stream["position"]
211		preserve_fn(pusher_pool.on_new_notifications)(
212		min_stream_id, max_stream_id
213		)
214
215		stream = results.get("receipts")
216		if stream and stream["rows"]:
217		rows = stream["rows"]
218		affected_room_ids = set(row[1] for row in rows)
219		min_stream_id = rows[0][0]
220		max_stream_id = stream["position"]
221		preserve_fn(pusher_pool.on_new_receipts)(
222		min_stream_id, max_stream_id, affected_room_ids
223		)
224
225		while True:
226		try:
227		args = store.stream_positions()
228		args["timeout"] = 30000
229		result = yield http_client.get_json(replication_url, args=args)
230		yield store.process_replication(result)
231		poke_pushers(result)
232		except:
233		logger.exception("Error replicating from %r", replication_url)
234		yield sleep(30)
	190
	191	def stop_pusher(self, user_id, app_id, pushkey):
	192	key = "%s:%s" % (app_id, pushkey)
	193	pushers_for_user = self.pusher_pool.pushers.get(user_id, {})
	194	pusher = pushers_for_user.pop(key, None)
	195	if pusher is None:
	196	return
	197	logger.info("Stopping pusher %r / %r", user_id, key)
	198	pusher.on_stop()
	199
	200	def start_pusher(self, user_id, app_id, pushkey):
	201	key = "%s:%s" % (app_id, pushkey)
	202	logger.info("Starting pusher %r / %r", user_id, key)
	203	return self.pusher_pool._refresh_pusher(app_id, pushkey, user_id)
235	204
236	205
237	206	def start(config_options):

287	256	reactor.run()
288	257
289	258	def start():
290		ps.replicate()
291	259	ps.get_pusherpool().start()
292	260	ps.get_datastore().start_profiling()
293	261	ps.get_state_handler().start_caching()

+161

-222

synapse/app/synchrotron.py less more

15	15
16	16	import synapse
17	17
18		from synapse.api.constants import EventTypes, PresenceState
	18	from synapse.api.constants import EventTypes
19	19	from synapse.config._base import ConfigError
20	20	from synapse.config.homeserver import HomeServerConfig
21	21	from synapse.config.logger import setup_logging
22		from synapse.handlers.presence import PresenceHandler
	22	from synapse.handlers.presence import PresenceHandler, get_interested_parties
23	23	from synapse.http.site import SynapseSite
24	24	from synapse.http.server import JsonResource
25	25	from synapse.metrics.resource import MetricsResource, METRICS_PREFIX

39	39	from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
40	40	from synapse.replication.slave.storage.devices import SlavedDeviceStore
41	41	from synapse.replication.slave.storage.room import RoomStore
	42	from synapse.replication.tcp.client import ReplicationClientHandler
42	43	from synapse.server import HomeServer
43	44	from synapse.storage.client_ips import ClientIpStore
44	45	from synapse.storage.engines import create_engine
45		from synapse.storage.presence import PresenceStore, UserPresenceState
	46	from synapse.storage.presence import UserPresenceState
46	47	from synapse.storage.roommember import RoomMemberStore
47		from synapse.util.async import sleep
48	48	from synapse.util.httpresourcetree import create_resource_tree
49		from synapse.util.logcontext import LoggingContext, preserve_fn, \
50		PreserveLoggingContext
	49	from synapse.util.logcontext import LoggingContext, PreserveLoggingContext, preserve_fn
51	50	from synapse.util.manhole import manhole
52	51	from synapse.util.rlimit import change_resource_limit
53	52	from synapse.util.stringutils import random_string

62	61	import logging
63	62	import contextlib
64	63	import gc
65		import ujson as json
66	64
67	65	logger = logging.getLogger("synapse.app.synchrotron")
68	66

90	88	RoomMemberStore.__dict__["did_forget"]
91	89	)
92	90
93		# XXX: This is a bit broken because we don't persist the accepted list in a
94		# way that can be replicated. This means that we don't have a way to
95		# invalidate the cache correctly.
96		get_presence_list_accepted = PresenceStore.__dict__[
97		"get_presence_list_accepted"
98		]
99		get_presence_list_observers_accepted = PresenceStore.__dict__[
100		"get_presence_list_observers_accepted"
101		]
102
103	91
104	92	UPDATE_SYNCING_USERS_MS = 10 * 1000
105	93
106	94
107	95	class SynchrotronPresence(object):
108	96	def __init__(self, hs):
	97	self.hs = hs
109	98	self.is_mine_id = hs.is_mine_id
110	99	self.http_client = hs.get_simple_http_client()
111	100	self.store = hs.get_datastore()
112	101	self.user_to_num_current_syncs = {}
113		self.syncing_users_url = hs.config.worker_replication_url + "/syncing_users"
114	102	self.clock = hs.get_clock()
115	103	self.notifier = hs.get_notifier()
116	104

120	108	for state in active_presence
121	109	}
122	110
	111	# user_id -> last_sync_ms. Lists the users that have stopped syncing
	112	# but we haven't notified the master of that yet
	113	self.users_going_offline = {}
	114
	115	self._send_stop_syncing_loop = self.clock.looping_call(
	116	self.send_stop_syncing, 10 * 1000
	117	)
	118
123	119	self.process_id = random_string(16)
124	120	logger.info("Presence process_id is %r", self.process_id)
125	121
126		self._sending_sync = False
127		self._need_to_send_sync = False
128		self.clock.looping_call(
129		self._send_syncing_users_regularly,
130		UPDATE_SYNCING_USERS_MS,
131		)
132
133		reactor.addSystemEventTrigger("before", "shutdown", self._on_shutdown)
	122	def send_user_sync(self, user_id, is_syncing, last_sync_ms):
	123	self.hs.get_tcp_replication().send_user_sync(user_id, is_syncing, last_sync_ms)
	124
	125	def mark_as_coming_online(self, user_id):
	126	"""A user has started syncing. Send a UserSync to the master, unless they
	127	had recently stopped syncing.
	128
	129	Args:
	130	user_id (str)
	131	"""
	132	going_offline = self.users_going_offline.pop(user_id, None)
	133	if not going_offline:
	134	# Safe to skip because we haven't yet told the master they were offline
	135	self.send_user_sync(user_id, True, self.clock.time_msec())
	136
	137	def mark_as_going_offline(self, user_id):
	138	"""A user has stopped syncing. We wait before notifying the master as
	139	its likely they'll come back soon. This allows us to avoid sending
	140	a stopped syncing immediately followed by a started syncing notification
	141	to the master
	142
	143	Args:
	144	user_id (str)
	145	"""
	146	self.users_going_offline[user_id] = self.clock.time_msec()
	147
	148	def send_stop_syncing(self):
	149	"""Check if there are any users who have stopped syncing a while ago
	150	and haven't come back yet. If there are poke the master about them.
	151	"""
	152	now = self.clock.time_msec()
	153	for user_id, last_sync_ms in self.users_going_offline.items():
	154	if now - last_sync_ms > 10 * 1000:
	155	self.users_going_offline.pop(user_id, None)
	156	self.send_user_sync(user_id, False, last_sync_ms)
134	157
135	158	def set_state(self, user, state, ignore_status_msg=False):
136	159	# TODO Hows this supposed to work?

138	161
139	162	get_states = PresenceHandler.get_states.__func__
140	163	get_state = PresenceHandler.get_state.__func__
141		_get_interested_parties = PresenceHandler._get_interested_parties.__func__
142	164	current_state_for_users = PresenceHandler.current_state_for_users.__func__
143	165
144		@defer.inlineCallbacks
145	166	def user_syncing(self, user_id, affect_presence):
146	167	if affect_presence:
147	168	curr_sync = self.user_to_num_current_syncs.get(user_id, 0)
148	169	self.user_to_num_current_syncs[user_id] = curr_sync + 1
149		prev_states = yield self.current_state_for_users([user_id])
150		if prev_states[user_id].state == PresenceState.OFFLINE:
151		# TODO: Don't block the sync request on this HTTP hit.
152		yield self._send_syncing_users_now()
	170
	171	# If we went from no in flight sync to some, notify replication
	172	if self.user_to_num_current_syncs[user_id] == 1:
	173	self.mark_as_coming_online(user_id)
153	174
154	175	def _end():
155	176	# We check that the user_id is in user_to_num_current_syncs because

158	179	if affect_presence and user_id in self.user_to_num_current_syncs:
159	180	self.user_to_num_current_syncs[user_id] -= 1
160	181
	182	# If we went from one in flight sync to non, notify replication
	183	if self.user_to_num_current_syncs[user_id] == 0:
	184	self.mark_as_going_offline(user_id)
	185
161	186	@contextlib.contextmanager
162	187	def _user_syncing():
163	188	try:

165	190	finally:
166	191	_end()
167	192
168		defer.returnValue(_user_syncing())
169
170		@defer.inlineCallbacks
171		def _on_shutdown(self):
172		# When the synchrotron is shutdown tell the master to clear the in
173		# progress syncs for this process
174		self.user_to_num_current_syncs.clear()
175		yield self._send_syncing_users_now()
176
177		def _send_syncing_users_regularly(self):
178		# Only send an update if we aren't in the middle of sending one.
179		if not self._sending_sync:
180		preserve_fn(self._send_syncing_users_now)()
181
182		@defer.inlineCallbacks
183		def _send_syncing_users_now(self):
184		if self._sending_sync:
185		# We don't want to race with sending another update.
186		# Instead we wait for that update to finish and send another
187		# update afterwards.
188		self._need_to_send_sync = True
189		return
190
191		# Flag that we are sending an update.
192		self._sending_sync = True
193
194		yield self.http_client.post_json_get_json(self.syncing_users_url, {
195		"process_id": self.process_id,
196		"syncing_users": [
197		user_id for user_id, count in self.user_to_num_current_syncs.items()
198		if count > 0
199		],
200		})
201
202		# Unset the flag as we are no longer sending an update.
203		self._sending_sync = False
204		if self._need_to_send_sync:
205		# If something happened while we were sending the update then
206		# we might need to send another update.
207		# TODO: Check if the update that was sent matches the current state
208		# as we only need to send an update if they are different.
209		self._need_to_send_sync = False
210		yield self._send_syncing_users_now()
	193	return defer.succeed(_user_syncing())
211	194
212	195	@defer.inlineCallbacks
213	196	def notify_from_replication(self, states, stream_id):
214		parties = yield self._get_interested_parties(
215		states, calculate_remote_hosts=False
216		)
217		room_ids_to_states, users_to_states, _ = parties
	197	parties = yield get_interested_parties(self.store, states)
	198	room_ids_to_states, users_to_states = parties
218	199
219	200	self.notifier.on_new_event(
220	201	"presence_key", stream_id, rooms=room_ids_to_states.keys(),

222	203	)
223	204
224	205	@defer.inlineCallbacks
225		def process_replication(self, result):
226		stream = result.get("presence", {"rows": []})
227		states = []
228		for row in stream["rows"]:
229		(
230		position, user_id, state, last_active_ts,
231		last_federation_update_ts, last_user_sync_ts, status_msg,
232		currently_active
233		) = row
234		state = UserPresenceState(
235		user_id, state, last_active_ts,
236		last_federation_update_ts, last_user_sync_ts, status_msg,
237		currently_active
238		)
239		self.user_to_current_state[user_id] = state
240		states.append(state)
241
242		if states and "position" in stream:
243		stream_id = int(stream["position"])
244		yield self.notify_from_replication(states, stream_id)
	206	def process_replication_rows(self, token, rows):
	207	states = [UserPresenceState(
	208	row.user_id, row.state, row.last_active_ts,
	209	row.last_federation_update_ts, row.last_user_sync_ts, row.status_msg,
	210	row.currently_active
	211	) for row in rows]
	212
	213	for state in states:
	214	self.user_to_current_state[row.user_id] = state
	215
	216	stream_id = token
	217	yield self.notify_from_replication(states, stream_id)
	218
	219	def get_currently_syncing_users(self):
	220	return [
	221	user_id for user_id, count in self.user_to_num_current_syncs.iteritems()
	222	if count > 0
	223	]
245	224
246	225
247	226	class SynchrotronTyping(object):

256	235	# value which we must use for the next replication request.
257	236	return {"typing": self._latest_room_serial}
258	237
259		def process_replication(self, result):
260		stream = result.get("typing")
261		if stream:
262		self._latest_room_serial = int(stream["position"])
263
264		for row in stream["rows"]:
265		position, room_id, typing_json = row
266		typing = json.loads(typing_json)
267		self._room_serials[room_id] = position
268		self._room_typing[room_id] = typing
	238	def process_replication_rows(self, token, rows):
	239	self._latest_room_serial = token
	240
	241	for row in rows:
	242	self._room_serials[row.room_id] = token
	243	self._room_typing[row.room_id] = row.user_ids
269	244
270	245
271	246	class SynchrotronApplicationService(object):

350	325	else:
351	326	logger.warn("Unrecognized listener type: %s", listener["type"])
352	327
353		@defer.inlineCallbacks
354		def replicate(self):
355		http_client = self.get_simple_http_client()
356		store = self.get_datastore()
357		replication_url = self.config.worker_replication_url
358		notifier = self.get_notifier()
359		presence_handler = self.get_presence_handler()
360		typing_handler = self.get_typing_handler()
361
362		def notify_from_stream(
363		result, stream_name, stream_key, room=None, user=None
364		):
365		stream = result.get(stream_name)
366		if stream:
367		position_index = stream["field_names"].index("position")
368		if room:
369		room_index = stream["field_names"].index(room)
370		if user:
371		user_index = stream["field_names"].index(user)
372
373		users = ()
374		rooms = ()
375		for row in stream["rows"]:
376		position = row[position_index]
377
378		if user:
379		users = (row[user_index],)
380
381		if room:
382		rooms = (row[room_index],)
383
384		notifier.on_new_event(
385		stream_key, position, users=users, rooms=rooms
386		)
387
388		@defer.inlineCallbacks
389		def notify_device_list_update(result):
390		stream = result.get("device_lists")
391		if not stream:
392		return
393
394		position_index = stream["field_names"].index("position")
395		user_index = stream["field_names"].index("user_id")
396
397		for row in stream["rows"]:
398		position = row[position_index]
399		user_id = row[user_index]
400
401		room_ids = yield store.get_rooms_for_user(user_id)
402
403		notifier.on_new_event(
404		"device_list_key", position, rooms=room_ids,
405		)
406
407		@defer.inlineCallbacks
408		def notify(result):
409		stream = result.get("events")
410		if stream:
411		max_position = stream["position"]
412
413		event_map = yield store.get_events([row[1] for row in stream["rows"]])
414
415		for row in stream["rows"]:
416		position = row[0]
417		event_id = row[1]
418		event = event_map.get(event_id, None)
419		if not event:
420		continue
421
422		extra_users = ()
423		if event.type == EventTypes.Member:
424		extra_users = (event.state_key,)
425		notifier.on_new_room_event(
426		event, position, max_position, extra_users
427		)
428
429		notify_from_stream(
430		result, "push_rules", "push_rules_key", user="user_id"
431		)
432		notify_from_stream(
433		result, "user_account_data", "account_data_key", user="user_id"
434		)
435		notify_from_stream(
436		result, "room_account_data", "account_data_key", user="user_id"
437		)
438		notify_from_stream(
439		result, "tag_account_data", "account_data_key", user="user_id"
440		)
441		notify_from_stream(
442		result, "receipts", "receipt_key", room="room_id"
443		)
444		notify_from_stream(
445		result, "typing", "typing_key", room="room_id"
446		)
447		notify_from_stream(
448		result, "to_device", "to_device_key", user="user_id"
449		)
450		yield notify_device_list_update(result)
451
452		while True:
453		try:
454		args = store.stream_positions()
455		args.update(typing_handler.stream_positions())
456		args["timeout"] = 30000
457		result = yield http_client.get_json(replication_url, args=args)
458		yield store.process_replication(result)
459		typing_handler.process_replication(result)
460		yield presence_handler.process_replication(result)
461		yield notify(result)
462		except:
463		logger.exception("Error replicating from %r", replication_url)
464		yield sleep(5)
	328	self.get_tcp_replication().start_replication(self)
	329
	330	def build_tcp_replication(self):
	331	return SyncReplicationHandler(self)
465	332
466	333	def build_presence_handler(self):
467	334	return SynchrotronPresence(self)
468	335
469	336	def build_typing_handler(self):
470	337	return SynchrotronTyping(self)
	338
	339
	340	class SyncReplicationHandler(ReplicationClientHandler):
	341	def __init__(self, hs):
	342	super(SyncReplicationHandler, self).__init__(hs.get_datastore())
	343
	344	self.store = hs.get_datastore()
	345	self.typing_handler = hs.get_typing_handler()
	346	self.presence_handler = hs.get_presence_handler()
	347	self.notifier = hs.get_notifier()
	348
	349	self.presence_handler.sync_callback = self.send_user_sync
	350
	351	def on_rdata(self, stream_name, token, rows):
	352	super(SyncReplicationHandler, self).on_rdata(stream_name, token, rows)
	353
	354	preserve_fn(self.process_and_notify)(stream_name, token, rows)
	355
	356	def get_streams_to_replicate(self):
	357	args = super(SyncReplicationHandler, self).get_streams_to_replicate()
	358	args.update(self.typing_handler.stream_positions())
	359	return args
	360
	361	def get_currently_syncing_users(self):
	362	return self.presence_handler.get_currently_syncing_users()
	363
	364	@defer.inlineCallbacks
	365	def process_and_notify(self, stream_name, token, rows):
	366	if stream_name == "events":
	367	# We shouldn't get multiple rows per token for events stream, so
	368	# we don't need to optimise this for multiple rows.
	369	for row in rows:
	370	event = yield self.store.get_event(row.event_id)
	371	extra_users = ()
	372	if event.type == EventTypes.Member:
	373	extra_users = (event.state_key,)
	374	max_token = self.store.get_room_max_stream_ordering()
	375	self.notifier.on_new_room_event(
	376	event, token, max_token, extra_users
	377	)
	378	elif stream_name == "push_rules":
	379	self.notifier.on_new_event(
	380	"push_rules_key", token, users=[row.user_id for row in rows],
	381	)
	382	elif stream_name in ("account_data", "tag_account_data",):
	383	self.notifier.on_new_event(
	384	"account_data_key", token, users=[row.user_id for row in rows],
	385	)
	386	elif stream_name == "receipts":
	387	self.notifier.on_new_event(
	388	"receipt_key", token, rooms=[row.room_id for row in rows],
	389	)
	390	elif stream_name == "typing":
	391	self.typing_handler.process_replication_rows(token, rows)
	392	self.notifier.on_new_event(
	393	"typing_key", token, rooms=[row.room_id for row in rows],
	394	)
	395	elif stream_name == "to_device":
	396	entities = [row.entity for row in rows if row.entity.startswith("@")]
	397	if entities:
	398	self.notifier.on_new_event(
	399	"to_device_key", token, users=entities,
	400	)
	401	elif stream_name == "device_lists":
	402	all_room_ids = set()
	403	for row in rows:
	404	room_ids = yield self.store.get_rooms_for_user(row.user_id)
	405	all_room_ids.update(room_ids)
	406	self.notifier.on_new_event(
	407	"device_list_key", token, rooms=all_room_ids,
	408	)
	409	elif stream_name == "presence":
	410	yield self.presence_handler.process_replication_rows(token, rows)
471	411
472	412
473	413	def start(config_options):

513	453
514	454	def start():
515	455	ss.get_datastore().start_profiling()
516		ss.replicate()
517	456	ss.get_state_handler().start_caching()
518	457
519	458	reactor.callWhenRunning(start)

+6

-2

synapse/app/synctl.py less more

124	124	"configfile",
125	125	nargs="?",
126	126	default="homeserver.yaml",
127		help="the homeserver config file, defaults to homserver.yaml",
	127	help="the homeserver config file, defaults to homeserver.yaml",
128	128	)
129	129	parser.add_argument(
130	130	"-w", "--worker",

201	201	worker_app = worker_config["worker_app"]
202	202	worker_pidfile = worker_config["worker_pid_file"]
203	203	worker_daemonize = worker_config["worker_daemonize"]
204		assert worker_daemonize # TODO print something more user friendly
	204	assert worker_daemonize, "In config %r: expected '%s' to be True" % (
	205	worker_configfile, "worker_daemonize")
205	206	worker_cache_factor = worker_config.get("synctl_cache_factor")
206	207	workers.append(Worker(
207	208	worker_app, worker_configfile, worker_pidfile, worker_cache_factor,

232	233
233	234	if action == "start" or action == "restart":
234	235	if start_stop_synapse:
	236	# Check if synapse is already running
	237	if os.path.exists(pidfile) and pid_running(int(open(pidfile).read())):
	238	abort("synapse.app.homeserver already running")
235	239	start(configfile)
236	240
237	241	for worker in workers:

+20

-18

synapse/appservice/__init__.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	from synapse.api.constants import EventTypes
	15	from synapse.util.caches.descriptors import cachedInlineCallbacks
15	16
16	17	from twisted.internet import defer
17	18

123	124	raise ValueError(
124	125	"Expected bool for 'exclusive' in ns '%s'" % ns
125	126	)
126		if not isinstance(regex_obj.get("regex"), basestring):
	127	regex = regex_obj.get("regex")
	128	if isinstance(regex, basestring):
	129	regex_obj["regex"] = re.compile(regex) # Pre-compile regex
	130	else:
127	131	raise ValueError(
128	132	"Expected string for 'regex' in ns '%s'" % ns
129	133	)
130	134	return namespaces
131	135
132		def _matches_regex(self, test_string, namespace_key, return_obj=False):
133		if not isinstance(test_string, basestring):
134		logger.error(
135		"Expected a string to test regex against, but got %s",
136		test_string
137		)
138		return False
139
	136	def _matches_regex(self, test_string, namespace_key):
140	137	for regex_obj in self.namespaces[namespace_key]:
141		if re.match(regex_obj["regex"], test_string):
142		if return_obj:
143		return regex_obj
144		return True
145		return False
	138	if regex_obj["regex"].match(test_string):
	139	return regex_obj
	140	return None
146	141
147	142	def _is_exclusive(self, ns_key, test_string):
148		regex_obj = self._matches_regex(test_string, ns_key, return_obj=True)
	143	regex_obj = self._matches_regex(test_string, ns_key)
149	144	if regex_obj:
150	145	return regex_obj["exclusive"]
151	146	return False

165	160	if not store:
166	161	defer.returnValue(False)
167	162
168		member_list = yield store.get_users_in_room(event.room_id)
	163	does_match = yield self._matches_user_in_member_list(event.room_id, store)
	164	defer.returnValue(does_match)
	165
	166	@cachedInlineCallbacks(num_args=1, cache_context=True)
	167	def _matches_user_in_member_list(self, room_id, store, cache_context):
	168	member_list = yield store.get_users_in_room(
	169	room_id, on_invalidate=cache_context.invalidate
	170	)
169	171
170	172	# check joined member events
171	173	for user_id in member_list:

218	220	)
219	221
220	222	def is_interested_in_alias(self, alias):
221		return self._matches_regex(alias, ApplicationService.NS_ALIASES)
	223	return bool(self._matches_regex(alias, ApplicationService.NS_ALIASES))
222	224
223	225	def is_interested_in_room(self, room_id):
224		return self._matches_regex(room_id, ApplicationService.NS_ROOMS)
	226	return bool(self._matches_regex(room_id, ApplicationService.NS_ROOMS))
225	227
226	228	def is_exclusive_user(self, user_id):
227	229	return (

+16

-0

synapse/config/emailconfig.py less more

70	70	self.email_riot_base_url = email_config.get(
71	71	"riot_base_url", None
72	72	)
	73	self.email_smtp_user = email_config.get(
	74	"smtp_user", None
	75	)
	76	self.email_smtp_pass = email_config.get(
	77	"smtp_pass", None
	78	)
	79	self.require_transport_security = email_config.get(
	80	"require_transport_security", False
	81	)
73	82	if "app_name" in email_config:
74	83	self.email_app_name = email_config["app_name"]
75	84	else:

90	99	# Defining a custom URL for Riot is only needed if email notifications
91	100	# should contain links to a self-hosted installation of Riot; when set
92	101	# the "app_name" setting is ignored.
	102	#
	103	# If your SMTP server requires authentication, the optional smtp_user &
	104	# smtp_pass variables should be used
	105	#
93	106	#email:
94	107	# enable_notifs: false
95	108	# smtp_host: "localhost"
96	109	# smtp_port: 25
	110	# smtp_user: "exampleusername"
	111	# smtp_pass: "examplepassword"
	112	# require_transport_security: False
97	113	# notif_from: "Your Friendly %(app)s Home Server <noreply@example.com>"
98	114	# app_name: Matrix
99	115	# template_dir: res/templates

+1

-0

synapse/config/registration.py less more

68	68	trusted_third_party_id_servers:
69	69	- matrix.org
70	70	- vector.im
	71	- riot.im
71	72	""" % locals()
72	73
73	74	def add_arguments(self, parser):

+12

-0

synapse/config/server.py less more

33	33	# applies to some federation traffic, and so shouldn't be used to
34	34	# "disable" federation
35	35	self.send_federation = config.get("send_federation", True)
	36
	37	self.filter_timeline_limit = config.get("filter_timeline_limit", -1)
36	38
37	39	if self.public_baseurl is not None:
38	40	if self.public_baseurl[-1] != '/':

143	145	# Whether to serve a web client from the HTTP/HTTPS root resource.
144	146	web_client: True
145	147
	148	# The root directory to server for the above web client.
	149	# If left undefined, synapse will serve the matrix-angular-sdk web client.
	150	# Make sure matrix-angular-sdk is installed with pip if web_client is True
	151	# and web_client_location is undefined
	152	# web_client_location: "/path/to/web/root"
	153
146	154	# The public-facing base URL for the client API (not including _matrix/...)
147	155	# public_baseurl: https://example.com:8448/
148	156

153	161
154	162	# The GC threshold parameters to pass to `gc.set_threshold`, if defined
155	163	# gc_thresholds: [700, 10, 10]
	164
	165	# Set the limit on the returned events in the timeline in the get
	166	# and sync operations. The default value is -1, means no upper limit.
	167	# filter_timeline_limit: 5000
156	168
157	169	# List of ports that Synapse should listen on, their purpose and their
158	170	# configuration.

+8

-0

synapse/config/voip.py less more

22	22	self.turn_username = config.get("turn_username")
23	23	self.turn_password = config.get("turn_password")
24	24	self.turn_user_lifetime = self.parse_duration(config["turn_user_lifetime"])
	25	self.turn_allow_guests = config.get("turn_allow_guests", True)
25	26
26	27	def default_config(self, **kwargs):
27	28	return """\

40	41
41	42	# How long generated TURN credentials last
42	43	turn_user_lifetime: "1h"
	44
	45	# Whether guests should be allowed to use the TURN server.
	46	# This defaults to True, otherwise VoIP will be unreliable for guests.
	47	# However, it does introduce a slight security risk as it allows users to
	48	# connect to arbitrary endpoints without having first signed up for a
	49	# valid account (e.g. by passing a CAPTCHA).
	50	turn_allow_guests: True
43	51	"""

+3

-1

synapse/config/workers.py less more

27	27	self.worker_pid_file = config.get("worker_pid_file")
28	28	self.worker_log_file = config.get("worker_log_file")
29	29	self.worker_log_config = config.get("worker_log_config")
30		self.worker_replication_url = config.get("worker_replication_url")
	30	self.worker_replication_host = config.get("worker_replication_host", None)
	31	self.worker_replication_port = config.get("worker_replication_port", None)
	32	self.worker_name = config.get("worker_name", self.worker_app)
31	33
32	34	if self.worker_listeners:
33	35	for listener in self.worker_listeners:

+3

-0

synapse/events/snapshot.py less more

49	49	"prev_group",
50	50	"delta_ids",
51	51	"prev_state_events",
	52	"app_service",
52	53	]
53	54
54	55	def __init__(self):

67	68	self.delta_ids = None
68	69
69	70	self.prev_state_events = None
	71
	72	self.app_service = None

+22

-1

synapse/events/utils.py less more

224	224
225	225	def serialize_event(e, time_now_ms, as_client_event=True,
226	226	event_format=format_event_for_client_v1,
227		token_id=None, only_event_fields=None):
	227	token_id=None, only_event_fields=None, is_invite=False):
	228	"""Serialize event for clients
	229
	230	Args:
	231	e (EventBase)
	232	time_now_ms (int)
	233	as_client_event (bool)
	234	event_format
	235	token_id
	236	only_event_fields
	237	is_invite (bool): Whether this is an invite that is being sent to the
	238	invitee
	239
	240	Returns:
	241	dict
	242	"""
228	243	# FIXME(erikj): To handle the case of presence events and the like
229	244	if not isinstance(e, EventBase):
230	245	return e

250	265	if txn_id is not None:
251	266	d["unsigned"]["transaction_id"] = txn_id
252	267
	268	# If this is an invite for somebody else, then we don't care about the
	269	# invite_room_state as that's meant solely for the invitee. Other clients
	270	# will already have the state since they're in the room.
	271	if not is_invite:
	272	d["unsigned"].pop("invite_room_state", None)
	273
253	274	if as_client_event:
254	275	d = event_format(d)
255	276

+48

-2

synapse/federation/federation_client.py less more

473	473	content (object): Any additional data to put into the content field
474	474	of the event.
475	475	Return:
476		A tuple of (origin (str), event (object)) where origin is the remote
477		homeserver which generated the event.
	476	Deferred: resolves to a tuple of (origin (str), event (object))
	477	where origin is the remote homeserver which generated the event.
	478
	479	Fails with a ``CodeMessageException`` if the chosen remote server
	480	returns a 300/400 code.
	481
	482	Fails with a ``RuntimeError`` if no servers were reachable.
478	483	"""
479	484	valid_memberships = {Membership.JOIN, Membership.LEAVE}
480	485	if membership not in valid_memberships:

527	532
528	533	@defer.inlineCallbacks
529	534	def send_join(self, destinations, pdu):
	535	"""Sends a join event to one of a list of homeservers.
	536
	537	Doing so will cause the remote server to add the event to the graph,
	538	and send the event out to the rest of the federation.
	539
	540	Args:
	541	destinations (str): Candidate homeservers which are probably
	542	participating in the room.
	543	pdu (BaseEvent): event to be sent
	544
	545	Return:
	546	Deferred: resolves to a dict with members ``origin`` (a string
	547	giving the serer the event was sent to, ``state`` (?) and
	548	``auth_chain``.
	549
	550	Fails with a ``CodeMessageException`` if the chosen remote server
	551	returns a 300/400 code.
	552
	553	Fails with a ``RuntimeError`` if no servers were reachable.
	554	"""
	555
530	556	for destination in destinations:
531	557	if destination == self.server_name:
532	558	continue

634	660
635	661	@defer.inlineCallbacks
636	662	def send_leave(self, destinations, pdu):
	663	"""Sends a leave event to one of a list of homeservers.
	664
	665	Doing so will cause the remote server to add the event to the graph,
	666	and send the event out to the rest of the federation.
	667
	668	This is mostly useful to reject received invites.
	669
	670	Args:
	671	destinations (str): Candidate homeservers which are probably
	672	participating in the room.
	673	pdu (BaseEvent): event to be sent
	674
	675	Return:
	676	Deferred: resolves to None.
	677
	678	Fails with a ``CodeMessageException`` if the chosen remote server
	679	returns a non-200 code.
	680
	681	Fails with a ``RuntimeError`` if no servers were reachable.
	682	"""
637	683	for destination in destinations:
638	684	if destination == self.server_name:
639	685	continue

+10

-0

synapse/federation/federation_server.py less more

439	439	key_id: json.loads(json_bytes)
440	440	}
441	441
	442	logger.info(
	443	"Claimed one-time-keys: %s",
	444	",".join((
	445	"%s for %s:%s" % (key_id, user_id, device_id)
	446	for user_id, user_keys in json_result.iteritems()
	447	for device_id, device_keys in user_keys.iteritems()
	448	for key_id, _ in device_keys.iteritems()
	449	)),
	450	)
	451
442	452	defer.returnValue({"one_time_keys": json_result})
443	453
444	454	@defer.inlineCallbacks

+319

-76

synapse/federation/send_queue.py less more

30	30
31	31	from .units import Edu
32	32
	33	from synapse.storage.presence import UserPresenceState
33	34	from synapse.util.metrics import Measure
34	35	import synapse.metrics
35	36
36	37	from blist import sorteddict
37		import ujson
	38	from collections import namedtuple
	39
	40	import logging
	41
	42	logger = logging.getLogger(__name__)
38	43
39	44
40	45	metrics = synapse.metrics.get_metrics_for(__name__)
41
42
43		PRESENCE_TYPE = "p"
44		KEYED_EDU_TYPE = "k"
45		EDU_TYPE = "e"
46		FAILURE_TYPE = "f"
47		DEVICE_MESSAGE_TYPE = "d"
48	46
49	47
50	48	class FederationRemoteSendQueue(object):

54	52	self.server_name = hs.hostname
55	53	self.clock = hs.get_clock()
56	54	self.notifier = hs.get_notifier()
57
58		self.presence_map = {}
59		self.presence_changed = sorteddict()
60
61		self.keyed_edu = {}
62		self.keyed_edu_changed = sorteddict()
63
64		self.edus = sorteddict()
65
66		self.failures = sorteddict()
67
68		self.device_messages = sorteddict()
	55	self.is_mine_id = hs.is_mine_id
	56
	57	self.presence_map = {} # Pending presence map user_id -> UserPresenceState
	58	self.presence_changed = sorteddict() # Stream position -> user_id
	59
	60	self.keyed_edu = {} # (destination, key) -> EDU
	61	self.keyed_edu_changed = sorteddict() # stream position -> (destination, key)
	62
	63	self.edus = sorteddict() # stream position -> Edu
	64
	65	self.failures = sorteddict() # stream position -> (destination, Failure)
	66
	67	self.device_messages = sorteddict() # stream position -> destination
69	68
70	69	self.pos = 1
71	70	self.pos_time = sorteddict()

121	120	del self.presence_changed[key]
122	121
123	122	user_ids = set(
124		user_id for uids in self.presence_changed.values() for _, user_id in uids
	123	user_id
	124	for uids in self.presence_changed.itervalues()
	125	for user_id in uids
125	126	)
126	127
127	128	to_del = [

188	189
189	190	self.notifier.on_new_replication_data()
190	191
191		def send_presence(self, destination, states):
192		"""As per TransactionQueue"""
	192	def send_presence(self, states):
	193	"""As per TransactionQueue
	194
	195	Args:
	196	states (list(UserPresenceState))
	197	"""
193	198	pos = self._next_pos()
194	199
195		self.presence_map.update({
196		state.user_id: state
197		for state in states
198		})
199
200		self.presence_changed[pos] = [
201		(destination, state.user_id) for state in states
202		]
	200	# We only want to send presence for our own users, so lets always just
	201	# filter here just in case.
	202	local_states = filter(lambda s: self.is_mine_id(s.user_id), states)
	203
	204	self.presence_map.update({state.user_id: state for state in local_states})
	205	self.presence_changed[pos] = [state.user_id for state in local_states]
203	206
204	207	self.notifier.on_new_replication_data()
205	208

219	222	def get_current_token(self):
220	223	return self.pos - 1
221	224
222		def get_replication_rows(self, token, limit, federation_ack=None):
223		"""
	225	def federation_ack(self, token):
	226	self._clear_queue_before_pos(token)
	227
	228	def get_replication_rows(self, from_token, to_token, limit, federation_ack=None):
	229	"""Get rows to be sent over federation between the two tokens
	230
224	231	Args:
225		token (int)
	232	from_token (int)
	233	to_token(int)
226	234	limit (int)
227	235	federation_ack (int): Optional. The position where the worker is
228	236	explicitly acknowledged it has handled. Allows us to drop

231	239	# TODO: Handle limit.
232	240
233	241	# To handle restarts where we wrap around
234		if token > self.pos:
235		token = -1
236
	242	if from_token > self.pos:
	243	from_token = -1
	244
	245	# list of tuple(int, BaseFederationRow), where the first is the position
	246	# of the federation stream.
237	247	rows = []
238	248
239	249	# There should be only one reader, so lets delete everything its

243	253
244	254	# Fetch changed presence
245	255	keys = self.presence_changed.keys()
246		i = keys.bisect_right(token)
247		dest_user_ids = set(
248		(pos, dest_user_id)
249		for pos in keys[i:]
250		for dest_user_id in self.presence_changed[pos]
251		)
252
253		for (key, (dest, user_id)) in dest_user_ids:
254		rows.append((key, PRESENCE_TYPE, ujson.dumps({
255		"destination": dest,
256		"state": self.presence_map[user_id].as_dict(),
257		})))
	256	i = keys.bisect_right(from_token)
	257	j = keys.bisect_right(to_token) + 1
	258	dest_user_ids = [
	259	(pos, user_id)
	260	for pos in keys[i:j]
	261	for user_id in self.presence_changed[pos]
	262	]
	263
	264	for (key, user_id) in dest_user_ids:
	265	rows.append((key, PresenceRow(
	266	state=self.presence_map[user_id],
	267	)))
258	268
259	269	# Fetch changes keyed edus
260	270	keys = self.keyed_edu_changed.keys()
261		i = keys.bisect_right(token)
262		keyed_edus = set((k, self.keyed_edu_changed[k]) for k in keys[i:])
263
264		for (pos, (destination, edu_key)) in keyed_edus:
265		rows.append(
266		(pos, KEYED_EDU_TYPE, ujson.dumps({
267		"key": edu_key,
268		"edu": self.keyed_edu[(destination, edu_key)].get_internal_dict(),
269		}))
270		)
	271	i = keys.bisect_right(from_token)
	272	j = keys.bisect_right(to_token) + 1
	273	# We purposefully clobber based on the key here, python dict comprehensions
	274	# always use the last value, so this will correctly point to the last
	275	# stream position.
	276	keyed_edus = {self.keyed_edu_changed[k]: k for k in keys[i:j]}
	277
	278	for ((destination, edu_key), pos) in keyed_edus.iteritems():
	279	rows.append((pos, KeyedEduRow(
	280	key=edu_key,
	281	edu=self.keyed_edu[(destination, edu_key)],
	282	)))
271	283
272	284	# Fetch changed edus
273	285	keys = self.edus.keys()
274		i = keys.bisect_right(token)
275		edus = set((k, self.edus[k]) for k in keys[i:])
	286	i = keys.bisect_right(from_token)
	287	j = keys.bisect_right(to_token) + 1
	288	edus = ((k, self.edus[k]) for k in keys[i:j])
276	289
277	290	for (pos, edu) in edus:
278		rows.append((pos, EDU_TYPE, ujson.dumps(edu.get_internal_dict())))
	291	rows.append((pos, EduRow(edu)))
279	292
280	293	# Fetch changed failures
281	294	keys = self.failures.keys()
282		i = keys.bisect_right(token)
283		failures = set((k, self.failures[k]) for k in keys[i:])
	295	i = keys.bisect_right(from_token)
	296	j = keys.bisect_right(to_token) + 1
	297	failures = ((k, self.failures[k]) for k in keys[i:j])
284	298
285	299	for (pos, (destination, failure)) in failures:
286		rows.append((pos, FAILURE_TYPE, ujson.dumps({
287		"destination": destination,
288		"failure": failure,
289		})))
	300	rows.append((pos, FailureRow(
	301	destination=destination,
	302	failure=failure,
	303	)))
290	304
291	305	# Fetch changed device messages
292	306	keys = self.device_messages.keys()
293		i = keys.bisect_right(token)
294		device_messages = set((k, self.device_messages[k]) for k in keys[i:])
295
296		for (pos, destination) in device_messages:
297		rows.append((pos, DEVICE_MESSAGE_TYPE, ujson.dumps({
298		"destination": destination,
299		})))
	307	i = keys.bisect_right(from_token)
	308	j = keys.bisect_right(to_token) + 1
	309	device_messages = {self.device_messages[k]: k for k in keys[i:j]}
	310
	311	for (destination, pos) in device_messages.iteritems():
	312	rows.append((pos, DeviceRow(
	313	destination=destination,
	314	)))
300	315
301	316	# Sort rows based on pos
302	317	rows.sort()
303	318
304		return rows
	319	return [(pos, row.TypeId, row.to_data()) for pos, row in rows]
	320
	321
	322	class BaseFederationRow(object):
	323	"""Base class for rows to be sent in the federation stream.
	324
	325	Specifies how to identify, serialize and deserialize the different types.
	326	"""
	327
	328	TypeId = None # Unique string that ids the type. Must be overriden in sub classes.
	329
	330	@staticmethod
	331	def from_data(data):
	332	"""Parse the data from the federation stream into a row.
	333
	334	Args:
	335	data: The value of ``data`` from FederationStreamRow.data, type
	336	depends on the type of stream
	337	"""
	338	raise NotImplementedError()
	339
	340	def to_data(self):
	341	"""Serialize this row to be sent over the federation stream.
	342
	343	Returns:
	344	The value to be sent in FederationStreamRow.data. The type depends
	345	on the type of stream.
	346	"""
	347	raise NotImplementedError()
	348
	349	def add_to_buffer(self, buff):
	350	"""Add this row to the appropriate field in the buffer ready for this
	351	to be sent over federation.
	352
	353	We use a buffer so that we can batch up events that have come in at
	354	the same time and send them all at once.
	355
	356	Args:
	357	buff (BufferedToSend)
	358	"""
	359	raise NotImplementedError()
	360
	361
	362	class PresenceRow(BaseFederationRow, namedtuple("PresenceRow", (
	363	"state", # UserPresenceState
	364	))):
	365	TypeId = "p"
	366
	367	@staticmethod
	368	def from_data(data):
	369	return PresenceRow(
	370	state=UserPresenceState.from_dict(data)
	371	)
	372
	373	def to_data(self):
	374	return self.state.as_dict()
	375
	376	def add_to_buffer(self, buff):
	377	buff.presence.append(self.state)
	378
	379
	380	class KeyedEduRow(BaseFederationRow, namedtuple("KeyedEduRow", (
	381	"key", # tuple(str) - the edu key passed to send_edu
	382	"edu", # Edu
	383	))):
	384	"""Streams EDUs that have an associated key that is ued to clobber. For example,
	385	typing EDUs clobber based on room_id.
	386	"""
	387
	388	TypeId = "k"
	389
	390	@staticmethod
	391	def from_data(data):
	392	return KeyedEduRow(
	393	key=tuple(data["key"]),
	394	edu=Edu(**data["edu"]),
	395	)
	396
	397	def to_data(self):
	398	return {
	399	"key": self.key,
	400	"edu": self.edu.get_internal_dict(),
	401	}
	402
	403	def add_to_buffer(self, buff):
	404	buff.keyed_edus.setdefault(
	405	self.edu.destination, {}
	406	)[self.key] = self.edu
	407
	408
	409	class EduRow(BaseFederationRow, namedtuple("EduRow", (
	410	"edu", # Edu
	411	))):
	412	"""Streams EDUs that don't have keys. See KeyedEduRow
	413	"""
	414	TypeId = "e"
	415
	416	@staticmethod
	417	def from_data(data):
	418	return EduRow(Edu(**data))
	419
	420	def to_data(self):
	421	return self.edu.get_internal_dict()
	422
	423	def add_to_buffer(self, buff):
	424	buff.edus.setdefault(self.edu.destination, []).append(self.edu)
	425
	426
	427	class FailureRow(BaseFederationRow, namedtuple("FailureRow", (
	428	"destination", # str
	429	"failure",
	430	))):
	431	"""Streams failures to a remote server. Failures are issued when there was
	432	something wrong with a transaction the remote sent us, e.g. it included
	433	an event that was invalid.
	434	"""
	435
	436	TypeId = "f"
	437
	438	@staticmethod
	439	def from_data(data):
	440	return FailureRow(
	441	destination=data["destination"],
	442	failure=data["failure"],
	443	)
	444
	445	def to_data(self):
	446	return {
	447	"destination": self.destination,
	448	"failure": self.failure,
	449	}
	450
	451	def add_to_buffer(self, buff):
	452	buff.failures.setdefault(self.destination, []).append(self.failure)
	453
	454
	455	class DeviceRow(BaseFederationRow, namedtuple("DeviceRow", (
	456	"destination", # str
	457	))):
	458	"""Streams the fact that either a) there is pending to device messages for
	459	users on the remote, or b) a local users device has changed and needs to
	460	be sent to the remote.
	461	"""
	462	TypeId = "d"
	463
	464	@staticmethod
	465	def from_data(data):
	466	return DeviceRow(destination=data["destination"])
	467
	468	def to_data(self):
	469	return {"destination": self.destination}
	470
	471	def add_to_buffer(self, buff):
	472	buff.device_destinations.add(self.destination)
	473
	474
	475	TypeToRow = {
	476	Row.TypeId: Row
	477	for Row in (
	478	PresenceRow,
	479	KeyedEduRow,
	480	EduRow,
	481	FailureRow,
	482	DeviceRow,
	483	)
	484	}
	485
	486
	487	ParsedFederationStreamData = namedtuple("ParsedFederationStreamData", (
	488	"presence", # list(UserPresenceState)
	489	"keyed_edus", # dict of destination -> { key -> Edu }
	490	"edus", # dict of destination -> [Edu]
	491	"failures", # dict of destination -> [failures]
	492	"device_destinations", # set of destinations
	493	))
	494
	495
	496	def process_rows_for_federation(transaction_queue, rows):
	497	"""Parse a list of rows from the federation stream and put them in the
	498	transaction queue ready for sending to the relevant homeservers.
	499
	500	Args:
	501	transaction_queue (TransactionQueue)
	502	rows (list(synapse.replication.tcp.streams.FederationStreamRow))
	503	"""
	504
	505	# The federation stream contains a bunch of different types of
	506	# rows that need to be handled differently. We parse the rows, put
	507	# them into the appropriate collection and then send them off.
	508
	509	buff = ParsedFederationStreamData(
	510	presence=[],
	511	keyed_edus={},
	512	edus={},
	513	failures={},
	514	device_destinations=set(),
	515	)
	516
	517	# Parse the rows in the stream and add to the buffer
	518	for row in rows:
	519	if row.type not in TypeToRow:
	520	logger.error("Unrecognized federation row type %r", row.type)
	521	continue
	522
	523	RowType = TypeToRow[row.type]
	524	parsed_row = RowType.from_data(row.data)
	525	parsed_row.add_to_buffer(buff)
	526
	527	if buff.presence:
	528	transaction_queue.send_presence(buff.presence)
	529
	530	for destination, edu_map in buff.keyed_edus.iteritems():
	531	for key, edu in edu_map.items():
	532	transaction_queue.send_edu(
	533	edu.destination, edu.edu_type, edu.content, key=key,
	534	)
	535
	536	for destination, edu_list in buff.edus.iteritems():
	537	for edu in edu_list:
	538	transaction_queue.send_edu(
	539	edu.destination, edu.edu_type, edu.content, key=None,
	540	)
	541
	542	for destination, failure_list in buff.failures.iteritems():
	543	for failure in failure_list:
	544	transaction_queue.send_failure(destination, failure)
	545
	546	for destination in buff.device_destinations:
	547	transaction_queue.send_device_messages(destination)

+81

-16

synapse/federation/transaction_queue.py less more

20	20
21	21	from synapse.api.errors import HttpResponseException
22	22	from synapse.util.async import run_on_reactor
23		from synapse.util.logcontext import preserve_context_over_fn
	23	from synapse.util.logcontext import preserve_context_over_fn, preserve_fn
24	24	from synapse.util.retryutils import NotRetryingDestination, get_retry_limiter
25	25	from synapse.util.metrics import measure_func
26		from synapse.types import get_domain_from_id
27		from synapse.handlers.presence import format_user_presence_state
	26	from synapse.handlers.presence import format_user_presence_state, get_interested_remotes
28	27	import synapse.metrics
29	28
30	29	import logging

40	39	)
41	40	sent_edus_counter = client_metrics.register_counter("sent_edus")
42	41
	42	sent_transactions_counter = client_metrics.register_counter("sent_transactions")
	43
43	44
44	45	class TransactionQueue(object):
45	46	"""This class makes sure we only have one transaction in flight at

76	77	# destination -> list of tuple(edu, deferred)
77	78	self.pending_edus_by_dest = edus = {}
78	79
79		# Presence needs to be separate as we send single aggragate EDUs
	80	# Map of user_id -> UserPresenceState for all the pending presence
	81	# to be sent out by user_id. Entries here get processed and put in
	82	# pending_presence_by_dest
	83	self.pending_presence = {}
	84
	85	# Map of destination -> user_id -> UserPresenceState of pending presence
	86	# to be sent to each destinations
80	87	self.pending_presence_by_dest = presence = {}
	88
	89	# Pending EDUs by their "key". Keyed EDUs are EDUs that get clobbered
	90	# based on their key (e.g. typing events by room_id)
	91	# Map of destination -> (edu_type, key) -> Edu
81	92	self.pending_edus_keyed_by_dest = edus_keyed = {}
82	93
83	94	metrics.register_callback(

112	123	self._is_processing = False
113	124	self._last_poked_id = -1
114	125
	126	self._processing_pending_presence = False
	127
115	128	def can_send_to(self, destination):
116	129	"""Can we send messages to the given server?
117	130

168	181	# Otherwise if the last member on a server in a room is
169	182	# banned then it won't receive the event because it won't
170	183	# be in the room after the ban.
171		users_in_room = yield self.state.get_current_user_in_room(
	184	destinations = yield self.state.get_current_hosts_in_room(
172	185	event.room_id, latest_event_ids=[
173	186	prev_id for prev_id, _ in event.prev_events
174	187	],
175	188	)
176	189
177		destinations = set(
178		get_domain_from_id(user_id) for user_id in users_in_room
179		)
180	190	if send_on_behalf_of is not None:
181	191	# If we are sending the event on behalf of another server
182	192	# then it already has the event and there is no reason to

223	233	self._attempt_new_transaction, destination
224	234	)
225	235
226		def send_presence(self, destination, states):
227		if not self.can_send_to(destination):
228		return
229
230		self.pending_presence_by_dest.setdefault(destination, {}).update({
	236	@preserve_fn # the caller should not yield on this
	237	@defer.inlineCallbacks
	238	def send_presence(self, states):
	239	"""Send the new presence states to the appropriate destinations.
	240
	241	This actually queues up the presence states ready for sending and
	242	triggers a background task to process them and send out the transactions.
	243
	244	Args:
	245	states (list(UserPresenceState))
	246	"""
	247
	248	# First we queue up the new presence by user ID, so multiple presence
	249	# updates in quick successtion are correctly handled
	250	# We only want to send presence for our own users, so lets always just
	251	# filter here just in case.
	252	self.pending_presence.update({
231	253	state.user_id: state for state in states
	254	if self.is_mine_id(state.user_id)
232	255	})
233	256
234		preserve_context_over_fn(
235		self._attempt_new_transaction, destination
236		)
	257	# We then handle the new pending presence in batches, first figuring
	258	# out the destinations we need to send each state to and then poking it
	259	# to attempt a new transaction. We linearize this so that we don't
	260	# accidentally mess up the ordering and send multiple presence updates
	261	# in the wrong order
	262	if self._processing_pending_presence:
	263	return
	264
	265	self._processing_pending_presence = True
	266	try:
	267	while True:
	268	states_map = self.pending_presence
	269	self.pending_presence = {}
	270
	271	if not states_map:
	272	break
	273
	274	yield self._process_presence_inner(states_map.values())
	275	finally:
	276	self._processing_pending_presence = False
	277
	278	@measure_func("txnqueue._process_presence")
	279	@defer.inlineCallbacks
	280	def _process_presence_inner(self, states):
	281	"""Given a list of states populate self.pending_presence_by_dest and
	282	poke to send a new transaction to each destination
	283
	284	Args:
	285	states (list(UserPresenceState))
	286	"""
	287	hosts_and_states = yield get_interested_remotes(self.store, states, self.state)
	288
	289	for destinations, states in hosts_and_states:
	290	for destination in destinations:
	291	if not self.can_send_to(destination):
	292	continue
	293
	294	self.pending_presence_by_dest.setdefault(
	295	destination, {}
	296	).update({
	297	state.user_id: state for state in states
	298	})
	299
	300	preserve_fn(self._attempt_new_transaction)(destination)
237	301
238	302	def send_edu(self, destination, edu_type, content, key=None):
239	303	edu = Edu(

373	437	destination, pending_pdus, pending_edus, pending_failures,
374	438	)
375	439	if success:
	440	sent_transactions_counter.inc()
376	441	# Remove the acknowledged device messages from the database
377	442	# Only bother if we actually sent some device messages
378	443	if device_message_edus:

+39

-1

synapse/federation/transport/client.py less more

192	192	@defer.inlineCallbacks
193	193	@log_function
194	194	def make_membership_event(self, destination, room_id, user_id, membership):
	195	"""Asks a remote server to build and sign us a membership event
	196
	197	Note that this does not append any events to any graphs.
	198
	199	Args:
	200	destination (str): address of remote homeserver
	201	room_id (str): room to join/leave
	202	user_id (str): user to be joined/left
	203	membership (str): one of join/leave
	204
	205	Returns:
	206	Deferred: Succeeds when we get a 2xx HTTP response. The result
	207	will be the decoded JSON body (ie, the new event).
	208
	209	Fails with ``HTTPRequestException`` if we get an HTTP response
	210	code >= 300.
	211
	212	Fails with ``NotRetryingDestination`` if we are not yet ready
	213	to retry this server.
	214	"""
195	215	valid_memberships = {Membership.JOIN, Membership.LEAVE}
196	216	if membership not in valid_memberships:
197	217	raise RuntimeError(

200	220	)
201	221	path = PREFIX + "/make_%s/%s/%s" % (membership, room_id, user_id)
202	222
	223	ignore_backoff = False
	224	retry_on_dns_fail = False
	225
	226	if membership == Membership.LEAVE:
	227	# we particularly want to do our best to send leave events. The
	228	# problem is that if it fails, we won't retry it later, so if the
	229	# remote server was just having a momentary blip, the room will be
	230	# out of sync.
	231	ignore_backoff = True
	232	retry_on_dns_fail = True
	233
203	234	content = yield self.client.get_json(
204	235	destination=destination,
205	236	path=path,
206		retry_on_dns_fail=False,
	237	retry_on_dns_fail=retry_on_dns_fail,
207	238	timeout=20000,
	239	ignore_backoff=ignore_backoff,
208	240	)
209	241
210	242	defer.returnValue(content)

231	263	destination=destination,
232	264	path=path,
233	265	data=content,
	266
	267	# we want to do our best to send this through. The problem is
	268	# that if it fails, we won't retry it later, so if the remote
	269	# server was just having a momentary blip, the room will be out of
	270	# sync.
	271	ignore_backoff=True,
234	272	)
235	273
236	274	defer.returnValue(response)

+9

-0

synapse/federation/transport/server.py less more

23	23	)
24	24	from synapse.util.ratelimitutils import FederationRateLimiter
25	25	from synapse.util.versionstring import get_version_string
	26	from synapse.util.logcontext import preserve_fn
26	27	from synapse.types import ThirdPartyInstanceID
27	28
28	29	import functools

78	79	def __init__(self, hs):
79	80	self.keyring = hs.get_keyring()
80	81	self.server_name = hs.hostname
	82	self.store = hs.get_datastore()
81	83
82	84	# A method just so we can pass 'self' as the authenticator to the Servlets
83	85	@defer.inlineCallbacks

136	138
137	139	logger.info("Request from %s", origin)
138	140	request.authenticated_entity = origin
	141
	142	# If we get a valid signed request from the other side, its probably
	143	# alive
	144	retry_timings = yield self.store.get_destination_retry_timings(origin)
	145	if retry_timings and retry_timings["retry_last_ts"]:
	146	logger.info("Marking origin %r as up", origin)
	147	preserve_fn(self.store.set_destination_retry_timings)(origin, 0, 0)
139	148
140	149	defer.returnValue(origin)
141	150

+31

-3

synapse/handlers/_base.py less more

52	52
53	53	self.event_builder_factory = hs.get_event_builder_factory()
54	54
55		def ratelimit(self, requester):
	55	@defer.inlineCallbacks
	56	def ratelimit(self, requester, update=True):
	57	"""Ratelimits requests.
	58
	59	Args:
	60	requester (Requester)
	61	update (bool): Whether to record that a request is being processed.
	62	Set to False when doing multiple checks for one request (e.g.
	63	to check up front if we would reject the request), and set to
	64	True for the last call for a given request.
	65
	66	Raises:
	67	LimitExceededError if the request should be ratelimited
	68	"""
56	69	time_now = self.clock.time()
57	70	user_id = requester.user.to_string()
58	71

66	79	if requester.app_service and not requester.app_service.is_rate_limited():
67	80	return
68	81
	82	# Check if there is a per user override in the DB.
	83	override = yield self.store.get_ratelimit_for_user(user_id)
	84	if override:
	85	# If overriden with a null Hz then ratelimiting has been entirely
	86	# disabled for the user
	87	if not override.messages_per_second:
	88	return
	89
	90	messages_per_second = override.messages_per_second
	91	burst_count = override.burst_count
	92	else:
	93	messages_per_second = self.hs.config.rc_messages_per_second
	94	burst_count = self.hs.config.rc_message_burst_count
	95
69	96	allowed, time_allowed = self.ratelimiter.send_message(
70	97	user_id, time_now,
71		msg_rate_hz=self.hs.config.rc_messages_per_second,
72		burst_count=self.hs.config.rc_message_burst_count,
	98	msg_rate_hz=messages_per_second,
	99	burst_count=burst_count,
	100	update=update,
73	101	)
74	102	if not allowed:
75	103	raise LimitExceededError(

+28

-1

synapse/handlers/device.py less more

16	16	from synapse.util import stringutils
17	17	from synapse.util.async import Linearizer
18	18	from synapse.util.caches.expiringcache import ExpiringCache
	19	from synapse.util.retryutils import NotRetryingDestination
19	20	from synapse.util.metrics import measure_func
20	21	from synapse.types import get_domain_from_id, RoomStreamToken
21	22	from twisted.internet import defer

424	425	# This can happen since we batch updates
425	426	return
426	427
	428	# Given a list of updates we check if we need to resync. This
	429	# happens if we've missed updates.
427	430	resync = yield self._need_to_do_resync(user_id, pending_updates)
428	431
429	432	if resync:
430	433	# Fetch all devices for the user.
431	434	origin = get_domain_from_id(user_id)
432		result = yield self.federation.query_user_devices(origin, user_id)
	435	try:
	436	result = yield self.federation.query_user_devices(origin, user_id)
	437	except NotRetryingDestination:
	438	# TODO: Remember that we are now out of sync and try again
	439	# later
	440	logger.warn(
	441	"Failed to handle device list update for %s,"
	442	" we're not retrying the remote",
	443	user_id,
	444	)
	445	# We abort on exceptions rather than accepting the update
	446	# as otherwise synapse will 'forget' that its device list
	447	# is out of date. If we bail then we will retry the resync
	448	# next time we get a device list update for this user_id.
	449	# This makes it more likely that the device lists will
	450	# eventually become consistent.
	451	return
	452	except Exception:
	453	# TODO: Remember that we are now out of sync and try again
	454	# later
	455	logger.exception(
	456	"Failed to handle device list update for %s", user_id
	457	)
	458	return
	459
433	460	stream_id = result["stream_id"]
434	461	devices = result["devices"]
435	462	yield self.store.update_remote_device_list_cache(

+70

-16

synapse/handlers/e2e_keys.py less more

20	20
21	21	from synapse.api.errors import SynapseError, CodeMessageException
22	22	from synapse.types import get_domain_from_id
23		from synapse.util.logcontext import preserve_fn, preserve_context_over_deferred
	23	from synapse.util.logcontext import preserve_fn, make_deferred_yieldable
24	24	from synapse.util.retryutils import NotRetryingDestination
25	25
26	26	logger = logging.getLogger(__name__)

144	144	"status": 503, "message": e.message
145	145	}
146	146
147		yield preserve_context_over_deferred(defer.gatherResults([
	147	yield make_deferred_yieldable(defer.gatherResults([
148	148	preserve_fn(do_remote_query)(destination)
149	149	for destination in remote_queries_not_in_cache
150	150	]))

256	256	"status": 503, "message": e.message
257	257	}
258	258
259		yield preserve_context_over_deferred(defer.gatherResults([
	259	yield make_deferred_yieldable(defer.gatherResults([
260	260	preserve_fn(claim_client_keys)(destination)
261	261	for destination in remote_queries
262	262	]))
	263
	264	logger.info(
	265	"Claimed one-time-keys: %s",
	266	",".join((
	267	"%s for %s:%s" % (key_id, user_id, device_id)
	268	for user_id, user_keys in json_result.iteritems()
	269	for device_id, device_keys in user_keys.iteritems()
	270	for key_id, _ in device_keys.iteritems()
	271	)),
	272	)
263	273
264	274	defer.returnValue({
265	275	"one_time_keys": json_result,

287	297
288	298	one_time_keys = keys.get("one_time_keys", None)
289	299	if one_time_keys:
290		logger.info(
291		"Adding %d one_time_keys for device %r for user %r at %d",
292		len(one_time_keys), device_id, user_id, time_now
293		)
294		key_list = []
295		for key_id, key_json in one_time_keys.items():
296		algorithm, key_id = key_id.split(":")
297		key_list.append((
298		algorithm, key_id, encode_canonical_json(key_json)
299		))
300
301		yield self.store.add_e2e_one_time_keys(
302		user_id, device_id, time_now, key_list
	300	yield self._upload_one_time_keys_for_user(
	301	user_id, device_id, time_now, one_time_keys,
303	302	)
304	303
305	304	# the device should have been registered already, but it may have been

312	311	result = yield self.store.count_e2e_one_time_keys(user_id, device_id)
313	312
314	313	defer.returnValue({"one_time_key_counts": result})
	314
	315	@defer.inlineCallbacks
	316	def _upload_one_time_keys_for_user(self, user_id, device_id, time_now,
	317	one_time_keys):
	318	logger.info(
	319	"Adding one_time_keys %r for device %r for user %r at %d",
	320	one_time_keys.keys(), device_id, user_id, time_now,
	321	)
	322
	323	# make a list of (alg, id, key) tuples
	324	key_list = []
	325	for key_id, key_obj in one_time_keys.items():
	326	algorithm, key_id = key_id.split(":")
	327	key_list.append((
	328	algorithm, key_id, key_obj
	329	))
	330
	331	# First we check if we have already persisted any of the keys.
	332	existing_key_map = yield self.store.get_e2e_one_time_keys(
	333	user_id, device_id, [k_id for _, k_id, _ in key_list]
	334	)
	335
	336	new_keys = [] # Keys that we need to insert. (alg, id, json) tuples.
	337	for algorithm, key_id, key in key_list:
	338	ex_json = existing_key_map.get((algorithm, key_id), None)
	339	if ex_json:
	340	if not _one_time_keys_match(ex_json, key):
	341	raise SynapseError(
	342	400,
	343	("One time key %s:%s already exists. "
	344	"Old key: %s; new key: %r") %
	345	(algorithm, key_id, ex_json, key)
	346	)
	347	else:
	348	new_keys.append((algorithm, key_id, encode_canonical_json(key)))
	349
	350	yield self.store.add_e2e_one_time_keys(
	351	user_id, device_id, time_now, new_keys
	352	)
	353
	354
	355	def _one_time_keys_match(old_key_json, new_key):
	356	old_key = json.loads(old_key_json)
	357
	358	# if either is a string rather than an object, they must match exactly
	359	if not isinstance(old_key, dict) or not isinstance(new_key, dict):
	360	return old_key == new_key
	361
	362	# otherwise, we strip off the 'signatures' if any, because it's legitimate
	363	# for different upload attempts to have different signatures.
	364	old_key.pop("signatures", None)
	365	new_key_copy = dict(new_key)
	366	new_key_copy.pop("signatures", None)
	367
	368	return old_key == new_key_copy

+56

-69

synapse/handlers/federation.py less more

27	27	from synapse.events.validator import EventValidator
28	28	from synapse.util import unwrapFirstError
29	29	from synapse.util.logcontext import (
30		PreserveLoggingContext, preserve_fn, preserve_context_over_deferred
	30	preserve_fn, preserve_context_over_deferred
31	31	)
32	32	from synapse.util.metrics import measure_func
33	33	from synapse.util.logutils import log_function

171	171	origin, pdu, prevs, min_depth
172	172	)
173	173
174		prevs = {e_id for e_id, _ in pdu.prev_events}
175		seen = set(have_seen.keys())
	174	# Update the set of things we've seen after trying to
	175	# fetch the missing stuff
	176	have_seen = yield self.store.have_events(prevs)
	177	seen = set(have_seen.iterkeys())
	178
	179	if not prevs - seen:
	180	logger.info(
	181	"Found all missing prev events for %s", pdu.event_id
	182	)
	183	elif prevs - seen:
	184	logger.info(
	185	"Not fetching %d missing events for room %r,event %s: %r...",
	186	len(prevs - seen), pdu.room_id, pdu.event_id,
	187	list(prevs - seen)[:5],
	188	)
	189
176	190	if prevs - seen:
177	191	logger.info(
178	192	"Still missing %d events for room %r: %r...",

207	221	Args:
208	222	origin (str): Origin of the pdu. Will be called to get the missing events
209	223	pdu: received pdu
210		prevs (str[]): List of event ids which we are missing
	224	prevs (set(str)): List of event ids which we are missing
211	225	min_depth (int): Minimum depth of events to return.
212
213		Returns:
214		Deferred<dict(str, str?)>: updated have_seen dictionary
215	226	"""
216	227	# We recalculate seen, since it may have changed.
217	228	have_seen = yield self.store.have_events(prevs)
218	229	seen = set(have_seen.keys())
219	230
220	231	if not prevs - seen:
221		# nothing left to do
222		defer.returnValue(have_seen)
	232	return
223	233
224	234	latest = yield self.store.get_latest_event_ids_in_room(
225	235	pdu.room_id

231	241	latest \|= seen
232	242
233	243	logger.info(
234		"Missing %d events for room %r: %r...",
235		len(prevs - seen), pdu.room_id, list(prevs - seen)[:5]
	244	"Missing %d events for room %r pdu %s: %r...",
	245	len(prevs - seen), pdu.room_id, pdu.event_id, list(prevs - seen)[:5]
236	246	)
237	247
238	248	# XXX: we set timeout to 10s to help workaround

264	274	timeout=10000,
265	275	)
266	276
	277	logger.info(
	278	"Got %d events: %r...",
	279	len(missing_events), [e.event_id for e in missing_events[:5]]
	280	)
	281
267	282	# We want to sort these by depth so we process them and
268	283	# tell clients about them in order.
269	284	missing_events.sort(key=lambda x: x.depth)
270	285
271	286	for e in missing_events:
	287	logger.info("Handling found event %s", e.event_id)
272	288	yield self.on_receive_pdu(
273	289	origin,
274	290	e,
275	291	get_missing=False
276	292	)
277
278		have_seen = yield self.store.have_events(
279		[ev for ev, _ in pdu.prev_events]
280		)
281		defer.returnValue(have_seen)
282	293
283	294	@log_function
284	295	@defer.inlineCallbacks

368	379	affected=event.event_id,
369	380	)
370	381
371		# if we're receiving valid events from an origin,
372		# it's probably a good idea to mark it as not in retry-state
373		# for sending (although this is a bit of a leap)
374		retry_timings = yield self.store.get_destination_retry_timings(origin)
375		if retry_timings and retry_timings["retry_last_ts"]:
376		self.store.set_destination_retry_timings(origin, 0, 0)
377
378	382	room = yield self.store.get_room(event.room_id)
379	383
380	384	if not room:

393	397	target_user = UserID.from_string(target_user_id)
394	398	extra_users.append(target_user)
395	399
396		with PreserveLoggingContext():
397		self.notifier.on_new_room_event(
398		event, event_stream_id, max_stream_id,
399		extra_users=extra_users
400		)
	400	self.notifier.on_new_room_event(
	401	event, event_stream_id, max_stream_id,
	402	extra_users=extra_users
	403	)
401	404
402	405	if event.type == EventTypes.Member:
403	406	if event.membership == Membership.JOIN:

915	918	origin, auth_chain, state, event
916	919	)
917	920
918		with PreserveLoggingContext():
919		self.notifier.on_new_room_event(
920		event, event_stream_id, max_stream_id,
921		extra_users=[joinee]
922		)
	921	self.notifier.on_new_room_event(
	922	event, event_stream_id, max_stream_id,
	923	extra_users=[joinee]
	924	)
923	925
924	926	logger.debug("Finished joining %s to %s", joinee, room_id)
925	927	finally:

1034	1036	target_user = UserID.from_string(target_user_id)
1035	1037	extra_users.append(target_user)
1036	1038
1037		with PreserveLoggingContext():
1038		self.notifier.on_new_room_event(
1039		event, event_stream_id, max_stream_id, extra_users=extra_users
1040		)
	1039	self.notifier.on_new_room_event(
	1040	event, event_stream_id, max_stream_id, extra_users=extra_users
	1041	)
1041	1042
1042	1043	if event.type == EventTypes.Member:
1043	1044	if event.content["membership"] == Membership.JOIN:

1083	1084	)
1084	1085
1085	1086	target_user = UserID.from_string(event.state_key)
1086		with PreserveLoggingContext():
1087		self.notifier.on_new_room_event(
1088		event, event_stream_id, max_stream_id,
1089		extra_users=[target_user],
1090		)
	1087	self.notifier.on_new_room_event(
	1088	event, event_stream_id, max_stream_id,
	1089	extra_users=[target_user],
	1090	)
1091	1091
1092	1092	defer.returnValue(event)
1093	1093
1094	1094	@defer.inlineCallbacks
1095	1095	def do_remotely_reject_invite(self, target_hosts, room_id, user_id):
1096		try:
1097		origin, event = yield self._make_and_verify_event(
1098		target_hosts,
1099		room_id,
1100		user_id,
1101		"leave"
1102		)
1103		event = self._sign_event(event)
1104		except SynapseError:
1105		raise
1106		except CodeMessageException as e:
1107		logger.warn("Failed to reject invite: %s", e)
1108		raise SynapseError(500, "Failed to reject invite")
	1096	origin, event = yield self._make_and_verify_event(
	1097	target_hosts,
	1098	room_id,
	1099	user_id,
	1100	"leave"
	1101	)
	1102	event = self._sign_event(event)
1109	1103
1110	1104	# Try the host that we succesfully called /make_leave/ on first for
1111	1105	# the /send_leave/ request.

1115	1109	except ValueError:
1116	1110	pass
1117	1111
1118		try:
1119		yield self.replication_layer.send_leave(
1120		target_hosts,
1121		event
1122		)
1123		except SynapseError:
1124		raise
1125		except CodeMessageException as e:
1126		logger.warn("Failed to reject invite: %s", e)
1127		raise SynapseError(500, "Failed to reject invite")
	1112	yield self.replication_layer.send_leave(
	1113	target_hosts,
	1114	event
	1115	)
1128	1116
1129	1117	context = yield self.state_handler.compute_event_context(event)
1130	1118

1245	1233	target_user = UserID.from_string(target_user_id)
1246	1234	extra_users.append(target_user)
1247	1235
1248		with PreserveLoggingContext():
1249		self.notifier.on_new_room_event(
1250		event, event_stream_id, max_stream_id, extra_users=extra_users
1251		)
	1236	self.notifier.on_new_room_event(
	1237	event, event_stream_id, max_stream_id, extra_users=extra_users
	1238	)
1252	1239
1253	1240	defer.returnValue(None)
1254	1241

+10

-1

synapse/handlers/identity.py less more

17	17	from twisted.internet import defer
18	18
19	19	from synapse.api.errors import (
20		CodeMessageException
	20	MatrixCodeMessageException, CodeMessageException
21	21	)
22	22	from ._base import BaseHandler
23	23	from synapse.util.async import run_on_reactor

89	89	),
90	90	{'sid': creds['sid'], 'client_secret': client_secret}
91	91	)
	92	except MatrixCodeMessageException as e:
	93	logger.info("getValidated3pid failed with Matrix error: %r", e)
	94	raise SynapseError(e.code, e.msg, e.errcode)
92	95	except CodeMessageException as e:
93	96	data = json.loads(e.msg)
94	97

158	161	params
159	162	)
160	163	defer.returnValue(data)
	164	except MatrixCodeMessageException as e:
	165	logger.info("Proxied requestToken failed with Matrix error: %r", e)
	166	raise SynapseError(e.code, e.msg, e.errcode)
161	167	except CodeMessageException as e:
162	168	logger.info("Proxied requestToken failed: %r", e)
163	169	raise e

192	198	params
193	199	)
194	200	defer.returnValue(data)
	201	except MatrixCodeMessageException as e:
	202	logger.info("Proxied requestToken failed with Matrix error: %r", e)
	203	raise SynapseError(e.code, e.msg, e.errcode)
195	204	except CodeMessageException as e:
196	205	logger.info("Proxied requestToken failed: %r", e)
197	206	raise e

+15

-22

synapse/handlers/message.py less more

15	15	from twisted.internet import defer
16	16
17	17	from synapse.api.constants import EventTypes, Membership
18		from synapse.api.errors import AuthError, Codes, SynapseError, LimitExceededError
	18	from synapse.api.errors import AuthError, Codes, SynapseError
19	19	from synapse.crypto.event_signing import add_hashes_and_signatures
20	20	from synapse.events.utils import serialize_event
21	21	from synapse.events.validator import EventValidator

174	174	defer.returnValue(chunk)
175	175
176	176	@defer.inlineCallbacks
177		def create_event(self, event_dict, token_id=None, txn_id=None, prev_event_ids=None):
	177	def create_event(self, requester, event_dict, token_id=None, txn_id=None,
	178	prev_event_ids=None):
178	179	"""
179	180	Given a dict from a client, create a new event.
180	181

184	185	Adds display names to Join membership events.
185	186
186	187	Args:
	188	requester
187	189	event_dict (dict): An entire event
188	190	token_id (str)
189	191	txn_id (str)

225	227
226	228	event, context = yield self._create_new_client_event(
227	229	builder=builder,
	230	requester=requester,
228	231	prev_event_ids=prev_event_ids,
229	232	)
230	233

250	253	# We check here if we are currently being rate limited, so that we
251	254	# don't do unnecessary work. We check again just before we actually
252	255	# send the event.
253		time_now = self.clock.time()
254		allowed, time_allowed = self.ratelimiter.send_message(
255		event.sender, time_now,
256		msg_rate_hz=self.hs.config.rc_messages_per_second,
257		burst_count=self.hs.config.rc_message_burst_count,
258		update=False,
259		)
260		if not allowed:
261		raise LimitExceededError(
262		retry_after_ms=int(1000 * (time_allowed - time_now)),
263		)
	256	yield self.ratelimit(requester, update=False)
264	257
265	258	user = UserID.from_string(event.sender)
266	259

318	311	See self.create_event and self.send_nonmember_event.
319	312	"""
320	313	event, context = yield self.create_event(
	314	requester,
321	315	event_dict,
322	316	token_id=requester.access_token_id,
323	317	txn_id=txn_id

415	409
416	410	@measure_func("_create_new_client_event")
417	411	@defer.inlineCallbacks
418		def _create_new_client_event(self, builder, prev_event_ids=None):
	412	def _create_new_client_event(self, builder, requester=None, prev_event_ids=None):
419	413	if prev_event_ids:
420	414	prev_events = yield self.store.add_event_hashes(prev_event_ids)
421	415	prev_max_depth = yield self.store.get_max_depth_of_events(prev_event_ids)

455	449	state_handler = self.state_handler
456	450
457	451	context = yield state_handler.compute_event_context(builder)
	452	if requester:
	453	context.app_service = requester.app_service
458	454
459	455	if builder.is_state():
460	456	builder.prev_state = yield self.store.add_event_hashes(

492	488	# We now need to go and hit out to wherever we need to hit out to.
493	489
494	490	if ratelimit:
495		self.ratelimit(requester)
	491	yield self.ratelimit(requester)
496	492
497	493	try:
498	494	yield self.auth.check_from_context(event, context)

530	526
531	527	state_to_include_ids = [
532	528	e_id
533		for k, e_id in context.current_state_ids.items()
	529	for k, e_id in context.current_state_ids.iteritems()
534	530	if k[0] in self.hs.config.room_invite_state_types
535		or k[0] == EventTypes.Member and k[1] == event.sender
	531	or k == (EventTypes.Member, event.sender)
536	532	]
537	533
538	534	state_to_include = yield self.store.get_events(state_to_include_ids)

544	540	"content": e.content,
545	541	"sender": e.sender,
546	542	}
547		for e in state_to_include.values()
	543	for e in state_to_include.itervalues()
548	544	]
549	545
550	546	invitee = UserID.from_string(event.state_key)

611	607	@defer.inlineCallbacks
612	608	def _notify():
613	609	yield run_on_reactor()
614		yield self.notifier.on_new_room_event(
	610	self.notifier.on_new_room_event(
615	611	event, event_stream_id, max_stream_id,
616	612	extra_users=extra_users
617	613	)
618	614
619	615	preserve_fn(_notify)()
620
621		# If invite, remove room_state from unsigned before sending.
622		event.unsigned.pop("invite_room_state", None)

+153

-75

synapse/handlers/presence.py less more

29	29	from synapse.storage.presence import UserPresenceState
30	30
31	31	from synapse.util.caches.descriptors import cachedInlineCallbacks
	32	from synapse.util.async import Linearizer
32	33	from synapse.util.logcontext import preserve_fn
33	34	from synapse.util.logutils import log_function
34	35	from synapse.util.metrics import Measure

186	187	# process_id to millisecond timestamp last updated.
187	188	self.external_process_to_current_syncs = {}
188	189	self.external_process_last_updated_ms = {}
	190	self.external_sync_linearizer = Linearizer(name="external_sync_linearizer")
189	191
190	192	# Start a LoopingCall in 30s that fires every 5s.
191	193	# The initial delay is to allow disconnected clients a chance to

315	317	if to_federation_ping:
316	318	federation_presence_out_counter.inc_by(len(to_federation_ping))
317	319
318		_, _, hosts_to_states = yield self._get_interested_parties(
319		to_federation_ping.values()
320		)
321
322		self._push_to_remotes(hosts_to_states)
	320	self._push_to_remotes(to_federation_ping.values())
323	321
324	322	def _handle_timeouts(self):
325	323	"""Checks the presence of users that have timed out and updates as

508	506	self.external_process_to_current_syncs[process_id] = syncing_user_ids
509	507
510	508	@defer.inlineCallbacks
	509	def update_external_syncs_row(self, process_id, user_id, is_syncing, sync_time_msec):
	510	"""Update the syncing users for an external process as a delta.
	511
	512	Args:
	513	process_id (str): An identifier for the process the users are
	514	syncing against. This allows synapse to process updates
	515	as user start and stop syncing against a given process.
	516	user_id (str): The user who has started or stopped syncing
	517	is_syncing (bool): Whether or not the user is now syncing
	518	sync_time_msec(int): Time in ms when the user was last syncing
	519	"""
	520	with (yield self.external_sync_linearizer.queue(process_id)):
	521	prev_state = yield self.current_state_for_user(user_id)
	522
	523	process_presence = self.external_process_to_current_syncs.setdefault(
	524	process_id, set()
	525	)
	526
	527	updates = []
	528	if is_syncing and user_id not in process_presence:
	529	if prev_state.state == PresenceState.OFFLINE:
	530	updates.append(prev_state.copy_and_replace(
	531	state=PresenceState.ONLINE,
	532	last_active_ts=sync_time_msec,
	533	last_user_sync_ts=sync_time_msec,
	534	))
	535	else:
	536	updates.append(prev_state.copy_and_replace(
	537	last_user_sync_ts=sync_time_msec,
	538	))
	539	process_presence.add(user_id)
	540	elif user_id in process_presence:
	541	updates.append(prev_state.copy_and_replace(
	542	last_user_sync_ts=sync_time_msec,
	543	))
	544
	545	if not is_syncing:
	546	process_presence.discard(user_id)
	547
	548	if updates:
	549	yield self._update_states(updates)
	550
	551	self.external_process_last_updated_ms[process_id] = self.clock.time_msec()
	552
	553	@defer.inlineCallbacks
	554	def update_external_syncs_clear(self, process_id):
	555	"""Marks all users that had been marked as syncing by a given process
	556	as offline.
	557
	558	Used when the process has stopped/disappeared.
	559	"""
	560	with (yield self.external_sync_linearizer.queue(process_id)):
	561	process_presence = self.external_process_to_current_syncs.pop(
	562	process_id, set()
	563	)
	564	prev_states = yield self.current_state_for_users(process_presence)
	565	time_now_ms = self.clock.time_msec()
	566
	567	yield self._update_states([
	568	prev_state.copy_and_replace(
	569	last_user_sync_ts=time_now_ms,
	570	)
	571	for prev_state in prev_states.itervalues()
	572	])
	573	self.external_process_last_updated_ms.pop(process_id, None)
	574
	575	@defer.inlineCallbacks
511	576	def current_state_for_user(self, user_id):
512	577	"""Get the current presence state for a user.
513	578	"""

526	591	for user_id in user_ids
527	592	}
528	593
529		missing = [user_id for user_id, state in states.items() if not state]
	594	missing = [user_id for user_id, state in states.iteritems() if not state]
530	595	if missing:
531	596	# There are things not in our in memory cache. Lets pull them out of
532	597	# the database.
533	598	res = yield self.store.get_presence_for_users(missing)
534	599	states.update(res)
535	600
536		missing = [user_id for user_id, state in states.items() if not state]
	601	missing = [user_id for user_id, state in states.iteritems() if not state]
537	602	if missing:
538	603	new = {
539	604	user_id: UserPresenceState.default(user_id)

545	610	defer.returnValue(states)
546	611
547	612	@defer.inlineCallbacks
548		def _get_interested_parties(self, states, calculate_remote_hosts=True):
549		"""Given a list of states return which entities (rooms, users, servers)
550		are interested in the given states.
551
552		Returns:
553		3-tuple: `(room_ids_to_states, users_to_states, hosts_to_states)`,
554		with each item being a dict of `entity_name` -> `[UserPresenceState]`
555		"""
556		room_ids_to_states = {}
557		users_to_states = {}
558		for state in states:
559		room_ids = yield self.store.get_rooms_for_user(state.user_id)
560		for room_id in room_ids:
561		room_ids_to_states.setdefault(room_id, []).append(state)
562
563		plist = yield self.store.get_presence_list_observers_accepted(state.user_id)
564		for u in plist:
565		users_to_states.setdefault(u, []).append(state)
566
567		# Always notify self
568		users_to_states.setdefault(state.user_id, []).append(state)
569
570		hosts_to_states = {}
571		if calculate_remote_hosts:
572		for room_id, states in room_ids_to_states.items():
573		local_states = filter(lambda s: self.is_mine_id(s.user_id), states)
574		if not local_states:
575		continue
576
577		hosts = yield self.store.get_hosts_in_room(room_id)
578
579		for host in hosts:
580		hosts_to_states.setdefault(host, []).extend(local_states)
581
582		for user_id, states in users_to_states.items():
583		local_states = filter(lambda s: self.is_mine_id(s.user_id), states)
584		if not local_states:
585		continue
586
587		host = get_domain_from_id(user_id)
588		hosts_to_states.setdefault(host, []).extend(local_states)
589
590		# TODO: de-dup hosts_to_states, as a single host might have multiple
591		# of same presence
592
593		defer.returnValue((room_ids_to_states, users_to_states, hosts_to_states))
594
595		@defer.inlineCallbacks
596	613	def _persist_and_notify(self, states):
597	614	"""Persist states in the database, poke the notifier and send to
598	615	interested remote servers
599	616	"""
600	617	stream_id, max_token = yield self.store.update_presence(states)
601	618
602		parties = yield self._get_interested_parties(states)
603		room_ids_to_states, users_to_states, hosts_to_states = parties
	619	parties = yield get_interested_parties(self.store, states)
	620	room_ids_to_states, users_to_states = parties
604	621
605	622	self.notifier.on_new_event(
606	623	"presence_key", stream_id, rooms=room_ids_to_states.keys(),
607		users=[UserID.from_string(u) for u in users_to_states.keys()]
608		)
609
610		self._push_to_remotes(hosts_to_states)
	624	users=[UserID.from_string(u) for u in users_to_states]
	625	)
	626
	627	self._push_to_remotes(states)
611	628
612	629	@defer.inlineCallbacks
613	630	def notify_for_states(self, state, stream_id):
614		parties = yield self._get_interested_parties([state])
615		room_ids_to_states, users_to_states, hosts_to_states = parties
	631	parties = yield get_interested_parties(self.store, [state])
	632	room_ids_to_states, users_to_states = parties
616	633
617	634	self.notifier.on_new_event(
618	635	"presence_key", stream_id, rooms=room_ids_to_states.keys(),
619		users=[UserID.from_string(u) for u in users_to_states.keys()]
620		)
621
622		def _push_to_remotes(self, hosts_to_states):
	636	users=[UserID.from_string(u) for u in users_to_states]
	637	)
	638
	639	def _push_to_remotes(self, states):
623	640	"""Sends state updates to remote servers.
624	641
625	642	Args:
626		hosts_to_states (dict): Mapping `server_name` -> `[UserPresenceState]`
627		"""
628		for host, states in hosts_to_states.items():
629		self.federation.send_presence(host, states)
	643	states (list(UserPresenceState))
	644	"""
	645	self.federation.send_presence(states)
630	646
631	647	@defer.inlineCallbacks
632	648	def incoming_presence(self, origin, content):

763	779	# don't need to send to local clients here, as that is done as part
764	780	# of the event stream/sync.
765	781	# TODO: Only send to servers not already in the room.
766		user_ids = yield self.store.get_users_in_room(room_id)
767	782	if self.is_mine(user):
768	783	state = yield self.current_state_for_user(user.to_string())
769	784
770		hosts = set(get_domain_from_id(u) for u in user_ids)
771		self._push_to_remotes({host: (state,) for host in hosts})
	785	self._push_to_remotes([state])
772	786	else:
	787	user_ids = yield self.store.get_users_in_room(room_id)
773	788	user_ids = filter(self.is_mine_id, user_ids)
774	789
775	790	states = yield self.current_state_for_users(user_ids)
776	791
777		self._push_to_remotes({user.domain: states.values()})
	792	self._push_to_remotes(states.values())
778	793
779	794	@defer.inlineCallbacks
780	795	def get_presence_list(self, observer_user, accepted=None):

1274	1289	persist_and_notify = True
1275	1290
1276	1291	return new_state, persist_and_notify, federation_ping
	1292
	1293
	1294	@defer.inlineCallbacks
	1295	def get_interested_parties(store, states):
	1296	"""Given a list of states return which entities (rooms, users)
	1297	are interested in the given states.
	1298
	1299	Args:
	1300	states (list(UserPresenceState))
	1301
	1302	Returns:
	1303	2-tuple: `(room_ids_to_states, users_to_states)`,
	1304	with each item being a dict of `entity_name` -> `[UserPresenceState]`
	1305	"""
	1306	room_ids_to_states = {}
	1307	users_to_states = {}
	1308	for state in states:
	1309	room_ids = yield store.get_rooms_for_user(state.user_id)
	1310	for room_id in room_ids:
	1311	room_ids_to_states.setdefault(room_id, []).append(state)
	1312
	1313	plist = yield store.get_presence_list_observers_accepted(state.user_id)
	1314	for u in plist:
	1315	users_to_states.setdefault(u, []).append(state)
	1316
	1317	# Always notify self
	1318	users_to_states.setdefault(state.user_id, []).append(state)
	1319
	1320	defer.returnValue((room_ids_to_states, users_to_states))
	1321
	1322
	1323	@defer.inlineCallbacks
	1324	def get_interested_remotes(store, states, state_handler):
	1325	"""Given a list of presence states figure out which remote servers
	1326	should be sent which.
	1327
	1328	All the presence states should be for local users only.
	1329
	1330	Args:
	1331	store (DataStore)
	1332	states (list(UserPresenceState))
	1333
	1334	Returns:
	1335	Deferred list of ([destinations], [UserPresenceState]), where for
	1336	each row the list of UserPresenceState should be sent to each
	1337	destination
	1338	"""
	1339	hosts_and_states = []
	1340
	1341	# First we look up the rooms each user is in (as well as any explicit
	1342	# subscriptions), then for each distinct room we look up the remote
	1343	# hosts in those rooms.
	1344	room_ids_to_states, users_to_states = yield get_interested_parties(store, states)
	1345
	1346	for room_id, states in room_ids_to_states.iteritems():
	1347	hosts = yield state_handler.get_current_hosts_in_room(room_id)
	1348	hosts_and_states.append((hosts, states))
	1349
	1350	for user_id, states in users_to_states.iteritems():
	1351	host = get_domain_from_id(user_id)
	1352	hosts_and_states.append(([host], states))
	1353
	1354	defer.returnValue(hosts_and_states)

+1

-1

synapse/handlers/profile.py less more

155	155	if not self.hs.is_mine(user):
156	156	return
157	157
158		self.ratelimit(requester)
	158	yield self.ratelimit(requester)
159	159
160	160	room_ids = yield self.store.get_rooms_for_user(
161	161	user.to_string(),

+64

-0

synapse/handlers/read_marker.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2017 Vector Creations Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from ._base import BaseHandler
	16
	17	from twisted.internet import defer
	18
	19	from synapse.util.async import Linearizer
	20
	21	import logging
	22	logger = logging.getLogger(__name__)
	23
	24
	25	class ReadMarkerHandler(BaseHandler):
	26	def __init__(self, hs):
	27	super(ReadMarkerHandler, self).__init__(hs)
	28	self.server_name = hs.config.server_name
	29	self.store = hs.get_datastore()
	30	self.read_marker_linearizer = Linearizer(name="read_marker")
	31	self.notifier = hs.get_notifier()
	32
	33	@defer.inlineCallbacks
	34	def received_client_read_marker(self, room_id, user_id, event_id):
	35	"""Updates the read marker for a given user in a given room if the event ID given
	36	is ahead in the stream relative to the current read marker.
	37
	38	This uses a notifier to indicate that account data should be sent down /sync if
	39	the read marker has changed.
	40	"""
	41
	42	with (yield self.read_marker_linearizer.queue((room_id, user_id))):
	43	account_data = yield self.store.get_account_data_for_room(user_id, room_id)
	44
	45	existing_read_marker = account_data.get("m.fully_read", None)
	46
	47	should_update = True
	48
	49	if existing_read_marker:
	50	# Only update if the new marker is ahead in the stream
	51	should_update = yield self.store.is_event_after(
	52	event_id,
	53	existing_read_marker['event_id']
	54	)
	55
	56	if should_update:
	57	content = {
	58	"event_id": event_id
	59	}
	60	max_id = yield self.store.add_account_data_to_room(
	61	user_id, room_id, "m.fully_read", content
	62	)
	63	self.notifier.on_new_event("account_data_key", max_id, users=[user_id])

+7

-0

synapse/handlers/register.py less more

50	50	raise SynapseError(
51	51	400,
52	52	"User ID can only contain characters a-z, 0-9, or '_-./'",
	53	Codes.INVALID_USERNAME
	54	)
	55
	56	if not localpart:
	57	raise SynapseError(
	58	400,
	59	"User ID cannot be empty",
53	60	Codes.INVALID_USERNAME
54	61	)
55	62

+1

-1

synapse/handlers/room.py less more

74	74	"""
75	75	user_id = requester.user.to_string()
76	76
77		self.ratelimit(requester)
	77	yield self.ratelimit(requester)
78	78
79	79	if "room_alias_name" in config:
80	80	for wchar in string.whitespace:

+15

-12

synapse/handlers/room_member.py less more

69	69	content["kind"] = "guest"
70	70
71	71	event, context = yield msg_handler.create_event(
	72	requester,
72	73	{
73	74	"type": EventTypes.Member,
74	75	"content": content,

137	138	content,
138	139	)
139	140	yield user_joined_room(self.distributor, user, room_id)
140
141		def reject_remote_invite(self, user_id, room_id, remote_room_hosts):
142		return self.hs.get_handlers().federation_handler.do_remotely_reject_invite(
143		remote_room_hosts,
144		room_id,
145		user_id
146		)
147	141
148	142	@defer.inlineCallbacks
149	143	def update_membership(

285	279	else:
286	280	# send the rejection to the inviter's HS.
287	281	remote_room_hosts = remote_room_hosts + [inviter.domain]
288
	282	fed_handler = self.hs.get_handlers().federation_handler
289	283	try:
290		ret = yield self.reject_remote_invite(
291		target.to_string(), room_id, remote_room_hosts
	284	ret = yield fed_handler.do_remotely_reject_invite(
	285	remote_room_hosts,
	286	room_id,
	287	target.to_string(),
292	288	)
293	289	defer.returnValue(ret)
294		except SynapseError as e:
	290	except Exception as e:
	291	# if we were unable to reject the exception, just mark
	292	# it as rejected on our end and plough ahead.
	293	#
	294	# The 'except' clause is very broad, but we need to
	295	# capture everything from DNS failures upwards
	296	#
295	297	logger.warn("Failed to reject invite: %s", e)
296	298
297	299	yield self.store.locally_reject_invite(

736	738	if len(current_state_ids) == 1 and create_event_id:
737	739	defer.returnValue(self.hs.is_mine_id(create_event_id))
738	740
739		for (etype, state_key), event_id in current_state_ids.items():
	741	for etype, state_key in current_state_ids:
740	742	if etype != EventTypes.Member or not self.hs.is_mine_id(state_key):
741	743	continue
742	744
	745	event_id = current_state_ids[(etype, state_key)]
743	746	event = yield self.store.get_event(event_id, allow_none=True)
744	747	if not event:
745	748	continue

+4

-3

synapse/handlers/typing.py less more

23	23	import logging
24	24
25	25	from collections import namedtuple
26		import ujson as json
27	26
28	27	logger = logging.getLogger(__name__)
29	28

287	286	for room_id, serial in self._room_serials.items():
288	287	if last_id < serial and serial <= current_id:
289	288	typing = self._room_typing[room_id]
290		typing_bytes = json.dumps(list(typing), ensure_ascii=False)
291		rows.append((serial, room_id, typing_bytes))
	289	rows.append((serial, room_id, list(typing)))
292	290	rows.sort()
293	291	return rows
	292
	293	def get_current_token(self):
	294	return self._latest_room_serial
294	295
295	296
296	297	class TypingNotificationEventSource(object):

+42

-18

synapse/http/client.py less more

15	15	from OpenSSL.SSL import VERIFY_NONE
16	16
17	17	from synapse.api.errors import (
18		CodeMessageException, SynapseError, Codes,
	18	CodeMessageException, MatrixCodeMessageException, SynapseError, Codes,
19	19	)
20	20	from synapse.util.logcontext import preserve_context_over_fn
	21	from synapse.util import logcontext
21	22	import synapse.metrics
22	23	from synapse.http.endpoint import SpiderEndpoint
23	24

71	72	contextFactory=hs.get_http_client_context_factory()
72	73	)
73	74	self.user_agent = hs.version_string
	75	self.clock = hs.get_clock()
74	76	if hs.config.user_agent_suffix:
75	77	self.user_agent = "%s %s" % (self.user_agent, hs.config.user_agent_suffix,)
76	78
	79	@defer.inlineCallbacks
77	80	def request(self, method, uri, args, *kwargs):
78	81	# A small wrapper around self.agent.request() so we can easily attach
79	82	# counters to it
80	83	outgoing_requests_counter.inc(method)
81		d = preserve_context_over_fn(
82		self.agent.request,
83		method, uri, args, *kwargs
84		)
	84
	85	def send_request():
	86	request_deferred = self.agent.request(
	87	method, uri, args, *kwargs
	88	)
	89
	90	return self.clock.time_bound_deferred(
	91	request_deferred,
	92	time_out=60,
	93	)
85	94
86	95	logger.info("Sending request %s %s", method, uri)
87	96
88		def _cb(response):
	97	try:
	98	with logcontext.PreserveLoggingContext():
	99	response = yield send_request()
	100
89	101	incoming_responses_counter.inc(method, response.code)
90	102	logger.info(
91	103	"Received response to %s %s: %s",
92	104	method, uri, response.code
93	105	)
94		return response
95
96		def _eb(failure):
	106	defer.returnValue(response)
	107	except Exception as e:
97	108	incoming_responses_counter.inc(method, "ERR")
98	109	logger.info(
99	110	"Error sending request to %s %s: %s %s",
100		method, uri, failure.type, failure.getErrorMessage()
101		)
102		return failure
103
104		d.addCallbacks(_cb, _eb)
105
106		return d
	111	method, uri, type(e).__name__, e.message
	112	)
	113	raise e
107	114
108	115	@defer.inlineCallbacks
109	116	def post_urlencoded_get_json(self, uri, args={}):

143	150	)
144	151
145	152	body = yield preserve_context_over_fn(readBody, response)
	153
	154	if 200 <= response.code < 300:
	155	defer.returnValue(json.loads(body))
	156	else:
	157	raise self._exceptionFromFailedRequest(response, body)
146	158
147	159	defer.returnValue(json.loads(body))
148	160

163	175	On a non-2xx HTTP response. The response body will be used as the
164	176	error message.
165	177	"""
166		body = yield self.get_raw(uri, args)
167		defer.returnValue(json.loads(body))
	178	try:
	179	body = yield self.get_raw(uri, args)
	180	defer.returnValue(json.loads(body))
	181	except CodeMessageException as e:
	182	raise self._exceptionFromFailedRequest(e.code, e.msg)
168	183
169	184	@defer.inlineCallbacks
170	185	def put_json(self, uri, json_body, args={}):

245	260	else:
246	261	raise CodeMessageException(response.code, body)
247	262
	263	def _exceptionFromFailedRequest(self, response, body):
	264	try:
	265	jsonBody = json.loads(body)
	266	errcode = jsonBody['errcode']
	267	error = jsonBody['error']
	268	return MatrixCodeMessageException(response.code, error, errcode)
	269	except (ValueError, KeyError):
	270	return CodeMessageException(response.code, body)
	271
248	272	# XXX: FIXME: This is horribly copy-pasted from matrixfederationclient.
249	273	# The two should be factored out.
250	274

+15

-8

synapse/http/matrixfederationclient.py less more

124	124	code >= 300.
125	125	Fails with ``NotRetryingDestination`` if we are not yet ready
126	126	to retry this server.
	127	(May also fail with plenty of other Exceptions for things like DNS
	128	failures, connection failures, SSL failures.)
127	129	"""
128	130	limiter = yield synapse.util.retryutils.get_retry_limiter(
129	131	destination,

301	303
302	304	Returns:
303	305	Deferred: Succeeds when we get a 2xx HTTP response. The result
304		will be the decoded JSON body. On a 4xx or 5xx error response a
305		CodeMessageException is raised.
	306	will be the decoded JSON body.
	307
	308	Fails with ``HTTPRequestException`` if we get an HTTP response
	309	code >= 300.
306	310
307	311	Fails with ``NotRetryingDestination`` if we are not yet ready
308	312	to retry this server.

359	363	try the request anyway.
360	364	Returns:
361	365	Deferred: Succeeds when we get a 2xx HTTP response. The result
362		will be the decoded JSON body. On a 4xx or 5xx error response a
363		CodeMessageException is raised.
	366	will be the decoded JSON body.
	367
	368	Fails with ``HTTPRequestException`` if we get an HTTP response
	369	code >= 300.
364	370
365	371	Fails with ``NotRetryingDestination`` if we are not yet ready
366	372	to retry this server.

409	415	ignore_backoff (bool): true to ignore the historical backoff data
410	416	and try the request anyway.
411	417	Returns:
412		Deferred: Succeeds when we get any HTTP response.
413
414		The result of the deferred is a tuple of `(code, response)`,
415		where `response` is a dict representing the decoded JSON body.
	418	Deferred: Succeeds when we get a 2xx HTTP response. The result
	419	will be the decoded JSON body.
	420
	421	Fails with ``HTTPRequestException`` if we get an HTTP response
	422	code >= 300.
416	423
417	424	Fails with ``NotRetryingDestination`` if we are not yet ready
418	425	to retry this server.

+23

-15

synapse/notifier.py less more

162	162	self.store = hs.get_datastore()
163	163	self.pending_new_room_events = []
164	164
	165	self.replication_callbacks = []
	166
165	167	self.clock = hs.get_clock()
166	168	self.appservice_handler = hs.get_application_service_handler()
167	169

201	203	lambda: len(self.user_to_user_stream),
202	204	)
203	205
204		@preserve_fn
	206	def add_replication_callback(self, cb):
	207	"""Add a callback that will be called when some new data is available.
	208	Callback is not given any arguments.
	209	"""
	210	self.replication_callbacks.append(cb)
	211
205	212	def on_new_room_event(self, event, room_stream_id, max_room_stream_id,
206	213	extra_users=[]):
207	214	""" Used by handlers to inform the notifier something has happened

215	222	until all previous events have been persisted before notifying
216	223	the client streams.
217	224	"""
218		with PreserveLoggingContext():
219		self.pending_new_room_events.append((
220		room_stream_id, event, extra_users
221		))
222		self._notify_pending_new_room_events(max_room_stream_id)
223
224		self.notify_replication()
225
226		@preserve_fn
	225	self.pending_new_room_events.append((
	226	room_stream_id, event, extra_users
	227	))
	228	self._notify_pending_new_room_events(max_room_stream_id)
	229
	230	self.notify_replication()
	231
227	232	def _notify_pending_new_room_events(self, max_room_stream_id):
228	233	"""Notify for the room events that were queued waiting for a previous
229	234	event to be persisted.

241	246	else:
242	247	self._on_new_room_event(event, room_stream_id, extra_users)
243	248
244		@preserve_fn
245	249	def _on_new_room_event(self, event, room_stream_id, extra_users=[]):
246	250	"""Notify any user streams that are interested in this room event"""
247	251	# poke any interested application service.
248		self.appservice_handler.notify_interested_services(room_stream_id)
	252	preserve_fn(self.appservice_handler.notify_interested_services)(
	253	room_stream_id)
249	254
250	255	if self.federation_sender:
251		self.federation_sender.notify_new_events(room_stream_id)
	256	preserve_fn(self.federation_sender.notify_new_events)(
	257	room_stream_id
	258	)
252	259
253	260	if event.type == EventTypes.Member and event.membership == Membership.JOIN:
254	261	self._user_joined_room(event.state_key, event.room_id)

259	266	rooms=[event.room_id],
260	267	)
261	268
262		@preserve_fn
263	269	def on_new_event(self, stream_key, new_token, users=[], rooms=[]):
264	270	""" Used to inform listeners that something has happend event wise.
265	271

286	292
287	293	self.notify_replication()
288	294
289		@preserve_fn
290	295	def on_new_replication_data(self):
291	296	"""Used to inform replication listeners that something has happend
292	297	without waking up any of the normal user event streams"""

509	514	self.replication_deferred = ObservableDeferred(defer.Deferred())
510	515	deferred.callback(None)
511	516
	517	for cb in self.replication_callbacks:
	518	preserve_fn(cb)()
	519
512	520	@defer.inlineCallbacks
513	521	def wait_for_replication(self, callback, timeout):
514	522	"""Wait for an event to happen.

+5

-1

synapse/push/bulk_push_rule_evaluator.py less more

86	86	condition_cache = {}
87	87
88	88	for uid, rules in self.rules_by_user.items():
89		display_name = room_members.get(uid, {}).get("display_name", None)
	89	display_name = None
	90	profile_info = room_members.get(uid)
	91	if profile_info:
	92	display_name = profile_info.display_name
	93
90	94	if not display_name:
91	95	# Handle the case where we are pushing a membership event to
92	96	# that user, as they might not be already joined.

+5

-1

synapse/push/mailer.py less more

199	199	yield sendmail(
200	200	self.hs.config.email_smtp_host,
201	201	raw_from, raw_to, multipart_msg.as_string(),
202		port=self.hs.config.email_smtp_port
	202	port=self.hs.config.email_smtp_port,
	203	requireAuthentication=self.hs.config.email_smtp_user is not None,
	204	username=self.hs.config.email_smtp_user,
	205	password=self.hs.config.email_smtp_pass,
	206	requireTransportSecurity=self.hs.config.require_transport_security
203	207	)
204	208
205	209	@defer.inlineCallbacks

+2

-5

synapse/push/push_tools.py less more

16	16	from synapse.push.presentable_names import (
17	17	calculate_room_name, name_from_member_event
18	18	)
19		from synapse.util.logcontext import preserve_fn, preserve_context_over_deferred
20	19
21	20
22	21	@defer.inlineCallbacks
23	22	def get_badge_count(store, user_id):
24		invites, joins = yield preserve_context_over_deferred(defer.gatherResults([
25		preserve_fn(store.get_invited_rooms_for_user)(user_id),
26		preserve_fn(store.get_rooms_for_user)(user_id),
27		], consumeErrors=True))
	23	invites = yield store.get_invited_rooms_for_user(user_id)
	24	joins = yield store.get_rooms_for_user(user_id)
28	25
29	26	my_receipts_by_room = yield store.get_receipts_for_user(
30	27	user_id, "m.read",

+0

-60

~~synapse/replication/expire_cache.py~~ less more

0		# Copyright 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		from synapse.http.server import respond_with_json_bytes, request_handler
15		from synapse.http.servlet import parse_json_object_from_request
16
17		from twisted.web.resource import Resource
18		from twisted.web.server import NOT_DONE_YET
19
20
21		class ExpireCacheResource(Resource):
22		"""
23		HTTP endpoint for expiring storage caches.
24
25		POST /_synapse/replication/expire_cache HTTP/1.1
26		Content-Type: application/json
27
28		{
29		"invalidate": [
30		{
31		"name": "func_name",
32		"keys": ["key1", "key2"]
33		}
34		]
35		}
36		"""
37
38		def __init__(self, hs):
39		Resource.__init__(self) # Resource is old-style, so no super()
40
41		self.store = hs.get_datastore()
42		self.version_string = hs.version_string
43		self.clock = hs.get_clock()
44
45		def render_POST(self, request):
46		self._async_render_POST(request)
47		return NOT_DONE_YET
48
49		@request_handler()
50		def _async_render_POST(self, request):
51		content = parse_json_object_from_request(request)
52
53		for row in content["invalidate"]:
54		name = row["name"]
55		keys = tuple(row["keys"])
56
57		getattr(self.store, name).invalidate(keys)
58
59		respond_with_json_bytes(request, 200, "{}")

+0

-59

~~synapse/replication/presence_resource.py~~ less more

0		# Copyright 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		from synapse.http.server import respond_with_json_bytes, request_handler
15		from synapse.http.servlet import parse_json_object_from_request
16
17		from twisted.web.resource import Resource
18		from twisted.web.server import NOT_DONE_YET
19		from twisted.internet import defer
20
21
22		class PresenceResource(Resource):
23		"""
24		HTTP endpoint for marking users as syncing.
25
26		POST /_synapse/replication/presence HTTP/1.1
27		Content-Type: application/json
28
29		{
30		"process_id": "<process_id>",
31		"syncing_users": ["<user_id>"]
32		}
33		"""
34
35		def __init__(self, hs):
36		Resource.__init__(self) # Resource is old-style, so no super()
37
38		self.version_string = hs.version_string
39		self.clock = hs.get_clock()
40		self.presence_handler = hs.get_presence_handler()
41
42		def render_POST(self, request):
43		self._async_render_POST(request)
44		return NOT_DONE_YET
45
46		@request_handler()
47		@defer.inlineCallbacks
48		def _async_render_POST(self, request):
49		content = parse_json_object_from_request(request)
50
51		process_id = content["process_id"]
52		syncing_user_ids = content["syncing_users"]
53
54		yield self.presence_handler.update_external_syncs(
55		process_id, set(syncing_user_ids)
56		)
57
58		respond_with_json_bytes(request, 200, "{}")

+0

-54

~~synapse/replication/pusher_resource.py~~ less more

0		# Copyright 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		from synapse.http.server import respond_with_json_bytes, request_handler
15		from synapse.http.servlet import parse_json_object_from_request
16
17		from twisted.web.resource import Resource
18		from twisted.web.server import NOT_DONE_YET
19		from twisted.internet import defer
20
21
22		class PusherResource(Resource):
23		"""
24		HTTP endpoint for deleting rejected pushers
25		"""
26
27		def __init__(self, hs):
28		Resource.__init__(self) # Resource is old-style, so no super()
29
30		self.version_string = hs.version_string
31		self.store = hs.get_datastore()
32		self.notifier = hs.get_notifier()
33		self.clock = hs.get_clock()
34
35		def render_POST(self, request):
36		self._async_render_POST(request)
37		return NOT_DONE_YET
38
39		@request_handler()
40		@defer.inlineCallbacks
41		def _async_render_POST(self, request):
42		content = parse_json_object_from_request(request)
43
44		for remove in content["remove"]:
45		yield self.store.delete_pusher_by_app_id_pushkey_user_id(
46		remove["app_id"],
47		remove["push_key"],
48		remove["user_id"],
49		)
50
51		self.notifier.on_new_replication_data()
52
53		respond_with_json_bytes(request, 200, "{}")

+0

-576

~~synapse/replication/resource.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2015 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		from synapse.http.servlet import parse_integer, parse_string
16		from synapse.http.server import request_handler, finish_request
17		from synapse.replication.pusher_resource import PusherResource
18		from synapse.replication.presence_resource import PresenceResource
19		from synapse.replication.expire_cache import ExpireCacheResource
20		from synapse.api.errors import SynapseError
21
22		from twisted.web.resource import Resource
23		from twisted.web.server import NOT_DONE_YET
24		from twisted.internet import defer
25
26		import ujson as json
27
28		import collections
29		import logging
30
31		logger = logging.getLogger(__name__)
32
33		REPLICATION_PREFIX = "/_synapse/replication"
34
35		STREAM_NAMES = (
36		("events",),
37		("presence",),
38		("typing",),
39		("receipts",),
40		("user_account_data", "room_account_data", "tag_account_data",),
41		("backfill",),
42		("push_rules",),
43		("pushers",),
44		("caches",),
45		("to_device",),
46		("public_rooms",),
47		("federation",),
48		("device_lists",),
49		)
50
51
52		class ReplicationResource(Resource):
53		"""
54		HTTP endpoint for extracting data from synapse.
55
56		The streams of data returned by the endpoint are controlled by the
57		parameters given to the API. To return a given stream pass a query
58		parameter with a position in the stream to return data from or the
59		special value "-1" to return data from the start of the stream.
60
61		If there is no data for any of the supplied streams after the given
62		position then the request will block until there is data for one
63		of the streams. This allows clients to long-poll this API.
64
65		The possible streams are:
66
67		* "streams": A special stream returing the positions of other streams.
68		* "events": The new events seen on the server.
69		* "presence": Presence updates.
70		* "typing": Typing updates.
71		* "receipts": Receipt updates.
72		* "user_account_data": Top-level per user account data.
73		* "room_account_data: Per room per user account data.
74		* "tag_account_data": Per room per user tags.
75		* "backfill": Old events that have been backfilled from other servers.
76		* "push_rules": Per user changes to push rules.
77		* "pushers": Per user changes to their pushers.
78		* "caches": Cache invalidations.
79
80		The API takes two additional query parameters:
81
82		* "timeout": How long to wait before returning an empty response.
83		* "limit": The maximum number of rows to return for the selected streams.
84
85		The response is a JSON object with keys for each stream with updates. Under
86		each key is a JSON object with:
87
88		* "position": The current position of the stream.
89		* "field_names": The names of the fields in each row.
90		* "rows": The updates as an array of arrays.
91
92		There are a number of ways this API could be used:
93
94		1) To replicate the contents of the backing database to another database.
95		2) To be notified when the contents of a shared backing database changes.
96		3) To "tail" the activity happening on a server for debugging.
97
98		In the first case the client would track all of the streams and store it's
99		own copy of the data.
100
101		In the second case the client might theoretically just be able to follow
102		the "streams" stream to track where the other streams are. However in
103		practise it will probably need to get the contents of the streams in
104		order to expire the any in-memory caches. Whether it gets the contents
105		of the streams from this replication API or directly from the backing
106		store is a matter of taste.
107
108		In the third case the client would use the "streams" stream to find what
109		streams are available and their current positions. Then it can start
110		long-polling this replication API for new data on those streams.
111		"""
112
113		def __init__(self, hs):
114		Resource.__init__(self) # Resource is old-style, so no super()
115
116		self.version_string = hs.version_string
117		self.store = hs.get_datastore()
118		self.sources = hs.get_event_sources()
119		self.presence_handler = hs.get_presence_handler()
120		self.typing_handler = hs.get_typing_handler()
121		self.federation_sender = hs.get_federation_sender()
122		self.notifier = hs.notifier
123		self.clock = hs.get_clock()
124		self.config = hs.get_config()
125
126		self.putChild("remove_pushers", PusherResource(hs))
127		self.putChild("syncing_users", PresenceResource(hs))
128		self.putChild("expire_cache", ExpireCacheResource(hs))
129
130		def render_GET(self, request):
131		self._async_render_GET(request)
132		return NOT_DONE_YET
133
134		@defer.inlineCallbacks
135		def current_replication_token(self):
136		stream_token = yield self.sources.get_current_token()
137		backfill_token = yield self.store.get_current_backfill_token()
138		push_rules_token, room_stream_token = self.store.get_push_rules_stream_token()
139		pushers_token = self.store.get_pushers_stream_token()
140		caches_token = self.store.get_cache_stream_token()
141		public_rooms_token = self.store.get_current_public_room_stream_id()
142		federation_token = self.federation_sender.get_current_token()
143		device_list_token = self.store.get_device_stream_token()
144
145		defer.returnValue(_ReplicationToken(
146		room_stream_token,
147		int(stream_token.presence_key),
148		int(stream_token.typing_key),
149		int(stream_token.receipt_key),
150		int(stream_token.account_data_key),
151		backfill_token,
152		push_rules_token,
153		pushers_token,
154		0, # State stream is no longer a thing
155		caches_token,
156		int(stream_token.to_device_key),
157		int(public_rooms_token),
158		int(federation_token),
159		int(device_list_token),
160		))
161
162		@request_handler()
163		@defer.inlineCallbacks
164		def _async_render_GET(self, request):
165		limit = parse_integer(request, "limit", 100)
166		timeout = parse_integer(request, "timeout", 10 * 1000)
167
168		request.setHeader(b"Content-Type", b"application/json")
169
170		request_streams = {
171		name: parse_integer(request, name)
172		for names in STREAM_NAMES for name in names
173		}
174		request_streams["streams"] = parse_string(request, "streams")
175
176		federation_ack = parse_integer(request, "federation_ack", None)
177
178		def replicate():
179		return self.replicate(
180		request_streams, limit,
181		federation_ack=federation_ack
182		)
183
184		writer = yield self.notifier.wait_for_replication(replicate, timeout)
185		result = writer.finish()
186
187		for stream_name, stream_content in result.items():
188		logger.info(
189		"Replicating %d rows of %s from %s -> %s",
190		len(stream_content["rows"]),
191		stream_name,
192		request_streams.get(stream_name),
193		stream_content["position"],
194		)
195
196		request.write(json.dumps(result, ensure_ascii=False))
197		finish_request(request)
198
199		@defer.inlineCallbacks
200		def replicate(self, request_streams, limit, federation_ack=None):
201		writer = _Writer()
202		current_token = yield self.current_replication_token()
203		logger.debug("Replicating up to %r", current_token)
204
205		if limit == 0:
206		raise SynapseError(400, "Limit cannot be 0")
207
208		yield self.account_data(writer, current_token, limit, request_streams)
209		yield self.events(writer, current_token, limit, request_streams)
210		# TODO: implement limit
211		yield self.presence(writer, current_token, request_streams)
212		yield self.typing(writer, current_token, request_streams)
213		yield self.receipts(writer, current_token, limit, request_streams)
214		yield self.push_rules(writer, current_token, limit, request_streams)
215		yield self.pushers(writer, current_token, limit, request_streams)
216		yield self.caches(writer, current_token, limit, request_streams)
217		yield self.to_device(writer, current_token, limit, request_streams)
218		yield self.public_rooms(writer, current_token, limit, request_streams)
219		yield self.device_lists(writer, current_token, limit, request_streams)
220		self.federation(writer, current_token, limit, request_streams, federation_ack)
221		self.streams(writer, current_token, request_streams)
222
223		logger.debug("Replicated %d rows", writer.total)
224		defer.returnValue(writer)
225
226		def streams(self, writer, current_token, request_streams):
227		request_token = request_streams.get("streams")
228
229		streams = []
230
231		if request_token is not None:
232		if request_token == "-1":
233		for names, position in zip(STREAM_NAMES, current_token):
234		streams.extend((name, position) for name in names)
235		else:
236		items = zip(
237		STREAM_NAMES,
238		current_token,
239		_ReplicationToken(request_token)
240		)
241		for names, current_id, last_id in items:
242		if last_id < current_id:
243		streams.extend((name, current_id) for name in names)
244
245		if streams:
246		writer.write_header_and_rows(
247		"streams", streams, ("name", "position"),
248		position=str(current_token)
249		)
250
251		@defer.inlineCallbacks
252		def events(self, writer, current_token, limit, request_streams):
253		request_events = request_streams.get("events")
254		request_backfill = request_streams.get("backfill")
255
256		if request_events is not None or request_backfill is not None:
257		if request_events is None:
258		request_events = current_token.events
259		if request_backfill is None:
260		request_backfill = current_token.backfill
261
262		no_new_tokens = (
263		request_events == current_token.events
264		and request_backfill == current_token.backfill
265		)
266		if no_new_tokens:
267		return
268
269		res = yield self.store.get_all_new_events(
270		request_backfill, request_events,
271		current_token.backfill, current_token.events,
272		limit
273		)
274
275		upto_events_token = _position_from_rows(
276		res.new_forward_events, current_token.events
277		)
278
279		upto_backfill_token = _position_from_rows(
280		res.new_backfill_events, current_token.backfill
281		)
282
283		if request_events != upto_events_token:
284		writer.write_header_and_rows("events", res.new_forward_events, (
285		"position", "event_id", "room_id", "type", "state_key",
286		), position=upto_events_token)
287
288		if request_backfill != upto_backfill_token:
289		writer.write_header_and_rows("backfill", res.new_backfill_events, (
290		"position", "event_id", "room_id", "type", "state_key", "redacts",
291		), position=upto_backfill_token)
292
293		writer.write_header_and_rows(
294		"forward_ex_outliers", res.forward_ex_outliers,
295		("position", "event_id", "state_group"),
296		)
297		writer.write_header_and_rows(
298		"backward_ex_outliers", res.backward_ex_outliers,
299		("position", "event_id", "state_group"),
300		)
301
302		@defer.inlineCallbacks
303		def presence(self, writer, current_token, request_streams):
304		current_position = current_token.presence
305
306		request_presence = request_streams.get("presence")
307
308		if request_presence is not None and request_presence != current_position:
309		presence_rows = yield self.presence_handler.get_all_presence_updates(
310		request_presence, current_position
311		)
312		upto_token = _position_from_rows(presence_rows, current_position)
313		writer.write_header_and_rows("presence", presence_rows, (
314		"position", "user_id", "state", "last_active_ts",
315		"last_federation_update_ts", "last_user_sync_ts",
316		"status_msg", "currently_active",
317		), position=upto_token)
318
319		@defer.inlineCallbacks
320		def typing(self, writer, current_token, request_streams):
321		current_position = current_token.typing
322
323		request_typing = request_streams.get("typing")
324
325		if request_typing is not None and request_typing != current_position:
326		# If they have a higher token than current max, we can assume that
327		# they had been talking to a previous instance of the master. Since
328		# we reset the token on restart, the best (but hacky) thing we can
329		# do is to simply resend down all the typing notifications.
330		if request_typing > current_position:
331		request_typing = 0
332
333		typing_rows = yield self.typing_handler.get_all_typing_updates(
334		request_typing, current_position
335		)
336		upto_token = _position_from_rows(typing_rows, current_position)
337		writer.write_header_and_rows("typing", typing_rows, (
338		"position", "room_id", "typing"
339		), position=upto_token)
340
341		@defer.inlineCallbacks
342		def receipts(self, writer, current_token, limit, request_streams):
343		current_position = current_token.receipts
344
345		request_receipts = request_streams.get("receipts")
346
347		if request_receipts is not None and request_receipts != current_position:
348		receipts_rows = yield self.store.get_all_updated_receipts(
349		request_receipts, current_position, limit
350		)
351		upto_token = _position_from_rows(receipts_rows, current_position)
352		writer.write_header_and_rows("receipts", receipts_rows, (
353		"position", "room_id", "receipt_type", "user_id", "event_id", "data"
354		), position=upto_token)
355
356		@defer.inlineCallbacks
357		def account_data(self, writer, current_token, limit, request_streams):
358		current_position = current_token.account_data
359
360		user_account_data = request_streams.get("user_account_data")
361		room_account_data = request_streams.get("room_account_data")
362		tag_account_data = request_streams.get("tag_account_data")
363
364		if user_account_data is not None or room_account_data is not None:
365		if user_account_data is None:
366		user_account_data = current_position
367		if room_account_data is None:
368		room_account_data = current_position
369
370		no_new_tokens = (
371		user_account_data == current_position
372		and room_account_data == current_position
373		)
374		if no_new_tokens:
375		return
376
377		user_rows, room_rows = yield self.store.get_all_updated_account_data(
378		user_account_data, room_account_data, current_position, limit
379		)
380
381		upto_users_token = _position_from_rows(user_rows, current_position)
382		upto_rooms_token = _position_from_rows(room_rows, current_position)
383
384		writer.write_header_and_rows("user_account_data", user_rows, (
385		"position", "user_id", "type", "content"
386		), position=upto_users_token)
387		writer.write_header_and_rows("room_account_data", room_rows, (
388		"position", "user_id", "room_id", "type", "content"
389		), position=upto_rooms_token)
390
391		if tag_account_data is not None:
392		tag_rows = yield self.store.get_all_updated_tags(
393		tag_account_data, current_position, limit
394		)
395		upto_tag_token = _position_from_rows(tag_rows, current_position)
396		writer.write_header_and_rows("tag_account_data", tag_rows, (
397		"position", "user_id", "room_id", "tags"
398		), position=upto_tag_token)
399
400		@defer.inlineCallbacks
401		def push_rules(self, writer, current_token, limit, request_streams):
402		current_position = current_token.push_rules
403
404		push_rules = request_streams.get("push_rules")
405
406		if push_rules is not None and push_rules != current_position:
407		rows = yield self.store.get_all_push_rule_updates(
408		push_rules, current_position, limit
409		)
410		upto_token = _position_from_rows(rows, current_position)
411		writer.write_header_and_rows("push_rules", rows, (
412		"position", "event_stream_ordering", "user_id", "rule_id", "op",
413		"priority_class", "priority", "conditions", "actions"
414		), position=upto_token)
415
416		@defer.inlineCallbacks
417		def pushers(self, writer, current_token, limit, request_streams):
418		current_position = current_token.pushers
419
420		pushers = request_streams.get("pushers")
421
422		if pushers is not None and pushers != current_position:
423		updated, deleted = yield self.store.get_all_updated_pushers(
424		pushers, current_position, limit
425		)
426		upto_token = _position_from_rows(updated, current_position)
427		writer.write_header_and_rows("pushers", updated, (
428		"position", "user_id", "access_token", "profile_tag", "kind",
429		"app_id", "app_display_name", "device_display_name", "pushkey",
430		"ts", "lang", "data"
431		), position=upto_token)
432		writer.write_header_and_rows("deleted_pushers", deleted, (
433		"position", "user_id", "app_id", "pushkey"
434		), position=upto_token)
435
436		@defer.inlineCallbacks
437		def caches(self, writer, current_token, limit, request_streams):
438		current_position = current_token.caches
439
440		caches = request_streams.get("caches")
441
442		if caches is not None and caches != current_position:
443		updated_caches = yield self.store.get_all_updated_caches(
444		caches, current_position, limit
445		)
446		upto_token = _position_from_rows(updated_caches, current_position)
447		writer.write_header_and_rows("caches", updated_caches, (
448		"position", "cache_func", "keys", "invalidation_ts"
449		), position=upto_token)
450
451		@defer.inlineCallbacks
452		def to_device(self, writer, current_token, limit, request_streams):
453		current_position = current_token.to_device
454
455		to_device = request_streams.get("to_device")
456
457		if to_device is not None and to_device != current_position:
458		to_device_rows = yield self.store.get_all_new_device_messages(
459		to_device, current_position, limit
460		)
461		upto_token = _position_from_rows(to_device_rows, current_position)
462		writer.write_header_and_rows("to_device", to_device_rows, (
463		"position", "user_id", "device_id", "message_json"
464		), position=upto_token)
465
466		@defer.inlineCallbacks
467		def public_rooms(self, writer, current_token, limit, request_streams):
468		current_position = current_token.public_rooms
469
470		public_rooms = request_streams.get("public_rooms")
471
472		if public_rooms is not None and public_rooms != current_position:
473		public_rooms_rows = yield self.store.get_all_new_public_rooms(
474		public_rooms, current_position, limit
475		)
476		upto_token = _position_from_rows(public_rooms_rows, current_position)
477		writer.write_header_and_rows("public_rooms", public_rooms_rows, (
478		"position", "room_id", "visibility", "appservice_id", "network_id",
479		), position=upto_token)
480
481		def federation(self, writer, current_token, limit, request_streams, federation_ack):
482		if self.config.send_federation:
483		return
484
485		current_position = current_token.federation
486
487		federation = request_streams.get("federation")
488
489		if federation is not None and federation != current_position:
490		federation_rows = self.federation_sender.get_replication_rows(
491		federation, limit, federation_ack=federation_ack,
492		)
493		upto_token = _position_from_rows(federation_rows, current_position)
494		writer.write_header_and_rows("federation", federation_rows, (
495		"position", "type", "content",
496		), position=upto_token)
497
498		@defer.inlineCallbacks
499		def device_lists(self, writer, current_token, limit, request_streams):
500		current_position = current_token.device_lists
501
502		device_lists = request_streams.get("device_lists")
503
504		if device_lists is not None and device_lists != current_position:
505		changes = yield self.store.get_all_device_list_changes_for_remotes(
506		device_lists,
507		)
508		writer.write_header_and_rows("device_lists", changes, (
509		"position", "user_id", "destination",
510		), position=current_position)
511
512
513		class _Writer(object):
514		"""Writes the streams as a JSON object as the response to the request"""
515		def __init__(self):
516		self.streams = {}
517		self.total = 0
518
519		def write_header_and_rows(self, name, rows, fields, position=None):
520		if position is None:
521		if rows:
522		position = rows[-1][0]
523		else:
524		return
525
526		self.streams[name] = {
527		"position": position if type(position) is int else str(position),
528		"field_names": fields,
529		"rows": rows,
530		}
531
532		self.total += len(rows)
533
534		def __nonzero__(self):
535		return bool(self.total)
536
537		def finish(self):
538		return self.streams
539
540
541		class _ReplicationToken(collections.namedtuple("_ReplicationToken", (
542		"events", "presence", "typing", "receipts", "account_data", "backfill",
543		"push_rules", "pushers", "state", "caches", "to_device", "public_rooms",
544		"federation", "device_lists",
545		))):
546		__slots__ = []
547
548		def __new__(cls, *args):
549		if len(args) == 1:
550		streams = [int(value) for value in args[0].split("_")]
551		if len(streams) < len(cls._fields):
552		streams.extend([0] * (len(cls._fields) - len(streams)))
553		return cls(*streams)
554		else:
555		return super(_ReplicationToken, cls).__new__(cls, *args)
556
557		def __str__(self):
558		return "_".join(str(value) for value in self)
559
560
561		def _position_from_rows(rows, current_position):
562		"""Calculates a position to return for a stream. Ideally we want to return the
563		position of the last row, as that will be the most correct. However, if there
564		are no rows we fall back to using the current position to stop us from
565		repeatedly hitting the storage layer unncessarily thinking there are updates.
566		(Not all advances of the token correspond to an actual update)
567
568		We can't just always return the current position, as we often limit the
569		number of rows we replicate, and so the stream may lag. The assumption is
570		that if the storage layer returns no new rows then we are not lagging and
571		we are at the `current_position`.
572		"""
573		if rows:
574		return rows[-1][0]
575		return current_position

+7

-24

synapse/replication/slave/storage/_base.py less more

14	14
15	15	from synapse.storage._base import SQLBaseStore
16	16	from synapse.storage.engines import PostgresEngine
17		from twisted.internet import defer
18	17
19	18	from ._slaved_id_tracker import SlavedIdTracker
20	19

33	32	else:
34	33	self._cache_id_gen = None
35	34
36		self.expire_cache_url = hs.config.worker_replication_url + "/expire_cache"
37		self.http_client = hs.get_simple_http_client()
	35	self.hs = hs
38	36
39	37	def stream_positions(self):
40	38	pos = {}

42	40	pos["caches"] = self._cache_id_gen.get_current_token()
43	41	return pos
44	42
45		def process_replication(self, result):
46		stream = result.get("caches")
47		if stream:
48		for row in stream["rows"]:
49		(
50		position, cache_func, keys, invalidation_ts,
51		) = row
52
	43	def process_replication_rows(self, stream_name, token, rows):
	44	if stream_name == "caches":
	45	self._cache_id_gen.advance(token)
	46	for row in rows:
53	47	try:
54		getattr(self, cache_func).invalidate(tuple(keys))
	48	getattr(self, row.cache_func).invalidate(tuple(row.keys))
55	49	except AttributeError:
56	50	# We probably haven't pulled in the cache in this worker,
57	51	# which is fine.
58	52	pass
59		self._cache_id_gen.advance(int(stream["position"]))
60		return defer.succeed(None)
61	53
62	54	def _invalidate_cache_and_stream(self, txn, cache_func, keys):
63	55	txn.call_after(cache_func.invalidate, keys)
64	56	txn.call_after(self._send_invalidation_poke, cache_func, keys)
65	57
66		@defer.inlineCallbacks
67	58	def _send_invalidation_poke(self, cache_func, keys):
68		try:
69		yield self.http_client.post_json_get_json(self.expire_cache_url, {
70		"invalidate": [{
71		"name": cache_func.__name__,
72		"keys": list(keys),
73		}]
74		})
75		except:
76		logger.exception("Failed to poke on expire_cache")
	59	self.hs.get_tcp_replication().send_invalidate_cache(cache_func, keys)

+19

-32

synapse/replication/slave/storage/account_data.py less more

68	68	result["tag_account_data"] = position
69	69	return result
70	70
71		def process_replication(self, result):
72		stream = result.get("user_account_data")
73		if stream:
74		self._account_data_id_gen.advance(int(stream["position"]))
75		for row in stream["rows"]:
76		position, user_id, data_type = row[:3]
77		self.get_global_account_data_by_type_for_user.invalidate(
78		(data_type, user_id,)
	71	def process_replication_rows(self, stream_name, token, rows):
	72	if stream_name == "tag_account_data":
	73	self._account_data_id_gen.advance(token)
	74	for row in rows:
	75	self.get_tags_for_user.invalidate((row.user_id,))
	76	self._account_data_stream_cache.entity_has_changed(
	77	row.user_id, token
79	78	)
80		self.get_account_data_for_user.invalidate((user_id,))
	79	elif stream_name == "account_data":
	80	self._account_data_id_gen.advance(token)
	81	for row in rows:
	82	if not row.room_id:
	83	self.get_global_account_data_by_type_for_user.invalidate(
	84	(row.data_type, row.user_id,)
	85	)
	86	self.get_account_data_for_user.invalidate((row.user_id,))
81	87	self._account_data_stream_cache.entity_has_changed(
82		user_id, position
	88	row.user_id, token
83	89	)
84
85		stream = result.get("room_account_data")
86		if stream:
87		self._account_data_id_gen.advance(int(stream["position"]))
88		for row in stream["rows"]:
89		position, user_id = row[:2]
90		self.get_account_data_for_user.invalidate((user_id,))
91		self._account_data_stream_cache.entity_has_changed(
92		user_id, position
93		)
94
95		stream = result.get("tag_account_data")
96		if stream:
97		self._account_data_id_gen.advance(int(stream["position"]))
98		for row in stream["rows"]:
99		position, user_id = row[:2]
100		self.get_tags_for_user.invalidate((user_id,))
101		self._account_data_stream_cache.entity_has_changed(
102		user_id, position
103		)
104
105		return super(SlavedAccountDataStore, self).process_replication(result)
	90	return super(SlavedAccountDataStore, self).process_replication_rows(
	91	stream_name, token, rows
	92	)

+10

-13

synapse/replication/slave/storage/deviceinbox.py less more

52	52	result["to_device"] = self._device_inbox_id_gen.get_current_token()
53	53	return result
54	54
55		def process_replication(self, result):
56		stream = result.get("to_device")
57		if stream:
58		self._device_inbox_id_gen.advance(int(stream["position"]))
59		for row in stream["rows"]:
60		stream_id = row[0]
61		entity = row[1]
62
63		if entity.startswith("@"):
	55	def process_replication_rows(self, stream_name, token, rows):
	56	if stream_name == "to_device":
	57	self._device_inbox_id_gen.advance(token)
	58	for row in rows:
	59	if row.entity.startswith("@"):
64	60	self._device_inbox_stream_cache.entity_has_changed(
65		entity, stream_id
	61	row.entity, token
66	62	)
67	63	else:
68	64	self._device_federation_outbox_stream_cache.entity_has_changed(
69		entity, stream_id
	65	row.entity, token
70	66	)
71
72		return super(SlavedDeviceInboxStore, self).process_replication(result)
	67	return super(SlavedDeviceInboxStore, self).process_replication_rows(
	68	stream_name, token, rows
	69	)

+10

-14

synapse/replication/slave/storage/devices.py less more

50	50	result["device_lists"] = self._device_list_id_gen.get_current_token()
51	51	return result
52	52
53		def process_replication(self, result):
54		stream = result.get("device_lists")
55		if stream:
56		self._device_list_id_gen.advance(int(stream["position"]))
57		for row in stream["rows"]:
58		stream_id = row[0]
59		user_id = row[1]
60		destination = row[2]
61
	53	def process_replication_rows(self, stream_name, token, rows):
	54	if stream_name == "device_lists":
	55	self._device_list_id_gen.advance(token)
	56	for row in rows:
62	57	self._device_list_stream_cache.entity_has_changed(
63		user_id, stream_id
	58	row.user_id, token
64	59	)
65	60
66		if destination:
	61	if row.destination:
67	62	self._device_list_federation_stream_cache.entity_has_changed(
68		destination, stream_id
	63	row.destination, token
69	64	)
70
71		return super(SlavedDeviceStore, self).process_replication(result)
	65	return super(SlavedDeviceStore, self).process_replication_rows(
	66	stream_name, token, rows
	67	)

+21

-43

synapse/replication/slave/storage/events.py less more

70	70	# to reach inside the __dict__ to extract them.
71	71	get_rooms_for_user = RoomMemberStore.__dict__["get_rooms_for_user"]
72	72	get_users_in_room = RoomMemberStore.__dict__["get_users_in_room"]
	73	get_hosts_in_room = RoomMemberStore.__dict__["get_hosts_in_room"]
73	74	get_users_who_share_room_with_user = (
74	75	RoomMemberStore.__dict__["get_users_who_share_room_with_user"]
75	76	)

99	100	)
100	101	_get_state_groups_from_groups_txn = (
101	102	DataStore._get_state_groups_from_groups_txn.__func__
102		)
103		_get_state_group_from_group = (
104		StateStore.__dict__["_get_state_group_from_group"]
105	103	)
106	104	get_recent_event_ids_for_room = (
107	105	StreamStore.__dict__["get_recent_event_ids_for_room"]

145	143	RoomMemberStore.__dict__["_get_joined_users_from_context"]
146	144	)
147	145
	146	get_joined_hosts = DataStore.get_joined_hosts.__func__
	147	_get_joined_hosts = RoomMemberStore.__dict__["_get_joined_hosts"]
	148
148	149	get_recent_events_for_room = DataStore.get_recent_events_for_room.__func__
149	150	get_room_events_stream_for_rooms = (
150	151	DataStore.get_room_events_stream_for_rooms.__func__

200	201	result["backfill"] = -self._backfill_id_gen.get_current_token()
201	202	return result
202	203
203		def process_replication(self, result):
204		stream = result.get("events")
205		if stream:
206		self._stream_id_gen.advance(int(stream["position"]))
207
208		if stream["rows"]:
209		logger.info("Got %d event rows", len(stream["rows"]))
210
211		for row in stream["rows"]:
212		self._process_replication_row(
213		row, backfilled=False,
	204	def process_replication_rows(self, stream_name, token, rows):
	205	if stream_name == "events":
	206	self._stream_id_gen.advance(token)
	207	for row in rows:
	208	self.invalidate_caches_for_event(
	209	token, row.event_id, row.room_id, row.type, row.state_key,
	210	row.redacts,
	211	backfilled=False,
214	212	)
215
216		stream = result.get("backfill")
217		if stream:
218		self._backfill_id_gen.advance(-int(stream["position"]))
219		for row in stream["rows"]:
220		self._process_replication_row(
221		row, backfilled=True,
	213	elif stream_name == "backfill":
	214	self._backfill_id_gen.advance(-token)
	215	for row in rows:
	216	self.invalidate_caches_for_event(
	217	-token, row.event_id, row.room_id, row.type, row.state_key,
	218	row.redacts,
	219	backfilled=True,
222	220	)
223
224		stream = result.get("forward_ex_outliers")
225		if stream:
226		self._stream_id_gen.advance(int(stream["position"]))
227		for row in stream["rows"]:
228		event_id = row[1]
229		self._invalidate_get_event_cache(event_id)
230
231		stream = result.get("backward_ex_outliers")
232		if stream:
233		self._backfill_id_gen.advance(-int(stream["position"]))
234		for row in stream["rows"]:
235		event_id = row[1]
236		self._invalidate_get_event_cache(event_id)
237
238		return super(SlavedEventStore, self).process_replication(result)
239
240		def _process_replication_row(self, row, backfilled):
241		stream_ordering = row[0] if not backfilled else -row[0]
242		self.invalidate_caches_for_event(
243		stream_ordering, row[1], row[2], row[3], row[4], row[5],
244		backfilled=backfilled,
	221	return super(SlavedEventStore, self).process_replication_rows(
	222	stream_name, token, rows
245	223	)
246	224
247	225	def invalidate_caches_for_event(self, stream_ordering, event_id, room_id,

+19

-10

synapse/replication/slave/storage/presence.py less more

38	38	_get_presence_for_user = PresenceStore.__dict__["_get_presence_for_user"]
39	39	get_presence_for_users = PresenceStore.__dict__["get_presence_for_users"]
40	40
	41	# XXX: This is a bit broken because we don't persist the accepted list in a
	42	# way that can be replicated. This means that we don't have a way to
	43	# invalidate the cache correctly.
	44	get_presence_list_accepted = PresenceStore.__dict__[
	45	"get_presence_list_accepted"
	46	]
	47	get_presence_list_observers_accepted = PresenceStore.__dict__[
	48	"get_presence_list_observers_accepted"
	49	]
	50
41	51	def get_current_presence_token(self):
42	52	return self._presence_id_gen.get_current_token()
43	53

47	57	result["presence"] = position
48	58	return result
49	59
50		def process_replication(self, result):
51		stream = result.get("presence")
52		if stream:
53		self._presence_id_gen.advance(int(stream["position"]))
54		for row in stream["rows"]:
55		position, user_id = row[:2]
	60	def process_replication_rows(self, stream_name, token, rows):
	61	if stream_name == "presence":
	62	self._presence_id_gen.advance(token)
	63	for row in rows:
56	64	self.presence_stream_cache.entity_has_changed(
57		user_id, position
	65	row.user_id, token
58	66	)
59		self._get_presence_for_user.invalidate((user_id,))
60
61		return super(SlavedPresenceStore, self).process_replication(result)
	67	self._get_presence_for_user.invalidate((row.user_id,))
	68	return super(SlavedPresenceStore, self).process_replication_rows(
	69	stream_name, token, rows
	70	)

+10

-13

synapse/replication/slave/storage/push_rule.py less more

49	49	result["push_rules"] = self._push_rules_stream_id_gen.get_current_token()
50	50	return result
51	51
52		def process_replication(self, result):
53		stream = result.get("push_rules")
54		if stream:
55		for row in stream["rows"]:
56		position = row[0]
57		user_id = row[2]
58		self.get_push_rules_for_user.invalidate((user_id,))
59		self.get_push_rules_enabled_for_user.invalidate((user_id,))
	52	def process_replication_rows(self, stream_name, token, rows):
	53	if stream_name == "push_rules":
	54	self._push_rules_stream_id_gen.advance(token)
	55	for row in rows:
	56	self.get_push_rules_for_user.invalidate((row.user_id,))
	57	self.get_push_rules_enabled_for_user.invalidate((row.user_id,))
60	58	self.push_rules_stream_cache.entity_has_changed(
61		user_id, position
	59	row.user_id, token
62	60	)
63
64		self._push_rules_stream_id_gen.advance(int(stream["position"]))
65
66		return super(SlavedPushRuleStore, self).process_replication(result)
	61	return super(SlavedPushRuleStore, self).process_replication_rows(
	62	stream_name, token, rows
	63	)

+6

-10

synapse/replication/slave/storage/pushers.py less more

39	39	result["pushers"] = self._pushers_id_gen.get_current_token()
40	40	return result
41	41
42		def process_replication(self, result):
43		stream = result.get("pushers")
44		if stream:
45		self._pushers_id_gen.advance(int(stream["position"]))
46
47		stream = result.get("deleted_pushers")
48		if stream:
49		self._pushers_id_gen.advance(int(stream["position"]))
50
51		return super(SlavedPusherStore, self).process_replication(result)
	42	def process_replication_rows(self, stream_name, token, rows):
	43	if stream_name == "pushers":
	44	self._pushers_id_gen.advance(token)
	45	return super(SlavedPusherStore, self).process_replication_rows(
	46	stream_name, token, rows
	47	)

+13

-11

synapse/replication/slave/storage/receipts.py less more

64	64	result["receipts"] = self._receipts_id_gen.get_current_token()
65	65	return result
66	66
67		def process_replication(self, result):
68		stream = result.get("receipts")
69		if stream:
70		self._receipts_id_gen.advance(int(stream["position"]))
71		for row in stream["rows"]:
72		position, room_id, receipt_type, user_id = row[:4]
73		self.invalidate_caches_for_receipt(room_id, receipt_type, user_id)
74		self._receipts_stream_cache.entity_has_changed(room_id, position)
75
76		return super(SlavedReceiptsStore, self).process_replication(result)
77
78	67	def invalidate_caches_for_receipt(self, room_id, receipt_type, user_id):
79	68	self.get_receipts_for_user.invalidate((user_id, receipt_type))
80	69	self.get_linearized_receipts_for_room.invalidate_many((room_id,))
81	70	self.get_last_receipt_event_id_for_user.invalidate(
82	71	(user_id, room_id, receipt_type)
83	72	)
	73
	74	def process_replication_rows(self, stream_name, token, rows):
	75	if stream_name == "receipts":
	76	self._receipts_id_gen.advance(token)
	77	for row in rows:
	78	self.invalidate_caches_for_receipt(
	79	row.room_id, row.receipt_type, row.user_id
	80	)
	81	self._receipts_stream_cache.entity_has_changed(row.room_id, token)
	82
	83	return super(SlavedReceiptsStore, self).process_replication_rows(
	84	stream_name, token, rows
	85	)

+6

-5

synapse/replication/slave/storage/room.py less more

45	45	result["public_rooms"] = self._public_room_id_gen.get_current_token()
46	46	return result
47	47
48		def process_replication(self, result):
49		stream = result.get("public_rooms")
50		if stream:
51		self._public_room_id_gen.advance(int(stream["position"]))
	48	def process_replication_rows(self, stream_name, token, rows):
	49	if stream_name == "public_rooms":
	50	self._public_room_id_gen.advance(token)
52	51
53		return super(RoomStore, self).process_replication(result)
	52	return super(RoomStore, self).process_replication_rows(
	53	stream_name, token, rows
	54	)

+30

-0

synapse/replication/tcp/__init__.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2017 Vector Creations Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	"""This module implements the TCP replication protocol used by synapse to
	16	communicate between the master process and its workers (when they're enabled).
	17
	18	Further details can be found in docs/tcp_replication.rst
	19
	20
	21	Structure of the module:
	22	* client.py - the client classes used for workers to connect to master
	23	* command.py - the definitions of all the valid commands
	24	* protocol.py - contains bot the client and server protocol implementations,
	25	these should not be used directly
	26	* resource.py - the server classes that accepts and handle client connections
	27	* streams.py - the definitons of all the valid streams
	28
	29	"""

+196

-0

synapse/replication/tcp/client.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2017 Vector Creations Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	"""A replication client for use by synapse workers.
	15	"""
	16
	17	from twisted.internet import reactor, defer
	18	from twisted.internet.protocol import ReconnectingClientFactory
	19
	20	from .commands import (
	21	FederationAckCommand, UserSyncCommand, RemovePusherCommand, InvalidateCacheCommand,
	22	)
	23	from .protocol import ClientReplicationStreamProtocol
	24
	25	import logging
	26
	27	logger = logging.getLogger(__name__)
	28
	29
	30	class ReplicationClientFactory(ReconnectingClientFactory):
	31	"""Factory for building connections to the master. Will reconnect if the
	32	connection is lost.
	33
	34	Accepts a handler that will be called when new data is available or data
	35	is required.
	36	"""
	37	maxDelay = 5 # Try at least once every N seconds
	38
	39	def __init__(self, hs, client_name, handler):
	40	self.client_name = client_name
	41	self.handler = handler
	42	self.server_name = hs.config.server_name
	43	self._clock = hs.get_clock() # As self.clock is defined in super class
	44
	45	reactor.addSystemEventTrigger("before", "shutdown", self.stopTrying)
	46
	47	def startedConnecting(self, connector):
	48	logger.info("Connecting to replication: %r", connector.getDestination())
	49
	50	def buildProtocol(self, addr):
	51	logger.info("Connected to replication: %r", addr)
	52	self.resetDelay()
	53	return ClientReplicationStreamProtocol(
	54	self.client_name, self.server_name, self._clock, self.handler
	55	)
	56
	57	def clientConnectionLost(self, connector, reason):
	58	logger.error("Lost replication conn: %r", reason)
	59	ReconnectingClientFactory.clientConnectionLost(self, connector, reason)
	60
	61	def clientConnectionFailed(self, connector, reason):
	62	logger.error("Failed to connect to replication: %r", reason)
	63	ReconnectingClientFactory.clientConnectionFailed(
	64	self, connector, reason
	65	)
	66
	67
	68	class ReplicationClientHandler(object):
	69	"""A base handler that can be passed to the ReplicationClientFactory.
	70
	71	By default proxies incoming replication data to the SlaveStore.
	72	"""
	73	def __init__(self, store):
	74	self.store = store
	75
	76	# The current connection. None if we are currently (re)connecting
	77	self.connection = None
	78
	79	# Any pending commands to be sent once a new connection has been
	80	# established
	81	self.pending_commands = []
	82
	83	# Map from string -> deferred, to wake up when receiveing a SYNC with
	84	# the given string.
	85	# Used for tests.
	86	self.awaiting_syncs = {}
	87
	88	def start_replication(self, hs):
	89	"""Helper method to start a replication connection to the remote server
	90	using TCP.
	91	"""
	92	client_name = hs.config.worker_name
	93	factory = ReplicationClientFactory(hs, client_name, self)
	94	host = hs.config.worker_replication_host
	95	port = hs.config.worker_replication_port
	96	reactor.connectTCP(host, port, factory)
	97
	98	def on_rdata(self, stream_name, token, rows):
	99	"""Called when we get new replication data. By default this just pokes
	100	the slave store.
	101
	102	Can be overriden in subclasses to handle more.
	103	"""
	104	logger.info("Received rdata %s -> %s", stream_name, token)
	105	self.store.process_replication_rows(stream_name, token, rows)
	106
	107	def on_position(self, stream_name, token):
	108	"""Called when we get new position data. By default this just pokes
	109	the slave store.
	110
	111	Can be overriden in subclasses to handle more.
	112	"""
	113	self.store.process_replication_rows(stream_name, token, [])
	114
	115	def on_sync(self, data):
	116	"""When we received a SYNC we wake up any deferreds that were waiting
	117	for the sync with the given data.
	118
	119	Used by tests.
	120	"""
	121	d = self.awaiting_syncs.pop(data, None)
	122	if d:
	123	d.callback(data)
	124
	125	def get_streams_to_replicate(self):
	126	"""Called when a new connection has been established and we need to
	127	subscribe to streams.
	128
	129	Returns a dictionary of stream name to token.
	130	"""
	131	args = self.store.stream_positions()
	132	user_account_data = args.pop("user_account_data", None)
	133	room_account_data = args.pop("room_account_data", None)
	134	if user_account_data:
	135	args["account_data"] = user_account_data
	136	elif room_account_data:
	137	args["account_data"] = room_account_data
	138	return args
	139
	140	def get_currently_syncing_users(self):
	141	"""Get the list of currently syncing users (if any). This is called
	142	when a connection has been established and we need to send the
	143	currently syncing users. (Overriden by the synchrotron's only)
	144	"""
	145	return []
	146
	147	def send_command(self, cmd):
	148	"""Send a command to master (when we get establish a connection if we
	149	don't have one already.)
	150	"""
	151	if self.connection:
	152	self.connection.send_command(cmd)
	153	else:
	154	logger.warn("Queuing command as not connected: %r", cmd.NAME)
	155	self.pending_commands.append(cmd)
	156
	157	def send_federation_ack(self, token):
	158	"""Ack data for the federation stream. This allows the master to drop
	159	data stored purely in memory.
	160	"""
	161	self.send_command(FederationAckCommand(token))
	162
	163	def send_user_sync(self, user_id, is_syncing, last_sync_ms):
	164	"""Poke the master that a user has started/stopped syncing.
	165	"""
	166	self.send_command(UserSyncCommand(user_id, is_syncing, last_sync_ms))
	167
	168	def send_remove_pusher(self, app_id, push_key, user_id):
	169	"""Poke the master to remove a pusher for a user
	170	"""
	171	cmd = RemovePusherCommand(app_id, push_key, user_id)
	172	self.send_command(cmd)
	173
	174	def send_invalidate_cache(self, cache_func, keys):
	175	"""Poke the master to invalidate a cache.
	176	"""
	177	cmd = InvalidateCacheCommand(cache_func.__name__, keys)
	178	self.send_command(cmd)
	179
	180	def await_sync(self, data):
	181	"""Returns a deferred that is resolved when we receive a SYNC command
	182	with given data.
	183
	184	Used by tests.
	185	"""
	186	return self.awaiting_syncs.setdefault(data, defer.Deferred())
	187
	188	def update_connection(self, connection):
	189	"""Called when a connection has been established (or lost with None).
	190	"""
	191	self.connection = connection
	192	if connection:
	193	for cmd in self.pending_commands:
	194	connection.send_command(cmd)
	195	self.pending_commands = []

+346

-0

synapse/replication/tcp/commands.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2017 Vector Creations Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	"""Defines the various valid commands
	15
	16	The VALID_SERVER_COMMANDS and VALID_CLIENT_COMMANDS define which commands are
	17	allowed to be sent by which side.
	18	"""
	19
	20	import logging
	21	import ujson as json
	22
	23
	24	logger = logging.getLogger(__name__)
	25
	26
	27	class Command(object):
	28	"""The base command class.
	29
	30	All subclasses must set the NAME variable which equates to the name of the
	31	command on the wire.
	32
	33	A full command line on the wire is constructed from `NAME + " " + to_line()`
	34
	35	The default implementation creates a command of form `<NAME> <data>`
	36	"""
	37	NAME = None
	38
	39	def __init__(self, data):
	40	self.data = data
	41
	42	@classmethod
	43	def from_line(cls, line):
	44	"""Deserialises a line from the wire into this command. `line` does not
	45	include the command.
	46	"""
	47	return cls(line)
	48
	49	def to_line(self):
	50	"""Serialises the comamnd for the wire. Does not include the command
	51	prefix.
	52	"""
	53	return self.data
	54
	55
	56	class ServerCommand(Command):
	57	"""Sent by the server on new connection and includes the server_name.
	58
	59	Format::
	60
	61	SERVER <server_name>
	62	"""
	63	NAME = "SERVER"
	64
	65
	66	class RdataCommand(Command):
	67	"""Sent by server when a subscribed stream has an update.
	68
	69	Format::
	70
	71	RDATA <stream_name> <token> <row_json>
	72
	73	The `<token>` may either be a numeric stream id OR "batch". The latter case
	74	is used to support sending multiple updates with the same stream ID. This
	75	is done by sending an RDATA for each row, with all but the last RDATA having
	76	a token of "batch" and the last having the final stream ID.
	77
	78	The client should batch all incoming RDATA with a token of "batch" (per
	79	stream_name) until it sees an RDATA with a numeric stream ID.
	80
	81	`<token>` of "batch" maps to the instance variable `token` being None.
	82
	83	An example of a batched series of RDATA::
	84
	85	RDATA presence batch ["@foo:example.com", "online", ...]
	86	RDATA presence batch ["@bar:example.com", "online", ...]
	87	RDATA presence 59 ["@baz:example.com", "online", ...]
	88	"""
	89	NAME = "RDATA"
	90
	91	def __init__(self, stream_name, token, row):
	92	self.stream_name = stream_name
	93	self.token = token
	94	self.row = row
	95
	96	@classmethod
	97	def from_line(cls, line):
	98	stream_name, token, row_json = line.split(" ", 2)
	99	return cls(
	100	stream_name,
	101	None if token == "batch" else int(token),
	102	json.loads(row_json)
	103	)
	104
	105	def to_line(self):
	106	return " ".join((
	107	self.stream_name,
	108	str(self.token) if self.token is not None else "batch",
	109	json.dumps(self.row),
	110	))
	111
	112
	113	class PositionCommand(Command):
	114	"""Sent by the client to tell the client the stream postition without
	115	needing to send an RDATA.
	116	"""
	117	NAME = "POSITION"
	118
	119	def __init__(self, stream_name, token):
	120	self.stream_name = stream_name
	121	self.token = token
	122
	123	@classmethod
	124	def from_line(cls, line):
	125	stream_name, token = line.split(" ", 1)
	126	return cls(stream_name, int(token))
	127
	128	def to_line(self):
	129	return " ".join((self.stream_name, str(self.token),))
	130
	131
	132	class ErrorCommand(Command):
	133	"""Sent by either side if there was an ERROR. The data is a string describing
	134	the error.
	135	"""
	136	NAME = "ERROR"
	137
	138
	139	class PingCommand(Command):
	140	"""Sent by either side as a keep alive. The data is arbitary (often timestamp)
	141	"""
	142	NAME = "PING"
	143
	144
	145	class NameCommand(Command):
	146	"""Sent by client to inform the server of the client's identity. The data
	147	is the name
	148	"""
	149	NAME = "NAME"
	150
	151
	152	class ReplicateCommand(Command):
	153	"""Sent by the client to subscribe to the stream.
	154
	155	Format::
	156
	157	REPLICATE <stream_name> <token>
	158
	159	Where <token> may be either:
	160	* a numeric stream_id to stream updates from
	161	* "NOW" to stream all subsequent updates.
	162
	163	The <stream_name> can be "ALL" to subscribe to all known streams, in which
	164	case the <token> must be set to "NOW", i.e.::
	165
	166	REPLICATE ALL NOW
	167	"""
	168	NAME = "REPLICATE"
	169
	170	def __init__(self, stream_name, token):
	171	self.stream_name = stream_name
	172	self.token = token
	173
	174	@classmethod
	175	def from_line(cls, line):
	176	stream_name, token = line.split(" ", 1)
	177	if token in ("NOW", "now"):
	178	token = "NOW"
	179	else:
	180	token = int(token)
	181	return cls(stream_name, token)
	182
	183	def to_line(self):
	184	return " ".join((self.stream_name, str(self.token),))
	185
	186
	187	class UserSyncCommand(Command):
	188	"""Sent by the client to inform the server that a user has started or
	189	stopped syncing. Used to calculate presence on the master.
	190
	191	Includes a timestamp of when the last user sync was.
	192
	193	Format::
	194
	195	USER_SYNC <user_id> <state> <last_sync_ms>
	196
	197	Where <state> is either "start" or "stop"
	198	"""
	199	NAME = "USER_SYNC"
	200
	201	def __init__(self, user_id, is_syncing, last_sync_ms):
	202	self.user_id = user_id
	203	self.is_syncing = is_syncing
	204	self.last_sync_ms = last_sync_ms
	205
	206	@classmethod
	207	def from_line(cls, line):
	208	user_id, state, last_sync_ms = line.split(" ", 2)
	209
	210	if state not in ("start", "end"):
	211	raise Exception("Invalid USER_SYNC state %r" % (state,))
	212
	213	return cls(user_id, state == "start", int(last_sync_ms))
	214
	215	def to_line(self):
	216	return " ".join((
	217	self.user_id, "start" if self.is_syncing else "end", str(self.last_sync_ms),
	218	))
	219
	220
	221	class FederationAckCommand(Command):
	222	"""Sent by the client when it has processed up to a given point in the
	223	federation stream. This allows the master to drop in-memory caches of the
	224	federation stream.
	225
	226	This must only be sent from one worker (i.e. the one sending federation)
	227
	228	Format::
	229
	230	FEDERATION_ACK <token>
	231	"""
	232	NAME = "FEDERATION_ACK"
	233
	234	def __init__(self, token):
	235	self.token = token
	236
	237	@classmethod
	238	def from_line(cls, line):
	239	return cls(int(line))
	240
	241	def to_line(self):
	242	return str(self.token)
	243
	244
	245	class SyncCommand(Command):
	246	"""Used for testing. The client protocol implementation allows waiting
	247	on a SYNC command with a specified data.
	248	"""
	249	NAME = "SYNC"
	250
	251
	252	class RemovePusherCommand(Command):
	253	"""Sent by the client to request the master remove the given pusher.
	254
	255	Format::
	256
	257	REMOVE_PUSHER <app_id> <push_key> <user_id>
	258	"""
	259	NAME = "REMOVE_PUSHER"
	260
	261	def __init__(self, app_id, push_key, user_id):
	262	self.user_id = user_id
	263	self.app_id = app_id
	264	self.push_key = push_key
	265
	266	@classmethod
	267	def from_line(cls, line):
	268	app_id, push_key, user_id = line.split(" ", 2)
	269
	270	return cls(app_id, push_key, user_id)
	271
	272	def to_line(self):
	273	return " ".join((self.app_id, self.push_key, self.user_id))
	274
	275
	276	class InvalidateCacheCommand(Command):
	277	"""Sent by the client to invalidate an upstream cache.
	278
	279	THIS IS NOT RELIABLE, AND SHOULD NOT BE USED ACCEPT FOR THINGS THAT ARE
	280	NOT DISASTROUS IF WE DROP ON THE FLOOR.
	281
	282	Mainly used to invalidate destination retry timing caches.
	283
	284	Format::
	285
	286	INVALIDATE_CACHE <cache_func> <keys_json>
	287
	288	Where <keys_json> is a json list.
	289	"""
	290	NAME = "INVALIDATE_CACHE"
	291
	292	def __init__(self, cache_func, keys):
	293	self.cache_func = cache_func
	294	self.keys = keys
	295
	296	@classmethod
	297	def from_line(cls, line):
	298	cache_func, keys_json = line.split(" ", 1)
	299
	300	return cls(cache_func, json.loads(keys_json))
	301
	302	def to_line(self):
	303	return " ".join((self.cache_func, json.dumps(self.keys)))
	304
	305
	306	# Map of command name to command type.
	307	COMMAND_MAP = {
	308	cmd.NAME: cmd
	309	for cmd in (
	310	ServerCommand,
	311	RdataCommand,
	312	PositionCommand,
	313	ErrorCommand,
	314	PingCommand,
	315	NameCommand,
	316	ReplicateCommand,
	317	UserSyncCommand,
	318	FederationAckCommand,
	319	SyncCommand,
	320	RemovePusherCommand,
	321	InvalidateCacheCommand,
	322	)
	323	}
	324
	325	# The commands the server is allowed to send
	326	VALID_SERVER_COMMANDS = (
	327	ServerCommand.NAME,
	328	RdataCommand.NAME,
	329	PositionCommand.NAME,
	330	ErrorCommand.NAME,
	331	PingCommand.NAME,
	332	SyncCommand.NAME,
	333	)
	334
	335	# The commands the client is allowed to send
	336	VALID_CLIENT_COMMANDS = (
	337	NameCommand.NAME,
	338	ReplicateCommand.NAME,
	339	PingCommand.NAME,
	340	UserSyncCommand.NAME,
	341	FederationAckCommand.NAME,
	342	RemovePusherCommand.NAME,
	343	InvalidateCacheCommand.NAME,
	344	ErrorCommand.NAME,
	345	)

+640

-0

synapse/replication/tcp/protocol.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2017 Vector Creations Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	"""This module contains the implementation of both the client and server
	15	protocols.
	16
	17	The basic structure of the protocol is line based, where the initial word of
	18	each line specifies the command. The rest of the line is parsed based on the
	19	command. For example, the `RDATA` command is defined as::
	20
	21	RDATA <stream_name> <token> <row_json>
	22
	23	(Note that `<row_json>` may contains spaces, but cannot contain newlines.)
	24
	25	Blank lines are ignored.
	26
	27	# Example
	28
	29	An example iteraction is shown below. Each line is prefixed with '>' or '<' to
	30	indicate which side is sending, these are not included on the wire::
	31
	32	* connection established *
	33	> SERVER localhost:8823
	34	> PING 1490197665618
	35	< NAME synapse.app.appservice
	36	< PING 1490197665618
	37	< REPLICATE events 1
	38	< REPLICATE backfill 1
	39	< REPLICATE caches 1
	40	> POSITION events 1
	41	> POSITION backfill 1
	42	> POSITION caches 1
	43	> RDATA caches 2 ["get_user_by_id",["@01register-user:localhost:8823"],1490197670513]
	44	> RDATA events 14 ["$149019767112vOHxz:localhost:8823",
	45	"!AFDCvgApUmpdfVjIXm:localhost:8823","m.room.guest_access","",null]
	46	< PING 1490197675618
	47	> ERROR server stopping
	48	* connection closed by server *
	49	"""
	50
	51	from twisted.internet import defer
	52	from twisted.protocols.basic import LineOnlyReceiver
	53	from twisted.python.failure import Failure
	54
	55	from commands import (
	56	COMMAND_MAP, VALID_CLIENT_COMMANDS, VALID_SERVER_COMMANDS,
	57	ErrorCommand, ServerCommand, RdataCommand, PositionCommand, PingCommand,
	58	NameCommand, ReplicateCommand, UserSyncCommand, SyncCommand,
	59	)
	60	from streams import STREAMS_MAP
	61
	62	from synapse.util.stringutils import random_string
	63	from synapse.metrics.metric import CounterMetric
	64
	65	import logging
	66	import synapse.metrics
	67	import struct
	68	import fcntl
	69
	70
	71	metrics = synapse.metrics.get_metrics_for(__name__)
	72
	73	connection_close_counter = metrics.register_counter(
	74	"close_reason", labels=["reason_type"],
	75	)
	76
	77
	78	# A list of all connected protocols. This allows us to send metrics about the
	79	# connections.
	80	connected_connections = []
	81
	82
	83	logger = logging.getLogger(__name__)
	84
	85
	86	PING_TIME = 5000
	87	PING_TIMEOUT_MULTIPLIER = 5
	88	PING_TIMEOUT_MS = PING_TIME * PING_TIMEOUT_MULTIPLIER
	89
	90
	91	class ConnectionStates(object):
	92	CONNECTING = "connecting"
	93	ESTABLISHED = "established"
	94	PAUSED = "paused"
	95	CLOSED = "closed"
	96
	97
	98	class BaseReplicationStreamProtocol(LineOnlyReceiver):
	99	"""Base replication protocol shared between client and server.
	100
	101	Reads lines (ignoring blank ones) and parses them into command classes,
	102	asserting that they are valid for the given direction, i.e. server commands
	103	are only sent by the server.
	104
	105	On receiving a new command it calls `on_<COMMAND_NAME>` with the parsed
	106	command.
	107
	108	It also sends `PING` periodically, and correctly times out remote connections
	109	(if they send a `PING` command)
	110	"""
	111	delimiter = b'\n'
	112
	113	VALID_INBOUND_COMMANDS = [] # Valid commands we expect to receive
	114	VALID_OUTBOUND_COMMANDS = [] # Valid commans we can send
	115
	116	max_line_buffer = 10000
	117
	118	def __init__(self, clock):
	119	self.clock = clock
	120
	121	self.last_received_command = self.clock.time_msec()
	122	self.last_sent_command = 0
	123	self.time_we_closed = None # When we requested the connection be closed
	124
	125	self.received_ping = False # Have we reecived a ping from the other side
	126
	127	self.state = ConnectionStates.CONNECTING
	128
	129	self.name = "anon" # The name sent by a client.
	130	self.conn_id = random_string(5) # To dedupe in case of name clashes.
	131
	132	# List of pending commands to send once we've established the connection
	133	self.pending_commands = []
	134
	135	# The LoopingCall for sending pings.
	136	self._send_ping_loop = None
	137
	138	self.inbound_commands_counter = CounterMetric(
	139	"inbound_commands", labels=["command"],
	140	)
	141	self.outbound_commands_counter = CounterMetric(
	142	"outbound_commands", labels=["command"],
	143	)
	144
	145	def connectionMade(self):
	146	logger.info("[%s] Connection established", self.id())
	147
	148	self.state = ConnectionStates.ESTABLISHED
	149
	150	connected_connections.append(self) # Register connection for metrics
	151
	152	self.transport.registerProducer(self, True) # For the *Producing callbacks
	153
	154	self._send_pending_commands()
	155
	156	# Starts sending pings
	157	self._send_ping_loop = self.clock.looping_call(self.send_ping, 5000)
	158
	159	# Always send the initial PING so that the other side knows that they
	160	# can time us out.
	161	self.send_command(PingCommand(self.clock.time_msec()))
	162
	163	def send_ping(self):
	164	"""Periodically sends a ping and checks if we should close the connection
	165	due to the other side timing out.
	166	"""
	167	now = self.clock.time_msec()
	168
	169	if self.time_we_closed:
	170	if now - self.time_we_closed > PING_TIMEOUT_MS:
	171	logger.info(
	172	"[%s] Failed to close connection gracefully, aborting", self.id()
	173	)
	174	self.transport.abortConnection()
	175	else:
	176	if now - self.last_sent_command >= PING_TIME:
	177	self.send_command(PingCommand(now))
	178
	179	if self.received_ping and now - self.last_received_command > PING_TIMEOUT_MS:
	180	logger.info(
	181	"[%s] Connection hasn't received command in %r ms. Closing.",
	182	self.id(), now - self.last_received_command
	183	)
	184	self.send_error("ping timeout")
	185
	186	def lineReceived(self, line):
	187	"""Called when we've received a line
	188	"""
	189	if line.strip() == "":
	190	# Ignore blank lines
	191	return
	192
	193	line = line.decode("utf-8")
	194	cmd_name, rest_of_line = line.split(" ", 1)
	195
	196	if cmd_name not in self.VALID_INBOUND_COMMANDS:
	197	logger.error("[%s] invalid command %s", self.id(), cmd_name)
	198	self.send_error("invalid command: %s", cmd_name)
	199	return
	200
	201	self.last_received_command = self.clock.time_msec()
	202
	203	self.inbound_commands_counter.inc(cmd_name)
	204
	205	cmd_cls = COMMAND_MAP[cmd_name]
	206	try:
	207	cmd = cmd_cls.from_line(rest_of_line)
	208	except Exception as e:
	209	logger.exception(
	210	"[%s] failed to parse line %r: %r", self.id(), cmd_name, rest_of_line
	211	)
	212	self.send_error(
	213	"failed to parse line for %r: %r (%r):" % (cmd_name, e, rest_of_line)
	214	)
	215	return
	216
	217	# Now lets try and call on_<CMD_NAME> function
	218	try:
	219	getattr(self, "on_%s" % (cmd_name,))(cmd)
	220	except Exception:
	221	logger.exception("[%s] Failed to handle line: %r", self.id(), line)
	222
	223	def close(self):
	224	logger.warn("[%s] Closing connection", self.id())
	225	self.time_we_closed = self.clock.time_msec()
	226	self.transport.loseConnection()
	227	self.on_connection_closed()
	228
	229	def send_error(self, error_string, *args):
	230	"""Send an error to remote and close the connection.
	231	"""
	232	self.send_command(ErrorCommand(error_string % args))
	233	self.close()
	234
	235	def send_command(self, cmd, do_buffer=True):
	236	"""Send a command if connection has been established.
	237
	238	Args:
	239	cmd (Command)
	240	do_buffer (bool): Whether to buffer the message or always attempt
	241	to send the command. This is mostly used to send an error
	242	message if we're about to close the connection due our buffers
	243	becoming full.
	244	"""
	245	if self.state == ConnectionStates.CLOSED:
	246	logger.info("[%s] Not sending, connection closed", self.id())
	247	return
	248
	249	if do_buffer and self.state != ConnectionStates.ESTABLISHED:
	250	self._queue_command(cmd)
	251	return
	252
	253	self.outbound_commands_counter.inc(cmd.NAME)
	254
	255	string = "%s %s" % (cmd.NAME, cmd.to_line(),)
	256	if "\n" in string:
	257	raise Exception("Unexpected newline in command: %r", string)
	258
	259	self.sendLine(string.encode("utf-8"))
	260
	261	self.last_sent_command = self.clock.time_msec()
	262
	263	def _queue_command(self, cmd):
	264	"""Queue the command until the connection is ready to write to again.
	265	"""
	266	logger.info("[%s] Queing as conn %r, cmd: %r", self.id(), self.state, cmd)
	267	self.pending_commands.append(cmd)
	268
	269	if len(self.pending_commands) > self.max_line_buffer:
	270	# The other side is failing to keep up and out buffers are becoming
	271	# full, so lets close the connection.
	272	# XXX: should we squawk more loudly?
	273	logger.error("[%s] Remote failed to keep up", self.id())
	274	self.send_command(ErrorCommand("Failed to keep up"), do_buffer=False)
	275	self.close()
	276
	277	def _send_pending_commands(self):
	278	"""Send any queued commandes
	279	"""
	280	pending = self.pending_commands
	281	self.pending_commands = []
	282	for cmd in pending:
	283	self.send_command(cmd)
	284
	285	def on_PING(self, line):
	286	self.received_ping = True
	287
	288	def on_ERROR(self, cmd):
	289	logger.error("[%s] Remote reported error: %r", self.id(), cmd.data)
	290
	291	def pauseProducing(self):
	292	"""This is called when both the kernel send buffer and the twisted
	293	tcp connection send buffers have become full.
	294
	295	We don't actually have any control over those sizes, so we buffer some
	296	commands ourselves before knifing the connection due to the remote
	297	failing to keep up.
	298	"""
	299	logger.info("[%s] Pause producing", self.id())
	300	self.state = ConnectionStates.PAUSED
	301
	302	def resumeProducing(self):
	303	"""The remote has caught up after we started buffering!
	304	"""
	305	logger.info("[%s] Resume producing", self.id())
	306	self.state = ConnectionStates.ESTABLISHED
	307	self._send_pending_commands()
	308
	309	def stopProducing(self):
	310	"""We're never going to send any more data (normally because either
	311	we or the remote has closed the connection)
	312	"""
	313	logger.info("[%s] Stop producing", self.id())
	314	self.on_connection_closed()
	315
	316	def connectionLost(self, reason):
	317	logger.info("[%s] Replication connection closed: %r", self.id(), reason)
	318	if isinstance(reason, Failure):
	319	connection_close_counter.inc(reason.type.__name__)
	320	else:
	321	connection_close_counter.inc(reason.__class__.__name__)
	322
	323	try:
	324	# Remove us from list of connections to be monitored
	325	connected_connections.remove(self)
	326	except ValueError:
	327	pass
	328
	329	# Stop the looping call sending pings.
	330	if self._send_ping_loop and self._send_ping_loop.running:
	331	self._send_ping_loop.stop()
	332
	333	self.on_connection_closed()
	334
	335	def on_connection_closed(self):
	336	logger.info("[%s] Connection was closed", self.id())
	337
	338	self.state = ConnectionStates.CLOSED
	339	self.pending_commands = []
	340
	341	if self.transport:
	342	self.transport.unregisterProducer()
	343
	344	def __str__(self):
	345	return "ReplicationConnection<name=%s,conn_id=%s,addr=%s>" % (
	346	self.name, self.conn_id, self.addr,
	347	)
	348
	349	def id(self):
	350	return "%s-%s" % (self.name, self.conn_id)
	351
	352
	353	class ServerReplicationStreamProtocol(BaseReplicationStreamProtocol):
	354	VALID_INBOUND_COMMANDS = VALID_CLIENT_COMMANDS
	355	VALID_OUTBOUND_COMMANDS = VALID_SERVER_COMMANDS
	356
	357	def __init__(self, server_name, clock, streamer, addr):
	358	BaseReplicationStreamProtocol.__init__(self, clock) # Old style class
	359
	360	self.server_name = server_name
	361	self.streamer = streamer
	362	self.addr = addr
	363
	364	# The streams the client has subscribed to and is up to date with
	365	self.replication_streams = set()
	366
	367	# The streams the client is currently subscribing to.
	368	self.connecting_streams = set()
	369
	370	# Map from stream name to list of updates to send once we've finished
	371	# subscribing the client to the stream.
	372	self.pending_rdata = {}
	373
	374	def connectionMade(self):
	375	self.send_command(ServerCommand(self.server_name))
	376	BaseReplicationStreamProtocol.connectionMade(self)
	377	self.streamer.new_connection(self)
	378
	379	def on_NAME(self, cmd):
	380	logger.info("[%s] Renamed to %r", self.id(), cmd.data)
	381	self.name = cmd.data
	382
	383	def on_USER_SYNC(self, cmd):
	384	self.streamer.on_user_sync(
	385	self.conn_id, cmd.user_id, cmd.is_syncing, cmd.last_sync_ms,
	386	)
	387
	388	def on_REPLICATE(self, cmd):
	389	stream_name = cmd.stream_name
	390	token = cmd.token
	391
	392	if stream_name == "ALL":
	393	# Subscribe to all streams we're publishing to.
	394	for stream in self.streamer.streams_by_name.iterkeys():
	395	self.subscribe_to_stream(stream, token)
	396	else:
	397	self.subscribe_to_stream(stream_name, token)
	398
	399	def on_FEDERATION_ACK(self, cmd):
	400	self.streamer.federation_ack(cmd.token)
	401
	402	def on_REMOVE_PUSHER(self, cmd):
	403	self.streamer.on_remove_pusher(cmd.app_id, cmd.push_key, cmd.user_id)
	404
	405	def on_INVALIDATE_CACHE(self, cmd):
	406	self.streamer.on_invalidate_cache(cmd.cache_func, cmd.keys)
	407
	408	@defer.inlineCallbacks
	409	def subscribe_to_stream(self, stream_name, token):
	410	"""Subscribe the remote to a streams.
	411
	412	This invloves checking if they've missed anything and sending those
	413	updates down if they have. During that time new updates for the stream
	414	are queued and sent once we've sent down any missed updates.
	415	"""
	416	self.replication_streams.discard(stream_name)
	417	self.connecting_streams.add(stream_name)
	418
	419	try:
	420	# Get missing updates
	421	updates, current_token = yield self.streamer.get_stream_updates(
	422	stream_name, token,
	423	)
	424
	425	# Send all the missing updates
	426	for update in updates:
	427	token, row = update[0], update[1]
	428	self.send_command(RdataCommand(stream_name, token, row))
	429
	430	# We send a POSITION command to ensure that they have an up to
	431	# date token (especially useful if we didn't send any updates
	432	# above)
	433	self.send_command(PositionCommand(stream_name, current_token))
	434
	435	# Now we can send any updates that came in while we were subscribing
	436	pending_rdata = self.pending_rdata.pop(stream_name, [])
	437	for token, update in pending_rdata:
	438	# Only send updates newer than the current token
	439	if token > current_token:
	440	self.send_command(RdataCommand(stream_name, token, update))
	441
	442	# They're now fully subscribed
	443	self.replication_streams.add(stream_name)
	444	except Exception as e:
	445	logger.exception("[%s] Failed to handle REPLICATE command", self.id())
	446	self.send_error("failed to handle replicate: %r", e)
	447	finally:
	448	self.connecting_streams.discard(stream_name)
	449
	450	def stream_update(self, stream_name, token, data):
	451	"""Called when a new update is available to stream to clients.
	452
	453	We need to check if the client is interested in the stream or not
	454	"""
	455	if stream_name in self.replication_streams:
	456	# The client is subscribed to the stream
	457	self.send_command(RdataCommand(stream_name, token, data))
	458	elif stream_name in self.connecting_streams:
	459	# The client is being subscribed to the stream
	460	logger.debug("[%s] Queuing RDATA %r %r", self.id(), stream_name, token)
	461	self.pending_rdata.setdefault(stream_name, []).append((token, data))
	462	else:
	463	# The client isn't subscribed
	464	logger.debug("[%s] Dropping RDATA %r %r", self.id(), stream_name, token)
	465
	466	def send_sync(self, data):
	467	self.send_command(SyncCommand(data))
	468
	469	def on_connection_closed(self):
	470	BaseReplicationStreamProtocol.on_connection_closed(self)
	471	self.streamer.lost_connection(self)
	472
	473
	474	class ClientReplicationStreamProtocol(BaseReplicationStreamProtocol):
	475	VALID_INBOUND_COMMANDS = VALID_SERVER_COMMANDS
	476	VALID_OUTBOUND_COMMANDS = VALID_CLIENT_COMMANDS
	477
	478	def __init__(self, client_name, server_name, clock, handler):
	479	BaseReplicationStreamProtocol.__init__(self, clock)
	480
	481	self.client_name = client_name
	482	self.server_name = server_name
	483	self.handler = handler
	484
	485	# Map of stream to batched updates. See RdataCommand for info on how
	486	# batching works.
	487	self.pending_batches = {}
	488
	489	def connectionMade(self):
	490	self.send_command(NameCommand(self.client_name))
	491	BaseReplicationStreamProtocol.connectionMade(self)
	492
	493	# Once we've connected subscribe to the necessary streams
	494	for stream_name, token in self.handler.get_streams_to_replicate().iteritems():
	495	self.replicate(stream_name, token)
	496
	497	# Tell the server if we have any users currently syncing (should only
	498	# happen on synchrotrons)
	499	currently_syncing = self.handler.get_currently_syncing_users()
	500	now = self.clock.time_msec()
	501	for user_id in currently_syncing:
	502	self.send_command(UserSyncCommand(user_id, True, now))
	503
	504	# We've now finished connecting to so inform the client handler
	505	self.handler.update_connection(self)
	506
	507	def on_SERVER(self, cmd):
	508	if cmd.data != self.server_name:
	509	logger.error("[%s] Connected to wrong remote: %r", self.id(), cmd.data)
	510	self.send_error("Wrong remote")
	511
	512	def on_RDATA(self, cmd):
	513	try:
	514	row = STREAMS_MAP[cmd.stream_name].ROW_TYPE(*cmd.row)
	515	except Exception:
	516	logger.exception(
	517	"[%s] Failed to parse RDATA: %r %r",
	518	self.id(), cmd.stream_name, cmd.row
	519	)
	520	raise
	521
	522	if cmd.token is None:
	523	# I.e. this is part of a batch of updates for this stream. Batch
	524	# until we get an update for the stream with a non None token
	525	self.pending_batches.setdefault(cmd.stream_name, []).append(row)
	526	else:
	527	# Check if this is the last of a batch of updates
	528	rows = self.pending_batches.pop(cmd.stream_name, [])
	529	rows.append(row)
	530
	531	self.handler.on_rdata(cmd.stream_name, cmd.token, rows)
	532
	533	def on_POSITION(self, cmd):
	534	self.handler.on_position(cmd.stream_name, cmd.token)
	535
	536	def on_SYNC(self, cmd):
	537	self.handler.on_sync(cmd.data)
	538
	539	def replicate(self, stream_name, token):
	540	"""Send the subscription request to the server
	541	"""
	542	if stream_name not in STREAMS_MAP:
	543	raise Exception("Invalid stream name %r" % (stream_name,))
	544
	545	logger.info(
	546	"[%s] Subscribing to replication stream: %r from %r",
	547	self.id(), stream_name, token
	548	)
	549
	550	self.send_command(ReplicateCommand(stream_name, token))
	551
	552	def on_connection_closed(self):
	553	BaseReplicationStreamProtocol.on_connection_closed(self)
	554	self.handler.update_connection(None)
	555
	556
	557	# The following simply registers metrics for the replication connections
	558
	559	metrics.register_callback(
	560	"pending_commands",
	561	lambda: {
	562	(p.name, p.conn_id): len(p.pending_commands)
	563	for p in connected_connections
	564	},
	565	labels=["name", "conn_id"],
	566	)
	567
	568
	569	def transport_buffer_size(protocol):
	570	if protocol.transport:
	571	size = len(protocol.transport.dataBuffer) + protocol.transport._tempDataLen
	572	return size
	573	return 0
	574
	575
	576	metrics.register_callback(
	577	"transport_send_buffer",
	578	lambda: {
	579	(p.name, p.conn_id): transport_buffer_size(p)
	580	for p in connected_connections
	581	},
	582	labels=["name", "conn_id"],
	583	)
	584
	585
	586	def transport_kernel_read_buffer_size(protocol, read=True):
	587	SIOCINQ = 0x541B
	588	SIOCOUTQ = 0x5411
	589
	590	if protocol.transport:
	591	fileno = protocol.transport.getHandle().fileno()
	592	if read:
	593	op = SIOCINQ
	594	else:
	595	op = SIOCOUTQ
	596	size = struct.unpack("I", fcntl.ioctl(fileno, op, '\0\0\0\0'))[0]
	597	return size
	598	return 0
	599
	600
	601	metrics.register_callback(
	602	"transport_kernel_send_buffer",
	603	lambda: {
	604	(p.name, p.conn_id): transport_kernel_read_buffer_size(p, False)
	605	for p in connected_connections
	606	},
	607	labels=["name", "conn_id"],
	608	)
	609
	610
	611	metrics.register_callback(
	612	"transport_kernel_read_buffer",
	613	lambda: {
	614	(p.name, p.conn_id): transport_kernel_read_buffer_size(p, True)
	615	for p in connected_connections
	616	},
	617	labels=["name", "conn_id"],
	618	)
	619
	620
	621	metrics.register_callback(
	622	"inbound_commands",
	623	lambda: {
	624	(k[0], p.name, p.conn_id): count
	625	for p in connected_connections
	626	for k, count in p.inbound_commands_counter.counts.iteritems()
	627	},
	628	labels=["command", "name", "conn_id"],
	629	)
	630
	631	metrics.register_callback(
	632	"outbound_commands",
	633	lambda: {
	634	(k[0], p.name, p.conn_id): count
	635	for p in connected_connections
	636	for k, count in p.outbound_commands_counter.counts.iteritems()
	637	},
	638	labels=["command", "name", "conn_id"],
	639	)

+290

-0

synapse/replication/tcp/resource.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2017 Vector Creations Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	"""The server side of the replication stream.
	15	"""
	16
	17	from twisted.internet import defer, reactor
	18	from twisted.internet.protocol import Factory
	19
	20	from streams import STREAMS_MAP, FederationStream
	21	from protocol import ServerReplicationStreamProtocol
	22
	23	from synapse.util.metrics import Measure, measure_func
	24
	25	import logging
	26	import synapse.metrics
	27
	28
	29	metrics = synapse.metrics.get_metrics_for(__name__)
	30	stream_updates_counter = metrics.register_counter(
	31	"stream_updates", labels=["stream_name"]
	32	)
	33	user_sync_counter = metrics.register_counter("user_sync")
	34	federation_ack_counter = metrics.register_counter("federation_ack")
	35	remove_pusher_counter = metrics.register_counter("remove_pusher")
	36	invalidate_cache_counter = metrics.register_counter("invalidate_cache")
	37
	38	logger = logging.getLogger(__name__)
	39
	40
	41	class ReplicationStreamProtocolFactory(Factory):
	42	"""Factory for new replication connections.
	43	"""
	44	def __init__(self, hs):
	45	self.streamer = ReplicationStreamer(hs)
	46	self.clock = hs.get_clock()
	47	self.server_name = hs.config.server_name
	48
	49	def buildProtocol(self, addr):
	50	return ServerReplicationStreamProtocol(
	51	self.server_name,
	52	self.clock,
	53	self.streamer,
	54	addr
	55	)
	56
	57
	58	class ReplicationStreamer(object):
	59	"""Handles replication connections.
	60
	61	This needs to be poked when new replication data may be available. When new
	62	data is available it will propagate to all connected clients.
	63	"""
	64
	65	def __init__(self, hs):
	66	self.store = hs.get_datastore()
	67	self.presence_handler = hs.get_presence_handler()
	68	self.clock = hs.get_clock()
	69
	70	# Current connections.
	71	self.connections = []
	72
	73	metrics.register_callback("total_connections", lambda: len(self.connections))
	74
	75	# List of streams that clients can subscribe to.
	76	# We only support federation stream if federation sending hase been
	77	# disabled on the master.
	78	self.streams = [
	79	stream(hs) for stream in STREAMS_MAP.itervalues()
	80	if stream != FederationStream or not hs.config.send_federation
	81	]
	82
	83	self.streams_by_name = {stream.NAME: stream for stream in self.streams}
	84
	85	metrics.register_callback(
	86	"connections_per_stream",
	87	lambda: {
	88	(stream_name,): len([
	89	conn for conn in self.connections
	90	if stream_name in conn.replication_streams
	91	])
	92	for stream_name in self.streams_by_name
	93	},
	94	labels=["stream_name"],
	95	)
	96
	97	self.federation_sender = None
	98	if not hs.config.send_federation:
	99	self.federation_sender = hs.get_federation_sender()
	100
	101	hs.get_notifier().add_replication_callback(self.on_notifier_poke)
	102
	103	# Keeps track of whether we are currently checking for updates
	104	self.is_looping = False
	105	self.pending_updates = False
	106
	107	reactor.addSystemEventTrigger("before", "shutdown", self.on_shutdown)
	108
	109	def on_shutdown(self):
	110	# close all connections on shutdown
	111	for conn in self.connections:
	112	conn.send_error("server shutting down")
	113
	114	@defer.inlineCallbacks
	115	def on_notifier_poke(self):
	116	"""Checks if there is actually any new data and sends it to the
	117	connections if there are.
	118
	119	This should get called each time new data is available, even if it
	120	is currently being executed, so that nothing gets missed
	121	"""
	122	if not self.connections:
	123	# Don't bother if nothing is listening. We still need to advance
	124	# the stream tokens otherwise they'll fall beihind forever
	125	for stream in self.streams:
	126	stream.discard_updates_and_advance()
	127	return
	128
	129	# If we're in the process of checking for new updates, mark that fact
	130	# and return
	131	if self.is_looping:
	132	logger.debug("Noitifier poke loop already running")
	133	self.pending_updates = True
	134	return
	135
	136	self.pending_updates = True
	137	self.is_looping = True
	138
	139	try:
	140	# Keep looping while there have been pokes about potential updates.
	141	# This protects against the race where a stream we already checked
	142	# gets an update while we're handling other streams.
	143	while self.pending_updates:
	144	self.pending_updates = False
	145
	146	with Measure(self.clock, "repl.stream.get_updates"):
	147	# First we tell the streams that they should update their
	148	# current tokens.
	149	for stream in self.streams:
	150	stream.advance_current_token()
	151
	152	for stream in self.streams:
	153	if stream.last_token == stream.upto_token:
	154	continue
	155
	156	logger.debug(
	157	"Getting stream: %s: %s -> %s",
	158	stream.NAME, stream.last_token, stream.upto_token
	159	)
	160	updates, current_token = yield stream.get_updates()
	161
	162	logger.debug(
	163	"Sending %d updates to %d connections",
	164	len(updates), len(self.connections),
	165	)
	166
	167	if updates:
	168	logger.info(
	169	"Streaming: %s -> %s", stream.NAME, updates[-1][0]
	170	)
	171	stream_updates_counter.inc_by(len(updates), stream.NAME)
	172
	173	# Some streams return multiple rows with the same stream IDs,
	174	# we need to make sure they get sent out in batches. We do
	175	# this by setting the current token to all but the last of
	176	# a series of updates with the same token to have a None
	177	# token. See RdataCommand for more details.
	178	batched_updates = _batch_updates(updates)
	179
	180	for conn in self.connections:
	181	for token, row in batched_updates:
	182	try:
	183	conn.stream_update(stream.NAME, token, row)
	184	except Exception:
	185	logger.exception("Failed to replicate")
	186
	187	logger.debug("No more pending updates, breaking poke loop")
	188	finally:
	189	self.pending_updates = False
	190	self.is_looping = False
	191
	192	@measure_func("repl.get_stream_updates")
	193	def get_stream_updates(self, stream_name, token):
	194	"""For a given stream get all updates since token. This is called when
	195	a client first subscribes to a stream.
	196	"""
	197	stream = self.streams_by_name.get(stream_name, None)
	198	if not stream:
	199	raise Exception("unknown stream %s", stream_name)
	200
	201	return stream.get_updates_since(token)
	202
	203	@measure_func("repl.federation_ack")
	204	def federation_ack(self, token):
	205	"""We've received an ack for federation stream from a client.
	206	"""
	207	federation_ack_counter.inc()
	208	if self.federation_sender:
	209	self.federation_sender.federation_ack(token)
	210
	211	@measure_func("repl.on_user_sync")
	212	def on_user_sync(self, conn_id, user_id, is_syncing, last_sync_ms):
	213	"""A client has started/stopped syncing on a worker.
	214	"""
	215	user_sync_counter.inc()
	216	self.presence_handler.update_external_syncs_row(
	217	conn_id, user_id, is_syncing, last_sync_ms,
	218	)
	219
	220	@measure_func("repl.on_remove_pusher")
	221	@defer.inlineCallbacks
	222	def on_remove_pusher(self, app_id, push_key, user_id):
	223	"""A client has asked us to remove a pusher
	224	"""
	225	remove_pusher_counter.inc()
	226	yield self.store.delete_pusher_by_app_id_pushkey_user_id(
	227	app_id=app_id, pushkey=push_key, user_id=user_id
	228	)
	229
	230	self.notifier.on_new_replication_data()
	231
	232	@measure_func("repl.on_invalidate_cache")
	233	def on_invalidate_cache(self, cache_func, keys):
	234	"""The client has asked us to invalidate a cache
	235	"""
	236	invalidate_cache_counter.inc()
	237	getattr(self.store, cache_func).invalidate(tuple(keys))
	238
	239	def send_sync_to_all_connections(self, data):
	240	"""Sends a SYNC command to all clients.
	241
	242	Used in tests.
	243	"""
	244	for conn in self.connections:
	245	conn.send_sync(data)
	246
	247	def new_connection(self, connection):
	248	"""A new client connection has been established
	249	"""
	250	self.connections.append(connection)
	251
	252	def lost_connection(self, connection):
	253	"""A client connection has been lost
	254	"""
	255	try:
	256	self.connections.remove(connection)
	257	except ValueError:
	258	pass
	259
	260	# We need to tell the presence handler that the connection has been
	261	# lost so that it can handle any ongoing syncs on that connection.
	262	self.presence_handler.update_external_syncs_clear(connection.conn_id)
	263
	264
	265	def _batch_updates(updates):
	266	"""Takes a list of updates of form [(token, row)] and sets the token to
	267	None for all rows where the next row has the same token. This is used to
	268	implement batching.
	269
	270	For example:
	271
	272	[(1, _), (1, _), (2, _), (3, _), (3, _)]
	273
	274	becomes:
	275
	276	[(None, _), (1, _), (2, _), (None, _), (3, _)]
	277	"""
	278	if not updates:
	279	return []
	280
	281	new_updates = []
	282	for i, update in enumerate(updates[:-1]):
	283	if update[0] == updates[i + 1][0]:
	284	new_updates.append((None, update[1]))
	285	else:
	286	new_updates.append(update)
	287
	288	new_updates.append(updates[-1])
	289	return new_updates

+464

-0

synapse/replication/tcp/streams.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2017 Vector Creations Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	"""Defines all the valid streams that clients can subscribe to, and the format
	16	of the rows returned by each stream.
	17
	18	Each stream is defined by the following information:
	19
	20	stream name: The name of the stream
	21	row type: The type that is used to serialise/deserialse the row
	22	current_token: The function that returns the current token for the stream
	23	update_function: The function that returns a list of updates between two tokens
	24	"""
	25
	26	from twisted.internet import defer
	27	from collections import namedtuple
	28
	29	import logging
	30
	31
	32	logger = logging.getLogger(__name__)
	33
	34
	35	MAX_EVENTS_BEHIND = 10000
	36
	37
	38	EventStreamRow = namedtuple("EventStreamRow", (
	39	"event_id", # str
	40	"room_id", # str
	41	"type", # str
	42	"state_key", # str, optional
	43	"redacts", # str, optional
	44	))
	45	BackfillStreamRow = namedtuple("BackfillStreamRow", (
	46	"event_id", # str
	47	"room_id", # str
	48	"type", # str
	49	"state_key", # str, optional
	50	"redacts", # str, optional
	51	))
	52	PresenceStreamRow = namedtuple("PresenceStreamRow", (
	53	"user_id", # str
	54	"state", # str
	55	"last_active_ts", # int
	56	"last_federation_update_ts", # int
	57	"last_user_sync_ts", # int
	58	"status_msg", # str
	59	"currently_active", # bool
	60	))
	61	TypingStreamRow = namedtuple("TypingStreamRow", (
	62	"room_id", # str
	63	"user_ids", # list(str)
	64	))
	65	ReceiptsStreamRow = namedtuple("ReceiptsStreamRow", (
	66	"room_id", # str
	67	"receipt_type", # str
	68	"user_id", # str
	69	"event_id", # str
	70	"data", # dict
	71	))
	72	PushRulesStreamRow = namedtuple("PushRulesStreamRow", (
	73	"user_id", # str
	74	))
	75	PushersStreamRow = namedtuple("PushersStreamRow", (
	76	"user_id", # str
	77	"app_id", # str
	78	"pushkey", # str
	79	"deleted", # bool
	80	))
	81	CachesStreamRow = namedtuple("CachesStreamRow", (
	82	"cache_func", # str
	83	"keys", # list(str)
	84	"invalidation_ts", # int
	85	))
	86	PublicRoomsStreamRow = namedtuple("PublicRoomsStreamRow", (
	87	"room_id", # str
	88	"visibility", # str
	89	"appservice_id", # str, optional
	90	"network_id", # str, optional
	91	))
	92	DeviceListsStreamRow = namedtuple("DeviceListsStreamRow", (
	93	"user_id", # str
	94	"destination", # str
	95	))
	96	ToDeviceStreamRow = namedtuple("ToDeviceStreamRow", (
	97	"entity", # str
	98	))
	99	FederationStreamRow = namedtuple("FederationStreamRow", (
	100	"type", # str, the type of data as defined in the BaseFederationRows
	101	"data", # dict, serialization of a federation.send_queue.BaseFederationRow
	102	))
	103	TagAccountDataStreamRow = namedtuple("TagAccountDataStreamRow", (
	104	"user_id", # str
	105	"room_id", # str
	106	"data", # dict
	107	))
	108	AccountDataStreamRow = namedtuple("AccountDataStream", (
	109	"user_id", # str
	110	"room_id", # str
	111	"data_type", # str
	112	"data", # dict
	113	))
	114
	115
	116	class Stream(object):
	117	"""Base class for the streams.
	118
	119	Provides a `get_updates()` function that returns new updates since the last
	120	time it was called up until the point `advance_current_token` was called.
	121	"""
	122	NAME = None # The name of the stream
	123	ROW_TYPE = None # The type of the row
	124	_LIMITED = True # Whether the update function takes a limit
	125
	126	def __init__(self, hs):
	127	# The token from which we last asked for updates
	128	self.last_token = self.current_token()
	129
	130	# The token that we will get updates up to
	131	self.upto_token = self.current_token()
	132
	133	def advance_current_token(self):
	134	"""Updates `upto_token` to "now", which updates up until which point
	135	get_updates[_since] will fetch rows till.
	136	"""
	137	self.upto_token = self.current_token()
	138
	139	def discard_updates_and_advance(self):
	140	"""Called when the stream should advance but the updates would be discarded,
	141	e.g. when there are no currently connected workers.
	142	"""
	143	self.upto_token = self.current_token()
	144	self.last_token = self.upto_token
	145
	146	@defer.inlineCallbacks
	147	def get_updates(self):
	148	"""Gets all updates since the last time this function was called (or
	149	since the stream was constructed if it hadn't been called before),
	150	until the `upto_token`
	151
	152	Returns:
	153	(list(ROW_TYPE), int): list of updates plus the token used as an
	154	upper bound of the updates (i.e. the "current token")
	155	"""
	156	updates, current_token = yield self.get_updates_since(self.last_token)
	157	self.last_token = current_token
	158
	159	defer.returnValue((updates, current_token))
	160
	161	@defer.inlineCallbacks
	162	def get_updates_since(self, from_token):
	163	"""Like get_updates except allows specifying from when we should
	164	stream updates
	165
	166	Returns:
	167	(list(ROW_TYPE), int): list of updates plus the token used as an
	168	upper bound of the updates (i.e. the "current token")
	169	"""
	170	if from_token in ("NOW", "now"):
	171	defer.returnValue(([], self.upto_token))
	172
	173	current_token = self.upto_token
	174
	175	from_token = int(from_token)
	176
	177	if from_token == current_token:
	178	defer.returnValue(([], current_token))
	179
	180	if self._LIMITED:
	181	rows = yield self.update_function(
	182	from_token, current_token,
	183	limit=MAX_EVENTS_BEHIND + 1,
	184	)
	185
	186	if len(rows) >= MAX_EVENTS_BEHIND:
	187	raise Exception("stream %s has fallen behined" % (self.NAME))
	188	else:
	189	rows = yield self.update_function(
	190	from_token, current_token,
	191	)
	192
	193	updates = [(row[0], self.ROW_TYPE(*row[1:])) for row in rows]
	194
	195	defer.returnValue((updates, current_token))
	196
	197	def current_token(self):
	198	"""Gets the current token of the underlying streams. Should be provided
	199	by the sub classes
	200
	201	Returns:
	202	int
	203	"""
	204	raise NotImplementedError()
	205
	206	def update_function(self, from_token, current_token, limit=None):
	207	"""Get updates between from_token and to_token. If Stream._LIMITED is
	208	True then limit is provided, otherwise it's not.
	209
	210	Returns:
	211	Deferred(list(tuple)): the first entry in the tuple is the token for
	212	that update, and the rest of the tuple gets used to construct
	213	a ``ROW_TYPE`` instance
	214	"""
	215	raise NotImplementedError()
	216
	217
	218	class EventsStream(Stream):
	219	"""We received a new event, or an event went from being an outlier to not
	220	"""
	221	NAME = "events"
	222	ROW_TYPE = EventStreamRow
	223
	224	def __init__(self, hs):
	225	store = hs.get_datastore()
	226	self.current_token = store.get_current_events_token
	227	self.update_function = store.get_all_new_forward_event_rows
	228
	229	super(EventsStream, self).__init__(hs)
	230
	231
	232	class BackfillStream(Stream):
	233	"""We fetched some old events and either we had never seen that event before
	234	or it went from being an outlier to not.
	235	"""
	236	NAME = "backfill"
	237	ROW_TYPE = BackfillStreamRow
	238
	239	def __init__(self, hs):
	240	store = hs.get_datastore()
	241	self.current_token = store.get_current_backfill_token
	242	self.update_function = store.get_all_new_backfill_event_rows
	243
	244	super(BackfillStream, self).__init__(hs)
	245
	246
	247	class PresenceStream(Stream):
	248	NAME = "presence"
	249	_LIMITED = False
	250	ROW_TYPE = PresenceStreamRow
	251
	252	def __init__(self, hs):
	253	store = hs.get_datastore()
	254	presence_handler = hs.get_presence_handler()
	255
	256	self.current_token = store.get_current_presence_token
	257	self.update_function = presence_handler.get_all_presence_updates
	258
	259	super(PresenceStream, self).__init__(hs)
	260
	261
	262	class TypingStream(Stream):
	263	NAME = "typing"
	264	_LIMITED = False
	265	ROW_TYPE = TypingStreamRow
	266
	267	def __init__(self, hs):
	268	typing_handler = hs.get_typing_handler()
	269
	270	self.current_token = typing_handler.get_current_token
	271	self.update_function = typing_handler.get_all_typing_updates
	272
	273	super(TypingStream, self).__init__(hs)
	274
	275
	276	class ReceiptsStream(Stream):
	277	NAME = "receipts"
	278	ROW_TYPE = ReceiptsStreamRow
	279
	280	def __init__(self, hs):
	281	store = hs.get_datastore()
	282
	283	self.current_token = store.get_max_receipt_stream_id
	284	self.update_function = store.get_all_updated_receipts
	285
	286	super(ReceiptsStream, self).__init__(hs)
	287
	288
	289	class PushRulesStream(Stream):
	290	"""A user has changed their push rules
	291	"""
	292	NAME = "push_rules"
	293	ROW_TYPE = PushRulesStreamRow
	294
	295	def __init__(self, hs):
	296	self.store = hs.get_datastore()
	297	super(PushRulesStream, self).__init__(hs)
	298
	299	def current_token(self):
	300	push_rules_token, _ = self.store.get_push_rules_stream_token()
	301	return push_rules_token
	302
	303	@defer.inlineCallbacks
	304	def update_function(self, from_token, to_token, limit):
	305	rows = yield self.store.get_all_push_rule_updates(from_token, to_token, limit)
	306	defer.returnValue([(row[0], row[2]) for row in rows])
	307
	308
	309	class PushersStream(Stream):
	310	"""A user has added/changed/removed a pusher
	311	"""
	312	NAME = "pushers"
	313	ROW_TYPE = PushersStreamRow
	314
	315	def __init__(self, hs):
	316	store = hs.get_datastore()
	317
	318	self.current_token = store.get_pushers_stream_token
	319	self.update_function = store.get_all_updated_pushers_rows
	320
	321	super(PushersStream, self).__init__(hs)
	322
	323
	324	class CachesStream(Stream):
	325	"""A cache was invalidated on the master and no other stream would invalidate
	326	the cache on the workers
	327	"""
	328	NAME = "caches"
	329	ROW_TYPE = CachesStreamRow
	330
	331	def __init__(self, hs):
	332	store = hs.get_datastore()
	333
	334	self.current_token = store.get_cache_stream_token
	335	self.update_function = store.get_all_updated_caches
	336
	337	super(CachesStream, self).__init__(hs)
	338
	339
	340	class PublicRoomsStream(Stream):
	341	"""The public rooms list changed
	342	"""
	343	NAME = "public_rooms"
	344	ROW_TYPE = PublicRoomsStreamRow
	345
	346	def __init__(self, hs):
	347	store = hs.get_datastore()
	348
	349	self.current_token = store.get_current_public_room_stream_id
	350	self.update_function = store.get_all_new_public_rooms
	351
	352	super(PublicRoomsStream, self).__init__(hs)
	353
	354
	355	class DeviceListsStream(Stream):
	356	"""Someone added/changed/removed a device
	357	"""
	358	NAME = "device_lists"
	359	_LIMITED = False
	360	ROW_TYPE = DeviceListsStreamRow
	361
	362	def __init__(self, hs):
	363	store = hs.get_datastore()
	364
	365	self.current_token = store.get_device_stream_token
	366	self.update_function = store.get_all_device_list_changes_for_remotes
	367
	368	super(DeviceListsStream, self).__init__(hs)
	369
	370
	371	class ToDeviceStream(Stream):
	372	"""New to_device messages for a client
	373	"""
	374	NAME = "to_device"
	375	ROW_TYPE = ToDeviceStreamRow
	376
	377	def __init__(self, hs):
	378	store = hs.get_datastore()
	379
	380	self.current_token = store.get_to_device_stream_token
	381	self.update_function = store.get_all_new_device_messages
	382
	383	super(ToDeviceStream, self).__init__(hs)
	384
	385
	386	class FederationStream(Stream):
	387	"""Data to be sent over federation. Only available when master has federation
	388	sending disabled.
	389	"""
	390	NAME = "federation"
	391	ROW_TYPE = FederationStreamRow
	392
	393	def __init__(self, hs):
	394	federation_sender = hs.get_federation_sender()
	395
	396	self.current_token = federation_sender.get_current_token
	397	self.update_function = federation_sender.get_replication_rows
	398
	399	super(FederationStream, self).__init__(hs)
	400
	401
	402	class TagAccountDataStream(Stream):
	403	"""Someone added/removed a tag for a room
	404	"""
	405	NAME = "tag_account_data"
	406	ROW_TYPE = TagAccountDataStreamRow
	407
	408	def __init__(self, hs):
	409	store = hs.get_datastore()
	410
	411	self.current_token = store.get_max_account_data_stream_id
	412	self.update_function = store.get_all_updated_tags
	413
	414	super(TagAccountDataStream, self).__init__(hs)
	415
	416
	417	class AccountDataStream(Stream):
	418	"""Global or per room account data was changed
	419	"""
	420	NAME = "account_data"
	421	ROW_TYPE = AccountDataStreamRow
	422
	423	def __init__(self, hs):
	424	self.store = hs.get_datastore()
	425
	426	self.current_token = self.store.get_max_account_data_stream_id
	427
	428	super(AccountDataStream, self).__init__(hs)
	429
	430	@defer.inlineCallbacks
	431	def update_function(self, from_token, to_token, limit):
	432	global_results, room_results = yield self.store.get_all_updated_account_data(
	433	from_token, from_token, to_token, limit
	434	)
	435
	436	results = list(room_results)
	437	results.extend(
	438	(stream_id, user_id, None, account_data_type, content,)
	439	for stream_id, user_id, account_data_type, content in global_results
	440	)
	441
	442	defer.returnValue(results)
	443
	444
	445	STREAMS_MAP = {
	446	stream.NAME: stream
	447	for stream in (
	448	EventsStream,
	449	BackfillStream,
	450	PresenceStream,
	451	TypingStream,
	452	ReceiptsStream,
	453	PushRulesStream,
	454	PushersStream,
	455	CachesStream,
	456	PublicRoomsStream,
	457	DeviceListsStream,
	458	ToDeviceStream,
	459	FederationStream,
	460	TagAccountDataStream,
	461	AccountDataStream,
	462	)
	463	}

+2

-0

synapse/rest/__init__.py less more

39	39	register,
40	40	auth,
41	41	receipts,
	42	read_marker,
42	43	keys,
43	44	tokenrefresh,
44	45	tags,

87	88	register.register_servlets(hs, client_resource)
88	89	auth.register_servlets(hs, client_resource)
89	90	receipts.register_servlets(hs, client_resource)
	91	read_marker.register_servlets(hs, client_resource)
90	92	keys.register_servlets(hs, client_resource)
91	93	tokenrefresh.register_servlets(hs, client_resource)
92	94	tags.register_servlets(hs, client_resource)

+5

-1

synapse/rest/client/v1/directory.py less more

38	38
39	39	def __init__(self, hs):
40	40	super(ClientDirectoryServer, self).__init__(hs)
	41	self.store = hs.get_datastore()
41	42	self.handlers = hs.get_handlers()
42	43
43	44	@defer.inlineCallbacks

69	70	logger.debug("Got servers: %s", servers)
70	71
71	72	# TODO(erikj): Check types.
72		# TODO(erikj): Check that room exists
	73
	74	room = yield self.store.get_room(room_id)
	75	if room is None:
	76	raise SynapseError(400, "Room does not exist")
73	77
74	78	dir_handler = self.handlers.directory_handler
75	79

+8

-1

synapse/rest/client/v1/room.py less more

163	163	else:
164	164	msg_handler = self.handlers.message_handler
165	165	event, context = yield msg_handler.create_event(
	166	requester,
166	167	event_dict,
167	168	token_id=requester.access_token_id,
168	169	txn_id=txn_id,

405	406	users_with_profile = yield self.state.get_current_user_in_room(room_id)
406	407
407	408	defer.returnValue((200, {
408		"joined": users_with_profile
	409	"joined": {
	410	user_id: {
	411	"avatar_url": profile.avatar_url,
	412	"display_name": profile.display_name,
	413	}
	414	for user_id, profile in users_with_profile.iteritems()
	415	}
409	416	}))
410	417
411	418

+4

-1

synapse/rest/client/v1/voip.py less more

27	27
28	28	@defer.inlineCallbacks
29	29	def on_GET(self, request):
30		requester = yield self.auth.get_user_by_req(request)
	30	requester = yield self.auth.get_user_by_req(
	31	request,
	32	self.hs.config.turn_allow_guests
	33	)
31	34
32	35	turnUris = self.hs.config.turn_uris
33	36	turnSecret = self.hs.config.turn_shared_secret

+10

-0

synapse/rest/client/v2_alpha/_base.py less more

46	46	new_prefix = CLIENT_V2_ALPHA_PREFIX.replace("/v2_alpha", "/r%d" % release)
47	47	patterns.append(re.compile("^" + new_prefix + path_regex))
48	48	return patterns
	49
	50
	51	def set_timeline_upper_limit(filter_json, filter_timeline_limit):
	52	if filter_timeline_limit < 0:
	53	return # no upper limits
	54	timeline = filter_json.get('room', {}).get('timeline', {})
	55	if 'limit' in timeline:
	56	filter_json['room']['timeline']["limit"] = min(
	57	filter_json['room']['timeline']['limit'],
	58	filter_timeline_limit)

+8

-1

synapse/rest/client/v2_alpha/account_data.py less more

15	15	from ._base import client_v2_patterns
16	16
17	17	from synapse.http.servlet import RestServlet, parse_json_object_from_request
18		from synapse.api.errors import AuthError
	18	from synapse.api.errors import AuthError, SynapseError
19	19
20	20	from twisted.internet import defer
21	21

81	81
82	82	body = parse_json_object_from_request(request)
83	83
	84	if account_data_type == "m.fully_read":
	85	raise SynapseError(
	86	405,
	87	"Cannot set m.fully_read through this API."
	88	" Use /rooms/!roomId:server.name/read_markers"
	89	)
	90
84	91	max_id = yield self.store.add_account_data_to_room(
85	92	user_id, room_id, account_data_type, body
86	93	)

+6

-0

synapse/rest/client/v2_alpha/filter.py less more

19	19	from synapse.types import UserID
20	20
21	21	from ._base import client_v2_patterns
	22	from ._base import set_timeline_upper_limit
22	23
23	24	import logging
24	25

84	85	raise AuthError(403, "Can only create filters for local users")
85	86
86	87	content = parse_json_object_from_request(request)
	88	set_timeline_upper_limit(
	89	content,
	90	self.hs.config.filter_timeline_limit
	91	)
	92
87	93	filter_id = yield self.filtering.add_user_filter(
88	94	user_localpart=target_user.localpart,
89	95	user_filter=content,

+66

-0

synapse/rest/client/v2_alpha/read_marker.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2017 Vector Creations Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from twisted.internet import defer
	16
	17	from synapse.http.servlet import RestServlet, parse_json_object_from_request
	18	from ._base import client_v2_patterns
	19
	20	import logging
	21
	22
	23	logger = logging.getLogger(__name__)
	24
	25
	26	class ReadMarkerRestServlet(RestServlet):
	27	PATTERNS = client_v2_patterns("/rooms/(?P<room_id>[^/]*)/read_markers$")
	28
	29	def __init__(self, hs):
	30	super(ReadMarkerRestServlet, self).__init__()
	31	self.auth = hs.get_auth()
	32	self.receipts_handler = hs.get_receipts_handler()
	33	self.read_marker_handler = hs.get_read_marker_handler()
	34	self.presence_handler = hs.get_presence_handler()
	35
	36	@defer.inlineCallbacks
	37	def on_POST(self, request, room_id):
	38	requester = yield self.auth.get_user_by_req(request)
	39
	40	yield self.presence_handler.bump_presence_active_time(requester.user)
	41
	42	body = parse_json_object_from_request(request)
	43
	44	read_event_id = body.get("m.read", None)
	45	if read_event_id:
	46	yield self.receipts_handler.received_client_receipt(
	47	room_id,
	48	"m.read",
	49	user_id=requester.user.to_string(),
	50	event_id=read_event_id
	51	)
	52
	53	read_marker_event_id = body.get("m.fully_read", None)
	54	if read_marker_event_id:
	55	yield self.read_marker_handler.received_client_read_marker(
	56	room_id,
	57	user_id=requester.user.to_string(),
	58	event_id=read_marker_event_id
	59	)
	60
	61	defer.returnValue((200, {}))
	62
	63
	64	def register_servlets(hs, http_server):
	65	ReadMarkerRestServlet(hs).register(http_server)

+41

-1

synapse/rest/client/v2_alpha/register.py less more

20	20	from synapse.api.constants import LoginType
21	21	from synapse.api.errors import SynapseError, Codes, UnrecognizedRequestError
22	22	from synapse.http.servlet import (
23		RestServlet, parse_json_object_from_request, assert_params_in_request
	23	RestServlet, parse_json_object_from_request, assert_params_in_request, parse_string
24	24	)
25	25	from synapse.util.msisdn import phone_number_to_msisdn
26	26

30	30	import hmac
31	31	from hashlib import sha1
32	32	from synapse.util.async import run_on_reactor
	33	from synapse.util.ratelimitutils import FederationRateLimiter
33	34
34	35
35	36	# We ought to be using hmac.compare_digest() but on older pythons it doesn't

112	113
113	114	ret = yield self.identity_handler.requestMsisdnToken(**body)
114	115	defer.returnValue((200, ret))
	116
	117
	118	class UsernameAvailabilityRestServlet(RestServlet):
	119	PATTERNS = client_v2_patterns("/register/available")
	120
	121	def __init__(self, hs):
	122	"""
	123	Args:
	124	hs (synapse.server.HomeServer): server
	125	"""
	126	super(UsernameAvailabilityRestServlet, self).__init__()
	127	self.hs = hs
	128	self.registration_handler = hs.get_handlers().registration_handler
	129	self.ratelimiter = FederationRateLimiter(
	130	hs.get_clock(),
	131	# Time window of 2s
	132	window_size=2000,
	133	# Artificially delay requests if rate > sleep_limit/window_size
	134	sleep_limit=1,
	135	# Amount of artificial delay to apply
	136	sleep_msec=1000,
	137	# Error with 429 if more than reject_limit requests are queued
	138	reject_limit=1,
	139	# Allow 1 request at a time
	140	concurrent_requests=1,
	141	)
	142
	143	@defer.inlineCallbacks
	144	def on_GET(self, request):
	145	ip = self.hs.get_ip_from_request(request)
	146	with self.ratelimiter.ratelimit(ip) as wait_deferred:
	147	yield wait_deferred
	148
	149	username = parse_string(request, "username", required=True)
	150
	151	yield self.registration_handler.check_username(username)
	152
	153	defer.returnValue((200, {"available": True}))
115	154
116	155
117	156	class RegisterRestServlet(RestServlet):

554	593	def register_servlets(hs, http_server):
555	594	EmailRegisterRequestTokenRestServlet(hs).register(http_server)
556	595	MsisdnRegisterRequestTokenRestServlet(hs).register(http_server)
	596	UsernameAvailabilityRestServlet(hs).register(http_server)
557	597	RegisterRestServlet(hs).register(http_server)

+5

-0

synapse/rest/client/v2_alpha/sync.py less more

27	27	from synapse.api.errors import SynapseError
28	28	from synapse.api.constants import PresenceState
29	29	from ._base import client_v2_patterns
	30	from ._base import set_timeline_upper_limit
30	31
31	32	import itertools
32	33	import logging

77	78
78	79	def __init__(self, hs):
79	80	super(SyncRestServlet, self).__init__()
	81	self.hs = hs
80	82	self.auth = hs.get_auth()
81	83	self.sync_handler = hs.get_sync_handler()
82	84	self.clock = hs.get_clock()

120	122	if filter_id.startswith('{'):
121	123	try:
122	124	filter_object = json.loads(filter_id)
	125	set_timeline_upper_limit(filter_object,
	126	self.hs.config.filter_timeline_limit)
123	127	except:
124	128	raise SynapseError(400, "Invalid filter JSON")
125	129	self.filtering.check_valid_filter(filter_object)

252	256	invite = serialize_event(
253	257	room.invite, time_now, token_id=token_id,
254	258	event_format=format_event_for_client_v2_without_room_id,
	259	is_invite=True,
255	260	)
256	261	unsigned = dict(invite.get("unsigned", {}))
257	262	invite["unsigned"] = unsigned

+4

-4

synapse/rest/client/v2_alpha/thirdparty.py less more

35	35
36	36	@defer.inlineCallbacks
37	37	def on_GET(self, request):
38		yield self.auth.get_user_by_req(request)
	38	yield self.auth.get_user_by_req(request, allow_guest=True)
39	39
40	40	protocols = yield self.appservice_handler.get_3pe_protocols()
41	41	defer.returnValue((200, protocols))

53	53
54	54	@defer.inlineCallbacks
55	55	def on_GET(self, request, protocol):
56		yield self.auth.get_user_by_req(request)
	56	yield self.auth.get_user_by_req(request, allow_guest=True)
57	57
58	58	protocols = yield self.appservice_handler.get_3pe_protocols(
59	59	only_protocol=protocol,

76	76
77	77	@defer.inlineCallbacks
78	78	def on_GET(self, request, protocol):
79		yield self.auth.get_user_by_req(request)
	79	yield self.auth.get_user_by_req(request, allow_guest=True)
80	80
81	81	fields = request.args
82	82	fields.pop("access_token", None)

100	100
101	101	@defer.inlineCallbacks
102	102	def on_GET(self, request, protocol):
103		yield self.auth.get_user_by_req(request)
	103	yield self.auth.get_user_by_req(request, allow_guest=True)
104	104
105	105	fields = request.args
106	106	fields.pop("access_token", None)

+2

-3

synapse/rest/key/v2/local_key_resource.py less more

83	83	}
84	84
85	85	old_verify_keys = {}
86		for key in self.config.old_signing_keys:
87		key_id = "%s:%s" % (key.alg, key.version)
	86	for key_id, key in self.config.old_signing_keys.items():
88	87	verify_key_bytes = key.encode()
89	88	old_verify_keys[key_id] = {
90	89	u"key": encode_base64(verify_key_bytes),
91		u"expired_ts": key.expired,
	90	u"expired_ts": key.expired_ts,
92	91	}
93	92
94	93	tls_fingerprints = self.config.tls_fingerprints

+3

-1

synapse/rest/media/v1/media_repository.py less more

33	33	from synapse.util.async import Linearizer
34	34	from synapse.util.stringutils import is_ascii
35	35	from synapse.util.logcontext import preserve_context_over_fn
	36	from synapse.util.retryutils import NotRetryingDestination
36	37
37	38	import os
38	39	import errno

180	181	logger.exception("Failed to fetch remote media %s/%s",
181	182	server_name, media_id)
182	183	raise
183
	184	except NotRetryingDestination:
	185	logger.warn("Not retrying destination %r", server_name)
184	186	except Exception:
185	187	logger.exception("Failed to fetch remote media %s/%s",
186	188	server_name, media_id)

+9

-0

synapse/server.py less more

47	47	from synapse.handlers.events import EventHandler, EventStreamHandler
48	48	from synapse.handlers.initial_sync import InitialSyncHandler
49	49	from synapse.handlers.receipts import ReceiptsHandler
	50	from synapse.handlers.read_marker import ReadMarkerHandler
50	51	from synapse.http.client import SimpleHttpClient, InsecureInterceptableContextFactory
51	52	from synapse.http.matrixfederationclient import MatrixFederationHttpClient
52	53	from synapse.notifier import Notifier

131	132	'federation_sender',
132	133	'receipts_handler',
133	134	'macaroon_generator',
	135	'tcp_replication',
	136	'read_marker_handler',
134	137	]
135	138
136	139	def __init__(self, hostname, **kwargs):

289	292	def build_receipts_handler(self):
290	293	return ReceiptsHandler(self)
291	294
	295	def build_read_marker_handler(self):
	296	return ReadMarkerHandler(self)
	297
	298	def build_tcp_replication(self):
	299	raise NotImplementedError()
	300
292	301	def remove_pusher(self, app_id, push_key, user_id):
293	302	return self.get_pusherpool().remove_pusher(app_id, push_key, user_id)
294	303

+11

-0

synapse/state.py less more

173	173	room_id, entry.state_id, entry.state
174	174	)
175	175	defer.returnValue(joined_users)
	176
	177	@defer.inlineCallbacks
	178	def get_current_hosts_in_room(self, room_id, latest_event_ids=None):
	179	if not latest_event_ids:
	180	latest_event_ids = yield self.store.get_latest_event_ids_in_room(room_id)
	181	logger.debug("calling resolve_state_groups from get_current_hosts_in_room")
	182	entry = yield self.resolve_state_groups(room_id, latest_event_ids)
	183	joined_hosts = yield self.store.get_joined_hosts(
	184	room_id, entry.state_id, entry.state
	185	)
	186	defer.returnValue(joined_hosts)
176	187
177	188	@defer.inlineCallbacks
178	189	def compute_event_context(self, event, old_state=None):

+4

-4

synapse/storage/_base.py less more

59	59	object.__setattr__(self, "database_engine", database_engine)
60	60	object.__setattr__(self, "after_callbacks", after_callbacks)
61	61
62		def call_after(self, callback, *args):
	62	def call_after(self, callback, args, *kwargs):
63	63	"""Call the given callback on the main twisted thread after the
64	64	transaction has finished. Used to invalidate the caches on the
65	65	correct thread.
66	66	"""
67		self.after_callbacks.append((callback, args))
	67	self.after_callbacks.append((callback, args, kwargs))
68	68
69	69	def __getattr__(self, name):
70	70	return getattr(self.txn, name)

318	318	inner_func, args, *kwargs
319	319	)
320	320	finally:
321		for after_callback, after_args in after_callbacks:
322		after_callback(*after_args)
	321	for after_callback, after_args, after_kwargs in after_callbacks:
	322	after_callback(after_args, *after_kwargs)
323	323	defer.returnValue(result)
324	324
325	325	@defer.inlineCallbacks

+69

-35

synapse/storage/background_updates.py less more

209	209	self._background_update_handlers[update_name] = update_handler
210	210
211	211	def register_background_index_update(self, update_name, index_name,
212		table, columns, where_clause=None):
	212	table, columns, where_clause=None,
	213	unique=False,
	214	psql_only=False):
213	215	"""Helper for store classes to do a background index addition
214	216
215	217	To use:

225	227	index_name (str): name of index to add
226	228	table (str): table to add index to
227	229	columns (list[str]): columns/expressions to include in index
228		"""
229
230		# if this is postgres, we add the indexes concurrently. Otherwise
231		# we fall back to doing it inline
232		if isinstance(self.database_engine, engines.PostgresEngine):
233		conc = True
234		else:
235		conc = False
236		# We don't use partial indices on SQLite as it wasn't introduced
237		# until 3.8, and wheezy has 3.7
238		where_clause = None
239
240		sql = (
241		"CREATE INDEX %(conc)s %(name)s ON %(table)s (%(columns)s)"
242		" %(where_clause)s"
243		) % {
244		"conc": "CONCURRENTLY" if conc else "",
245		"name": index_name,
246		"table": table,
247		"columns": ", ".join(columns),
248		"where_clause": "WHERE " + where_clause if where_clause else ""
249		}
250
251		def create_index_concurrently(conn):
	230	unique (bool): true to make a UNIQUE index
	231	psql_only: true to only create this index on psql databases (useful
	232	for virtual sqlite tables)
	233	"""
	234
	235	def create_index_psql(conn):
252	236	conn.rollback()
253	237	# postgres insists on autocommit for the index
254	238	conn.set_session(autocommit=True)
	239
	240	try:
	241	c = conn.cursor()
	242
	243	# If a previous attempt to create the index was interrupted,
	244	# we may already have a half-built index. Let's just drop it
	245	# before trying to create it again.
	246
	247	sql = "DROP INDEX IF EXISTS %s" % (index_name,)
	248	logger.debug("[SQL] %s", sql)
	249	c.execute(sql)
	250
	251	sql = (
	252	"CREATE %(unique)s INDEX CONCURRENTLY %(name)s"
	253	" ON %(table)s"
	254	" (%(columns)s) %(where_clause)s"
	255	) % {
	256	"unique": "UNIQUE" if unique else "",
	257	"name": index_name,
	258	"table": table,
	259	"columns": ", ".join(columns),
	260	"where_clause": "WHERE " + where_clause if where_clause else ""
	261	}
	262	logger.debug("[SQL] %s", sql)
	263	c.execute(sql)
	264	finally:
	265	conn.set_session(autocommit=False)
	266
	267	def create_index_sqlite(conn):
	268	# Sqlite doesn't support concurrent creation of indexes.
	269	#
	270	# We don't use partial indices on SQLite as it wasn't introduced
	271	# until 3.8, and wheezy has 3.7
	272	#
	273	# We assume that sqlite doesn't give us invalid indices; however
	274	# we may still end up with the index existing but the
	275	# background_updates not having been recorded if synapse got shut
	276	# down at the wrong moment - hance we use IF NOT EXISTS. (SQLite
	277	# has supported CREATE TABLE\|INDEX IF NOT EXISTS since 3.3.0.)
	278	sql = (
	279	"CREATE %(unique)s INDEX IF NOT EXISTS %(name)s ON %(table)s"
	280	" (%(columns)s)"
	281	) % {
	282	"unique": "UNIQUE" if unique else "",
	283	"name": index_name,
	284	"table": table,
	285	"columns": ", ".join(columns),
	286	}
	287
255	288	c = conn.cursor()
	289	logger.debug("[SQL] %s", sql)
256	290	c.execute(sql)
257		conn.set_session(autocommit=False)
258
259		def create_index(conn):
260		c = conn.cursor()
261		c.execute(sql)
	291
	292	if isinstance(self.database_engine, engines.PostgresEngine):
	293	runner = create_index_psql
	294	elif psql_only:
	295	runner = None
	296	else:
	297	runner = create_index_sqlite
262	298
263	299	@defer.inlineCallbacks
264	300	def updater(progress, batch_size):
265		logger.info("Adding index %s to %s", index_name, table)
266		if conc:
267		yield self.runWithConnection(create_index_concurrently)
268		else:
269		yield self.runWithConnection(create_index)
	301	if runner is not None:
	302	logger.info("Adding index %s to %s", index_name, table)
	303	yield self.runWithConnection(runner)
270	304	yield self._end_background_update(update_name)
271	305	defer.returnValue(1)
272	306

+4

-0

synapse/storage/client_ips.py less more

32	32	self.client_ip_last_seen = Cache(
33	33	name="client_ip_last_seen",
34	34	keylen=4,
	35	max_entries=5000,
35	36	)
36	37
37	38	super(ClientIpStore, self).__init__(hs)

119	120	where_clauses.append("(user_id = ? AND device_id = ?)")
120	121	bindings.extend((user_id, device_id))
121	122
	123	if not where_clauses:
	124	return []
	125
122	126	inner_select = (
123	127	"SELECT MAX(last_seen) mls, user_id, device_id FROM user_ips "
124	128	"WHERE %(where)s "

+7

-4

synapse/storage/deviceinbox.py less more

324	324	# we return.
325	325	upper_pos = min(current_pos, last_pos + limit)
326	326	sql = (
327		"SELECT stream_id, user_id"
	327	"SELECT max(stream_id), user_id"
328	328	" FROM device_inbox"
329	329	" WHERE ? < stream_id AND stream_id <= ?"
330		" ORDER BY stream_id ASC"
	330	" GROUP BY user_id"
331	331	)
332	332	txn.execute(sql, (last_pos, upper_pos))
333	333	rows = txn.fetchall()
334	334
335	335	sql = (
336		"SELECT stream_id, destination"
	336	"SELECT max(stream_id), destination"
337	337	" FROM device_federation_outbox"
338	338	" WHERE ? < stream_id AND stream_id <= ?"
339		" ORDER BY stream_id ASC"
	339	" GROUP BY destination"
340	340	)
341	341	txn.execute(sql, (last_pos, upper_pos))
342	342	rows.extend(txn)
	343
	344	# Order by ascending stream ordering
	345	rows.sort()
343	346
344	347	return rows
345	348

+25

-6

synapse/storage/devices.py less more

17	17	from twisted.internet import defer
18	18
19	19	from synapse.api.errors import StoreError
20		from ._base import SQLBaseStore
	20	from ._base import SQLBaseStore, Cache
21	21	from synapse.util.caches.descriptors import cached, cachedList, cachedInlineCallbacks
22	22
23	23

27	27	class DeviceStore(SQLBaseStore):
28	28	def __init__(self, hs):
29	29	super(DeviceStore, self).__init__(hs)
	30
	31	# Map of (user_id, device_id) -> bool. If there is an entry that implies
	32	# the device exists.
	33	self.device_id_exists_cache = Cache(
	34	name="device_id_exists",
	35	keylen=2,
	36	max_entries=10000,
	37	)
30	38
31	39	self._clock.looping_call(
32	40	self._prune_old_outbound_device_pokes, 60 * 60 * 1000

53	61	defer.Deferred: boolean whether the device was inserted or an
54	62	existing device existed with that ID.
55	63	"""
	64	key = (user_id, device_id)
	65	if self.device_id_exists_cache.get(key, None):
	66	defer.returnValue(False)
	67
56	68	try:
57	69	inserted = yield self._simple_insert(
58	70	"devices",

64	76	desc="store_device",
65	77	or_ignore=True,
66	78	)
	79	self.device_id_exists_cache.prefill(key, True)
67	80	defer.returnValue(inserted)
68	81	except Exception as e:
69	82	logger.error("store_device with device_id=%s(%r) user_id=%s(%r)"

92	105	desc="get_device",
93	106	)
94	107
	108	@defer.inlineCallbacks
95	109	def delete_device(self, user_id, device_id):
96	110	"""Delete a device.
97	111

101	115	Returns:
102	116	defer.Deferred
103	117	"""
104		return self._simple_delete_one(
	118	yield self._simple_delete_one(
105	119	table="devices",
106	120	keyvalues={"user_id": user_id, "device_id": device_id},
107	121	desc="delete_device",
108	122	)
109	123
	124	self.device_id_exists_cache.invalidate((user_id, device_id))
	125
	126	@defer.inlineCallbacks
110	127	def delete_devices(self, user_id, device_ids):
111	128	"""Deletes several devices.
112	129

116	133	Returns:
117	134	defer.Deferred
118	135	"""
119		return self._simple_delete_many(
	136	yield self._simple_delete_many(
120	137	table="devices",
121	138	column="device_id",
122	139	iterable=device_ids,
123	140	keyvalues={"user_id": user_id},
124	141	desc="delete_devices",
125	142	)
	143	for device_id in device_ids:
	144	self.device_id_exists_cache.invalidate((user_id, device_id))
126	145
127	146	def update_device(self, user_id, device_id, new_display_name=None):
128	147	"""Update a device.

532	551	rows = yield self._execute("get_user_whose_devices_changed", None, sql, from_key)
533	552	defer.returnValue(set(row[0] for row in rows))
534	553
535		def get_all_device_list_changes_for_remotes(self, from_key):
	554	def get_all_device_list_changes_for_remotes(self, from_key, to_key):
536	555	"""Return a list of `(stream_id, user_id, destination)` which is the
537	556	combined list of changes to devices, and which destinations need to be
538	557	poked. `destination` may be None if no destinations need to be poked.

540	559	sql = """
541	560	SELECT stream_id, user_id, destination FROM device_lists_stream
542	561	LEFT JOIN device_lists_outbound_pokes USING (stream_id, user_id, device_id)
543		WHERE stream_id > ?
	562	WHERE ? < stream_id AND stream_id <= ?
544	563	"""
545	564	return self._execute(
546	565	"get_all_device_list_changes_for_remotes", None,
547		sql, from_key,
	566	sql, from_key, to_key
548	567	)
549	568
550	569	@defer.inlineCallbacks

+39

-23

synapse/storage/end_to_end_keys.py less more

13	13	# limitations under the License.
14	14	from twisted.internet import defer
15	15
16		from synapse.api.errors import SynapseError
	16	from synapse.util.caches.descriptors import cached
17	17
18	18	from canonicaljson import encode_canonical_json
19	19	import ujson as json

122	122	return result
123	123
124	124	@defer.inlineCallbacks
125		def add_e2e_one_time_keys(self, user_id, device_id, time_now, key_list):
126		"""Insert some new one time keys for a device.
127
128		Checks if any of the keys are already inserted, if they are then check
129		if they match. If they don't then we raise an error.
130		"""
131
132		# First we check if we have already persisted any of the keys.
	125	def get_e2e_one_time_keys(self, user_id, device_id, key_ids):
	126	"""Retrieve a number of one-time keys for a user
	127
	128	Args:
	129	user_id(str): id of user to get keys for
	130	device_id(str): id of device to get keys for
	131	key_ids(list[str]): list of key ids (excluding algorithm) to
	132	retrieve
	133
	134	Returns:
	135	deferred resolving to Dict[(str, str), str]: map from (algorithm,
	136	key_id) to json string for key
	137	"""
	138
133	139	rows = yield self._simple_select_many_batch(
134	140	table="e2e_one_time_keys_json",
135	141	column="key_id",
136		iterable=[key_id for _, key_id, _ in key_list],
	142	iterable=key_ids,
137	143	retcols=("algorithm", "key_id", "key_json",),
138	144	keyvalues={
139	145	"user_id": user_id,

142	148	desc="add_e2e_one_time_keys_check",
143	149	)
144	150
145		existing_key_map = {
	151	defer.returnValue({
146	152	(row["algorithm"], row["key_id"]): row["key_json"] for row in rows
147		}
148
149		new_keys = [] # Keys that we need to insert
150		for algorithm, key_id, json_bytes in key_list:
151		ex_bytes = existing_key_map.get((algorithm, key_id), None)
152		if ex_bytes:
153		if json_bytes != ex_bytes:
154		raise SynapseError(
155		400, "One time key with key_id %r already exists" % (key_id,)
156		)
157		else:
158		new_keys.append((algorithm, key_id, json_bytes))
	153	})
	154
	155	@defer.inlineCallbacks
	156	def add_e2e_one_time_keys(self, user_id, device_id, time_now, new_keys):
	157	"""Insert some new one time keys for a device. Errors if any of the
	158	keys already exist.
	159
	160	Args:
	161	user_id(str): id of user to get keys for
	162	device_id(str): id of device to get keys for
	163	time_now(long): insertion time to record (ms since epoch)
	164	new_keys(iterable[(str, str, str)]: keys to add - each a tuple of
	165	(algorithm, key_id, key json)
	166	"""
159	167
160	168	def _add_e2e_one_time_keys(txn):
161	169	# We are protected from race between lookup and insertion due to

176	184	for algorithm, key_id, json_bytes in new_keys
177	185	],
178	186	)
	187	txn.call_after(
	188	self.count_e2e_one_time_keys.invalidate, (user_id, device_id,)
	189	)
179	190	yield self.runInteraction(
180	191	"add_e2e_one_time_keys_insert", _add_e2e_one_time_keys
181	192	)
182	193
	194	@cached(max_entries=10000)
183	195	def count_e2e_one_time_keys(self, user_id, device_id):
184	196	""" Count the number of one time keys the server has for a device
185	197	Returns:

224	236	)
225	237	for user_id, device_id, algorithm, key_id in delete:
226	238	txn.execute(sql, (user_id, device_id, algorithm, key_id))
	239	txn.call_after(
	240	self.count_e2e_one_time_keys.invalidate, (user_id, device_id,)
	241	)
227	242	return result
228	243	return self.runInteraction(
229	244	"claim_e2e_one_time_keys", _claim_e2e_one_time_keys

241	256	keyvalues={"user_id": user_id, "device_id": device_id},
242	257	desc="delete_e2e_one_time_keys_by_device"
243	258	)
	259	self.count_e2e_one_time_keys.invalidate((user_id, device_id,))

+220

-33

synapse/storage/events.py less more

28	28	from synapse.api.errors import SynapseError
29	29	from synapse.state import resolve_events
30	30	from synapse.util.caches.descriptors import cached
	31	from synapse.types import get_domain_from_id
31	32
32	33	from canonicaljson import encode_canonical_json
33	34	from collections import deque, namedtuple, OrderedDict

48	49
49	50	metrics = synapse.metrics.get_metrics_for(__name__)
50	51	persist_event_counter = metrics.register_counter("persisted_events")
	52	event_counter = metrics.register_counter(
	53	"persisted_events_sep", labels=["type", "origin_type", "origin_entity"]
	54	)
51	55
52	56
53	57	def encode_json(json_object):

202	206	where_clause="contains_url = true AND outlier = false",
203	207	)
204	208
	209	# an event_id index on event_search is useful for the purge_history
	210	# api. Plus it means we get to enforce some integrity with a UNIQUE
	211	# clause
	212	self.register_background_index_update(
	213	"event_search_event_id_idx",
	214	index_name="event_search_event_id_idx",
	215	table="event_search",
	216	columns=["event_id"],
	217	unique=True,
	218	psql_only=True,
	219	)
	220
205	221	self._event_persist_queue = _EventPeristenceQueue()
206	222
207	223	def persist_events(self, events_and_contexts, backfilled=False):

369	385	new_forward_extremeties=new_forward_extremeties,
370	386	)
371	387	persist_event_counter.inc_by(len(chunk))
	388	for event, context in chunk:
	389	if context.app_service:
	390	origin_type = "local"
	391	origin_entity = context.app_service.id
	392	elif self.hs.is_mine_id(event.sender):
	393	origin_type = "local"
	394	origin_entity = "client"
	395	else:
	396	origin_type = "remote"
	397	origin_entity = get_domain_from_id(event.sender)
	398
	399	event_counter.inc(event.type, origin_type, origin_entity)
	400
	401	for room_id, (_, _, new_state) in current_state_for_room.iteritems():
	402	self.get_current_state_ids.prefill(
	403	(room_id, ), new_state
	404	)
372	405
373	406	@defer.inlineCallbacks
374	407	def _calculate_new_extremeties(self, room_id, event_contexts, latest_event_ids):

418	451	Assumes that we are only persisting events for one room at a time.
419	452
420	453	Returns:
421		2-tuple (to_delete, to_insert) where both are state dicts, i.e.
422		(type, state_key) -> event_id. `to_delete` are the entries to
	454	3-tuple (to_delete, to_insert, new_state) where both are state dicts,
	455	i.e. (type, state_key) -> event_id. `to_delete` are the entries to
423	456	first be deleted from current_state_events, `to_insert` are entries
424		to insert.
	457	to insert. `new_state` is the full set of state.
425	458	May return None if there are no changes to be applied.
426	459	"""
427	460	# Now we need to work out the different state sets for

528	561	if ev_id in events_to_insert
529	562	}
530	563
531		defer.returnValue((to_delete, to_insert))
	564	defer.returnValue((to_delete, to_insert, current_state))
532	565
533	566	@defer.inlineCallbacks
534	567	def get_event(self, event_id, check_redacted=True,

681	714
682	715	def _update_current_state_txn(self, txn, state_delta_by_room):
683	716	for room_id, current_state_tuple in state_delta_by_room.iteritems():
684		to_delete, to_insert = current_state_tuple
	717	to_delete, to_insert, _ = current_state_tuple
685	718	txn.executemany(
686	719	"DELETE FROM current_state_events WHERE event_id = ?",
687	720	[(ev_id,) for ev_id in to_delete.itervalues()],

1326	1359	def _invalidate_get_event_cache(self, event_id):
1327	1360	self._get_event_cache.invalidate((event_id,))
1328	1361
1329		def _get_events_from_cache(self, events, allow_rejected):
	1362	def _get_events_from_cache(self, events, allow_rejected, update_metrics=True):
	1363	"""Fetch events from the caches
	1364
	1365	Args:
	1366	events (list(str)): list of event_ids to fetch
	1367	allow_rejected (bool): Whether to teturn events that were rejected
	1368	update_metrics (bool): Whether to update the cache hit ratio metrics
	1369
	1370	Returns:
	1371	dict of event_id -> _EventCacheEntry for each event_id in cache. If
	1372	allow_rejected is `False` then there will still be an entry but it
	1373	will be `None`
	1374	"""
1330	1375	event_map = {}
1331	1376
1332	1377	for event_id in events:
1333		ret = self._get_event_cache.get((event_id,), None)
	1378	ret = self._get_event_cache.get(
	1379	(event_id,), None,
	1380	update_metrics=update_metrics,
	1381	)
1334	1382	if not ret:
1335	1383	continue
1336	1384

1769	1817	def get_current_backfill_token(self):
1770	1818	"""The current minimum token that backfilled events have reached"""
1771	1819	return -self._backfill_id_gen.get_current_token()
	1820
	1821	def get_current_events_token(self):
	1822	"""The current maximum token that events have reached"""
	1823	return self._stream_id_gen.get_current_token()
	1824
	1825	def get_all_new_forward_event_rows(self, last_id, current_id, limit):
	1826	if last_id == current_id:
	1827	return defer.succeed([])
	1828
	1829	def get_all_new_forward_event_rows(txn):
	1830	sql = (
	1831	"SELECT e.stream_ordering, e.event_id, e.room_id, e.type,"
	1832	" state_key, redacts"
	1833	" FROM events AS e"
	1834	" LEFT JOIN redactions USING (event_id)"
	1835	" LEFT JOIN state_events USING (event_id)"
	1836	" WHERE ? < stream_ordering AND stream_ordering <= ?"
	1837	" ORDER BY stream_ordering ASC"
	1838	" LIMIT ?"
	1839	)
	1840	txn.execute(sql, (last_id, current_id, limit))
	1841	new_event_updates = txn.fetchall()
	1842
	1843	if len(new_event_updates) == limit:
	1844	upper_bound = new_event_updates[-1][0]
	1845	else:
	1846	upper_bound = current_id
	1847
	1848	sql = (
	1849	"SELECT event_stream_ordering, e.event_id, e.room_id, e.type,"
	1850	" state_key, redacts"
	1851	" FROM events AS e"
	1852	" INNER JOIN ex_outlier_stream USING (event_id)"
	1853	" LEFT JOIN redactions USING (event_id)"
	1854	" LEFT JOIN state_events USING (event_id)"
	1855	" WHERE ? < event_stream_ordering"
	1856	" AND event_stream_ordering <= ?"
	1857	" ORDER BY event_stream_ordering DESC"
	1858	)
	1859	txn.execute(sql, (last_id, upper_bound))
	1860	new_event_updates.extend(txn)
	1861
	1862	return new_event_updates
	1863	return self.runInteraction(
	1864	"get_all_new_forward_event_rows", get_all_new_forward_event_rows
	1865	)
	1866
	1867	def get_all_new_backfill_event_rows(self, last_id, current_id, limit):
	1868	if last_id == current_id:
	1869	return defer.succeed([])
	1870
	1871	def get_all_new_backfill_event_rows(txn):
	1872	sql = (
	1873	"SELECT -e.stream_ordering, e.event_id, e.room_id, e.type,"
	1874	" state_key, redacts"
	1875	" FROM events AS e"
	1876	" LEFT JOIN redactions USING (event_id)"
	1877	" LEFT JOIN state_events USING (event_id)"
	1878	" WHERE ? > stream_ordering AND stream_ordering >= ?"
	1879	" ORDER BY stream_ordering ASC"
	1880	" LIMIT ?"
	1881	)
	1882	txn.execute(sql, (-last_id, -current_id, limit))
	1883	new_event_updates = txn.fetchall()
	1884
	1885	if len(new_event_updates) == limit:
	1886	upper_bound = new_event_updates[-1][0]
	1887	else:
	1888	upper_bound = current_id
	1889
	1890	sql = (
	1891	"SELECT -event_stream_ordering, e.event_id, e.room_id, e.type,"
	1892	" state_key, redacts"
	1893	" FROM events AS e"
	1894	" INNER JOIN ex_outlier_stream USING (event_id)"
	1895	" LEFT JOIN redactions USING (event_id)"
	1896	" LEFT JOIN state_events USING (event_id)"
	1897	" WHERE ? > event_stream_ordering"
	1898	" AND event_stream_ordering >= ?"
	1899	" ORDER BY event_stream_ordering DESC"
	1900	)
	1901	txn.execute(sql, (-last_id, -upper_bound))
	1902	new_event_updates.extend(txn.fetchall())
	1903
	1904	return new_event_updates
	1905	return self.runInteraction(
	1906	"get_all_new_backfill_event_rows", get_all_new_backfill_event_rows
	1907	)
1772	1908
1773	1909	@cached(num_args=5, max_entries=10)
1774	1910	def get_all_new_events(self, last_backfill_id, last_forward_id,

1902	2038	400, "topological_ordering is greater than forward extremeties"
1903	2039	)
1904	2040
	2041	logger.debug("[purge] looking for events to delete")
	2042
1905	2043	txn.execute(
1906	2044	"SELECT event_id, state_key FROM events"
1907	2045	" LEFT JOIN state_events USING (room_id, event_id)"

1910	2048	)
1911	2049	event_rows = txn.fetchall()
1912	2050
	2051	to_delete = [
	2052	(event_id,) for event_id, state_key in event_rows
	2053	if state_key is None and not self.hs.is_mine_id(event_id)
	2054	]
	2055	logger.info(
	2056	"[purge] found %i events before cutoff, of which %i are remote"
	2057	" non-state events to delete", len(event_rows), len(to_delete))
	2058
1913	2059	for event_id, state_key in event_rows:
1914	2060	txn.call_after(self._get_state_group_for_event.invalidate, (event_id,))
	2061
	2062	logger.debug("[purge] Finding new backward extremities")
1915	2063
1916	2064	# We calculate the new entries for the backward extremeties by finding
1917	2065	# all events that point to events that are to be purged

1925	2073	)
1926	2074	new_backwards_extrems = txn.fetchall()
1927	2075
	2076	logger.debug("[purge] replacing backward extremities: %r", new_backwards_extrems)
	2077
1928	2078	txn.execute(
1929	2079	"DELETE FROM event_backward_extremities WHERE room_id = ?",
1930	2080	(room_id,)

1938	2088	(room_id, event_id) for event_id, in new_backwards_extrems
1939	2089	]
1940	2090	)
	2091
	2092	logger.debug("[purge] finding redundant state groups")
1941	2093
1942	2094	# Get all state groups that are only referenced by events that are
1943	2095	# to be deleted.

1954	2106	)
1955	2107
1956	2108	state_rows = txn.fetchall()
1957		state_groups_to_delete = [sg for sg, in state_rows]
	2109	logger.debug("[purge] found %i redundant state groups", len(state_rows))
	2110
	2111	# make a set of the redundant state groups, so that we can look them up
	2112	# efficiently
	2113	state_groups_to_delete = set([sg for sg, in state_rows])
1958	2114
1959	2115	# Now we get all the state groups that rely on these state groups
1960		new_state_edges = []
1961		chunks = [
1962		state_groups_to_delete[i:i + 100]
1963		for i in xrange(0, len(state_groups_to_delete), 100)
1964		]
1965		for chunk in chunks:
	2116	logger.debug("[purge] finding state groups which depend on redundant"
	2117	" state groups")
	2118	remaining_state_groups = []
	2119	for i in xrange(0, len(state_rows), 100):
	2120	chunk = [sg for sg, in state_rows[i:i + 100]]
	2121	# look for state groups whose prev_state_group is one we are about
	2122	# to delete
1966	2123	rows = self._simple_select_many_txn(
1967	2124	txn,
1968	2125	table="state_group_edges",

1971	2128	retcols=["state_group"],
1972	2129	keyvalues={},
1973	2130	)
1974		new_state_edges.extend(row["state_group"] for row in rows)
1975
1976		# Now we turn the state groups that reference to-be-deleted state groups
1977		# to non delta versions.
1978		for new_state_edge in new_state_edges:
	2131	remaining_state_groups.extend(
	2132	row["state_group"] for row in rows
	2133
	2134	# exclude state groups we are about to delete: no point in
	2135	# updating them
	2136	if row["state_group"] not in state_groups_to_delete
	2137	)
	2138
	2139	# Now we turn the state groups that reference to-be-deleted state
	2140	# groups to non delta versions.
	2141	for sg in remaining_state_groups:
	2142	logger.debug("[purge] de-delta-ing remaining state group %s", sg)
1979	2143	curr_state = self._get_state_groups_from_groups_txn(
1980		txn, [new_state_edge], types=None
1981		)
1982		curr_state = curr_state[new_state_edge]
	2144	txn, [sg], types=None
	2145	)
	2146	curr_state = curr_state[sg]
1983	2147
1984	2148	self._simple_delete_txn(
1985	2149	txn,
1986	2150	table="state_groups_state",
1987	2151	keyvalues={
1988		"state_group": new_state_edge,
	2152	"state_group": sg,
1989	2153	}
1990	2154	)
1991	2155

1993	2157	txn,
1994	2158	table="state_group_edges",
1995	2159	keyvalues={
1996		"state_group": new_state_edge,
	2160	"state_group": sg,
1997	2161	}
1998	2162	)
1999	2163

2002	2166	table="state_groups_state",
2003	2167	values=[
2004	2168	{
2005		"state_group": new_state_edge,
	2169	"state_group": sg,
2006	2170	"room_id": room_id,
2007	2171	"type": key[0],
2008	2172	"state_key": key[1],

2012	2176	],
2013	2177	)
2014	2178
	2179	logger.debug("[purge] removing redundant state groups")
2015	2180	txn.executemany(
2016	2181	"DELETE FROM state_groups_state WHERE state_group = ?",
2017	2182	state_rows

2020	2185	"DELETE FROM state_groups WHERE id = ?",
2021	2186	state_rows
2022	2187	)
	2188
2023	2189	# Delete all non-state
	2190	logger.debug("[purge] removing events from event_to_state_groups")
2024	2191	txn.executemany(
2025	2192	"DELETE FROM event_to_state_groups WHERE event_id = ?",
2026	2193	[(event_id,) for event_id, _ in event_rows]
2027	2194	)
2028	2195
	2196	logger.debug("[purge] updating room_depth")
2029	2197	txn.execute(
2030	2198	"UPDATE room_depth SET min_depth = ? WHERE room_id = ?",
2031	2199	(topological_ordering, room_id,)
2032	2200	)
2033	2201
2034	2202	# Delete all remote non-state events
2035		to_delete = [
2036		(event_id,) for event_id, state_key in event_rows
2037		if state_key is None and not self.hs.is_mine_id(event_id)
2038		]
2039	2203	for table in (
2040	2204	"events",
2041	2205	"event_json",

2051	2215	"event_signatures",
2052	2216	"rejections",
2053	2217	):
	2218	logger.debug("[purge] removing remote non-state events from %s", table)
	2219
2054	2220	txn.executemany(
2055	2221	"DELETE FROM %s WHERE event_id = ?" % (table,),
2056	2222	to_delete
2057	2223	)
2058	2224
2059		txn.executemany(
2060		"DELETE FROM events WHERE event_id = ?",
2061		to_delete
2062		)
2063	2225	# Mark all state and own events as outliers
	2226	logger.debug("[purge] marking remaining events as outliers")
2064	2227	txn.executemany(
2065	2228	"UPDATE events SET outlier = ?"
2066	2229	" WHERE event_id = ?",

2070	2233	]
2071	2234	)
2072	2235
	2236	logger.info("[purge] done")
	2237
	2238	@defer.inlineCallbacks
	2239	def is_event_after(self, event_id1, event_id2):
	2240	"""Returns True if event_id1 is after event_id2 in the stream
	2241	"""
	2242	to_1, so_1 = yield self._get_event_ordering(event_id1)
	2243	to_2, so_2 = yield self._get_event_ordering(event_id2)
	2244	defer.returnValue((to_1, so_1) > (to_2, so_2))
	2245
	2246	@defer.inlineCallbacks
	2247	def _get_event_ordering(self, event_id):
	2248	res = yield self._simple_select_one(
	2249	table="events",
	2250	retcols=["topological_ordering", "stream_ordering"],
	2251	keyvalues={"event_id": event_id},
	2252	allow_none=True
	2253	)
	2254
	2255	if not res:
	2256	raise SynapseError(404, "Could not find event %s" % (event_id,))
	2257
	2258	defer.returnValue((int(res["topological_ordering"]), int(res["stream_ordering"])))
	2259
2073	2260
2074	2261	AllNewEventsResult = namedtuple("AllNewEventsResult", [
2075	2262	"new_forward_events", "new_backfill_events",

+13

-0

synapse/storage/push_rule.py less more

15	15	from ._base import SQLBaseStore
16	16	from synapse.util.caches.descriptors import cachedInlineCallbacks, cachedList
17	17	from synapse.push.baserules import list_with_base_rules
	18	from synapse.api.constants import EventTypes
18	19	from twisted.internet import defer
19	20
20	21	import logging

182	183	for uid in users_with_receipts:
183	184	if uid in local_users_in_room:
184	185	user_ids.add(uid)
	186
	187	forgotten = yield self.who_forgot_in_room(
	188	event.room_id, on_invalidate=cache_context.invalidate,
	189	)
	190
	191	for row in forgotten:
	192	user_id = row["user_id"]
	193	event_id = row["event_id"]
	194
	195	mem_id = current_state_ids.get((EventTypes.Member, user_id), None)
	196	if event_id == mem_id:
	197	user_ids.discard(user_id)
185	198
186	199	rules_by_user = yield self.bulk_get_push_rules(
187	200	user_ids, on_invalidate=cache_context.invalidate,

+42

-0

synapse/storage/pusher.py less more

132	132	return (updated, deleted)
133	133	return self.runInteraction(
134	134	"get_all_updated_pushers", get_all_updated_pushers_txn
	135	)
	136
	137	def get_all_updated_pushers_rows(self, last_id, current_id, limit):
	138	"""Get all the pushers that have changed between the given tokens.
	139
	140	Returns:
	141	Deferred(list(tuple)): each tuple consists of:
	142	stream_id (str)
	143	user_id (str)
	144	app_id (str)
	145	pushkey (str)
	146	was_deleted (bool): whether the pusher was added/updated (False)
	147	or deleted (True)
	148	"""
	149
	150	if last_id == current_id:
	151	return defer.succeed([])
	152
	153	def get_all_updated_pushers_rows_txn(txn):
	154	sql = (
	155	"SELECT id, user_name, app_id, pushkey"
	156	" FROM pushers"
	157	" WHERE ? < id AND id <= ?"
	158	" ORDER BY id ASC LIMIT ?"
	159	)
	160	txn.execute(sql, (last_id, current_id, limit))
	161	results = [list(row) + [False] for row in txn]
	162
	163	sql = (
	164	"SELECT stream_id, user_id, app_id, pushkey"
	165	" FROM deleted_pushers"
	166	" WHERE ? < stream_id AND stream_id <= ?"
	167	" ORDER BY stream_id ASC LIMIT ?"
	168	)
	169	txn.execute(sql, (last_id, current_id, limit))
	170
	171	results.extend(list(row) + [True] for row in txn)
	172	results.sort() # Sort so that they're ordered by stream id
	173
	174	return results
	175	return self.runInteraction(
	176	"get_all_updated_pushers_rows", get_all_updated_pushers_rows_txn
135	177	)
136	178
137	179	@cachedInlineCallbacks(num_args=1, max_entries=15000)

+7

-4

synapse/storage/receipts.py less more

46	46	# Returns an ObservableDeferred
47	47	res = self.get_users_with_read_receipts_in_room.cache.get((room_id,), None)
48	48
49		if res and res.called and user_id in res.result:
50		# We'd only be adding to the set, so no point invalidating if the
51		# user is already there
52		return
	49	if res:
	50	if isinstance(res, defer.Deferred) and res.called:
	51	res = res.result
	52	if user_id in res:
	53	# We'd only be adding to the set, so no point invalidating if the
	54	# user is already there
	55	return
53	56
54	57	self.get_users_with_read_receipts_in_room.invalidate((room_id,))
55	58

+35

-1

synapse/storage/room.py less more

15	15	from twisted.internet import defer
16	16
17	17	from synapse.api.errors import StoreError
18		from synapse.util.caches.descriptors import cached
	18	from synapse.util.caches.descriptors import cached, cachedInlineCallbacks
19	19
20	20	from ._base import SQLBaseStore
21	21	from .engines import PostgresEngine, Sqlite3Engine

30	30	OpsLevel = collections.namedtuple(
31	31	"OpsLevel",
32	32	("ban_level", "kick_level", "redact_level",)
	33	)
	34
	35	RatelimitOverride = collections.namedtuple(
	36	"RatelimitOverride",
	37	("messages_per_second", "burst_count",)
33	38	)
34	39
35	40

472	477	return self.runInteraction(
473	478	"get_all_new_public_rooms", get_all_new_public_rooms
474	479	)
	480
	481	@cachedInlineCallbacks(max_entries=10000)
	482	def get_ratelimit_for_user(self, user_id):
	483	"""Check if there are any overrides for ratelimiting for the given
	484	user
	485
	486	Args:
	487	user_id (str)
	488
	489	Returns:
	490	RatelimitOverride if there is an override, else None. If the contents
	491	of RatelimitOverride are None or 0 then ratelimitng has been
	492	disabled for that user entirely.
	493	"""
	494	row = yield self._simple_select_one(
	495	table="ratelimit_override",
	496	keyvalues={"user_id": user_id},
	497	retcols=("messages_per_second", "burst_count"),
	498	allow_none=True,
	499	desc="get_ratelimit_for_user",
	500	)
	501
	502	if row:
	503	defer.returnValue(RatelimitOverride(
	504	messages_per_second=row["messages_per_second"],
	505	burst_count=row["burst_count"],
	506	))
	507	else:
	508	defer.returnValue(None)

+126

-27

synapse/storage/roommember.py less more

17	17	from collections import namedtuple
18	18
19	19	from ._base import SQLBaseStore
	20	from synapse.util.caches import intern_string
20	21	from synapse.util.caches.descriptors import cached, cachedInlineCallbacks
	22	from synapse.util.stringutils import to_ascii
21	23
22	24	from synapse.api.constants import Membership, EventTypes
23	25	from synapse.types import get_domain_from_id

31	33	RoomsForUser = namedtuple(
32	34	"RoomsForUser",
33	35	("room_id", "sender", "membership", "event_id", "stream_ordering")
	36	)
	37
	38
	39	# We store this using a namedtuple so that we save about 3x space over using a
	40	# dict.
	41	ProfileInfo = namedtuple(
	42	"ProfileInfo", ("avatar_url", "display_name")
34	43	)
35	44
36	45

138	147	hosts = frozenset(get_domain_from_id(user_id) for user_id in user_ids)
139	148	defer.returnValue(hosts)
140	149
141		@cached(max_entries=500000, iterable=True)
	150	@cached(max_entries=100000, iterable=True)
142	151	def get_users_in_room(self, room_id):
143	152	def f(txn):
144	153	sql = (

151	160	)
152	161
153	162	txn.execute(sql, (room_id, Membership.JOIN,))
154		return [r[0] for r in txn]
	163	return [to_ascii(r[0]) for r in txn]
155	164	return self.runInteraction("get_users_in_room", f)
156	165
157	166	@cached()

377	386	state_group = object()
378	387
379	388	return self._get_joined_users_from_context(
380		event.room_id, state_group, context.current_state_ids, event=event,
	389	event.room_id, state_group, context.current_state_ids,
	390	event=event,
	391	context=context,
381	392	)
382	393
383	394	def get_joined_users_from_state(self, room_id, state_group, state_ids):

395	406	@cachedInlineCallbacks(num_args=2, cache_context=True, iterable=True,
396	407	max_entries=100000)
397	408	def _get_joined_users_from_context(self, room_id, state_group, current_state_ids,
398		cache_context, event=None):
	409	cache_context, event=None, context=None):
399	410	# We don't use `state_group`, it's there so that we can cache based
400	411	# on it. However, it's important that it's never None, since two current_states
401	412	# with a state_group of None are likely to be different.
402	413	# See bulk_get_push_rules_for_room for how we work around this.
403	414	assert state_group is not None
404	415
	416	users_in_room = {}
405	417	member_event_ids = [
406	418	e_id
407	419	for key, e_id in current_state_ids.iteritems()
408	420	if key[0] == EventTypes.Member
409	421	]
410	422
411		rows = yield self._simple_select_many_batch(
412		table="room_memberships",
413		column="event_id",
414		iterable=member_event_ids,
415		retcols=['user_id', 'display_name', 'avatar_url'],
416		keyvalues={
417		"membership": Membership.JOIN,
418		},
419		batch_size=500,
420		desc="_get_joined_users_from_context",
421		)
422
423		users_in_room = {
424		row["user_id"]: {
425		"display_name": row["display_name"],
426		"avatar_url": row["avatar_url"],
427		}
428		for row in rows
429		}
	423	if context is not None:
	424	# If we have a context with a delta from a previous state group,
	425	# check if we also have the result from the previous group in cache.
	426	# If we do then we can reuse that result and simply update it with
	427	# any membership changes in `delta_ids`
	428	if context.prev_group and context.delta_ids:
	429	prev_res = self._get_joined_users_from_context.cache.get(
	430	(room_id, context.prev_group), None
	431	)
	432	if prev_res and isinstance(prev_res, dict):
	433	users_in_room = dict(prev_res)
	434	member_event_ids = [
	435	e_id
	436	for key, e_id in context.delta_ids.iteritems()
	437	if key[0] == EventTypes.Member
	438	]
	439	for etype, state_key in context.delta_ids:
	440	users_in_room.pop(state_key, None)
	441
	442	# We check if we have any of the member event ids in the event cache
	443	# before we ask the DB
	444
	445	# We don't update the event cache hit ratio as it completely throws off
	446	# the hit ratio counts. After all, we don't populate the cache if we
	447	# miss it here
	448	event_map = self._get_events_from_cache(
	449	member_event_ids,
	450	allow_rejected=False,
	451	update_metrics=False,
	452	)
	453
	454	missing_member_event_ids = []
	455	for event_id in member_event_ids:
	456	ev_entry = event_map.get(event_id)
	457	if ev_entry:
	458	if ev_entry.event.membership == Membership.JOIN:
	459	users_in_room[to_ascii(ev_entry.event.state_key)] = ProfileInfo(
	460	display_name=to_ascii(
	461	ev_entry.event.content.get("displayname", None)
	462	),
	463	avatar_url=to_ascii(
	464	ev_entry.event.content.get("avatar_url", None)
	465	),
	466	)
	467	else:
	468	missing_member_event_ids.append(event_id)
	469
	470	if missing_member_event_ids:
	471	rows = yield self._simple_select_many_batch(
	472	table="room_memberships",
	473	column="event_id",
	474	iterable=missing_member_event_ids,
	475	retcols=('user_id', 'display_name', 'avatar_url',),
	476	keyvalues={
	477	"membership": Membership.JOIN,
	478	},
	479	batch_size=500,
	480	desc="_get_joined_users_from_context",
	481	)
	482
	483	users_in_room.update({
	484	to_ascii(row["user_id"]): ProfileInfo(
	485	avatar_url=to_ascii(row["avatar_url"]),
	486	display_name=to_ascii(row["display_name"]),
	487	)
	488	for row in rows
	489	})
430	490
431	491	if event is not None and event.type == EventTypes.Member:
432	492	if event.membership == Membership.JOIN:
433	493	if event.event_id in member_event_ids:
434		users_in_room[event.state_key] = {
435		"display_name": event.content.get("displayname", None),
436		"avatar_url": event.content.get("avatar_url", None),
437		}
	494	users_in_room[to_ascii(event.state_key)] = ProfileInfo(
	495	display_name=to_ascii(event.content.get("displayname", None)),
	496	avatar_url=to_ascii(event.content.get("avatar_url", None)),
	497	)
438	498
439	499	defer.returnValue(users_in_room)
440	500

472	532	defer.returnValue(True)
473	533
474	534	defer.returnValue(False)
	535
	536	def get_joined_hosts(self, room_id, state_group, state_ids):
	537	if not state_group:
	538	# If state_group is None it means it has yet to be assigned a
	539	# state group, i.e. we need to make sure that calls with a state_group
	540	# of None don't hit previous cached calls with a None state_group.
	541	# To do this we set the state_group to a new object as object() != object()
	542	state_group = object()
	543
	544	return self._get_joined_hosts(
	545	room_id, state_group, state_ids
	546	)
	547
	548	@cachedInlineCallbacks(num_args=2, max_entries=10000, iterable=True)
	549	def _get_joined_hosts(self, room_id, state_group, current_state_ids):
	550	# We don't use `state_group`, its there so that we can cache based
	551	# on it. However, its important that its never None, since two current_state's
	552	# with a state_group of None are likely to be different.
	553	# See bulk_get_push_rules_for_room for how we work around this.
	554	assert state_group is not None
	555
	556	joined_hosts = set()
	557	for etype, state_key in current_state_ids:
	558	if etype == EventTypes.Member:
	559	try:
	560	host = get_domain_from_id(state_key)
	561	except:
	562	logger.warn("state_key not user_id: %s", state_key)
	563	continue
	564
	565	if host in joined_hosts:
	566	continue
	567
	568	event_id = current_state_ids[(etype, state_key)]
	569	event = yield self.get_event(event_id, allow_none=True)
	570	if event and event.content["membership"] == Membership.JOIN:
	571	joined_hosts.add(intern_string(host))
	572
	573	defer.returnValue(joined_hosts)
475	574
476	575	@defer.inlineCallbacks
477	576	def _background_add_membership_profile(self, progress, batch_size):

+4

-0

synapse/storage/schema/delta/37/remove_auth_idx.py less more

35	35	-- and is used incredibly rarely.
36	36	DROP INDEX IF EXISTS events_order_topo_stream_room;
37	37
	38	-- an equivalent index to this actually gets re-created in delta 41, because it
	39	-- turned out that deleting it wasn't a great plan :/. In any case, let's
	40	-- delete it here, and delta 41 will create a new one with an added UNIQUE
	41	-- constraint
38	42	DROP INDEX IF EXISTS event_search_ev_idx;
39	43	"""
40	44

+17

-0

synapse/storage/schema/delta/41/event_search_event_id_idx.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT into background_updates (update_name, progress_json)
	16	VALUES ('event_search_event_id_idx', '{}');

+22

-0

synapse/storage/schema/delta/41/ratelimit.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE ratelimit_override (
	16	user_id TEXT NOT NULL,
	17	messages_per_second BIGINT,
	18	burst_count BIGINT
	19	);
	20
	21	CREATE UNIQUE INDEX ratelimit_override_idx ON ratelimit_override(user_id);

+43

-19

synapse/storage/state.py less more

13	13	# limitations under the License.
14	14
15	15	from ._base import SQLBaseStore
16		from synapse.util.caches.descriptors import cached, cachedList, cachedInlineCallbacks
	16	from synapse.util.caches.descriptors import cached, cachedList
17	17	from synapse.util.caches import intern_string
	18	from synapse.util.stringutils import to_ascii
18	19	from synapse.storage.engines import PostgresEngine
19	20
20	21	from twisted.internet import defer

68	69	where_clause="type='m.room.member'",
69	70	)
70	71
71		@cachedInlineCallbacks(max_entries=100000, iterable=True)
	72	@cached(max_entries=100000, iterable=True)
72	73	def get_current_state_ids(self, room_id):
73		rows = yield self._simple_select_list(
74		table="current_state_events",
75		keyvalues={"room_id": room_id},
76		retcols=["event_id", "type", "state_key"],
77		desc="_calculate_state_delta",
78		)
79		defer.returnValue({
80		(r["type"], r["state_key"]): r["event_id"] for r in rows
81		})
	74	"""Get the current state event ids for a room based on the
	75	current_state_events table.
	76
	77	Args:
	78	room_id (str)
	79
	80	Returns:
	81	deferred: dict of (type, state_key) -> event_id
	82	"""
	83	def _get_current_state_ids_txn(txn):
	84	txn.execute(
	85	"""SELECT type, state_key, event_id FROM current_state_events
	86	WHERE room_id = ?
	87	""",
	88	(room_id,)
	89	)
	90
	91	return {
	92	(intern_string(r[0]), intern_string(r[1])): to_ascii(r[2]) for r in txn
	93	}
	94
	95	return self.runInteraction(
	96	"get_current_state_ids",
	97	_get_current_state_ids_txn,
	98	)
82	99
83	100	@defer.inlineCallbacks
84	101	def get_state_groups_ids(self, room_id, event_ids):

209	226	],
210	227	)
211	228
	229	# Prefill the state group cache with this group.
	230	# It's fine to use the sequence like this as the state group map
	231	# is immutable. (If the map wasn't immutable then this prefill could
	232	# race with another update)
	233	txn.call_after(
	234	self._state_group_cache.update,
	235	self._state_group_cache.sequence,
	236	key=context.state_group,
	237	value=dict(context.current_state_ids),
	238	full=True,
	239	)
	240
212	241	self._simple_insert_many_txn(
213	242	txn,
214	243	table="event_to_state_groups",

262	291
263	292	return count
264	293
265		@cached(num_args=2, max_entries=100000, iterable=True)
266		def _get_state_group_from_group(self, group, types):
267		raise NotImplementedError()
268
269		@cachedList(cached_method_name="_get_state_group_from_group",
270		list_name="groups", num_args=2, inlineCallbacks=True)
	294	@defer.inlineCallbacks
271	295	def _get_state_groups_from_groups(self, groups, types):
272	296	"""Returns dictionary state_group -> (dict of (type, state_key) -> event id)
273	297	"""

495	519	state_map = yield self.get_state_ids_for_events([event_id], types)
496	520	defer.returnValue(state_map[event_id])
497	521
498		@cached(num_args=2, max_entries=100000)
	522	@cached(num_args=2, max_entries=50000)
499	523	def _get_state_group_for_event(self, room_id, event_id):
500	524	return self._simple_select_one_onecol(
501	525	table="event_to_state_groups",

643	667	state_dict = results[group]
644	668
645	669	state_dict.update(
646		((intern_string(k[0]), intern_string(k[1])), v)
	670	((intern_string(k[0]), intern_string(k[1])), to_ascii(v))
647	671	for k, v in group_state_dict.iteritems()
648	672	)
649	673

+4

-6

synapse/types.py less more

55	55
56	56
57	57	def get_domain_from_id(string):
58		try:
59		return string.split(":", 1)[1]
60		except IndexError:
	58	idx = string.find(":")
	59	if idx == -1:
61	60	raise SynapseError(400, "Invalid ID: %r" % (string,))
	61	return string[idx + 1:]
62	62
63	63
64	64	class DomainSpecificString(

215	215	return self
216	216
217	217	def copy_and_replace(self, key, new_value):
218		d = self._asdict()
219		d[key] = new_value
220		return StreamToken(**d)
	218	return self._replace(**{key: new_value})
221	219
222	220
223	221	StreamToken.START = StreamToken(

+1

-1

synapse/util/__init__.py less more

25	25
26	26	class DeferredTimedOutError(SynapseError):
27	27	def __init__(self):
28		super(SynapseError, self).__init__(504, "Timed out")
	28	super(DeferredTimedOutError, self).__init__(504, "Timed out")
29	29
30	30
31	31	def unwrapFirstError(failure):

+6

-1

synapse/util/async.py less more

88	88	deferred.addCallbacks(callback, errback)
89	89
90	90	def observe(self):
	91	"""Observe the underlying deferred.
	92
	93	Can return either a deferred if the underlying deferred is still pending
	94	(or has failed), or the actual value. Callers may need to use maybeDeferred.
	95	"""
91	96	if not self._result:
92	97	d = defer.Deferred()
93	98

100	105	return d
101	106	else:
102	107	success, res = self._result
103		return defer.succeed(res) if success else defer.fail(res)
	108	return res if success else defer.fail(res)
104	109
105	110	def observers(self):
106	111	return self._observers

+11

-20

synapse/util/caches/__init__.py less more

13	13	# limitations under the License.
14	14
15	15	import synapse.metrics
16		from lrucache import LruCache
17	16	import os
18	17
19	18	CACHE_SIZE_FACTOR = float(os.environ.get("SYNAPSE_CACHE_FACTOR", 0.1))
20
21		DEBUG_CACHES = False
22	19
23	20	metrics = synapse.metrics.get_metrics_for("synapse.util.caches")
24	21

37	34	lambda: len(cache),
38	35	name,
39	36	)
40
41
42		_string_cache = LruCache(int(100000 * CACHE_SIZE_FACTOR))
43		_stirng_cache_metrics = register_cache("string_cache", _string_cache)
44	37
45	38
46	39	KNOWN_KEYS = {

66	59
67	60
68	61	def intern_string(string):
69		"""Takes a (potentially) unicode string and interns using custom cache
	62	"""Takes a (potentially) unicode string and interns it if it's ascii
70	63	"""
71		new_str = _string_cache.setdefault(string, string)
72		if new_str is string:
73		_stirng_cache_metrics.inc_hits()
74		else:
75		_stirng_cache_metrics.inc_misses()
76		return new_str
	64	if string is None:
	65	return None
	66
	67	try:
	68	string = string.encode("ascii")
	69	return intern(string)
	70	except UnicodeEncodeError:
	71	return string
77	72
78	73
79	74	def intern_dict(dictionary):

86	81
87	82
88	83	def _intern_known_values(key, value):
89		intern_str_keys = ("event_id", "room_id")
90		intern_unicode_keys = ("sender", "user_id", "type", "state_key")
	84	intern_keys = ("event_id", "room_id", "sender", "user_id", "type", "state_key",)
91	85
92		if key in intern_str_keys:
93		return intern(value.encode('ascii'))
94
95		if key in intern_unicode_keys:
	86	if key in intern_keys:
96	87	return intern_string(value)
97	88
98	89	return value

+95

-37

synapse/util/caches/descriptors.py less more

17	17	from synapse.util import unwrapFirstError, logcontext
18	18	from synapse.util.caches.lrucache import LruCache
19	19	from synapse.util.caches.treecache import TreeCache, iterate_tree_cache_entry
20
21		from . import DEBUG_CACHES, register_cache
	20	from synapse.util.stringutils import to_ascii
	21
	22	from . import register_cache
22	23
23	24	from twisted.internet import defer
24	25	from collections import namedtuple

75	76
76	77	self.cache = LruCache(
77	78	max_size=max_entries, keylen=keylen, cache_type=cache_type,
78		size_callback=(lambda d: len(d.result)) if iterable else None,
	79	size_callback=(lambda d: len(d)) if iterable else None,
79	80	)
80	81
81	82	self.name = name

94	95	"Cache objects can only be accessed from the main thread"
95	96	)
96	97
97		def get(self, key, default=_CacheSentinel, callback=None):
	98	def get(self, key, default=_CacheSentinel, callback=None, update_metrics=True):
	99	"""Looks the key up in the caches.
	100
	101	Args:
	102	key(tuple)
	103	default: What is returned if key is not in the caches. If not
	104	specified then function throws KeyError instead
	105	callback(fn): Gets called when the entry in the cache is invalidated
	106	update_metrics (bool): whether to update the cache hit rate metrics
	107
	108	Returns:
	109	Either a Deferred or the raw result
	110	"""
98	111	callbacks = [callback] if callback else []
99	112	val = self._pending_deferred_cache.get(key, _CacheSentinel)
100	113	if val is not _CacheSentinel:
101	114	if val.sequence == self.sequence:
102	115	val.callbacks.update(callbacks)
103		self.metrics.inc_hits()
	116	if update_metrics:
	117	self.metrics.inc_hits()
104	118	return val.deferred
105	119
106	120	val = self.cache.get(key, _CacheSentinel, callbacks=callbacks)

108	122	self.metrics.inc_hits()
109	123	return val
110	124
111		self.metrics.inc_misses()
	125	if update_metrics:
	126	self.metrics.inc_misses()
112	127
113	128	if default is _CacheSentinel:
114	129	raise KeyError()

136	151	if self.sequence == entry.sequence:
137	152	existing_entry = self._pending_deferred_cache.pop(key, None)
138	153	if existing_entry is entry:
139		self.cache.set(key, entry.deferred, entry.callbacks)
	154	self.cache.set(key, result, entry.callbacks)
140	155	else:
141	156	entry.invalidate()
142	157	else:

151	166
152	167	def invalidate(self, key):
153	168	self.check_thread()
154		if not isinstance(key, tuple):
155		raise TypeError(
156		"The cache key must be a tuple not %r" % (type(key),)
157		)
158	169
159	170	# Increment the sequence number so that any SELECT statements that
160	171	# raced with the INSERT don't update the cache (SYN-369)

223	234	)
224	235
225	236	self.num_args = num_args
	237
	238	# list of the names of the args used as the cache key
226	239	self.arg_names = all_args[1:num_args + 1]
	240
	241	# self.arg_defaults is a map of arg name to its default value for each
	242	# argument that has a default value
	243	if arg_spec.defaults:
	244	self.arg_defaults = dict(zip(
	245	all_args[-len(arg_spec.defaults):],
	246	arg_spec.defaults
	247	))
	248	else:
	249	self.arg_defaults = {}
227	250
228	251	if "cache_context" in self.arg_names:
229	252	raise Exception(

288	311	iterable=self.iterable,
289	312	)
290	313
	314	def get_cache_key_gen(args, kwargs):
	315	"""Given some args/kwargs return a generator that resolves into
	316	the cache_key.
	317
	318	We loop through each arg name, looking up if its in the `kwargs`,
	319	otherwise using the next argument in `args`. If there are no more
	320	args then we try looking the arg name up in the defaults
	321	"""
	322	pos = 0
	323	for nm in self.arg_names:
	324	if nm in kwargs:
	325	yield kwargs[nm]
	326	elif pos < len(args):
	327	yield args[pos]
	328	pos += 1
	329	else:
	330	yield self.arg_defaults[nm]
	331
	332	# By default our cache key is a tuple, but if there is only one item
	333	# then don't bother wrapping in a tuple. This is to save memory.
	334	if self.num_args == 1:
	335	nm = self.arg_names[0]
	336
	337	def get_cache_key(args, kwargs):
	338	if nm in kwargs:
	339	return kwargs[nm]
	340	elif len(args):
	341	return args[0]
	342	else:
	343	return self.arg_defaults[nm]
	344	else:
	345	def get_cache_key(args, kwargs):
	346	return tuple(get_cache_key_gen(args, kwargs))
	347
291	348	@functools.wraps(self.orig)
292	349	def wrapped(args, *kwargs):
293	350	# If we're passed a cache_context then we'll want to call its invalidate()
294	351	# whenever we are invalidated
295	352	invalidate_callback = kwargs.pop("on_invalidate", None)
296	353
297		# Add temp cache_context so inspect.getcallargs doesn't explode
298		if self.add_cache_context:
299		kwargs["cache_context"] = None
300
301		arg_dict = inspect.getcallargs(self.orig, obj, args, *kwargs)
302		cache_key = tuple(arg_dict[arg_nm] for arg_nm in self.arg_names)
	354	cache_key = get_cache_key(args, kwargs)
303	355
304	356	# Add our own `cache_context` to argument list if the wrapped function
305	357	# has asked for one

309	361	try:
310	362	cached_result_d = cache.get(cache_key, callback=invalidate_callback)
311	363
312		observer = cached_result_d.observe()
313		if DEBUG_CACHES:
314		@defer.inlineCallbacks
315		def check_result(cached_result):
316		actual_result = yield self.function_to_call(obj, args, *kwargs)
317		if actual_result != cached_result:
318		logger.error(
319		"Stale cache entry %s%r: cached: %r, actual %r",
320		self.orig.__name__, cache_key,
321		cached_result, actual_result,
322		)
323		raise ValueError("Stale cache entry")
324		defer.returnValue(cached_result)
325		observer.addCallback(check_result)
	364	if isinstance(cached_result_d, ObservableDeferred):
	365	observer = cached_result_d.observe()
	366	else:
	367	observer = cached_result_d
326	368
327	369	except KeyError:
328	370	ret = defer.maybeDeferred(

336	378
337	379	ret.addErrback(onErr)
338	380
	381	# If our cache_key is a string, try to convert to ascii to save
	382	# a bit of space in large caches
	383	if isinstance(cache_key, basestring):
	384	cache_key = to_ascii(cache_key)
	385
339	386	result_d = ObservableDeferred(ret, consumeErrors=True)
340	387	cache.set(cache_key, result_d, callback=invalidate_callback)
341	388	observer = result_d.observe()
342	389
343		return logcontext.make_deferred_yieldable(observer)
344
345		wrapped.invalidate = cache.invalidate
	390	if isinstance(observer, defer.Deferred):
	391	return logcontext.make_deferred_yieldable(observer)
	392	else:
	393	return observer
	394
	395	if self.num_args == 1:
	396	wrapped.invalidate = lambda key: cache.invalidate(key[0])
	397	wrapped.prefill = lambda key, val: cache.prefill(key[0], val)
	398	else:
	399	wrapped.invalidate = cache.invalidate
	400	wrapped.invalidate_all = cache.invalidate_all
	401	wrapped.invalidate_many = cache.invalidate_many
	402	wrapped.prefill = cache.prefill
	403
346	404	wrapped.invalidate_all = cache.invalidate_all
347		wrapped.invalidate_many = cache.invalidate_many
348		wrapped.prefill = cache.prefill
349	405	wrapped.cache = cache
350	406
351	407	obj.__dict__[self.orig.__name__] = wrapped

418	474
419	475	try:
420	476	res = cache.get(tuple(key), callback=invalidate_callback)
421		if not res.has_succeeded():
	477	if not isinstance(res, ObservableDeferred):
	478	results[arg] = res
	479	elif not res.has_succeeded():
422	480	res = res.observe()
423	481	res.addCallback(lambda r, arg: (arg, r), arg)
424	482	cached_defers[arg] = res

+1

-5

synapse/util/logcontext.py less more

333	333	LoggingContext.set_current_context(LoggingContext.sentinel)
334	334	return result
335	335
336		# XXX: why is this here rather than inside g? surely we want to preserve
337		# the context from the time the function was called, not when it was
338		# wrapped?
339		current = LoggingContext.current_context()
340
341	336	def g(args, *kwargs):
	337	current = LoggingContext.current_context()
342	338	res = f(args, *kwargs)
343	339	if isinstance(res, defer.Deferred) and not res.called:
344	340	# The function will have reset the context before returning, so

+14

-0

synapse/util/stringutils.py less more

39	39	return False
40	40	else:
41	41	return True
	42
	43
	44	def to_ascii(s):
	45	"""Converts a string to ascii if it is ascii, otherwise leave it alone.
	46
	47	If given None then will return None.
	48	"""
	49	if s is None:
	50	return None
	51
	52	try:
	53	return s.encode("ascii")
	54	except UnicodeEncodeError:
	55	return s

+2

-1

synapse/visibility.py less more

55	55	events ([synapse.events.EventBase]): list of events to filter
56	56	"""
57	57	forgotten = yield preserve_context_over_deferred(defer.gatherResults([
58		preserve_fn(store.who_forgot_in_room)(
	58	defer.maybeDeferred(
	59	preserve_fn(store.who_forgot_in_room),
59	60	room_id,
60	61	)
61	62	for room_id in frozenset(e.room_id for e in events)

+3

-1

tests/appservice/test_appservice.py less more

18	18	from mock import Mock
19	19	from tests import unittest
20	20
	21	import re
	22
21	23
22	24	def _regex(regex, exclusive=True):
23	25	return {
24		"regex": regex,
	26	"regex": re.compile(regex),
25	27	"exclusive": exclusive
26	28	}
27	29

+132

-0

tests/handlers/test_e2e_keys.py less more

13	13	# limitations under the License.
14	14
15	15	import mock
	16	from synapse.api import errors
16	17	from twisted.internet import defer
17	18
18	19	import synapse.api.errors

43	44	local_user = "@boris:" + self.hs.hostname
44	45	res = yield self.handler.query_local_devices({local_user: None})
45	46	self.assertDictEqual(res, {local_user: {}})
	47
	48	@defer.inlineCallbacks
	49	def test_reupload_one_time_keys(self):
	50	"""we should be able to re-upload the same keys"""
	51	local_user = "@boris:" + self.hs.hostname
	52	device_id = "xyz"
	53	keys = {
	54	"alg1:k1": "key1",
	55	"alg2:k2": {
	56	"key": "key2",
	57	"signatures": {"k1": "sig1"}
	58	},
	59	"alg2:k3": {
	60	"key": "key3",
	61	},
	62	}
	63
	64	res = yield self.handler.upload_keys_for_user(
	65	local_user, device_id, {"one_time_keys": keys},
	66	)
	67	self.assertDictEqual(res, {
	68	"one_time_key_counts": {"alg1": 1, "alg2": 2}
	69	})
	70
	71	# we should be able to change the signature without a problem
	72	keys["alg2:k2"]["signatures"]["k1"] = "sig2"
	73	res = yield self.handler.upload_keys_for_user(
	74	local_user, device_id, {"one_time_keys": keys},
	75	)
	76	self.assertDictEqual(res, {
	77	"one_time_key_counts": {"alg1": 1, "alg2": 2}
	78	})
	79
	80	@defer.inlineCallbacks
	81	def test_change_one_time_keys(self):
	82	"""attempts to change one-time-keys should be rejected"""
	83
	84	local_user = "@boris:" + self.hs.hostname
	85	device_id = "xyz"
	86	keys = {
	87	"alg1:k1": "key1",
	88	"alg2:k2": {
	89	"key": "key2",
	90	"signatures": {"k1": "sig1"}
	91	},
	92	"alg2:k3": {
	93	"key": "key3",
	94	},
	95	}
	96
	97	res = yield self.handler.upload_keys_for_user(
	98	local_user, device_id, {"one_time_keys": keys},
	99	)
	100	self.assertDictEqual(res, {
	101	"one_time_key_counts": {"alg1": 1, "alg2": 2}
	102	})
	103
	104	try:
	105	yield self.handler.upload_keys_for_user(
	106	local_user, device_id, {"one_time_keys": {"alg1:k1": "key2"}},
	107	)
	108	self.fail("No error when changing string key")
	109	except errors.SynapseError:
	110	pass
	111
	112	try:
	113	yield self.handler.upload_keys_for_user(
	114	local_user, device_id, {"one_time_keys": {"alg2:k3": "key2"}},
	115	)
	116	self.fail("No error when replacing dict key with string")
	117	except errors.SynapseError:
	118	pass
	119
	120	try:
	121	yield self.handler.upload_keys_for_user(
	122	local_user, device_id, {
	123	"one_time_keys": {"alg1:k1": {"key": "key"}}
	124	},
	125	)
	126	self.fail("No error when replacing string key with dict")
	127	except errors.SynapseError:
	128	pass
	129
	130	try:
	131	yield self.handler.upload_keys_for_user(
	132	local_user, device_id, {
	133	"one_time_keys": {
	134	"alg2:k2": {
	135	"key": "key3",
	136	"signatures": {"k1": "sig1"},
	137	}
	138	},
	139	},
	140	)
	141	self.fail("No error when replacing dict key")
	142	except errors.SynapseError:
	143	pass
	144
	145	@unittest.DEBUG
	146	@defer.inlineCallbacks
	147	def test_claim_one_time_key(self):
	148	local_user = "@boris:" + self.hs.hostname
	149	device_id = "xyz"
	150	keys = {
	151	"alg1:k1": "key1",
	152	}
	153
	154	res = yield self.handler.upload_keys_for_user(
	155	local_user, device_id, {"one_time_keys": keys},
	156	)
	157	self.assertDictEqual(res, {
	158	"one_time_key_counts": {"alg1": 1}
	159	})
	160
	161	res2 = yield self.handler.claim_one_time_keys({
	162	"one_time_keys": {
	163	local_user: {
	164	device_id: "alg1"
	165	}
	166	}
	167	}, timeout=None)
	168	self.assertEqual(res2, {
	169	"failures": {},
	170	"one_time_keys": {
	171	local_user: {
	172	device_id: {
	173	"alg1:k1": "key1"
	174	}
	175	}
	176	}
	177	})

+22

-8

tests/replication/slave/storage/_base.py less more

11	11	# See the License for the specific language governing permissions and
12	12	# limitations under the License.
13	13
14		from twisted.internet import defer
	14	from twisted.internet import defer, reactor
15	15	from tests import unittest
16	16
17	17	from mock import Mock, NonCallableMock
18	18	from tests.utils import setup_test_homeserver
19		from synapse.replication.resource import ReplicationResource
	19	from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
	20	from synapse.replication.tcp.client import (
	21	ReplicationClientHandler, ReplicationClientFactory,
	22	)
20	23
21	24
22	25	class BaseSlavedStoreTestCase(unittest.TestCase):

32	35	)
33	36	self.hs.get_ratelimiter().send_message.return_value = (True, 0)
34	37
35		self.replication = ReplicationResource(self.hs)
36
37	38	self.master_store = self.hs.get_datastore()
38	39	self.slaved_store = self.STORE_TYPE(self.hs.get_db_conn(), self.hs)
39	40	self.event_id = 0
40	41
	42	server_factory = ReplicationStreamProtocolFactory(self.hs)
	43	listener = reactor.listenUNIX("\0xxx", server_factory)
	44	self.addCleanup(listener.stopListening)
	45	self.streamer = server_factory.streamer
	46
	47	self.replication_handler = ReplicationClientHandler(self.slaved_store)
	48	client_factory = ReplicationClientFactory(
	49	self.hs, "client_name", self.replication_handler
	50	)
	51	client_connector = reactor.connectUNIX("\0xxx", client_factory)
	52	self.addCleanup(client_factory.stopTrying)
	53	self.addCleanup(client_connector.disconnect)
	54
41	55	@defer.inlineCallbacks
42	56	def replicate(self):
43		streams = self.slaved_store.stream_positions()
44		writer = yield self.replication.replicate(streams, 100)
45		result = writer.finish()
46		yield self.slaved_store.process_replication(result)
	57	yield self.streamer.on_notifier_poke()
	58	d = self.replication_handler.await_sync("replication_test")
	59	self.streamer.send_sync_to_all_connections("replication_test")
	60	yield d
47	61
48	62	@defer.inlineCallbacks
49	63	def check(self, method, args, expected_result=None):

+0

-204

~~tests/replication/test_resource.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import contextlib
16		import json
17
18		from mock import Mock, NonCallableMock
19		from twisted.internet import defer
20
21		import synapse.types
22		from synapse.replication.resource import ReplicationResource
23		from synapse.types import UserID
24		from tests import unittest
25		from tests.utils import setup_test_homeserver
26
27
28		class ReplicationResourceCase(unittest.TestCase):
29		@defer.inlineCallbacks
30		def setUp(self):
31		self.hs = yield setup_test_homeserver(
32		"red",
33		http_client=None,
34		replication_layer=Mock(),
35		ratelimiter=NonCallableMock(spec_set=[
36		"send_message",
37		]),
38		)
39		self.user_id = "@seeing:red"
40		self.user = UserID.from_string(self.user_id)
41
42		self.hs.get_ratelimiter().send_message.return_value = (True, 0)
43
44		self.resource = ReplicationResource(self.hs)
45
46		@defer.inlineCallbacks
47		def test_streams(self):
48		# Passing "-1" returns the current stream positions
49		code, body = yield self.get(streams="-1")
50		self.assertEquals(code, 200)
51		self.assertEquals(body["streams"]["field_names"], ["name", "position"])
52		position = body["streams"]["position"]
53		# Passing the current position returns an empty response after the
54		# timeout
55		get = self.get(streams=str(position), timeout="0")
56		self.hs.clock.advance_time_msec(1)
57		code, body = yield get
58		self.assertEquals(code, 200)
59		self.assertEquals(body, {})
60
61		@defer.inlineCallbacks
62		def test_events(self):
63		get = self.get(events="-1", timeout="0")
64		yield self.hs.get_handlers().room_creation_handler.create_room(
65		synapse.types.create_requester(self.user), {}
66		)
67		code, body = yield get
68		self.assertEquals(code, 200)
69		self.assertEquals(body["events"]["field_names"], [
70		"position", "event_id", "room_id", "type", "state_key",
71		])
72
73		@defer.inlineCallbacks
74		def test_presence(self):
75		get = self.get(presence="-1")
76		yield self.hs.get_presence_handler().set_state(
77		self.user, {"presence": "online"}
78		)
79		code, body = yield get
80		self.assertEquals(code, 200)
81		self.assertEquals(body["presence"]["field_names"], [
82		"position", "user_id", "state", "last_active_ts",
83		"last_federation_update_ts", "last_user_sync_ts",
84		"status_msg", "currently_active",
85		])
86
87		@defer.inlineCallbacks
88		def test_typing(self):
89		room_id = yield self.create_room()
90		get = self.get(typing="-1")
91		yield self.hs.get_typing_handler().started_typing(
92		self.user, self.user, room_id, timeout=2
93		)
94		code, body = yield get
95		self.assertEquals(code, 200)
96		self.assertEquals(body["typing"]["field_names"], [
97		"position", "room_id", "typing"
98		])
99
100		@defer.inlineCallbacks
101		def test_receipts(self):
102		room_id = yield self.create_room()
103		event_id = yield self.send_text_message(room_id, "Hello, World")
104		get = self.get(receipts="-1")
105		yield self.hs.get_receipts_handler().received_client_receipt(
106		room_id, "m.read", self.user_id, event_id
107		)
108		code, body = yield get
109		self.assertEquals(code, 200)
110		self.assertEquals(body["receipts"]["field_names"], [
111		"position", "room_id", "receipt_type", "user_id", "event_id", "data"
112		])
113
114		def _test_timeout(stream):
115		"""Check that a request for the given stream timesout"""
116		@defer.inlineCallbacks
117		def test_timeout(self):
118		get = self.get(**{stream: "-1", "timeout": "0"})
119		self.hs.clock.advance_time_msec(1)
120		code, body = yield get
121		self.assertEquals(code, 200)
122		self.assertEquals(body.get("rows", []), [])
123		test_timeout.__name__ = "test_timeout_%s" % (stream)
124		return test_timeout
125
126		test_timeout_events = _test_timeout("events")
127		test_timeout_presence = _test_timeout("presence")
128		test_timeout_typing = _test_timeout("typing")
129		test_timeout_receipts = _test_timeout("receipts")
130		test_timeout_user_account_data = _test_timeout("user_account_data")
131		test_timeout_room_account_data = _test_timeout("room_account_data")
132		test_timeout_tag_account_data = _test_timeout("tag_account_data")
133		test_timeout_backfill = _test_timeout("backfill")
134		test_timeout_push_rules = _test_timeout("push_rules")
135		test_timeout_pushers = _test_timeout("pushers")
136		test_timeout_state = _test_timeout("state")
137
138		@defer.inlineCallbacks
139		def send_text_message(self, room_id, message):
140		handler = self.hs.get_handlers().message_handler
141		event = yield handler.create_and_send_nonmember_event(
142		synapse.types.create_requester(self.user),
143		{
144		"type": "m.room.message",
145		"content": {"body": "message", "msgtype": "m.text"},
146		"room_id": room_id,
147		"sender": self.user.to_string(),
148		}
149		)
150		defer.returnValue(event.event_id)
151
152		@defer.inlineCallbacks
153		def create_room(self):
154		result = yield self.hs.get_handlers().room_creation_handler.create_room(
155		synapse.types.create_requester(self.user), {}
156		)
157		defer.returnValue(result["room_id"])
158
159		@defer.inlineCallbacks
160		def get(self, **params):
161		request = NonCallableMock(spec_set=[
162		"write", "finish", "setResponseCode", "setHeader", "args",
163		"method", "processing"
164		])
165
166		request.method = "GET"
167		request.args = {k: [v] for k, v in params.items()}
168
169		@contextlib.contextmanager
170		def processing():
171		yield
172		request.processing = processing
173
174		yield self.resource._async_render_GET(request)
175		self.assertTrue(request.finish.called)
176
177		if request.setResponseCode.called:
178		response_code = request.setResponseCode.call_args[0][0]
179		else:
180		response_code = 200
181
182		response_json = "".join(
183		call[0][0] for call in request.write.call_args_list
184		)
185		response_body = json.loads(response_json)
186
187		if response_code == 200:
188		self.check_response(response_body)
189
190		defer.returnValue((response_code, response_body))
191
192		def check_response(self, response_body):
193		for name, stream in response_body.items():
194		self.assertIn("field_names", stream)
195		field_names = stream["field_names"]
196		self.assertIn("rows", stream)
197		for row in stream["rows"]:
198		self.assertEquals(
199		len(row), len(field_names),
200		"%s: len(row = %r) == len(field_names = %r)" % (
201		name, row, field_names
202		)
203		)

+2

-2

tests/storage/event_injector.py less more

26	26	self.event_builder_factory = hs.get_event_builder_factory()
27	27
28	28	@defer.inlineCallbacks
29		def create_room(self, room):
	29	def create_room(self, room, user):
30	30	builder = self.event_builder_factory.new({
31	31	"type": EventTypes.Create,
32		"sender": "",
	32	"sender": user.to_string(),
33	33	"room_id": room.to_string(),
34	34	"content": {},
35	35	})

+1

-1

tests/storage/test__base.py less more

198	198
199	199	a.func.prefill(("foo",), ObservableDeferred(d))
200	200
201		self.assertEquals(a.func("foo").result, d.result)
	201	self.assertEquals(a.func("foo"), d.result)
202	202	self.assertEquals(callcount[0], 0)
203	203
204	204	@defer.inlineCallbacks

+1

-1

tests/storage/test_events.py less more

49	49	# Create something to report
50	50	room = RoomID.from_string("!abc123:test")
51	51	user = UserID.from_string("@raccoonlover:test")
52		yield self.event_injector.create_room(room)
	52	yield self.event_injector.create_room(room, user)
53	53
54	54	self.base_event = yield self._get_last_stream_token()
55	55

+38

-0

tests/util/caches/test_descriptors.py less more

174	174	logcontext.LoggingContext.sentinel)
175	175
176	176	return d1
	177
	178	@defer.inlineCallbacks
	179	def test_cache_default_args(self):
	180	class Cls(object):
	181	def __init__(self):
	182	self.mock = mock.Mock()
	183
	184	@descriptors.cached()
	185	def fn(self, arg1, arg2=2, arg3=3):
	186	return self.mock(arg1, arg2, arg3)
	187
	188	obj = Cls()
	189
	190	obj.mock.return_value = 'fish'
	191	r = yield obj.fn(1, 2, 3)
	192	self.assertEqual(r, 'fish')
	193	obj.mock.assert_called_once_with(1, 2, 3)
	194	obj.mock.reset_mock()
	195
	196	# a call with same params shouldn't call the mock again
	197	r = yield obj.fn(1, 2)
	198	self.assertEqual(r, 'fish')
	199	obj.mock.assert_not_called()
	200	obj.mock.reset_mock()
	201
	202	# a call with different params should call the mock again
	203	obj.mock.return_value = 'chips'
	204	r = yield obj.fn(2, 3)
	205	self.assertEqual(r, 'chips')
	206	obj.mock.assert_called_once_with(2, 3, 3)
	207	obj.mock.reset_mock()
	208
	209	# the two values should now be cached
	210	r = yield obj.fn(1, 2)
	211	self.assertEqual(r, 'fish')
	212	r = yield obj.fn(2, 3)
	213	self.assertEqual(r, 'chips')
	214	obj.mock.assert_not_called()

+3

-1

tests/util/test_snapshot_cache.py less more

52	52	# before the cache expires returns a resolved deferred.
53	53	get_result_at_11 = self.cache.get(11, "key")
54	54	self.assertIsNotNone(get_result_at_11)
55		self.assertTrue(get_result_at_11.called)
	55	if isinstance(get_result_at_11, Deferred):
	56	# The cache may return the actual result rather than a deferred
	57	self.assertTrue(get_result_at_11.called)
56	58
57	59	# Check that getting the key after the deferred has resolved
58	60	# after the cache expires returns None