Commit b5f99d18c37055bedcc966c16ee1e25013154799 - matrix-synapse

.buildkite/test_db.db less more

Binary diff not shown

+180

-0

CHANGES.md less more

	0	Synapse 1.20.1 (2020-09-24)
	1	===========================
	2
	3	Bugfixes
	4	--------
	5
	6	- Fix a bug introduced in v1.20.0 which caused the `synapse_port_db` script to fail. ([\#8386](https://github.com/matrix-org/synapse/issues/8386))
	7	- Fix a bug introduced in v1.20.0 which caused variables to be incorrectly escaped in Jinja2 templates. ([\#8394](https://github.com/matrix-org/synapse/issues/8394))
	8
	9
	10	Synapse 1.20.0 (2020-09-22)
	11	===========================
	12
	13	No significant changes since v1.20.0rc5.
	14
	15	Removal warning
	16	---------------
	17
	18	Historically, the [Synapse Admin
	19	API](https://github.com/matrix-org/synapse/tree/master/docs) has been
	20	accessible under the `/_matrix/client/api/v1/admin`,
	21	`/_matrix/client/unstable/admin`, `/_matrix/client/r0/admin` and
	22	`/_synapse/admin` prefixes. In a future release, we will be dropping support
	23	for accessing Synapse's Admin API using the `/_matrix/client/*` prefixes. This
	24	makes it easier for homeserver admins to lock down external access to the Admin
	25	API endpoints.
	26
	27	Synapse 1.20.0rc5 (2020-09-18)
	28	==============================
	29
	30	In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3.
	31
	32	Features
	33	--------
	34
	35	- Add flags to the `/versions` endpoint for whether new rooms default to using E2EE. ([\#8343](https://github.com/matrix-org/synapse/issues/8343))
	36
	37
	38	Bugfixes
	39	--------
	40
	41	- Fix rate limiting of federation `/send` requests. ([\#8342](https://github.com/matrix-org/synapse/issues/8342))
	42	- Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. ([\#8349](https://github.com/matrix-org/synapse/issues/8349))
	43
	44
	45	Internal Changes
	46	----------------
	47
	48	- Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented. ([\#8285](https://github.com/matrix-org/synapse/issues/8285))
	49
	50
	51	Synapse 1.19.3 (2020-09-18)
	52	===========================
	53
	54	Bugfixes
	55	--------
	56
	57	- Partially mitigate bug where newly joined servers couldn't get past events in a room when there is a malformed event. ([\#8350](https://github.com/matrix-org/synapse/issues/8350))
	58
	59
	60	Synapse 1.20.0rc4 (2020-09-16)
	61	==============================
	62
	63	Synapse 1.20.0rc4 is identical to 1.20.0rc3, with the addition of the security fix that was included in 1.19.2.
	64
	65
	66	Synapse 1.19.2 (2020-09-16)
	67	===========================
	68
	69	Due to the issue below server admins are encouraged to upgrade as soon as possible.
	70
	71	Bugfixes
	72	--------
	73
	74	- Fix joining rooms over federation that include malformed events. ([\#8324](https://github.com/matrix-org/synapse/issues/8324))
	75
	76
	77	Synapse 1.20.0rc3 (2020-09-11)
	78	==============================
	79
	80	Bugfixes
	81	--------
	82
	83	- Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
	84
	85
	86	Synapse 1.20.0rc2 (2020-09-09)
	87	==============================
	88
	89	Bugfixes
	90	--------
	91
	92	- Fix a bug introduced in v1.20.0rc1 causing some features related to notifications to misbehave following the implementation of unread counts. ([\#8280](https://github.com/matrix-org/synapse/issues/8280))
	93
	94
	95	Synapse 1.20.0rc1 (2020-09-08)
	96	==============================
	97
	98	Removal warning
	99	---------------
	100
	101	Some older clients used a [disallowed character](https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-register-email-requesttoken) (`:`) in the `client_secret` parameter of various endpoints. The incorrect behaviour was allowed for backwards compatibility, but is now being removed from Synapse as most users have updated their client. Further context can be found at [\#6766](https://github.com/matrix-org/synapse/issues/6766).
	102
	103	Features
	104	--------
	105
	106	- Add an endpoint to query your shared rooms with another user as an implementation of [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666). ([\#7785](https://github.com/matrix-org/synapse/issues/7785))
	107	- Iteratively encode JSON to avoid blocking the reactor. ([\#8013](https://github.com/matrix-org/synapse/issues/8013), [\#8116](https://github.com/matrix-org/synapse/issues/8116))
	108	- Add support for shadow-banning users (ignoring any message send requests). ([\#8034](https://github.com/matrix-org/synapse/issues/8034), [\#8092](https://github.com/matrix-org/synapse/issues/8092), [\#8095](https://github.com/matrix-org/synapse/issues/8095), [\#8142](https://github.com/matrix-org/synapse/issues/8142), [\#8152](https://github.com/matrix-org/synapse/issues/8152), [\#8157](https://github.com/matrix-org/synapse/issues/8157), [\#8158](https://github.com/matrix-org/synapse/issues/8158), [\#8176](https://github.com/matrix-org/synapse/issues/8176))
	109	- Use the default template file when its equivalent is not found in a custom template directory. ([\#8037](https://github.com/matrix-org/synapse/issues/8037), [\#8107](https://github.com/matrix-org/synapse/issues/8107), [\#8252](https://github.com/matrix-org/synapse/issues/8252))
	110	- Add unread messages count to sync responses, as specified in [MSC2654](https://github.com/matrix-org/matrix-doc/pull/2654). ([\#8059](https://github.com/matrix-org/synapse/issues/8059), [\#8254](https://github.com/matrix-org/synapse/issues/8254), [\#8270](https://github.com/matrix-org/synapse/issues/8270), [\#8274](https://github.com/matrix-org/synapse/issues/8274))
	111	- Optimise `/federation/v1/user/devices/` API by only returning devices with encryption keys. ([\#8198](https://github.com/matrix-org/synapse/issues/8198))
	112
	113
	114	Bugfixes
	115	--------
	116
	117	- Fix a memory leak by limiting the length of time that messages will be queued for a remote server that has been unreachable. ([\#7864](https://github.com/matrix-org/synapse/issues/7864))
	118	- Fix `Re-starting finished log context PUT-nnnn` warning when event persistence failed. ([\#8081](https://github.com/matrix-org/synapse/issues/8081))
	119	- Synapse now correctly enforces the valid characters in the `client_secret` parameter used in various endpoints. ([\#8101](https://github.com/matrix-org/synapse/issues/8101))
	120	- Fix a bug introduced in v1.7.2 impacting message retention policies that would allow federated homeservers to dictate a retention period that's lower than the configured minimum allowed duration in the configuration file. ([\#8104](https://github.com/matrix-org/synapse/issues/8104))
	121	- Fix a long-standing bug where invalid JSON would be accepted by Synapse. ([\#8106](https://github.com/matrix-org/synapse/issues/8106))
	122	- Fix a bug introduced in Synapse v1.12.0 which could cause `/sync` requests to fail with a 404 if you had a very old outstanding room invite. ([\#8110](https://github.com/matrix-org/synapse/issues/8110))
	123	- Return a proper error code when the rooms of an invalid group are requested. ([\#8129](https://github.com/matrix-org/synapse/issues/8129))
	124	- Fix a bug which could cause a leaked postgres connection if synapse was set to daemonize. ([\#8131](https://github.com/matrix-org/synapse/issues/8131))
	125	- Clarify the error code if a user tries to register with a numeric ID. This bug was introduced in v1.15.0. ([\#8135](https://github.com/matrix-org/synapse/issues/8135))
	126	- Fix a bug where appservices with ratelimiting disabled would still be ratelimited when joining rooms. This bug was introduced in v1.19.0. ([\#8139](https://github.com/matrix-org/synapse/issues/8139))
	127	- Fix logging in via OpenID Connect with a provider that uses integer user IDs. ([\#8190](https://github.com/matrix-org/synapse/issues/8190))
	128	- Fix a longstanding bug where user directory updates could break when unexpected profile data was included in events. ([\#8223](https://github.com/matrix-org/synapse/issues/8223))
	129	- Fix a longstanding bug where stats updates could break when unexpected profile data was included in events. ([\#8226](https://github.com/matrix-org/synapse/issues/8226))
	130	- Fix slow start times for large servers by removing a table scan of the `users` table from startup code. ([\#8271](https://github.com/matrix-org/synapse/issues/8271))
	131
	132
	133	Updates to the Docker image
	134	---------------------------
	135
	136	- Fix builds of the Docker image on non-x86 platforms. ([\#8144](https://github.com/matrix-org/synapse/issues/8144))
	137	- Added curl for healthcheck support and readme updates for the change. Contributed by @maquis196. ([\#8147](https://github.com/matrix-org/synapse/issues/8147))
	138
	139
	140	Improved Documentation
	141	----------------------
	142
	143	- Link to matrix-synapse-rest-password-provider in the password provider documentation. ([\#8111](https://github.com/matrix-org/synapse/issues/8111))
	144	- Updated documentation to note that Synapse does not follow `HTTP 308` redirects due to an upstream library not supporting them. Contributed by Ryan Cole. ([\#8120](https://github.com/matrix-org/synapse/issues/8120))
	145	- Explain better what GDPR-erased means when deactivating a user. ([\#8189](https://github.com/matrix-org/synapse/issues/8189))
	146
	147
	148	Internal Changes
	149	----------------
	150
	151	- Add filter `name` to the `/users` admin API, which filters by user ID or displayname. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#7377](https://github.com/matrix-org/synapse/issues/7377), [\#8163](https://github.com/matrix-org/synapse/issues/8163))
	152	- Reduce run times of some unit tests by advancing the reactor a fewer number of times. ([\#7757](https://github.com/matrix-org/synapse/issues/7757))
	153	- Don't fail `/submit_token` requests on incorrect session ID if `request_token_inhibit_3pid_errors` is turned on. ([\#7991](https://github.com/matrix-org/synapse/issues/7991))
	154	- Convert various parts of the codebase to async/await. ([\#8071](https://github.com/matrix-org/synapse/issues/8071), [\#8072](https://github.com/matrix-org/synapse/issues/8072), [\#8074](https://github.com/matrix-org/synapse/issues/8074), [\#8075](https://github.com/matrix-org/synapse/issues/8075), [\#8076](https://github.com/matrix-org/synapse/issues/8076), [\#8087](https://github.com/matrix-org/synapse/issues/8087), [\#8100](https://github.com/matrix-org/synapse/issues/8100), [\#8119](https://github.com/matrix-org/synapse/issues/8119), [\#8121](https://github.com/matrix-org/synapse/issues/8121), [\#8133](https://github.com/matrix-org/synapse/issues/8133), [\#8156](https://github.com/matrix-org/synapse/issues/8156), [\#8162](https://github.com/matrix-org/synapse/issues/8162), [\#8166](https://github.com/matrix-org/synapse/issues/8166), [\#8168](https://github.com/matrix-org/synapse/issues/8168), [\#8173](https://github.com/matrix-org/synapse/issues/8173), [\#8191](https://github.com/matrix-org/synapse/issues/8191), [\#8192](https://github.com/matrix-org/synapse/issues/8192), [\#8193](https://github.com/matrix-org/synapse/issues/8193), [\#8194](https://github.com/matrix-org/synapse/issues/8194), [\#8195](https://github.com/matrix-org/synapse/issues/8195), [\#8197](https://github.com/matrix-org/synapse/issues/8197), [\#8199](https://github.com/matrix-org/synapse/issues/8199), [\#8200](https://github.com/matrix-org/synapse/issues/8200), [\#8201](https://github.com/matrix-org/synapse/issues/8201), [\#8202](https://github.com/matrix-org/synapse/issues/8202), [\#8207](https://github.com/matrix-org/synapse/issues/8207), [\#8213](https://github.com/matrix-org/synapse/issues/8213), [\#8214](https://github.com/matrix-org/synapse/issues/8214))
	155	- Remove some unused database functions. ([\#8085](https://github.com/matrix-org/synapse/issues/8085))
	156	- Add type hints to various parts of the codebase. ([\#8090](https://github.com/matrix-org/synapse/issues/8090), [\#8127](https://github.com/matrix-org/synapse/issues/8127), [\#8187](https://github.com/matrix-org/synapse/issues/8187), [\#8241](https://github.com/matrix-org/synapse/issues/8241), [\#8140](https://github.com/matrix-org/synapse/issues/8140), [\#8183](https://github.com/matrix-org/synapse/issues/8183), [\#8232](https://github.com/matrix-org/synapse/issues/8232), [\#8235](https://github.com/matrix-org/synapse/issues/8235), [\#8237](https://github.com/matrix-org/synapse/issues/8237), [\#8244](https://github.com/matrix-org/synapse/issues/8244))
	157	- Return the previous stream token if a non-member event is a duplicate. ([\#8093](https://github.com/matrix-org/synapse/issues/8093), [\#8112](https://github.com/matrix-org/synapse/issues/8112))
	158	- Separate `get_current_token` into two since there are two different use cases for it. ([\#8113](https://github.com/matrix-org/synapse/issues/8113))
	159	- Remove `ChainedIdGenerator`. ([\#8123](https://github.com/matrix-org/synapse/issues/8123))
	160	- Reduce the amount of whitespace in JSON stored and sent in responses. ([\#8124](https://github.com/matrix-org/synapse/issues/8124))
	161	- Update the test federation client to handle streaming responses. ([\#8130](https://github.com/matrix-org/synapse/issues/8130))
	162	- Micro-optimisations to `get_auth_chain_ids`. ([\#8132](https://github.com/matrix-org/synapse/issues/8132))
	163	- Refactor `StreamIdGenerator` and `MultiWriterIdGenerator` to have the same interface. ([\#8161](https://github.com/matrix-org/synapse/issues/8161))
	164	- Add functions to `MultiWriterIdGen` used by events stream. ([\#8164](https://github.com/matrix-org/synapse/issues/8164), [\#8179](https://github.com/matrix-org/synapse/issues/8179))
	165	- Fix tests that were broken due to the merge of 1.19.1. ([\#8167](https://github.com/matrix-org/synapse/issues/8167))
	166	- Make `SlavedIdTracker.advance` have the same interface as `MultiWriterIDGenerator`. ([\#8171](https://github.com/matrix-org/synapse/issues/8171))
	167	- Remove unused `is_guest` parameter from, and add safeguard to, `MessageHandler.get_room_data`. ([\#8174](https://github.com/matrix-org/synapse/issues/8174), [\#8181](https://github.com/matrix-org/synapse/issues/8181))
	168	- Standardize the mypy configuration. ([\#8175](https://github.com/matrix-org/synapse/issues/8175))
	169	- Refactor some of `LoginRestServlet`'s helper methods, and move them to `AuthHandler` for easier reuse. ([\#8182](https://github.com/matrix-org/synapse/issues/8182))
	170	- Fix `wait_for_stream_position` to allow multiple waiters on same stream ID. ([\#8196](https://github.com/matrix-org/synapse/issues/8196))
	171	- Make `MultiWriterIDGenerator` work for streams that use negative values. ([\#8203](https://github.com/matrix-org/synapse/issues/8203))
	172	- Refactor queries for device keys and cross-signatures. ([\#8204](https://github.com/matrix-org/synapse/issues/8204), [\#8205](https://github.com/matrix-org/synapse/issues/8205), [\#8222](https://github.com/matrix-org/synapse/issues/8222), [\#8224](https://github.com/matrix-org/synapse/issues/8224), [\#8225](https://github.com/matrix-org/synapse/issues/8225), [\#8231](https://github.com/matrix-org/synapse/issues/8231), [\#8233](https://github.com/matrix-org/synapse/issues/8233), [\#8234](https://github.com/matrix-org/synapse/issues/8234))
	173	- Fix type hints for functions decorated with `@cached`. ([\#8240](https://github.com/matrix-org/synapse/issues/8240))
	174	- Remove obsolete `order` field from federation send queues. ([\#8245](https://github.com/matrix-org/synapse/issues/8245))
	175	- Stop sub-classing from object. ([\#8249](https://github.com/matrix-org/synapse/issues/8249))
	176	- Add more logging to debug slow startup. ([\#8264](https://github.com/matrix-org/synapse/issues/8264))
	177	- Do not attempt to upgrade database schema on worker processes. ([\#8266](https://github.com/matrix-org/synapse/issues/8266), [\#8276](https://github.com/matrix-org/synapse/issues/8276))
	178
	179
0	180	Synapse 1.19.1 (2020-08-27)
1	181	===========================
2	182

+3

-3

contrib/cmdclient/http.py less more

23	23	from twisted.web.http_headers import Headers
24	24
25	25
26		class HttpClient(object):
	26	class HttpClient:
27	27	""" Interface for talking json over http
28	28	"""
29	29

168	168	return d
169	169
170	170
171		class _RawProducer(object):
	171	class _RawProducer:
172	172	def __init__(self, data):
173	173	self.data = data
174	174	self.body = data

185	185	pass
186	186
187	187
188		class _JsonProducer(object):
	188	class _JsonProducer:
189	189	""" Used by the twisted http client to create the HTTP body from json
190	190	"""
191	191

+1

-1

contrib/experiments/cursesio.py less more

140	140	curses.endwin()
141	141
142	142
143		class Callback(object):
	143	class Callback:
144	144	def __init__(self, stdio):
145	145	self.stdio = stdio
146	146

+2

-2

contrib/experiments/test_messaging.py less more

54	54	logging.exception(failure)
55	55
56	56
57		class InputOutput(object):
	57	class InputOutput:
58	58	""" This is responsible for basic I/O so that a user can interact with
59	59	the example app.
60	60	"""

131	131	self.io.print_log(msg)
132	132
133	133
134		class Room(object):
	134	class Room:
135	135	""" Used to store (in memory) the current membership state of a room, and
136	136	which home servers we should send PDUs associated with the room to.
137	137	"""

+25

-0

debian/changelog less more

	0	matrix-synapse (1.20.1-1) unstable; urgency=medium
	1
	2	* New upstream release.
	3
	4	-- Andrej Shadura <andrewsh@debian.org> Fri, 25 Sep 2020 18:06:35 +0200
	5
	6	matrix-synapse (1.20.0-1) unstable; urgency=medium
	7
	8	* New upstream release.
	9	* Bump python3-canonicaljson version.
	10
	11	-- Andrej Shadura <andrewsh@debian.org> Wed, 23 Sep 2020 10:03:06 +0200
	12
	13	matrix-synapse (1.19.3-1) unstable; urgency=high
	14
	15	* New upstream release.
	16
	17	-- Andrej Shadura <andrewsh@debian.org> Sat, 19 Sep 2020 14:22:15 +0300
	18
	19	matrix-synapse (1.19.2-1) unstable; urgency=high
	20
	21	* New upstream release.
	22
	23	-- Andrej Shadura <andrewsh@debian.org> Wed, 16 Sep 2020 17:44:26 +0300
	24
0	25	matrix-synapse (1.19.1-1~bpo10+3) buster-backports; urgency=high
1	26
2	27	* Backport an upstream fix:

+1

-1

debian/control less more

12	12	python3-bcrypt,
13	13	python3-bleach (>= 1.4.2),
14	14	python3-blist,
15		python3-canonicaljson (>= 1.2.0),
	15	python3-canonicaljson (>= 1.3.0~),
16	16	python3-daemonize,
17	17	python3-frozendict (>= 1),
18	18	python3-idna,

+14

-3

docker/Dockerfile less more

18	18	FROM docker.io/python:${PYTHON_VERSION}-slim as builder
19	19
20	20	# install the OS build deps
21
22
23	21	RUN apt-get update && apt-get install -y \
24	22	build-essential \
	23	libffi-dev \
	24	libjpeg-dev \
25	25	libpq-dev \
	26	libssl-dev \
	27	libwebp-dev \
	28	libxml++2.6-dev \
	29	libxslt1-dev \
	30	zlib1g-dev \
26	31	&& rm -rf /var/lib/apt/lists/*
27	32
28	33	# Build dependencies that are not available as wheels, to speed up rebuilds

54	59	FROM docker.io/python:${PYTHON_VERSION}-slim
55	60
56	61	RUN apt-get update && apt-get install -y \
	62	curl \
	63	gosu \
	64	libjpeg62-turbo \
57	65	libpq5 \
	66	libwebp6 \
58	67	xmlsec1 \
59		gosu \
60	68	&& rm -rf /var/lib/apt/lists/*
61	69
62	70	COPY --from=builder /install /usr/local

68	76	EXPOSE 8008/tcp 8009/tcp 8448/tcp
69	77
70	78	ENTRYPOINT ["/start.py"]
	79
	80	HEALTHCHECK --interval=1m --timeout=5s \
	81	CMD curl -fSs http://localhost:8008/health \|\| exit 1

+29

-0

docker/README.md less more

161	161
162	162	You can choose to build a different docker image by changing the value of the `-f` flag to
163	163	point to another Dockerfile.
	164
	165	## Disabling the healthcheck
	166
	167	If you are using a non-standard port or tls inside docker you can disable the healthcheck
	168	whilst running the above `docker run` commands.
	169
	170	```
	171	--no-healthcheck
	172	```
	173	## Setting custom healthcheck on docker run
	174
	175	If you wish to point the healthcheck at a different port with docker command, add the following
	176
	177	```
	178	--health-cmd 'curl -fSs http://localhost:1234/health'
	179	```
	180
	181	## Setting the healthcheck in docker-compose file
	182
	183	You can add the following to set a custom healthcheck in a docker compose file.
	184	You will need version >2.1 for this to work.
	185
	186	```
	187	healthcheck:
	188	test: ["CMD", "curl", "-fSs", "http://localhost:8008/health"]
	189	interval: 1m
	190	timeout: 10s
	191	retries: 3
	192	```

+11

-6

docs/admin_api/user_admin_api.rst less more

107	107
108	108	GET /_synapse/admin/v2/users?from=0&limit=10&guests=false
109	109
110		To use it, you will need to authenticate by providing an `access_token` for a
	110	To use it, you will need to authenticate by providing an ``access_token`` for a
111	111	server admin: see `README.rst <README.rst>`_.
112	112
113	113	The parameter ``from`` is optional but used for pagination, denoting the

118	118	The parameter ``limit`` is optional but is used for pagination, denoting the
119	119	maximum number of items to return in this call. Defaults to ``100``.
120	120
121		The parameter ``user_id`` is optional and filters to only users with user IDs
122		that contain this value.
	121	The parameter ``user_id`` is optional and filters to only return users with user IDs
	122	that contain this value. This parameter is ignored when using the ``name`` parameter.
	123
	124	The parameter ``name`` is optional and filters to only return users with user ID localparts
	125	or displaynames that contain this value.
123	126
124	127	The parameter ``guests`` is optional and if ``false`` will exclude guest users.
125	128	Defaults to ``true`` to include guest users.

210	213
211	214	This API deactivates an account. It removes active access tokens, resets the
212	215	password, and deletes third-party IDs (to prevent the user requesting a
213		password reset). It can also mark the user as GDPR-erased (stopping their data
214		from distributed further, and deleting it entirely if there are no other
215		references to it).
	216	password reset).
	217
	218	It can also mark the user as GDPR-erased. This means messages sent by the
	219	user will still be visible by anyone that was in the room when these messages
	220	were sent, but hidden from users joining the room afterwards.
216	221
217	222	The api is::
218	223

+12

-0

docs/federate.md less more

46	46	proxy: see [reverse_proxy.md](<reverse_proxy.md>) for instructions on how to correctly
47	47	configure a reverse proxy.
48	48
	49	### Known issues
	50
	51	HTTP `308 Permanent Redirect` redirects are not followed: Due to missing features
	52	in the HTTP library used by Synapse, 308 redirects are currently not followed by
	53	federating servers, which can cause `M_UNKNOWN` or `401 Unauthorized` errors. This
	54	may affect users who are redirecting apex-to-www (e.g. `example.com` -> `www.example.com`),
	55	and especially users of the Kubernetes Nginx Ingress module, which uses 308 redirect
	56	codes by default. For those Kubernetes users, [this Stackoverflow post](https://stackoverflow.com/a/52617528/5096871)
	57	might be helpful. For other users, switching to a `301 Moved Permanently` code may be
	58	an option. 308 redirect codes will be supported properly in a future
	59	release of Synapse.
	60
49	61	## Running a demo federation of Synapses
50	62
51	63	If you want to get up and running quickly with a trio of homeservers in a

+1

-0

docs/password_auth_providers.md less more

13	13
14	14	* [matrix-synapse-ldap3](https://github.com/matrix-org/matrix-synapse-ldap3/)
15	15	* [matrix-synapse-shared-secret-auth](https://github.com/devture/matrix-synapse-shared-secret-auth)
	16	* [matrix-synapse-rest-password-provider](https://github.com/ma1uta/matrix-synapse-rest-password-provider)
16	17
17	18	## Required methods
18	19

+15

-11

docs/sample_config.yaml less more

377	377	# min_lifetime: 1d
378	378	# max_lifetime: 1y
379	379
380		# Retention policy limits. If set, a user won't be able to send a
381		# 'm.room.retention' event which features a 'min_lifetime' or a 'max_lifetime'
382		# that's not within this range. This is especially useful in closed federations,
383		# in which server admins can make sure every federating server applies the same
384		# rules.
	380	# Retention policy limits. If set, and the state of a room contains a
	381	# 'm.room.retention' event in its state which contains a 'min_lifetime' or a
	382	# 'max_lifetime' that's out of these bounds, Synapse will cap the room's policy
	383	# to these limits when running purge jobs.
385	384	#
386	385	#allowed_lifetime_min: 1d
387	386	#allowed_lifetime_max: 1y

407	406	# (e.g. every 12h), but not want that purge to be performed by a job that's
408	407	# iterating over every room it knows, which could be heavy on the server.
409	408	#
	409	# If any purge job is configured, it is strongly recommended to have at least
	410	# a single job with neither 'shortest_max_lifetime' nor 'longest_max_lifetime'
	411	# set, or one job without 'shortest_max_lifetime' and one job without
	412	# 'longest_max_lifetime' set. Otherwise some rooms might be ignored, even if
	413	# 'allowed_lifetime_min' and 'allowed_lifetime_max' are set, because capping a
	414	# room's policy to these values is done after the policies are retrieved from
	415	# Synapse's database (which is done using the range specified in a purge job's
	416	# configuration).
	417	#
410	418	#purge_jobs:
411		# - shortest_max_lifetime: 1d
412		# longest_max_lifetime: 3d
	419	# - longest_max_lifetime: 3d
413	420	# interval: 12h
414	421	# - shortest_max_lifetime: 3d
415		# longest_max_lifetime: 1y
416	422	# interval: 1d
417	423
418	424	# Inhibits the /requestToken endpoints from returning an error that might leak

2001	2007	# Directory in which Synapse will try to find the template files below.
2002	2008	# If not set, default templates from within the Synapse package will be used.
2003	2009	#
2004		# DO NOT UNCOMMENT THIS SETTING unless you want to customise the templates.
2005		# If you do uncomment it, you will need to make sure that all the templates
2006		# below are in the directory.
	2010	# Do not uncomment this setting unless you want to customise the templates.
2007	2011	#
2008	2012	# Synapse will look for the following templates in this directory:
2009	2013	#

+5

-0

docs/systemd-with-workers/system/matrix-synapse-worker@.service less more

0	0	[Unit]
1	1	Description=Synapse %i
2	2	AssertPathExists=/etc/matrix-synapse/workers/%i.yaml
	3
3	4	# This service should be restarted when the synapse target is restarted.
4	5	PartOf=matrix-synapse.target
	6
	7	# if this is started at the same time as the main, let the main process start
	8	# first, to initialise the database schema.
	9	After=matrix-synapse.service
5	10
6	11	[Service]
7	12	Type=notify

+56

-1

mypy.ini less more

0	0	[mypy]
1	1	namespace_packages = True
2		plugins = mypy_zope:plugin
	2	plugins = mypy_zope:plugin, scripts-dev/mypy_synapse_plugin.py
3	3	follow_imports = silent
4	4	check_untyped_defs = True
5	5	show_error_codes = True
6	6	show_traceback = True
7	7	mypy_path = stubs
	8	files =
	9	synapse/api,
	10	synapse/appservice,
	11	synapse/config,
	12	synapse/event_auth.py,
	13	synapse/events/builder.py,
	14	synapse/events/spamcheck.py,
	15	synapse/federation,
	16	synapse/handlers/auth.py,
	17	synapse/handlers/cas_handler.py,
	18	synapse/handlers/directory.py,
	19	synapse/handlers/events.py,
	20	synapse/handlers/federation.py,
	21	synapse/handlers/identity.py,
	22	synapse/handlers/initial_sync.py,
	23	synapse/handlers/message.py,
	24	synapse/handlers/oidc_handler.py,
	25	synapse/handlers/pagination.py,
	26	synapse/handlers/presence.py,
	27	synapse/handlers/room.py,
	28	synapse/handlers/room_member.py,
	29	synapse/handlers/room_member_worker.py,
	30	synapse/handlers/saml_handler.py,
	31	synapse/handlers/sync.py,
	32	synapse/handlers/ui_auth,
	33	synapse/http/federation/well_known_resolver.py,
	34	synapse/http/server.py,
	35	synapse/http/site.py,
	36	synapse/logging/,
	37	synapse/metrics,
	38	synapse/module_api,
	39	synapse/notifier.py,
	40	synapse/push/pusherpool.py,
	41	synapse/push/push_rule_evaluator.py,
	42	synapse/replication,
	43	synapse/rest,
	44	synapse/server.py,
	45	synapse/server_notices,
	46	synapse/spam_checker_api,
	47	synapse/state,
	48	synapse/storage/databases/main/stream.py,
	49	synapse/storage/databases/main/ui_auth.py,
	50	synapse/storage/database.py,
	51	synapse/storage/engines,
	52	synapse/storage/state.py,
	53	synapse/storage/util,
	54	synapse/streams,
	55	synapse/types.py,
	56	synapse/util/caches/descriptors.py,
	57	synapse/util/caches/stream_change_cache.py,
	58	synapse/util/metrics.py,
	59	tests/replication,
	60	tests/test_utils,
	61	tests/rest/client/v2_alpha/test_auth.py,
	62	tests/util/test_stream_change_cache.py
8	63
9	64	[mypy-pymacaroons.*]
10	65	ignore_missing_imports = True

+1

-0

scripts/synapse_port_db less more

88	88	"redactions": ["have_censored"],
89	89	"room_stats_state": ["is_federatable"],
90	90	"local_media_repository": ["safe_from_quarantine"],
	91	"users": ["shadow_banned"],
91	92	}
92	93
93	94

+28

-9

scripts-dev/federation_client.py less more

20	20	import base64
21	21	import json
22	22	import sys
	23	from typing import Any, Optional
23	24	from urllib import parse as urlparse
24	25
25	26	import nacl.signing
26	27	import requests
	28	import signedjson.types
27	29	import srvlookup
28	30	import yaml
29	31	from requests.adapters import HTTPAdapter

68	70	).encode("UTF-8")
69	71
70	72
71		def sign_json(json_object, signing_key, signing_name):
	73	def sign_json(
	74	json_object: Any, signing_key: signedjson.types.SigningKey, signing_name: str
	75	) -> Any:
72	76	signatures = json_object.pop("signatures", {})
73	77	unsigned = json_object.pop("unsigned", None)
74	78

121	125	return keys
122	126
123	127
124		def request_json(method, origin_name, origin_key, destination, path, content):
	128	def request(
	129	method: Optional[str],
	130	origin_name: str,
	131	origin_key: signedjson.types.SigningKey,
	132	destination: str,
	133	path: str,
	134	content: Optional[str],
	135	) -> requests.Response:
125	136	if method is None:
126	137	if content is None:
127	138	method = "GET"

158	169	if method == "POST":
159	170	headers["Content-Type"] = "application/json"
160	171
161		result = s.request(
162		method=method, url=dest, headers=headers, verify=False, data=content
163		)
164		sys.stderr.write("Status Code: %d\n" % (result.status_code,))
165		return result.json()
	172	return s.request(
	173	method=method,
	174	url=dest,
	175	headers=headers,
	176	verify=False,
	177	data=content,
	178	stream=True,
	179	)
166	180
167	181
168	182	def main():

221	235	with open(args.signing_key_path) as f:
222	236	key = read_signing_keys(f)[0]
223	237
224		result = request_json(
	238	result = request(
225	239	args.method,
226	240	args.server_name,
227	241	key,

230	244	content=args.body,
231	245	)
232	246
233		json.dump(result, sys.stdout)
	247	sys.stderr.write("Status Code: %d\n" % (result.status_code,))
	248
	249	for chunk in result.iter_content():
	250	# we write raw utf8 to stdout.
	251	sys.stdout.buffer.write(chunk)
	252
234	253	print("")
235	254
236	255

+1

-1

scripts-dev/hash_history.py less more

14	14	from synapse.storage.signatures import SignatureStore
15	15
16	16
17		class Store(object):
	17	class Store:
18	18	_get_pdu_tuples = PduStore.__dict__["_get_pdu_tuples"]
19	19	_get_pdu_content_hashes_txn = SignatureStore.__dict__["_get_pdu_content_hashes_txn"]
20	20	_get_prev_pdu_hashes_txn = SignatureStore.__dict__["_get_prev_pdu_hashes_txn"]

+85

-0

scripts-dev/mypy_synapse_plugin.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	"""This is a mypy plugin for Synpase to deal with some of the funky typing that
	16	can crop up, e.g the cache descriptors.
	17	"""
	18
	19	from typing import Callable, Optional
	20
	21	from mypy.plugin import MethodSigContext, Plugin
	22	from mypy.typeops import bind_self
	23	from mypy.types import CallableType
	24
	25
	26	class SynapsePlugin(Plugin):
	27	def get_method_signature_hook(
	28	self, fullname: str
	29	) -> Optional[Callable[[MethodSigContext], CallableType]]:
	30	if fullname.startswith(
	31	"synapse.util.caches.descriptors._CachedFunction.__call__"
	32	):
	33	return cached_function_method_signature
	34	return None
	35
	36
	37	def cached_function_method_signature(ctx: MethodSigContext) -> CallableType:
	38	"""Fixes the `_CachedFunction.__call__` signature to be correct.
	39
	40	It already has almost the correct signature, except:
	41
	42	1. the `self` argument needs to be marked as "bound"; and
	43	2. any `cache_context` argument should be removed.
	44	"""
	45
	46	# First we mark this as a bound function signature.
	47	signature = bind_self(ctx.default_signature)
	48
	49	# Secondly, we remove any "cache_context" args.
	50	#
	51	# Note: We should be only doing this if `cache_context=True` is set, but if
	52	# it isn't then the code will raise an exception when its called anyway, so
	53	# its not the end of the world.
	54	context_arg_index = None
	55	for idx, name in enumerate(signature.arg_names):
	56	if name == "cache_context":
	57	context_arg_index = idx
	58	break
	59
	60	if context_arg_index:
	61	arg_types = list(signature.arg_types)
	62	arg_types.pop(context_arg_index)
	63
	64	arg_names = list(signature.arg_names)
	65	arg_names.pop(context_arg_index)
	66
	67	arg_kinds = list(signature.arg_kinds)
	68	arg_kinds.pop(context_arg_index)
	69
	70	signature = signature.copy_modified(
	71	arg_types=arg_types, arg_names=arg_names, arg_kinds=arg_kinds,
	72	)
	73
	74	return signature
	75
	76
	77	def plugin(version: str):
	78	# This is the entry point of the plugin, and let's us deal with the fact
	79	# that the mypy plugin interface is not stable by looking at the version
	80	# string.
	81	#
	82	# However, since we pin the version of mypy Synapse uses in CI, we don't
	83	# really care.
	84	return SynapsePlugin

+47

-0

stubs/frozendict.pyi less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	# Stub for frozendict.
	16
	17	from typing import (
	18	Any,
	19	Hashable,
	20	Iterable,
	21	Iterator,
	22	Mapping,
	23	overload,
	24	Tuple,
	25	TypeVar,
	26	)
	27
	28	_KT = TypeVar("_KT", bound=Hashable) # Key type.
	29	_VT = TypeVar("_VT") # Value type.
	30
	31	class frozendict(Mapping[_KT, _VT]):
	32	@overload
	33	def __init__(self, **kwargs: _VT) -> None: ...
	34	@overload
	35	def __init__(self, __map: Mapping[_KT, _VT], **kwargs: _VT) -> None: ...
	36	@overload
	37	def __init__(
	38	self, __iterable: Iterable[Tuple[_KT, _VT]], **kwargs: _VT
	39	) -> None: ...
	40	def __getitem__(self, key: _KT) -> _VT: ...
	41	def __contains__(self, key: Any) -> bool: ...
	42	def copy(self, **add_or_replace: Any) -> frozendict: ...
	43	def __iter__(self) -> Iterator[_KT]: ...
	44	def __len__(self) -> int: ...
	45	def __repr__(self) -> str: ...
	46	def __hash__(self) -> int: ...

+1

-1

synapse/__init__.py less more

47	47	except ImportError:
48	48	pass
49	49
50		__version__ = "1.19.1"
	50	__version__ = "1.20.1"
51	51
52	52	if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
53	53	# We import here so that we don't have to install a bunch of deps when

+12

-2

synapse/api/auth.py less more

57	57	pass
58	58
59	59
60		class Auth(object):
	60	class Auth:
61	61	"""
62	62	FIXME: This class contains a mix of functions for authenticating users
63	63	of our client-server API and authenticating events added to room graphs.

212	212	user = user_info["user"]
213	213	token_id = user_info["token_id"]
214	214	is_guest = user_info["is_guest"]
	215	shadow_banned = user_info["shadow_banned"]
215	216
216	217	# Deny the request if the user account has expired.
217	218	if self._account_validity.enabled and not allow_expired:

251	252	opentracing.set_tag("device_id", device_id)
252	253
253	254	return synapse.types.create_requester(
254		user, token_id, is_guest, device_id, app_service=app_service
	255	user,
	256	token_id,
	257	is_guest,
	258	shadow_banned,
	259	device_id,
	260	app_service=app_service,
255	261	)
256	262	except KeyError:
257	263	raise MissingClientTokenError()

296	302	dict that includes:
297	303	`user` (UserID)
298	304	`is_guest` (bool)
	305	`shadow_banned` (bool)
299	306	`token_id` (int\|None): access token id. May be None if guest
300	307	`device_id` (str\|None): device corresponding to access token
301	308	Raises:

355	362	ret = {
356	363	"user": user,
357	364	"is_guest": True,
	365	"shadow_banned": False,
358	366	"token_id": None,
359	367	# all guests get the same device id
360	368	"device_id": GUEST_DEVICE_ID,

364	372	ret = {
365	373	"user": user,
366	374	"is_guest": False,
	375	"shadow_banned": False,
367	376	"token_id": None,
368	377	"device_id": None,
369	378	}

487	496	"user": UserID.from_string(ret.get("name")),
488	497	"token_id": ret.get("token_id", None),
489	498	"is_guest": False,
	499	"shadow_banned": ret.get("shadow_banned"),
490	500	"device_id": ret.get("device_id"),
491	501	"valid_until_ms": ret.get("valid_until_ms"),
492	502	}

+1

-1

synapse/api/auth_blocking.py less more

21	21	logger = logging.getLogger(__name__)
22	22
23	23
24		class AuthBlocking(object):
	24	class AuthBlocking:
25	25	def __init__(self, hs):
26	26	self.store = hs.get_datastore()
27	27

+13

-13

synapse/api/constants.py less more

27	27	MAX_USERID_LENGTH = 255
28	28
29	29
30		class Membership(object):
	30	class Membership:
31	31
32	32	"""Represents the membership states of a user in a room."""
33	33

39	39	LIST = (INVITE, JOIN, KNOCK, LEAVE, BAN)
40	40
41	41
42		class PresenceState(object):
	42	class PresenceState:
43	43	"""Represents the presence state of a user."""
44	44
45	45	OFFLINE = "offline"

47	47	ONLINE = "online"
48	48
49	49
50		class JoinRules(object):
	50	class JoinRules:
51	51	PUBLIC = "public"
52	52	KNOCK = "knock"
53	53	INVITE = "invite"
54	54	PRIVATE = "private"
55	55
56	56
57		class LoginType(object):
	57	class LoginType:
58	58	PASSWORD = "m.login.password"
59	59	EMAIL_IDENTITY = "m.login.email.identity"
60	60	MSISDN = "m.login.msisdn"

64	64	DUMMY = "m.login.dummy"
65	65
66	66
67		class EventTypes(object):
	67	class EventTypes:
68	68	Member = "m.room.member"
69	69	Create = "m.room.create"
70	70	Tombstone = "m.room.tombstone"

95	95	Presence = "m.presence"
96	96
97	97
98		class RejectedReason(object):
	98	class RejectedReason:
99	99	AUTH_ERROR = "auth_error"
100	100
101	101
102		class RoomCreationPreset(object):
	102	class RoomCreationPreset:
103	103	PRIVATE_CHAT = "private_chat"
104	104	PUBLIC_CHAT = "public_chat"
105	105	TRUSTED_PRIVATE_CHAT = "trusted_private_chat"
106	106
107	107
108		class ThirdPartyEntityKind(object):
	108	class ThirdPartyEntityKind:
109	109	USER = "user"
110	110	LOCATION = "location"
111	111

114	114	ServerNoticeLimitReached = "m.server_notice.usage_limit_reached"
115	115
116	116
117		class UserTypes(object):
	117	class UserTypes:
118	118	"""Allows for user type specific behaviour. With the benefit of hindsight
119	119	'admin' and 'guest' users should also be UserTypes. Normal users are type None
120	120	"""

124	124	ALL_USER_TYPES = (SUPPORT, BOT)
125	125
126	126
127		class RelationTypes(object):
	127	class RelationTypes:
128	128	"""The types of relations known to this server.
129	129	"""
130	130

133	133	REFERENCE = "m.reference"
134	134
135	135
136		class LimitBlockingTypes(object):
	136	class LimitBlockingTypes:
137	137	"""Reasons that a server may be blocked"""
138	138
139	139	MONTHLY_ACTIVE_USER = "monthly_active_user"
140	140	HS_DISABLED = "hs_disabled"
141	141
142	142
143		class EventContentFields(object):
	143	class EventContentFields:
144	144	"""Fields found in events' content, regardless of type."""
145	145
146	146	# Labels for the event, cf https://github.com/matrix-org/matrix-doc/pull/2326

151	151	SELF_DESTRUCT_AFTER = "org.matrix.self_destruct_after"
152	152
153	153
154		class RoomEncryptionAlgorithms(object):
	154	class RoomEncryptionAlgorithms:
155	155	MEGOLM_V1_AES_SHA2 = "m.megolm.v1.aes-sha2"
156	156	DEFAULT = MEGOLM_V1_AES_SHA2

+12

-4

synapse/api/errors.py less more

20	20	from http import HTTPStatus
21	21	from typing import Dict, List, Optional, Union
22	22
23		from canonicaljson import json
24
25	23	from twisted.web import http
	24
	25	from synapse.util import json_decoder
26	26
27	27	if typing.TYPE_CHECKING:
28	28	from synapse.types import JsonDict

30	30	logger = logging.getLogger(__name__)
31	31
32	32
33		class Codes(object):
	33	class Codes:
34	34	UNRECOGNIZED = "M_UNRECOGNIZED"
35	35	UNAUTHORIZED = "M_UNAUTHORIZED"
36	36	FORBIDDEN = "M_FORBIDDEN"

592	592	# try to parse the body as json, to get better errcode/msg, but
593	593	# default to M_UNKNOWN with the HTTP status as the error text
594	594	try:
595		j = json.loads(self.response.decode("utf-8"))
	595	j = json_decoder.decode(self.response.decode("utf-8"))
596	596	except ValueError:
597	597	j = {}
598	598

603	603	errmsg = j.pop("error", self.msg)
604	604
605	605	return ProxiedRequestError(self.code, errmsg, errcode, j)
	606
	607
	608	class ShadowBanError(Exception):
	609	"""
	610	Raised when a shadow-banned user attempts to perform an action.
	611
	612	This should be caught and a proper "fake" success response sent to the user.
	613	"""

+4

-4

synapse/api/filtering.py less more

22	22
23	23	from synapse.api.constants import EventContentFields
24	24	from synapse.api.errors import SynapseError
25		from synapse.storage.presence import UserPresenceState
	25	from synapse.api.presence import UserPresenceState
26	26	from synapse.types import RoomID, UserID
27	27
28	28	FILTER_SCHEMA = {

129	129	return UserID.from_string(user_id_str)
130	130
131	131
132		class Filtering(object):
	132	class Filtering:
133	133	def __init__(self, hs):
134	134	super(Filtering, self).__init__()
135	135	self.store = hs.get_datastore()

167	167	raise SynapseError(400, str(e))
168	168
169	169
170		class FilterCollection(object):
	170	class FilterCollection:
171	171	def __init__(self, filter_json):
172	172	self._filter_json = filter_json
173	173

248	248	)
249	249
250	250
251		class Filter(object):
	251	class Filter:
252	252	def __init__(self, filter_json):
253	253	self.filter_json = filter_json
254	254

+69

-0

synapse/api/presence.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from collections import namedtuple
	16
	17	from synapse.api.constants import PresenceState
	18
	19
	20	class UserPresenceState(
	21	namedtuple(
	22	"UserPresenceState",
	23	(
	24	"user_id",
	25	"state",
	26	"last_active_ts",
	27	"last_federation_update_ts",
	28	"last_user_sync_ts",
	29	"status_msg",
	30	"currently_active",
	31	),
	32	)
	33	):
	34	"""Represents the current presence state of the user.
	35
	36	user_id (str)
	37	last_active (int): Time in msec that the user last interacted with server.
	38	last_federation_update (int): Time in msec since either a) we sent a presence
	39	update to other servers or b) we received a presence update, depending
	40	on if is a local user or not.
	41	last_user_sync (int): Time in msec that the user last completed a sync
	42	(or event stream).
	43	status_msg (str): User set status message.
	44	"""
	45
	46	def as_dict(self):
	47	return dict(self._asdict())
	48
	49	@staticmethod
	50	def from_dict(d):
	51	return UserPresenceState(**d)
	52
	53	def copy_and_replace(self, **kwargs):
	54	return self._replace(**kwargs)
	55
	56	@classmethod
	57	def default(cls, user_id):
	58	"""Returns a default presence state.
	59	"""
	60	return cls(
	61	user_id=user_id,
	62	state=PresenceState.OFFLINE,
	63	last_active_ts=0,
	64	last_federation_update_ts=0,
	65	last_user_sync_ts=0,
	66	status_msg=None,
	67	currently_active=False,
	68	)

+1

-1

synapse/api/ratelimiting.py less more

20	20	from synapse.util import Clock
21	21
22	22
23		class Ratelimiter(object):
	23	class Ratelimiter:
24	24	"""
25	25	Ratelimit actions marked by arbitrary keys.
26	26

+5

-5

synapse/api/room_versions.py less more

17	17	import attr
18	18
19	19
20		class EventFormatVersions(object):
	20	class EventFormatVersions:
21	21	"""This is an internal enum for tracking the version of the event format,
22	22	independently from the room version.
23	23	"""

34	34	}
35	35
36	36
37		class StateResolutionVersions(object):
	37	class StateResolutionVersions:
38	38	"""Enum to identify the state resolution algorithms"""
39	39
40	40	V1 = 1 # room v1 state res
41	41	V2 = 2 # MSC1442 state res: room v2 and later
42	42
43	43
44		class RoomDisposition(object):
	44	class RoomDisposition:
45	45	STABLE = "stable"
46	46	UNSTABLE = "unstable"
47	47
48	48
49	49	@attr.s(slots=True, frozen=True)
50		class RoomVersion(object):
	50	class RoomVersion:
51	51	"""An object which describes the unique attributes of a room version."""
52	52
53	53	identifier = attr.ib() # str; the identifier for this version

68	68	limit_notifications_power_levels = attr.ib(type=bool)
69	69
70	70
71		class RoomVersions(object):
	71	class RoomVersions:
72	72	V1 = RoomVersion(
73	73	"1",
74	74	RoomDisposition.STABLE,

+1

-1

synapse/api/urls.py less more

32	32	LEGACY_MEDIA_PREFIX = "/_matrix/media/v1"
33	33
34	34
35		class ConsentURIBuilder(object):
	35	class ConsentURIBuilder:
36	36	def __init__(self, hs_config):
37	37	"""
38	38	Args:

+9

-2

synapse/app/_base.py less more

333	333	This is to workaround https://twistedmatrix.com/trac/ticket/9620, where we
334	334	can run out of file descriptors and infinite loop if we attempt to do too
335	335	many DNS queries at once
	336
	337	XXX: I'm confused by this. reactor.nameResolver does not use twisted.names unless
	338	you explicitly install twisted.names as the resolver; rather it uses a GAIResolver
	339	backed by the reactor's default threadpool (which is limited to 10 threads). So
	340	(a) I don't understand why twisted ticket 9620 is relevant, and (b) I don't
	341	understand why we would run out of FDs if we did too many lookups at once.
	342	-- richvdh 2020/08/29
336	343	"""
337	344	new_resolver = _LimitedHostnameResolver(
338	345	reactor.nameResolver, max_dns_requests_in_flight

341	348	reactor.installNameResolver(new_resolver)
342	349
343	350
344		class _LimitedHostnameResolver(object):
	351	class _LimitedHostnameResolver:
345	352	"""Wraps a IHostnameResolver, limiting the number of in-flight DNS lookups.
346	353	"""
347	354

401	408	yield deferred
402	409
403	410
404		class _DeferredResolutionReceiver(object):
	411	class _DeferredResolutionReceiver:
405	412	"""Wraps a IResolutionReceiver and simply resolves the given deferred when
406	413	resolution is complete
407	414	"""

+9

-11

synapse/app/admin_cmd.py less more

78	78	pass
79	79
80	80
81		@defer.inlineCallbacks
82		def export_data_command(hs, args):
	81	async def export_data_command(hs, args):
83	82	"""Export data for a user.
84	83
85	84	Args:

90	89	user_id = args.user_id
91	90	directory = args.output_directory
92	91
93		res = yield defer.ensureDeferred(
94		hs.get_handlers().admin_handler.export_user_data(
95		user_id, FileExfiltrationWriter(user_id, directory=directory)
96		)
	92	res = await hs.get_handlers().admin_handler.export_user_data(
	93	user_id, FileExfiltrationWriter(user_id, directory=directory)
97	94	)
98	95	print(res)
99	96

231	228	# We also make sure that `_base.start` gets run before we actually run the
232	229	# command.
233	230
234		@defer.inlineCallbacks
235		def run(_reactor):
	231	async def run():
236	232	with LoggingContext("command"):
237		yield _base.start(ss, [])
238		yield args.func(ss, args)
	233	_base.start(ss, [])
	234	await args.func(ss, args)
239	235
240	236	_base.start_worker_reactor(
241		"synapse-admin-cmd", config, run_command=lambda: task.react(run)
	237	"synapse-admin-cmd",
	238	config,
	239	run_command=lambda: task.react(lambda _reactor: defer.ensureDeferred(run())),
242	240	)
243	241
244	242

+1

-1

synapse/app/generic_worker.py less more

744	744	self.send_handler.wake_destination(server)
745	745
746	746
747		class FederationSenderHandler(object):
	747	class FederationSenderHandler:
748	748	"""Processes the fedration replication stream
749	749
750	750	This class is only instantiate on the worker responsible for sending outbound

+8

-10

synapse/app/homeserver.py less more

410	410
411	411	return provision
412	412
413		@defer.inlineCallbacks
414		def reprovision_acme():
	413	async def reprovision_acme():
415	414	"""
416	415	Provision a certificate from ACME, if required, and reload the TLS
417	416	certificate if it's renewed.
418	417	"""
419		reprovisioned = yield defer.ensureDeferred(do_acme())
	418	reprovisioned = await do_acme()
420	419	if reprovisioned:
421	420	_base.refresh_certificate(hs)
422	421
423		@defer.inlineCallbacks
424		def start():
	422	async def start():
425	423	try:
426	424	# Run the ACME provisioning code, if it's enabled.
427	425	if hs.config.acme_enabled:
428	426	acme = hs.get_acme_handler()
429	427	# Start up the webservices which we will respond to ACME
430	428	# challenges with, and then provision.
431		yield defer.ensureDeferred(acme.start_listening())
432		yield defer.ensureDeferred(do_acme())
	429	await acme.start_listening()
	430	await do_acme()
433	431
434	432	# Check if it needs to be reprovisioned every day.
435	433	hs.get_clock().looping_call(reprovision_acme, 24 * 60 * 60 * 1000)

438	436	if hs.config.oidc_enabled:
439	437	oidc = hs.get_oidc_handler()
440	438	# Loading the provider metadata also ensures the provider config is valid.
441		yield defer.ensureDeferred(oidc.load_metadata())
442		yield defer.ensureDeferred(oidc.load_jwks())
	439	await oidc.load_metadata()
	440	await oidc.load_jwks()
443	441
444	442	_base.start(hs, config.listeners)
445	443

455	453	reactor.stop()
456	454	sys.exit(1)
457	455
458		reactor.callWhenRunning(start)
	456	reactor.callWhenRunning(lambda: defer.ensureDeferred(start()))
459	457
460	458	return hs
461	459

+18

-15

synapse/appservice/__init__.py less more

13	13	# limitations under the License.
14	14	import logging
15	15	import re
	16	from typing import TYPE_CHECKING
16	17
17	18	from synapse.api.constants import EventTypes
	19	from synapse.appservice.api import ApplicationServiceApi
18	20	from synapse.types import GroupID, get_domain_from_id
19	21	from synapse.util.caches.descriptors import cached
20	22
	23	if TYPE_CHECKING:
	24	from synapse.storage.databases.main import DataStore
	25
21	26	logger = logging.getLogger(__name__)
22	27
23	28
24		class ApplicationServiceState(object):
	29	class ApplicationServiceState:
25	30	DOWN = "down"
26	31	UP = "up"
27	32
28	33
29		class AppServiceTransaction(object):
	34	class AppServiceTransaction:
30	35	"""Represents an application service transaction."""
31	36
32	37	def __init__(self, service, id, events):

34	39	self.id = id
35	40	self.events = events
36	41
37		def send(self, as_api):
	42	async def send(self, as_api: ApplicationServiceApi) -> bool:
38	43	"""Sends this transaction using the provided AS API interface.
39	44
40	45	Args:
41		as_api(ApplicationServiceApi): The API to use to send.
	46	as_api: The API to use to send.
42	47	Returns:
43		An Awaitable which resolves to True if the transaction was sent.
44		"""
45		return as_api.push_bulk(
	48	True if the transaction was sent.
	49	"""
	50	return await as_api.push_bulk(
46	51	service=self.service, events=self.events, txn_id=self.id
47	52	)
48	53
49		def complete(self, store):
	54	async def complete(self, store: "DataStore") -> None:
50	55	"""Completes this transaction as successful.
51	56
52	57	Marks this transaction ID on the application service and removes the

54	59
55	60	Args:
56	61	store: The database store to operate on.
57		Returns:
58		A Deferred which resolves to True if the transaction was completed.
59		"""
60		return store.complete_appservice_txn(service=self.service, txn_id=self.id)
61
62
63		class ApplicationService(object):
	62	"""
	63	await store.complete_appservice_txn(service=self.service, txn_id=self.id)
	64
	65
	66	class ApplicationService:
64	67	"""Defines an application service. This definition is mostly what is
65	68	provided to the /register AS API.
66	69

+11

-8

synapse/appservice/api.py less more

13	13	# limitations under the License.
14	14	import logging
15	15	import urllib
	16	from typing import TYPE_CHECKING, Optional
16	17
17	18	from prometheus_client import Counter
18
19		from twisted.internet import defer
20	19
21	20	from synapse.api.constants import EventTypes, ThirdPartyEntityKind
22	21	from synapse.api.errors import CodeMessageException
23	22	from synapse.events.utils import serialize_event
24	23	from synapse.http.client import SimpleHttpClient
25		from synapse.types import ThirdPartyInstanceID
	24	from synapse.types import JsonDict, ThirdPartyInstanceID
26	25	from synapse.util.caches.response_cache import ResponseCache
	26
	27	if TYPE_CHECKING:
	28	from synapse.appservice import ApplicationService
27	29
28	30	logger = logging.getLogger(__name__)
29	31

162	164	logger.warning("query_3pe to %s threw exception %s", uri, ex)
163	165	return []
164	166
165		def get_3pe_protocol(self, service, protocol):
	167	async def get_3pe_protocol(
	168	self, service: "ApplicationService", protocol: str
	169	) -> Optional[JsonDict]:
166	170	if service.url is None:
167	171	return {}
168	172
169		@defer.inlineCallbacks
170		def _get():
	173	async def _get() -> Optional[JsonDict]:
171	174	uri = "%s%s/thirdparty/protocol/%s" % (
172	175	service.url,
173	176	APP_SERVICE_PREFIX,
174	177	urllib.parse.quote(protocol),
175	178	)
176	179	try:
177		info = yield defer.ensureDeferred(self.get_json(uri, {}))
	180	info = await self.get_json(uri, {})
178	181
179	182	if not _is_valid_3pe_metadata(info):
180	183	logger.warning(

195	198	return None
196	199
197	200	key = (service.id, protocol)
198		return self.protocol_meta_cache.wrap(key, _get)
	201	return await self.protocol_meta_cache.wrap(key, _get)
199	202
200	203	async def push_bulk(self, service, events, txn_id=None):
201	204	if service.url is None:

+4

-4

synapse/appservice/scheduler.py less more

56	56	logger = logging.getLogger(__name__)
57	57
58	58
59		class ApplicationServiceScheduler(object):
	59	class ApplicationServiceScheduler:
60	60	""" Public facing API for this module. Does the required DI to tie the
61	61	components together. This also serves as the "event_pool", which in this
62	62	case is a simple array.

85	85	self.queuer.enqueue(service, event)
86	86
87	87
88		class _ServiceQueuer(object):
	88	class _ServiceQueuer:
89	89	"""Queue of events waiting to be sent to appservices.
90	90
91	91	Groups events into transactions per-appservice, and sends them on to the

132	132	self.requests_in_flight.discard(service.id)
133	133
134	134
135		class _TransactionController(object):
	135	class _TransactionController:
136	136	"""Transaction manager.
137	137
138	138	Builds AppServiceTransactions and runs their lifecycle. Also starts a Recoverer

208	208	return state == ApplicationServiceState.UP or state is None
209	209
210	210
211		class _Recoverer(object):
	211	class _Recoverer:
212	212	"""Manages retries and backoff for a DOWN appservice.
213	213
214	214	We have one of these for each appservice which is currently considered DOWN.

+108

-4

synapse/config/_base.py less more

17	17	import argparse
18	18	import errno
19	19	import os
	20	import time
	21	import urllib.parse
20	22	from collections import OrderedDict
21	23	from hashlib import sha256
22	24	from textwrap import dedent
23		from typing import Any, List, MutableMapping, Optional
	25	from typing import Any, Callable, List, MutableMapping, Optional
24	26
25	27	import attr
	28	import jinja2
	29	import pkg_resources
26	30	import yaml
27	31
28	32

83	87	return False
84	88
85	89
86		class Config(object):
	90	class Config:
87	91	"""
88	92	A configuration section, containing configuration keys and values.
89	93

98	102
99	103	def __init__(self, root_config=None):
100	104	self.root = root_config
	105
	106	# Get the path to the default Synapse template directory
	107	self.default_template_dir = pkg_resources.resource_filename(
	108	"synapse", "res/templates"
	109	)
101	110
102	111	def __getattr__(self, item: str) -> Any:
103	112	"""

183	192	with open(file_path) as file_stream:
184	193	return file_stream.read()
185	194
186
187		class RootConfig(object):
	195	def read_templates(
	196	self,
	197	filenames: List[str],
	198	custom_template_directory: Optional[str] = None,
	199	autoescape: bool = False,
	200	) -> List[jinja2.Template]:
	201	"""Load a list of template files from disk using the given variables.
	202
	203	This function will attempt to load the given templates from the default Synapse
	204	template directory. If `custom_template_directory` is supplied, that directory
	205	is tried first.
	206
	207	Files read are treated as Jinja templates. These templates are not rendered yet.
	208
	209	Args:
	210	filenames: A list of template filenames to read.
	211
	212	custom_template_directory: A directory to try to look for the templates
	213	before using the default Synapse template directory instead.
	214
	215	autoescape: Whether to autoescape variables before inserting them into the
	216	template.
	217
	218	Raises:
	219	ConfigError: if the file's path is incorrect or otherwise cannot be read.
	220
	221	Returns:
	222	A list of jinja2 templates.
	223	"""
	224	templates = []
	225	search_directories = [self.default_template_dir]
	226
	227	# The loader will first look in the custom template directory (if specified) for the
	228	# given filename. If it doesn't find it, it will use the default template dir instead
	229	if custom_template_directory:
	230	# Check that the given template directory exists
	231	if not self.path_exists(custom_template_directory):
	232	raise ConfigError(
	233	"Configured template directory does not exist: %s"
	234	% (custom_template_directory,)
	235	)
	236
	237	# Search the custom template directory as well
	238	search_directories.insert(0, custom_template_directory)
	239
	240	loader = jinja2.FileSystemLoader(search_directories)
	241	env = jinja2.Environment(loader=loader, autoescape=autoescape)
	242
	243	# Update the environment with our custom filters
	244	env.filters.update(
	245	{
	246	"format_ts": _format_ts_filter,
	247	"mxc_to_http": _create_mxc_to_http_filter(self.public_baseurl),
	248	}
	249	)
	250
	251	for filename in filenames:
	252	# Load the template
	253	template = env.get_template(filename)
	254	templates.append(template)
	255
	256	return templates
	257
	258
	259	def _format_ts_filter(value: int, format: str):
	260	return time.strftime(format, time.localtime(value / 1000))
	261
	262
	263	def _create_mxc_to_http_filter(public_baseurl: str) -> Callable:
	264	"""Create and return a jinja2 filter that converts MXC urls to HTTP
	265
	266	Args:
	267	public_baseurl: The public, accessible base URL of the homeserver
	268	"""
	269
	270	def mxc_to_http_filter(value, width, height, resize_method="crop"):
	271	if value[0:6] != "mxc://":
	272	return ""
	273
	274	server_and_media_id = value[6:]
	275	fragment = None
	276	if "#" in server_and_media_id:
	277	server_and_media_id, fragment = server_and_media_id.split("#", 1)
	278	fragment = "#" + fragment
	279
	280	params = {"width": width, "height": height, "method": resize_method}
	281	return "%s_matrix/media/v1/thumbnail/%s?%s%s" % (
	282	public_baseurl,
	283	server_and_media_id,
	284	urllib.parse.urlencode(params),
	285	fragment or "",
	286	)
	287
	288	return mxc_to_http_filter
	289
	290
	291	class RootConfig:
188	292	"""
189	293	Holder of an application's configuration.
190	294

+1

-1

synapse/config/cache.py less more

32	32	_DEFAULT_EVENT_CACHE_SIZE = "10K"
33	33
34	34
35		class CacheProperties(object):
	35	class CacheProperties:
36	36	def __init__(self):
37	37	# The default factor size for all caches
38	38	self.default_factor_size = float(

+70

-81

synapse/config/emailconfig.py less more

22	22	from typing import Optional
23	23
24	24	import attr
25		import pkg_resources
26	25
27	26	from ._base import Config, ConfigError
28	27

97	96	if parsed[1] == "":
98	97	raise RuntimeError("Invalid notif_from address")
99	98
	99	# A user-configurable template directory
100	100	template_dir = email_config.get("template_dir")
101		# we need an absolute path, because we change directory after starting (and
102		# we don't yet know what auxiliary templates like mail.css we will need).
103		# (Note that loading as package_resources with jinja.PackageLoader doesn't
104		# work for the same reason.)
105		if not template_dir:
106		template_dir = pkg_resources.resource_filename("synapse", "res/templates")
107
108		self.email_template_dir = os.path.abspath(template_dir)
	101	if isinstance(template_dir, str):
	102	# We need an absolute path, because we change directory after starting (and
	103	# we don't yet know what auxiliary templates like mail.css we will need).
	104	template_dir = os.path.abspath(template_dir)
	105	elif template_dir is not None:
	106	# If template_dir is something other than a str or None, warn the user
	107	raise ConfigError("Config option email.template_dir must be type str")
109	108
110	109	self.email_enable_notifs = email_config.get("enable_notifs", False)
111
112		account_validity_config = config.get("account_validity") or {}
113		account_validity_renewal_enabled = account_validity_config.get("renew_at")
114	110
115	111	self.threepid_behaviour_email = (
116	112	# Have Synapse handle the email sending if account_threepid_delegates.email

165	161	email_config.get("validation_token_lifetime", "1h")
166	162	)
167	163
168		if (
169		self.email_enable_notifs
170		or account_validity_renewal_enabled
171		or self.threepid_behaviour_email == ThreepidBehaviour.LOCAL
172		):
173		# make sure we can import the required deps
174		import bleach
175		import jinja2
176
177		# prevent unused warnings
178		jinja2
179		bleach
180
181	164	if self.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
182	165	missing = []
183	166	if not self.email_notif_from:

195	178
196	179	# These email templates have placeholders in them, and thus must be
197	180	# parsed using a templating engine during a request
198		self.email_password_reset_template_html = email_config.get(
	181	password_reset_template_html = email_config.get(
199	182	"password_reset_template_html", "password_reset.html"
200	183	)
201		self.email_password_reset_template_text = email_config.get(
	184	password_reset_template_text = email_config.get(
202	185	"password_reset_template_text", "password_reset.txt"
203	186	)
204		self.email_registration_template_html = email_config.get(
	187	registration_template_html = email_config.get(
205	188	"registration_template_html", "registration.html"
206	189	)
207		self.email_registration_template_text = email_config.get(
	190	registration_template_text = email_config.get(
208	191	"registration_template_text", "registration.txt"
209	192	)
210		self.email_add_threepid_template_html = email_config.get(
	193	add_threepid_template_html = email_config.get(
211	194	"add_threepid_template_html", "add_threepid.html"
212	195	)
213		self.email_add_threepid_template_text = email_config.get(
	196	add_threepid_template_text = email_config.get(
214	197	"add_threepid_template_text", "add_threepid.txt"
215	198	)
216	199
217		self.email_password_reset_template_failure_html = email_config.get(
	200	password_reset_template_failure_html = email_config.get(
218	201	"password_reset_template_failure_html", "password_reset_failure.html"
219	202	)
220		self.email_registration_template_failure_html = email_config.get(
	203	registration_template_failure_html = email_config.get(
221	204	"registration_template_failure_html", "registration_failure.html"
222	205	)
223		self.email_add_threepid_template_failure_html = email_config.get(
	206	add_threepid_template_failure_html = email_config.get(
224	207	"add_threepid_template_failure_html", "add_threepid_failure.html"
225	208	)
226	209
227	210	# These templates do not support any placeholder variables, so we
228	211	# will read them from disk once during setup
229		email_password_reset_template_success_html = email_config.get(
	212	password_reset_template_success_html = email_config.get(
230	213	"password_reset_template_success_html", "password_reset_success.html"
231	214	)
232		email_registration_template_success_html = email_config.get(
	215	registration_template_success_html = email_config.get(
233	216	"registration_template_success_html", "registration_success.html"
234	217	)
235		email_add_threepid_template_success_html = email_config.get(
	218	add_threepid_template_success_html = email_config.get(
236	219	"add_threepid_template_success_html", "add_threepid_success.html"
237	220	)
238	221
239		# Check templates exist
240		for f in [
	222	# Read all templates from disk
	223	(
241	224	self.email_password_reset_template_html,
242	225	self.email_password_reset_template_text,
243	226	self.email_registration_template_html,

247	230	self.email_password_reset_template_failure_html,
248	231	self.email_registration_template_failure_html,
249	232	self.email_add_threepid_template_failure_html,
250		email_password_reset_template_success_html,
251		email_registration_template_success_html,
252		email_add_threepid_template_success_html,
253		]:
254		p = os.path.join(self.email_template_dir, f)
255		if not os.path.isfile(p):
256		raise ConfigError("Unable to find template file %s" % (p,))
257
258		# Retrieve content of web templates
259		filepath = os.path.join(
260		self.email_template_dir, email_password_reset_template_success_html
261		)
262		self.email_password_reset_template_success_html = self.read_file(
263		filepath, "email.password_reset_template_success_html"
264		)
265		filepath = os.path.join(
266		self.email_template_dir, email_registration_template_success_html
267		)
268		self.email_registration_template_success_html_content = self.read_file(
269		filepath, "email.registration_template_success_html"
270		)
271		filepath = os.path.join(
272		self.email_template_dir, email_add_threepid_template_success_html
273		)
274		self.email_add_threepid_template_success_html_content = self.read_file(
275		filepath, "email.add_threepid_template_success_html"
	233	password_reset_template_success_html_template,
	234	registration_template_success_html_template,
	235	add_threepid_template_success_html_template,
	236	) = self.read_templates(
	237	[
	238	password_reset_template_html,
	239	password_reset_template_text,
	240	registration_template_html,
	241	registration_template_text,
	242	add_threepid_template_html,
	243	add_threepid_template_text,
	244	password_reset_template_failure_html,
	245	registration_template_failure_html,
	246	add_threepid_template_failure_html,
	247	password_reset_template_success_html,
	248	registration_template_success_html,
	249	add_threepid_template_success_html,
	250	],
	251	template_dir,
	252	)
	253
	254	# Render templates that do not contain any placeholders
	255	self.email_password_reset_template_success_html_content = (
	256	password_reset_template_success_html_template.render()
	257	)
	258	self.email_registration_template_success_html_content = (
	259	registration_template_success_html_template.render()
	260	)
	261	self.email_add_threepid_template_success_html_content = (
	262	add_threepid_template_success_html_template.render()
276	263	)
277	264
278	265	if self.email_enable_notifs:

289	276	% (", ".join(missing),)
290	277	)
291	278
292		self.email_notif_template_html = email_config.get(
	279	notif_template_html = email_config.get(
293	280	"notif_template_html", "notif_mail.html"
294	281	)
295		self.email_notif_template_text = email_config.get(
	282	notif_template_text = email_config.get(
296	283	"notif_template_text", "notif_mail.txt"
297	284	)
298	285
299		for f in self.email_notif_template_text, self.email_notif_template_html:
300		p = os.path.join(self.email_template_dir, f)
301		if not os.path.isfile(p):
302		raise ConfigError("Unable to find email template file %s" % (p,))
	286	(
	287	self.email_notif_template_html,
	288	self.email_notif_template_text,
	289	) = self.read_templates(
	290	[notif_template_html, notif_template_text], template_dir,
	291	)
303	292
304	293	self.email_notif_for_new_users = email_config.get(
305	294	"notif_for_new_users", True

308	297	"client_base_url", email_config.get("riot_base_url", None)
309	298	)
310	299
311		if account_validity_renewal_enabled:
312		self.email_expiry_template_html = email_config.get(
	300	if self.account_validity.renew_by_email_enabled:
	301	expiry_template_html = email_config.get(
313	302	"expiry_template_html", "notice_expiry.html"
314	303	)
315		self.email_expiry_template_text = email_config.get(
	304	expiry_template_text = email_config.get(
316	305	"expiry_template_text", "notice_expiry.txt"
317	306	)
318	307
319		for f in self.email_expiry_template_text, self.email_expiry_template_html:
320		p = os.path.join(self.email_template_dir, f)
321		if not os.path.isfile(p):
322		raise ConfigError("Unable to find email template file %s" % (p,))
	308	(
	309	self.account_validity_template_html,
	310	self.account_validity_template_text,
	311	) = self.read_templates(
	312	[expiry_template_html, expiry_template_text], template_dir,
	313	)
323	314
324	315	subjects_config = email_config.get("subjects", {})
325	316	subjects = {}

399	390	# Directory in which Synapse will try to find the template files below.
400	391	# If not set, default templates from within the Synapse package will be used.
401	392	#
402		# DO NOT UNCOMMENT THIS SETTING unless you want to customise the templates.
403		# If you do uncomment it, you will need to make sure that all the templates
404		# below are in the directory.
	393	# Do not uncomment this setting unless you want to customise the templates.
405	394	#
406	395	# Synapse will look for the following templates in this directory:
407	396	#

+1

-1

synapse/config/key.py less more

81	81
82	82
83	83	@attr.s
84		class TrustedKeyServer(object):
	84	class TrustedKeyServer:
85	85	# string: name of the server.
86	86	server_name = attr.ib()
87	87

+1

-1

synapse/config/metrics.py less more

21	21
22	22
23	23	@attr.s
24		class MetricsFlags(object):
	24	class MetricsFlags:
25	25	known_servers = attr.ib(default=False, validator=attr.validators.instance_of(bool))
26	26
27	27	@classmethod

+2

-2

synapse/config/ratelimiting.py less more

16	16	from ._base import Config
17	17
18	18
19		class RateLimitConfig(object):
	19	class RateLimitConfig:
20	20	def __init__(
21	21	self,
22	22	config: Dict[str, float],

26	26	self.burst_count = config.get("burst_count", defaults["burst_count"])
27	27
28	28
29		class FederationRateLimitConfig(object):
	29	class FederationRateLimitConfig:
30	30	_items_and_default = {
31	31	"window_size": 1000,
32	32	"sleep_limit": 10,

+1

-1

synapse/config/room.py less more

21	21	logger = logging.Logger(__name__)
22	22
23	23
24		class RoomDefaultEncryptionTypes(object):
	24	class RoomDefaultEncryptionTypes:
25	25	"""Possible values for the encryption_enabled_by_default_for_room_type config option"""
26	26
27	27	ALL = "all"

+1

-1

synapse/config/room_directory.py less more

148	148	return False
149	149
150	150
151		class _RoomDirectoryRule(object):
	151	class _RoomDirectoryRule:
152	152	"""Helper class to test whether a room directory action is allowed, like
153	153	creating an alias or publishing a room.
154	154	"""

+5

-11

synapse/config/saml2_config.py less more

17	17	from typing import Any, List
18	18
19	19	import attr
20		import jinja2
21		import pkg_resources
22	20
23	21	from synapse.python_dependencies import DependencyException, check_requirements
24	22	from synapse.util.module_loader import load_module, load_python_module

170	168	saml2_config.get("saml_session_lifetime", "15m")
171	169	)
172	170
173		template_dir = saml2_config.get("template_dir")
174		if not template_dir:
175		template_dir = pkg_resources.resource_filename("synapse", "res/templates",)
176
177		loader = jinja2.FileSystemLoader(template_dir)
178		# enable auto-escape here, to having to remember to escape manually in the
179		# template
180		env = jinja2.Environment(loader=loader, autoescape=True)
181		self.saml2_error_html_template = env.get_template("saml_error.html")
	171	# We enable autoescape here as the message may potentially come from a
	172	# remote resource
	173	self.saml2_error_html_template = self.read_templates(
	174	["saml_error.html"], saml2_config.get("template_dir"), autoescape=True
	175	)[0]
182	176
183	177	def _default_saml_config_dict(
184	178	self, required_attributes: set, optional_attributes: set

+15

-29

synapse/config/server.py less more

25	25
26	26	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
27	27	from synapse.http.endpoint import parse_and_validate_server_name
28		from synapse.python_dependencies import DependencyException, check_requirements
29	28
30	29	from ._base import Config, ConfigError
31	30

424	423	self.gc_thresholds = read_gc_thresholds(config.get("gc_thresholds", None))
425	424
426	425	@attr.s
427		class LimitRemoteRoomsConfig(object):
	426	class LimitRemoteRoomsConfig:
428	427	enabled = attr.ib(
429	428	validator=attr.validators.instance_of(bool), default=False
430	429	)

506	505	),
507	506	)
508	507	)
509
510		_check_resource_config(self.listeners)
511	508
512	509	self.cleanup_extremities_with_dummy_events = config.get(
513	510	"cleanup_extremities_with_dummy_events", True

963	960	# min_lifetime: 1d
964	961	# max_lifetime: 1y
965	962
966		# Retention policy limits. If set, a user won't be able to send a
967		# 'm.room.retention' event which features a 'min_lifetime' or a 'max_lifetime'
968		# that's not within this range. This is especially useful in closed federations,
969		# in which server admins can make sure every federating server applies the same
970		# rules.
	963	# Retention policy limits. If set, and the state of a room contains a
	964	# 'm.room.retention' event in its state which contains a 'min_lifetime' or a
	965	# 'max_lifetime' that's out of these bounds, Synapse will cap the room's policy
	966	# to these limits when running purge jobs.
971	967	#
972	968	#allowed_lifetime_min: 1d
973	969	#allowed_lifetime_max: 1y

993	989	# (e.g. every 12h), but not want that purge to be performed by a job that's
994	990	# iterating over every room it knows, which could be heavy on the server.
995	991	#
	992	# If any purge job is configured, it is strongly recommended to have at least
	993	# a single job with neither 'shortest_max_lifetime' nor 'longest_max_lifetime'
	994	# set, or one job without 'shortest_max_lifetime' and one job without
	995	# 'longest_max_lifetime' set. Otherwise some rooms might be ignored, even if
	996	# 'allowed_lifetime_min' and 'allowed_lifetime_max' are set, because capping a
	997	# room's policy to these values is done after the policies are retrieved from
	998	# Synapse's database (which is done using the range specified in a purge job's
	999	# configuration).
	1000	#
996	1001	#purge_jobs:
997		# - shortest_max_lifetime: 1d
998		# longest_max_lifetime: 3d
	1002	# - longest_max_lifetime: 3d
999	1003	# interval: 12h
1000	1004	# - shortest_max_lifetime: 3d
1001		# longest_max_lifetime: 1y
1002	1005	# interval: 1d
1003	1006
1004	1007	# Inhibits the /requestToken endpoints from returning an error that might leak

1132	1135	if name == "webclient":
1133	1136	logger.warning(NO_MORE_WEB_CLIENT_WARNING)
1134	1137	return
1135
1136
1137		def _check_resource_config(listeners: Iterable[ListenerConfig]) -> None:
1138		resource_names = {
1139		res_name
1140		for listener in listeners
1141		if listener.http_options
1142		for res in listener.http_options.resources
1143		for res_name in res.names
1144		}
1145
1146		for resource in resource_names:
1147		if resource == "consent":
1148		try:
1149		check_requirements("resources.consent")
1150		except DependencyException as e:
1151		raise ConfigError(e.message)

+22

-15

synapse/config/sso.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14		import os
15	14	from typing import Any, Dict
16
17		import pkg_resources
18	15
19	16	from ._base import Config
20	17

28	25	def read_config(self, config, **kwargs):
29	26	sso_config = config.get("sso") or {} # type: Dict[str, Any]
30	27
31		# Pick a template directory in order of:
32		# * The sso-specific template_dir
33		# * /path/to/synapse/install/res/templates
	28	# The sso-specific template_dir
34	29	template_dir = sso_config.get("template_dir")
35		if not template_dir:
36		template_dir = pkg_resources.resource_filename("synapse", "res/templates",)
37	30
38		self.sso_template_dir = template_dir
39		self.sso_account_deactivated_template = self.read_file(
40		os.path.join(self.sso_template_dir, "sso_account_deactivated.html"),
41		"sso_account_deactivated_template",
	31	# Read templates from disk
	32	(
	33	self.sso_redirect_confirm_template,
	34	self.sso_auth_confirm_template,
	35	self.sso_error_template,
	36	sso_account_deactivated_template,
	37	sso_auth_success_template,
	38	) = self.read_templates(
	39	[
	40	"sso_redirect_confirm.html",
	41	"sso_auth_confirm.html",
	42	"sso_error.html",
	43	"sso_account_deactivated.html",
	44	"sso_auth_success.html",
	45	],
	46	template_dir,
42	47	)
43		self.sso_auth_success_template = self.read_file(
44		os.path.join(self.sso_template_dir, "sso_auth_success.html"),
45		"sso_auth_success_template",
	48
	49	# These templates have no placeholders, so render them here
	50	self.sso_account_deactivated_template = (
	51	sso_account_deactivated_template.render()
46	52	)
	53	self.sso_auth_success_template = sso_auth_success_template.render()
47	54
48	55	self.sso_client_whitelist = sso_config.get("client_whitelist") or []
49	56

+4

-4

synapse/crypto/context_factory.py less more

82	82
83	83
84	84	@implementer(IPolicyForHTTPS)
85		class FederationPolicyForHTTPS(object):
	85	class FederationPolicyForHTTPS:
86	86	"""Factory for Twisted SSLClientConnectionCreators that are used to make connections
87	87	to remote servers for federation.
88	88

151	151
152	152
153	153	@implementer(IPolicyForHTTPS)
154		class RegularPolicyForHTTPS(object):
	154	class RegularPolicyForHTTPS:
155	155	"""Factory for Twisted SSLClientConnectionCreators that are used to make connections
156	156	to remote servers, for other than federation.
157	157

188	188
189	189
190	190	@implementer(IOpenSSLClientConnectionCreator)
191		class SSLClientConnectionCreator(object):
	191	class SSLClientConnectionCreator:
192	192	"""Creates openssl connection objects for client connections.
193	193
194	194	Replaces twisted.internet.ssl.ClientTLSOptions

213	213	return connection
214	214
215	215
216		class ConnectionVerifier(object):
	216	class ConnectionVerifier:
217	217	"""Set the SNI, and do cert verification
218	218
219	219	This is a thing which is attached to the TLSMemoryBIOProtocol, and is called by

+7

-8

synapse/crypto/keyring.py less more

56	56
57	57
58	58	@attr.s(slots=True, cmp=False)
59		class VerifyJsonRequest(object):
	59	class VerifyJsonRequest:
60	60	"""
61	61	A request to verify a JSON object.
62	62

95	95	pass
96	96
97	97
98		class Keyring(object):
	98	class Keyring:
99	99	def __init__(self, hs, key_fetchers=None):
100	100	self.clock = hs.get_clock()
101	101

419	419	remaining_requests.difference_update(completed)
420	420
421	421
422		class KeyFetcher(object):
	422	class KeyFetcher:
423	423	async def get_keys(self, keys_to_fetch):
424	424	"""
425	425	Args:

455	455	return keys
456	456
457	457
458		class BaseV2KeyFetcher(object):
	458	class BaseV2KeyFetcher:
459	459	def __init__(self, hs):
460	460	self.store = hs.get_datastore()
461	461	self.config = hs.get_config()

756	756	except Exception:
757	757	logger.exception("Error getting keys %s from %s", key_ids, server_name)
758	758
759		return await yieldable_gather_results(
760		get_key, keys_to_fetch.items()
761		).addCallback(lambda _: results)
	759	await yieldable_gather_results(get_key, keys_to_fetch.items())
	760	return results
762	761
763	762	async def get_server_verify_key_v2_direct(self, server_name, key_ids):
764	763	"""

768	767	key_ids (iterable[str]):
769	768
770	769	Returns:
771		Deferred[dict[str, FetchKeyResult]]: map from key ID to lookup result
	770	dict[str, FetchKeyResult]: map from key ID to lookup result
772	771
773	772	Raises:
774	773	KeyLookupError if there was a problem making the lookup

+1

-1

synapse/event_auth.py less more

46	46	Args:
47	47	room_version_obj: the version of the room
48	48	event: the event being checked.
49		auth_events (dict: event-key -> event): the existing room state.
	49	auth_events: the existing room state.
50	50
51	51	Raises:
52	52	AuthError if the checks fail

+5

-3

synapse/events/__init__.py less more

17	17	import abc
18	18	import os
19	19	from distutils.util import strtobool
20		from typing import Dict, Optional, Type
	20	from typing import Dict, Optional, Tuple, Type
21	21
22	22	from unpaddedbase64 import encode_base64
23	23

95	95	return instance._dict.get(self.key, self.default)
96	96
97	97
98		class _EventInternalMetadata(object):
	98	class _EventInternalMetadata:
99	99	__slots__ = ["_dict"]
100	100
101	101	def __init__(self, internal_metadata_dict: JsonDict):

119	119	# be here
120	120	before = DictProperty("before") # type: str
121	121	after = DictProperty("after") # type: str
122		order = DictProperty("order") # type: int
	122	order = DictProperty("order") # type: Tuple[int, int]
123	123
124	124	def get_dict(self) -> JsonDict:
125	125	return dict(self._dict)

132	132	rejection. This is needed as those events are marked as outliers, but
133	133	they still need to be processed as if they're new events (e.g. updating
134	134	invite state in the database, relaying to clients, etc).
	135
	136	(Added in synapse 0.99.0, so may be unreliable for events received before that)
135	137	"""
136	138	return self._dict.get("out_of_band_membership", False)
137	139

+14

-9

synapse/events/builder.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14		from typing import Optional
	14	from typing import Any, Dict, List, Optional, Tuple, Union
15	15
16	16	import attr
17	17	from nacl.signing import SigningKey

35	35
36	36
37	37	@attr.s(slots=True, cmp=False, frozen=True)
38		class EventBuilder(object):
	38	class EventBuilder:
39	39	"""A format independent event builder used to build up the event content
40	40	before signing the event.
41	41

96	96	def is_state(self):
97	97	return self._state_key is not None
98	98
99		async def build(self, prev_event_ids):
	99	async def build(self, prev_event_ids: List[str]) -> EventBase:
100	100	"""Transform into a fully signed and hashed event
101	101
102	102	Args:
103		prev_event_ids (list[str]): The event IDs to use as the prev events
	103	prev_event_ids: The event IDs to use as the prev events
104	104
105	105	Returns:
106		FrozenEvent
	106	The signed and hashed event.
107	107	"""
108	108
109	109	state_ids = await self._state.get_current_state_ids(

113	113
114	114	format_version = self.room_version.event_format
115	115	if format_version == EventFormatVersions.V1:
116		auth_events = await self._store.add_event_hashes(auth_ids)
117		prev_events = await self._store.add_event_hashes(prev_event_ids)
	116	# The types of auth/prev events changes between event versions.
	117	auth_events = await self._store.add_event_hashes(
	118	auth_ids
	119	) # type: Union[List[str], List[Tuple[str, Dict[str, str]]]]
	120	prev_events = await self._store.add_event_hashes(
	121	prev_event_ids
	122	) # type: Union[List[str], List[Tuple[str, Dict[str, str]]]]
118	123	else:
119	124	auth_events = auth_ids
120	125	prev_events = prev_event_ids

137	142	"unsigned": self.unsigned,
138	143	"depth": depth,
139	144	"prev_state": [],
140		}
	145	} # type: Dict[str, Any]
141	146
142	147	if self.is_state():
143	148	event_dict["state_key"] = self._state_key

158	163	)
159	164
160	165
161		class EventBuilderFactory(object):
	166	class EventBuilderFactory:
162	167	def __init__(self, hs):
163	168	self.clock = hs.get_clock()
164	169	self.hostname = hs.hostname

+34

-3

synapse/events/spamcheck.py less more

14	14	# limitations under the License.
15	15
16	16	import inspect
17		from typing import Any, Dict, List
	17	from typing import Any, Dict, List, Optional, Tuple
18	18
19		from synapse.spam_checker_api import SpamCheckerApi
	19	from synapse.spam_checker_api import RegistrationBehaviour, SpamCheckerApi
	20	from synapse.types import Collection
20	21
21	22	MYPY = False
22	23	if MYPY:
23	24	import synapse.server
24	25
25	26
26		class SpamChecker(object):
	27	class SpamChecker:
27	28	def __init__(self, hs: "synapse.server.HomeServer"):
28	29	self.spam_checkers = [] # type: List[Any]
29	30

159	160	return True
160	161
161	162	return False
	163
	164	def check_registration_for_spam(
	165	self,
	166	email_threepid: Optional[dict],
	167	username: Optional[str],
	168	request_info: Collection[Tuple[str, str]],
	169	) -> RegistrationBehaviour:
	170	"""Checks if we should allow the given registration request.
	171
	172	Args:
	173	email_threepid: The email threepid used for registering, if any
	174	username: The request user name, if any
	175	request_info: List of tuples of user agent and IP that
	176	were used during the registration process.
	177
	178	Returns:
	179	Enum for how the request should be handled
	180	"""
	181
	182	for spam_checker in self.spam_checkers:
	183	# For backwards compatibility, only run if the method exists on the
	184	# spam checker
	185	checker = getattr(spam_checker, "check_registration_for_spam", None)
	186	if checker:
	187	behaviour = checker(email_threepid, username, request_info)
	188	assert isinstance(behaviour, RegistrationBehaviour)
	189	if behaviour != RegistrationBehaviour.ALLOW:
	190	return behaviour
	191
	192	return RegistrationBehaviour.ALLOW

+1

-1

synapse/events/third_party_rules.py less more

17	17	from synapse.types import Requester
18	18
19	19
20		class ThirdPartyEventRules(object):
	20	class ThirdPartyEventRules:
21	21	"""Allows server admins to provide a Python module implementing an extra
22	22	set of rules to apply when processing events.
23	23

+1

-1

synapse/events/utils.py less more

321	321	return d
322	322
323	323
324		class EventClientSerializer(object):
	324	class EventClientSerializer:
325	325	"""Serializes events that are to be sent to clients.
326	326
327	327	This is used for bundling extra information with any events to be sent to

+4

-57

synapse/events/validator.py less more

19	19	from synapse.types import EventID, RoomID, UserID
20	20
21	21
22		class EventValidator(object):
	22	class EventValidator:
23	23	def validate_new(self, event, config):
24	24	"""Validates the event has roughly the right format
25	25

73	73	)
74	74
75	75	if event.type == EventTypes.Retention:
76		self._validate_retention(event, config)
	76	self._validate_retention(event)
77	77
78		def _validate_retention(self, event, config):
	78	def _validate_retention(self, event):
79	79	"""Checks that an event that defines the retention policy for a room respects the
80		boundaries imposed by the server's administrator.
	80	format enforced by the spec.
81	81
82	82	Args:
83	83	event (FrozenEvent): The event to validate.
84		config (Config): The homeserver's configuration.
85	84	"""
86	85	min_lifetime = event.content.get("min_lifetime")
87	86	max_lifetime = event.content.get("max_lifetime")

94	93	errcode=Codes.BAD_JSON,
95	94	)
96	95
97		if (
98		config.retention_allowed_lifetime_min is not None
99		and min_lifetime < config.retention_allowed_lifetime_min
100		):
101		raise SynapseError(
102		code=400,
103		msg=(
104		"'min_lifetime' can't be lower than the minimum allowed"
105		" value enforced by the server's administrator"
106		),
107		errcode=Codes.BAD_JSON,
108		)
109
110		if (
111		config.retention_allowed_lifetime_max is not None
112		and min_lifetime > config.retention_allowed_lifetime_max
113		):
114		raise SynapseError(
115		code=400,
116		msg=(
117		"'min_lifetime' can't be greater than the maximum allowed"
118		" value enforced by the server's administrator"
119		),
120		errcode=Codes.BAD_JSON,
121		)
122
123	96	if max_lifetime is not None:
124	97	if not isinstance(max_lifetime, int):
125	98	raise SynapseError(
126	99	code=400,
127	100	msg="'max_lifetime' must be an integer",
128		errcode=Codes.BAD_JSON,
129		)
130
131		if (
132		config.retention_allowed_lifetime_min is not None
133		and max_lifetime < config.retention_allowed_lifetime_min
134		):
135		raise SynapseError(
136		code=400,
137		msg=(
138		"'max_lifetime' can't be lower than the minimum allowed value"
139		" enforced by the server's administrator"
140		),
141		errcode=Codes.BAD_JSON,
142		)
143
144		if (
145		config.retention_allowed_lifetime_max is not None
146		and max_lifetime > config.retention_allowed_lifetime_max
147		):
148		raise SynapseError(
149		code=400,
150		msg=(
151		"'max_lifetime' can't be greater than the maximum allowed"
152		" value enforced by the server's administrator"
153		),
154	101	errcode=Codes.BAD_JSON,
155	102	)
156	103

+1

-1

synapse/federation/federation_base.py less more

38	38	logger = logging.getLogger(__name__)
39	39
40	40
41		class FederationBase(object):
	41	class FederationBase:
42	42	def __init__(self, hs):
43	43	self.hs = hs
44	44

+7

-8

synapse/federation/federation_client.py less more

53	53	from synapse.federation.federation_base import FederationBase, event_from_pdu_json
54	54	from synapse.logging.context import make_deferred_yieldable, preserve_fn
55	55	from synapse.logging.utils import log_function
56		from synapse.types import JsonDict
	56	from synapse.types import JsonDict, get_domain_from_id
57	57	from synapse.util import unwrapFirstError
58	58	from synapse.util.caches.expiringcache import ExpiringCache
59	59	from synapse.util.retryutils import NotRetryingDestination

216	216	for p in transaction_data["pdus"]
217	217	]
218	218
219		# FIXME: We should handle signature failures more gracefully.
220		pdus[:] = await make_deferred_yieldable(
221		defer.gatherResults(
222		self._check_sigs_and_hashes(room_version, pdus), consumeErrors=True,
223		).addErrback(unwrapFirstError)
	219	# Check signatures and hash of pdus, removing any from the list that fail checks
	220	pdus[:] = await self._check_sigs_and_hash_and_fetch(
	221	dest, pdus, outlier=True, room_version=room_version
224	222	)
225	223
226	224	return pdus

385	383	pdu.event_id, allow_rejected=True, allow_none=True
386	384	)
387	385
388		if not res and pdu.origin != origin:
	386	pdu_origin = get_domain_from_id(pdu.sender)
	387	if not res and pdu_origin != origin:
389	388	try:
390	389	res = await self.get_pdu(
391		destinations=[pdu.origin],
	390	destinations=[pdu_origin],
392	391	event_id=pdu.event_id,
393	392	room_version=room_version,
394	393	outlier=outlier,

+45

-18

synapse/federation/federation_server.py less more

27	27	Union,
28	28	)
29	29
30		from canonicaljson import json
31	30	from prometheus_client import Counter, Histogram
32	31
33	32	from twisted.internet import defer

62	61	ReplicationGetQueryRestServlet,
63	62	)
64	63	from synapse.types import JsonDict, get_domain_from_id
65		from synapse.util import glob_to_regex, unwrapFirstError
	64	from synapse.util import glob_to_regex, json_decoder, unwrapFirstError
66	65	from synapse.util.async_helpers import Linearizer, concurrently_execute
67	66	from synapse.util.caches.response_cache import ResponseCache
68	67

97	96	self.state = hs.get_state_handler()
98	97
99	98	self.device_handler = hs.get_device_handler()
	99	self._federation_ratelimiter = hs.get_federation_ratelimiter()
100	100
101	101	self._server_linearizer = Linearizer("fed_server")
102	102	self._transaction_linearizer = Linearizer("fed_txn_handler")
	103
	104	# We cache results for transaction with the same ID
	105	self._transaction_resp_cache = ResponseCache(
	106	hs, "fed_txn_handler", timeout_ms=30000
	107	)
103	108
104	109	self.transaction_actions = TransactionActions(self.store)
105	110

135	140	request_time = self._clock.time_msec()
136	141
137	142	transaction = Transaction(**transaction_data)
138
139		if not transaction.transaction_id: # type: ignore
	143	transaction_id = transaction.transaction_id # type: ignore
	144
	145	if not transaction_id:
140	146	raise Exception("Transaction missing transaction_id")
141	147
142		logger.debug("[%s] Got transaction", transaction.transaction_id) # type: ignore
143
144		# use a linearizer to ensure that we don't process the same transaction
145		# multiple times in parallel.
146		with (
147		await self._transaction_linearizer.queue(
148		(origin, transaction.transaction_id) # type: ignore
149		)
150		):
151		result = await self._handle_incoming_transaction(
152		origin, transaction, request_time
153		)
	148	logger.debug("[%s] Got transaction", transaction_id)
	149
	150	# We wrap in a ResponseCache so that we de-duplicate retried
	151	# transactions.
	152	return await self._transaction_resp_cache.wrap(
	153	(origin, transaction_id),
	154	self._on_incoming_transaction_inner,
	155	origin,
	156	transaction,
	157	request_time,
	158	)
	159
	160	async def _on_incoming_transaction_inner(
	161	self, origin: str, transaction: Transaction, request_time: int
	162	) -> Tuple[int, Dict[str, Any]]:
	163	# Use a linearizer to ensure that transactions from a remote are
	164	# processed in order.
	165	with await self._transaction_linearizer.queue(origin):
	166	# We rate limit here after we've queued up the incoming requests,
	167	# so that we don't fill up the ratelimiter with blocked requests.
	168	#
	169	# This is important as the ratelimiter allows N concurrent requests
	170	# at a time, and only starts ratelimiting if there are more requests
	171	# than that being processed at a time. If we queued up requests in
	172	# the linearizer/response cache after the ratelimiting then those
	173	# queued up requests would count as part of the allowed limit of N
	174	# concurrent requests.
	175	with self._federation_ratelimiter.ratelimit(origin) as d:
	176	await d
	177
	178	result = await self._handle_incoming_transaction(
	179	origin, transaction, request_time
	180	)
154	181
155	182	return result
156	183

550	577	for device_id, keys in device_keys.items():
551	578	for key_id, json_str in keys.items():
552	579	json_result.setdefault(user_id, {})[device_id] = {
553		key_id: json.loads(json_str)
	580	key_id: json_decoder.decode(json_str)
554	581	}
555	582
556	583	logger.info(

785	812	return regex.match(server_name)
786	813
787	814
788		class FederationHandlerRegistry(object):
	815	class FederationHandlerRegistry:
789	816	"""Allows classes to register themselves as handlers for a given EDU or
790	817	query type for incoming federation traffic.
791	818	"""

+21

-16

synapse/federation/persistence.py less more

19	19	"""
20	20
21	21	import logging
	22	from typing import Optional, Tuple
22	23
	24	from synapse.federation.units import Transaction
23	25	from synapse.logging.utils import log_function
	26	from synapse.types import JsonDict
24	27
25	28	logger = logging.getLogger(__name__)
26	29
27	30
28		class TransactionActions(object):
	31	class TransactionActions:
29	32	""" Defines persistence actions that relate to handling Transactions.
30	33	"""
31	34

33	36	self.store = datastore
34	37
35	38	@log_function
36		def have_responded(self, origin, transaction):
37		""" Have we already responded to a transaction with the same id and
	39	async def have_responded(
	40	self, origin: str, transaction: Transaction
	41	) -> Optional[Tuple[int, JsonDict]]:
	42	"""Have we already responded to a transaction with the same id and
38	43	origin?
39	44
40	45	Returns:
41		Deferred: Results in `None` if we have not previously responded to
42		this transaction or a 2-tuple of `(int, dict)` representing the
43		response code and response body.
	46	`None` if we have not previously responded to this transaction or a
	47	2-tuple of `(int, dict)` representing the response code and response body.
44	48	"""
45		if not transaction.transaction_id:
	49	transaction_id = transaction.transaction_id # type: ignore
	50	if not transaction_id:
46	51	raise RuntimeError("Cannot persist a transaction with no transaction_id")
47	52
48		return self.store.get_received_txn_response(transaction.transaction_id, origin)
	53	return await self.store.get_received_txn_response(transaction_id, origin)
49	54
50	55	@log_function
51		def set_response(self, origin, transaction, code, response):
52		""" Persist how we responded to a transaction.
53
54		Returns:
55		Deferred
	56	async def set_response(
	57	self, origin: str, transaction: Transaction, code: int, response: JsonDict
	58	) -> None:
	59	"""Persist how we responded to a transaction.
56	60	"""
57		if not transaction.transaction_id:
	61	transaction_id = transaction.transaction_id # type: ignore
	62	if not transaction_id:
58	63	raise RuntimeError("Cannot persist a transaction with no transaction_id")
59	64
60		return self.store.set_received_txn_response(
61		transaction.transaction_id, origin, code, response
	65	await self.store.set_received_txn_response(
	66	transaction_id, origin, code, response
62	67	)

+3

-3

synapse/federation/send_queue.py less more

36	36
37	37	from twisted.internet import defer
38	38
	39	from synapse.api.presence import UserPresenceState
39	40	from synapse.metrics import LaterGauge
40		from synapse.storage.presence import UserPresenceState
41	41	from synapse.util.metrics import Measure
42	42
43	43	from .units import Edu

45	45	logger = logging.getLogger(__name__)
46	46
47	47
48		class FederationRemoteSendQueue(object):
	48	class FederationRemoteSendQueue:
49	49	"""A drop in replacement for FederationSender"""
50	50
51	51	def __init__(self, hs):

364	364	)
365	365
366	366
367		class BaseFederationRow(object):
	367	class BaseFederationRow:
368	368	"""Base class for rows to be sent in the federation stream.
369	369
370	370	Specifies how to identify, serialize and deserialize the different types.

+5

-10

synapse/federation/sender/__init__.py less more

21	21
22	22	import synapse
23	23	import synapse.metrics
	24	from synapse.api.presence import UserPresenceState
24	25	from synapse.events import EventBase
25	26	from synapse.federation.sender.per_destination_queue import PerDestinationQueue
26	27	from synapse.federation.sender.transaction_manager import TransactionManager

38	39	events_processed_counter,
39	40	)
40	41	from synapse.metrics.background_process_metrics import run_as_background_process
41		from synapse.storage.presence import UserPresenceState
42	42	from synapse.types import ReadReceipt
43	43	from synapse.util.metrics import Measure, measure_func
44	44

55	55	)
56	56
57	57
58		class FederationSender(object):
	58	class FederationSender:
59	59	def __init__(self, hs: "synapse.server.HomeServer"):
60	60	self.hs = hs
61	61	self.server_name = hs.hostname

106	106	d.pending_edu_count() for d in self._per_destination_queues.values()
107	107	),
108	108	)
109
110		self._order = 1
111	109
112	110	self._is_processing = False
113	111	self._last_poked_id = -1

271	269	# a transaction in progress. If we do, stick it in the pending_pdus
272	270	# table and we'll get back to it later.
273	271
274		order = self._order
275		self._order += 1
276
277	272	destinations = set(destinations)
278	273	destinations.discard(self.server_name)
279	274	logger.debug("Sending to: %s", str(destinations))

285	280	sent_pdus_destination_dist_count.inc()
286	281
287	282	for destination in destinations:
288		self._get_per_destination_queue(destination).send_pdu(pdu, order)
	283	self._get_per_destination_queue(destination).send_pdu(pdu)
289	284
290	285	async def send_read_receipt(self, receipt: ReadReceipt) -> None:
291	286	"""Send a RR to any other servers in the room

328	323	room_id = receipt.room_id
329	324
330	325	# Work out which remote servers should be poked and poke them.
331		domains = await self.state.get_current_hosts_in_room(room_id)
	326	domains_set = await self.state.get_current_hosts_in_room(room_id)
332	327	domains = [
333	328	d
334		for d in domains
	329	for d in domains_set
335	330	if d != self.server_name
336	331	and self._federation_shard_config.should_handle(self._instance_name, d)
337	332	]

+32

-11

synapse/federation/sender/per_destination_queue.py less more

23	23	HttpResponseException,
24	24	RequestSendFailed,
25	25	)
	26	from synapse.api.presence import UserPresenceState
26	27	from synapse.events import EventBase
27	28	from synapse.federation.units import Edu
28	29	from synapse.handlers.presence import format_user_presence_state
29	30	from synapse.metrics import sent_transactions_counter
30	31	from synapse.metrics.background_process_metrics import run_as_background_process
31		from synapse.storage.presence import UserPresenceState
32	32	from synapse.types import ReadReceipt
33	33	from synapse.util.retryutils import NotRetryingDestination, get_retry_limiter
34	34

52	52	)
53	53
54	54
55		class PerDestinationQueue(object):
	55	class PerDestinationQueue:
56	56	"""
57	57	Manages the per-destination transmission queues.
58	58

91	91	self._destination = destination
92	92	self.transmission_loop_running = False
93	93
94		# a list of tuples of (pending pdu, order)
95		self._pending_pdus = [] # type: List[Tuple[EventBase, int]]
	94	# a list of pending PDUs
	95	self._pending_pdus = [] # type: List[EventBase]
96	96
97	97	# XXX this is never actually used: see
98	98	# https://github.com/matrix-org/synapse/issues/7549

131	131	+ len(self._pending_edus_keyed)
132	132	)
133	133
134		def send_pdu(self, pdu: EventBase, order: int) -> None:
	134	def send_pdu(self, pdu: EventBase) -> None:
135	135	"""Add a PDU to the queue, and start the transmission loop if necessary
136	136
137	137	Args:
138	138	pdu: pdu to send
139		order
140	139	"""
141		self._pending_pdus.append((pdu, order))
	140	self._pending_pdus.append(pdu)
142	141	self.attempt_new_transaction()
143	142
144	143	def send_presence(self, states: Iterable[UserPresenceState]) -> None:

184	183	returns immediately. Otherwise kicks off the process of sending a
185	184	transaction in the background.
186	185	"""
187		# list of (pending_pdu, deferred, order)
	186
188	187	if self.transmission_loop_running:
189	188	# XXX: this can get stuck on by a never-ending
190	189	# request at which point pending_pdus just keeps growing.

209	208	)
210	209
211	210	async def _transaction_transmission_loop(self) -> None:
212		pending_pdus = [] # type: List[Tuple[EventBase, int]]
	211	pending_pdus = [] # type: List[EventBase]
213	212	try:
214	213	self.transmission_loop_running = True
215	214

336	335	(e.retry_last_ts + e.retry_interval) / 1000.0
337	336	),
338	337	)
	338
	339	if e.retry_interval > 60 * 60 * 1000:
	340	# we won't retry for another hour!
	341	# (this suggests a significant outage)
	342	# We drop pending PDUs and EDUs because otherwise they will
	343	# rack up indefinitely.
	344	# Note that:
	345	# - the EDUs that are being dropped here are those that we can
	346	# afford to drop (specifically, only typing notifications,
	347	# read receipts and presence updates are being dropped here)
	348	# - Other EDUs such as to_device messages are queued with a
	349	# different mechanism
	350	# - this is all volatile state that would be lost if the
	351	# federation sender restarted anyway
	352
	353	# dropping read receipts is a bit sad but should be solved
	354	# through another mechanism, because this is all volatile!
	355	self._pending_pdus = []
	356	self._pending_edus = []
	357	self._pending_edus_keyed = {}
	358	self._pending_presence = {}
	359	self._pending_rrs = {}
339	360	except FederationDeniedError as e:
340	361	logger.info(e)
341	362	except HttpResponseException as e:

350	371	"TX [%s] Failed to send transaction: %s", self._destination, e
351	372	)
352	373
353		for p, _ in pending_pdus:
	374	for p in pending_pdus:
354	375	logger.info(
355	376	"Failed to send event %s to %s", p.event_id, self._destination
356	377	)
357	378	except Exception:
358	379	logger.exception("TX [%s] Failed to send transaction", self._destination)
359		for p, _ in pending_pdus:
	380	for p in pending_pdus:
360	381	logger.info(
361	382	"Failed to send event %s to %s", p.event_id, self._destination
362	383	)

+16

-17

synapse/federation/sender/transaction_manager.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
15		from typing import TYPE_CHECKING, List, Tuple
16
17		from canonicaljson import json
	15	from typing import TYPE_CHECKING, List
18	16
19	17	from synapse.api.errors import HttpResponseException
20	18	from synapse.events import EventBase

27	25	tags,
28	26	whitelisted_homeserver,
29	27	)
	28	from synapse.util import json_decoder
30	29	from synapse.util.metrics import measure_func
31	30
32	31	if TYPE_CHECKING:

35	34	logger = logging.getLogger(__name__)
36	35
37	36
38		class TransactionManager(object):
	37	class TransactionManager:
39	38	"""Helper class which handles building and sending transactions
40	39
41	40	shared between PerDestinationQueue objects

53	52
54	53	@measure_func("_send_new_transaction")
55	54	async def send_new_transaction(
56		self,
57		destination: str,
58		pending_pdus: List[Tuple[EventBase, int]],
59		pending_edus: List[Edu],
60		):
	55	self, destination: str, pdus: List[EventBase], edus: List[Edu],
	56	) -> bool:
	57	"""
	58	Args:
	59	destination: The destination to send to (e.g. 'example.org')
	60	pdus: In-order list of PDUs to send
	61	edus: List of EDUs to send
	62
	63	Returns:
	64	True iff the transaction was successful
	65	"""
61	66
62	67	# Make a transaction-sending opentracing span. This span follows on from
63	68	# all the edus in that transaction. This needs to be done since there is

67	72	span_contexts = []
68	73	keep_destination = whitelisted_homeserver(destination)
69	74
70		for edu in pending_edus:
	75	for edu in edus:
71	76	context = edu.get_context()
72	77	if context:
73		span_contexts.append(extract_text_map(json.loads(context)))
	78	span_contexts.append(extract_text_map(json_decoder.decode(context)))
74	79	if keep_destination:
75	80	edu.strip_context()
76	81
77	82	with start_active_span_follows_from("send_transaction", span_contexts):
78
79		# Sort based on the order field
80		pending_pdus.sort(key=lambda t: t[1])
81		pdus = [x[0] for x in pending_pdus]
82		edus = pending_edus
83
84	83	success = True
85	84
86	85	logger.debug("TX [%s] _attempt_new_transaction", destination)

+1

-1

synapse/federation/transport/client.py less more

29	29	logger = logging.getLogger(__name__)
30	30
31	31
32		class TransportLayerClient(object):
	32	class TransportLayerClient:
33	33	"""Sends federation HTTP requests to other servers"""
34	34
35	35	def __init__(self, hs):

+10

-7

synapse/federation/transport/server.py less more

44	44	)
45	45	from synapse.server import HomeServer
46	46	from synapse.types import ThirdPartyInstanceID, get_domain_from_id
47		from synapse.util.ratelimitutils import FederationRateLimiter
48	47	from synapse.util.versionstring import get_version_string
49	48
50	49	logger = logging.getLogger(__name__)

71	70	super(TransportLayerServer, self).__init__(hs, canonical_json=False)
72	71
73	72	self.authenticator = Authenticator(hs)
74		self.ratelimiter = FederationRateLimiter(
75		self.clock, config=hs.config.rc_federation
76		)
	73	self.ratelimiter = hs.get_federation_ratelimiter()
77	74
78	75	self.register_servlets()
79	76

99	96	pass
100	97
101	98
102		class Authenticator(object):
	99	class Authenticator:
103	100	def __init__(self, hs: HomeServer):
104	101	self._clock = hs.get_clock()
105	102	self.keyring = hs.get_keyring()

227	224	)
228	225
229	226
230		class BaseFederationServlet(object):
	227	class BaseFederationServlet:
231	228	"""Abstract base class for federation servlet classes.
232	229
233	230	The servlet object should have a PATH attribute which takes the form of a regexp to

270	267	REQUIRE_AUTH = True
271	268
272	269	PREFIX = FEDERATION_V1_PREFIX # Allows specifying the API version
	270
	271	RATELIMIT = True # Whether to rate limit requests or not
273	272
274	273	def __init__(self, handler, authenticator, ratelimiter, server_name):
275	274	self.handler = handler

334	333	)
335	334
336	335	with scope:
337		if origin:
	336	if origin and self.RATELIMIT:
338	337	with ratelimiter.ratelimit(origin) as d:
339	338	await d
340	339	if request._disconnected:

370	369
371	370	class FederationSendServlet(BaseFederationServlet):
372	371	PATH = "/send/(?P<transaction_id>[^/]*)/?"
	372
	373	# We ratelimit manually in the handler as we queue up the requests and we
	374	# don't want to fill up the ratelimiter with blocked requests.
	375	RATELIMIT = False
373	376
374	377	def __init__(self, handler, server_name, **kwargs):
375	378	super(FederationSendServlet, self).__init__(

+1

-3

synapse/federation/units.py less more

106	106	if "edus" in kwargs and not kwargs["edus"]:
107	107	del kwargs["edus"]
108	108
109		super(Transaction, self).__init__(
110		transaction_id=transaction_id, pdus=pdus, **kwargs
111		)
	109	super().__init__(transaction_id=transaction_id, pdus=pdus, **kwargs)
112	110
113	111	@staticmethod
114	112	def create_new(pdus, **kwargs):

+2

-2

synapse/groups/attestations.py less more

59	59	UPDATE_ATTESTATION_TIME_MS = 1 * 24 * 60 * 60 * 1000
60	60
61	61
62		class GroupAttestationSigning(object):
	62	class GroupAttestationSigning:
63	63	"""Creates and verifies group attestations.
64	64	"""
65	65

123	123	)
124	124
125	125
126		class GroupAttestionRenewer(object):
	126	class GroupAttestionRenewer:
127	127	"""Responsible for sending and receiving attestation updates.
128	128	"""
129	129

+1

-1

synapse/groups/groups_server.py less more

31	31	# TODO: Flairs
32	32
33	33
34		class GroupsServerWorkerHandler(object):
	34	class GroupsServerWorkerHandler:
35	35	def __init__(self, hs):
36	36	self.hs = hs
37	37	self.store = hs.get_datastore()

+1

-1

synapse/handlers/__init__.py less more

19	19	from .search import SearchHandler
20	20
21	21
22		class Handlers(object):
	22	class Handlers:
23	23
24	24	""" Deprecated. A collection of handlers.
25	25

+1

-1

synapse/handlers/_base.py less more

24	24	logger = logging.getLogger(__name__)
25	25
26	26
27		class BaseHandler(object):
	27	class BaseHandler:
28	28	"""
29	29	Common base class for the event handlers.
30	30	"""

+1

-1

synapse/handlers/account_data.py less more

13	13	# limitations under the License.
14	14
15	15
16		class AccountDataEventSource(object):
	16	class AccountDataEventSource:
17	17	def __init__(self, hs):
18	18	self.store = hs.get_datastore()
19	19

+4

-18

synapse/handlers/account_validity.py less more

25	25	from synapse.types import UserID
26	26	from synapse.util import stringutils
27	27
28		try:
29		from synapse.push.mailer import load_jinja2_templates
30		except ImportError:
31		load_jinja2_templates = None
32
33	28	logger = logging.getLogger(__name__)
34	29
35	30
36		class AccountValidityHandler(object):
	31	class AccountValidityHandler:
37	32	def __init__(self, hs):
38	33	self.hs = hs
39	34	self.config = hs.config

46	41	if (
47	42	self._account_validity.enabled
48	43	and self._account_validity.renew_by_email_enabled
49		and load_jinja2_templates
50	44	):
51	45	# Don't do email-specific configuration if renewal by email is disabled.
	46	self._template_html = self.config.account_validity_template_html
	47	self._template_text = self.config.account_validity_template_text
	48
52	49	try:
53	50	app_name = self.hs.config.email_app_name
54	51

63	60	self._from_string = self.hs.config.email_notif_from
64	61
65	62	self._raw_from = email.utils.parseaddr(self._from_string)[1]
66
67		self._template_html, self._template_text = load_jinja2_templates(
68		self.config.email_template_dir,
69		[
70		self.config.email_expiry_template_html,
71		self.config.email_expiry_template_text,
72		],
73		apply_format_ts_filter=True,
74		apply_mxc_to_http_filter=True,
75		public_baseurl=self.config.public_baseurl,
76		)
77	63
78	64	# Check the renewal emails to send and send them every 30min.
79	65	def send_emails():

+1

-1

synapse/handlers/acme.py less more

33	33	--------------------------------------------------------------------------------"""
34	34
35	35
36		class AcmeHandler(object):
	36	class AcmeHandler:
37	37	def __init__(self, hs):
38	38	self.hs = hs
39	39	self.reactor = hs.get_reactor()

+1

-1

synapse/handlers/acme_issuing_service.py less more

77	77
78	78	@attr.s
79	79	@implementer(ICertificateStore)
80		class ErsatzStore(object):
	80	class ErsatzStore:
81	81	"""
82	82	A store that only stores in memory.
83	83	"""

+1

-1

synapse/handlers/admin.py less more

196	196	return writer.finished()
197	197
198	198
199		class ExfiltrationWriter(object):
	199	class ExfiltrationWriter:
200	200	"""Interface used to specify how to write exported data.
201	201	"""
202	202

+1

-1

synapse/handlers/appservice.py less more

33	33	events_processed_counter = Counter("synapse_handlers_appservice_events_processed", "")
34	34
35	35
36		class ApplicationServicesHandler(object):
	36	class ApplicationServicesHandler:
37	37	def __init__(self, hs):
38	38	self.store = hs.get_datastore()
39	39	self.is_mine_id = hs.is_mine_id

+101

-9

synapse/handlers/auth.py less more

41	41	from synapse.logging.context import defer_to_thread
42	42	from synapse.metrics.background_process_metrics import run_as_background_process
43	43	from synapse.module_api import ModuleApi
44		from synapse.push.mailer import load_jinja2_templates
45		from synapse.types import Requester, UserID
	44	from synapse.types import JsonDict, Requester, UserID
46	45	from synapse.util import stringutils as stringutils
	46	from synapse.util.msisdn import phone_number_to_msisdn
47	47	from synapse.util.threepids import canonicalise_email
48	48
49	49	from ._base import BaseHandler
50	50
51	51	logger = logging.getLogger(__name__)
	52
	53
	54	def convert_client_dict_legacy_fields_to_identifier(
	55	submission: JsonDict,
	56	) -> Dict[str, str]:
	57	"""
	58	Convert a legacy-formatted login submission to an identifier dict.
	59
	60	Legacy login submissions (used in both login and user-interactive authentication)
	61	provide user-identifying information at the top-level instead.
	62
	63	These are now deprecated and replaced with identifiers:
	64	https://matrix.org/docs/spec/client_server/r0.6.1#identifier-types
	65
	66	Args:
	67	submission: The client dict to convert
	68
	69	Returns:
	70	The matching identifier dict
	71
	72	Raises:
	73	SynapseError: If the format of the client dict is invalid
	74	"""
	75	identifier = submission.get("identifier", {})
	76
	77	# Generate an m.id.user identifier if "user" parameter is present
	78	user = submission.get("user")
	79	if user:
	80	identifier = {"type": "m.id.user", "user": user}
	81
	82	# Generate an m.id.thirdparty identifier if "medium" and "address" parameters are present
	83	medium = submission.get("medium")
	84	address = submission.get("address")
	85	if medium and address:
	86	identifier = {
	87	"type": "m.id.thirdparty",
	88	"medium": medium,
	89	"address": address,
	90	}
	91
	92	# We've converted valid, legacy login submissions to an identifier. If the
	93	# submission still doesn't have an identifier, it's invalid
	94	if not identifier:
	95	raise SynapseError(400, "Invalid login submission", Codes.INVALID_PARAM)
	96
	97	# Ensure the identifier has a type
	98	if "type" not in identifier:
	99	raise SynapseError(
	100	400, "'identifier' dict has no key 'type'", errcode=Codes.MISSING_PARAM,
	101	)
	102
	103	return identifier
	104
	105
	106	def login_id_phone_to_thirdparty(identifier: JsonDict) -> Dict[str, str]:
	107	"""
	108	Convert a phone login identifier type to a generic threepid identifier.
	109
	110	Args:
	111	identifier: Login identifier dict of type 'm.id.phone'
	112
	113	Returns:
	114	An equivalent m.id.thirdparty identifier dict
	115	"""
	116	if "country" not in identifier or (
	117	# The specification requires a "phone" field, while Synapse used to require a "number"
	118	# field. Accept both for backwards compatibility.
	119	"phone" not in identifier
	120	and "number" not in identifier
	121	):
	122	raise SynapseError(
	123	400, "Invalid phone-type identifier", errcode=Codes.INVALID_PARAM
	124	)
	125
	126	# Accept both "phone" and "number" as valid keys in m.id.phone
	127	phone_number = identifier.get("phone", identifier["number"])
	128
	129	# Convert user-provided phone number to a consistent representation
	130	msisdn = phone_number_to_msisdn(identifier["country"], phone_number)
	131
	132	return {
	133	"type": "m.id.thirdparty",
	134	"medium": "msisdn",
	135	"address": msisdn,
	136	}
52	137
53	138
54	139	class AuthHandler(BaseHandler):

131	216	# after the SSO completes and before redirecting them back to their client.
132	217	# It notifies the user they are about to give access to their matrix account
133	218	# to the client.
134		self._sso_redirect_confirm_template = load_jinja2_templates(
135		hs.config.sso_template_dir, ["sso_redirect_confirm.html"],
136		)[0]
	219	self._sso_redirect_confirm_template = hs.config.sso_redirect_confirm_template
	220
137	221	# The following template is shown during user interactive authentication
138	222	# in the fallback auth scenario. It notifies the user that they are
139	223	# authenticating for an operation to occur on their account.
140		self._sso_auth_confirm_template = load_jinja2_templates(
141		hs.config.sso_template_dir, ["sso_auth_confirm.html"],
142		)[0]
	224	self._sso_auth_confirm_template = hs.config.sso_auth_confirm_template
	225
143	226	# The following template is shown after a successful user interactive
144	227	# authentication session. It tells the user they can close the window.
145	228	self._sso_auth_success_template = hs.config.sso_auth_success_template
	229
146	230	# The following template is shown during the SSO authentication process if
147	231	# the account is deactivated.
148	232	self._sso_account_deactivated_template = (

365	449	# authentication flow.
366	450	await self.store.set_ui_auth_clientdict(sid, clientdict)
367	451
	452	user_agent = request.requestHeaders.getRawHeaders(b"User-Agent", default=[b""])[
	453	0
	454	].decode("ascii", "surrogateescape")
	455
	456	await self.store.add_user_agent_ip_to_ui_auth_session(
	457	session.session_id, user_agent, clientip
	458	)
	459
368	460	if not authdict:
369	461	raise InteractiveAuthIncompleteError(
370	462	session.session_id, self._auth_dict_for_flows(flows, session.session_id)

1143	1235
1144	1236
1145	1237	@attr.s
1146		class MacaroonGenerator(object):
	1238	class MacaroonGenerator:
1147	1239
1148	1240	hs = attr.ib()
1149	1241

+10

-1

synapse/handlers/cas_handler.py less more

34	34	"""
35	35
36	36	def __init__(self, hs):
	37	self.hs = hs
37	38	self._hostname = hs.hostname
38	39	self._auth_handler = hs.get_auth_handler()
39	40	self._registration_handler = hs.get_registration_handler()

209	210
210	211	else:
211	212	if not registered_user_id:
	213	# Pull out the user-agent and IP from the request.
	214	user_agent = request.requestHeaders.getRawHeaders(
	215	b"User-Agent", default=[b""]
	216	)[0].decode("ascii", "surrogateescape")
	217	ip_address = self.hs.get_ip_from_request(request)
	218
212	219	registered_user_id = await self._registration_handler.register_user(
213		localpart=localpart, default_display_name=user_display_name
	220	localpart=localpart,
	221	default_display_name=user_display_name,
	222	user_agent_ips=(user_agent, ip_address),
214	223	)
215	224
216	225	await self._auth_handler.complete_sso_login(

+4

-2

synapse/handlers/device.py less more

233	233	return result
234	234
235	235	async def on_federation_query_user_devices(self, user_id):
236		stream_id, devices = await self.store.get_devices_with_keys_by_user(user_id)
	236	stream_id, devices = await self.store.get_e2e_device_keys_for_federation_query(
	237	user_id
	238	)
237	239	master_key = await self.store.get_e2e_cross_signing_key(user_id, "master")
238	240	self_signing_key = await self.store.get_e2e_cross_signing_key(
239	241	user_id, "self_signing"

494	496	device.update({"last_seen_ts": ip.get("last_seen"), "last_seen_ip": ip.get("ip")})
495	497
496	498
497		class DeviceListUpdater(object):
	499	class DeviceListUpdater:
498	500	"Handles incoming device list updates from federation and updates the DB"
499	501
500	502	def __init__(self, hs, device_handler):

+3

-4

synapse/handlers/devicemessage.py less more

15	15	import logging
16	16	from typing import Any, Dict
17	17
18		from canonicaljson import json
19
20	18	from synapse.api.errors import SynapseError
21	19	from synapse.logging.context import run_in_background
22	20	from synapse.logging.opentracing import (

26	24	start_active_span,
27	25	)
28	26	from synapse.types import UserID, get_domain_from_id
	27	from synapse.util import json_encoder
29	28	from synapse.util.stringutils import random_string
30	29
31	30	logger = logging.getLogger(__name__)
32	31
33	32
34		class DeviceMessageHandler(object):
	33	class DeviceMessageHandler:
35	34	def __init__(self, hs):
36	35	"""
37	36	Args:

173	172	"sender": sender_user_id,
174	173	"type": message_type,
175	174	"message_id": message_id,
176		"org.matrix.opentracing_context": json.dumps(context),
	175	"org.matrix.opentracing_context": json_encoder.encode(context),
177	176	}
178	177
179	178	log_kv({"local_messages": local_messages})

+6

-0

synapse/handlers/directory.py less more

22	22	CodeMessageException,
23	23	Codes,
24	24	NotFoundError,
	25	ShadowBanError,
25	26	StoreError,
26	27	SynapseError,
27	28	)

198	199
199	200	try:
200	201	await self._update_canonical_alias(requester, user_id, room_id, room_alias)
	202	except ShadowBanError as e:
	203	logger.info("Failed to update alias events due to shadow-ban: %s", e)
201	204	except AuthError as e:
202	205	logger.info("Failed to update alias events: %s", e)
203	206

291	294	"""
292	295	Send an updated canonical alias event if the removed alias was set as
293	296	the canonical alias or listed in the alt_aliases field.
	297
	298	Raises:
	299	ShadowBanError if the requester has been shadow-banned.
294	300	"""
295	301	alias_event = await self.state.get_current_state(
296	302	room_id, EventTypes.CanonicalAlias, ""

+8

-8

synapse/handlers/e2e_keys.py less more

18	18	from typing import Dict, List, Optional, Tuple
19	19
20	20	import attr
21		from canonicaljson import encode_canonical_json, json
	21	from canonicaljson import encode_canonical_json
22	22	from signedjson.key import VerifyKey, decode_verify_key_bytes
23	23	from signedjson.sign import SignatureVerifyException, verify_signed_json
24	24	from unpaddedbase64 import decode_base64

34	34	get_domain_from_id,
35	35	get_verify_key_from_cross_signing_key,
36	36	)
37		from synapse.util import unwrapFirstError
	37	from synapse.util import json_decoder, unwrapFirstError
38	38	from synapse.util.async_helpers import Linearizer
39	39	from synapse.util.caches.expiringcache import ExpiringCache
40	40	from synapse.util.retryutils import NotRetryingDestination

42	42	logger = logging.getLogger(__name__)
43	43
44	44
45		class E2eKeysHandler(object):
	45	class E2eKeysHandler:
46	46	def __init__(self, hs):
47	47	self.store = hs.get_datastore()
48	48	self.federation = hs.get_federation_client()

352	352	# make sure that each queried user appears in the result dict
353	353	result_dict[user_id] = {}
354	354
355		results = await self.store.get_e2e_device_keys(local_query)
	355	results = await self.store.get_e2e_device_keys_for_cs_api(local_query)
356	356
357	357	# Build the result structure
358	358	for user_id, device_keys in results.items():

403	403	for device_id, keys in device_keys.items():
404	404	for key_id, json_bytes in keys.items():
405	405	json_result.setdefault(user_id, {})[device_id] = {
406		key_id: json.loads(json_bytes)
	406	key_id: json_decoder.decode(json_bytes)
407	407	}
408	408
409	409	@trace

733	733	# fetch our stored devices. This is used to 1. verify
734	734	# signatures on the master key, and 2. to compare with what
735	735	# was sent if the device was signed
736		devices = await self.store.get_e2e_device_keys([(user_id, None)])
	736	devices = await self.store.get_e2e_device_keys_for_cs_api([(user_id, None)])
737	737
738	738	if user_id not in devices:
739	739	raise NotFoundError("No device keys found")

1185	1185
1186	1186
1187	1187	def _one_time_keys_match(old_key_json, new_key):
1188		old_key = json.loads(old_key_json)
	1188	old_key = json_decoder.decode(old_key_json)
1189	1189
1190	1190	# if either is a string rather than an object, they must match exactly
1191	1191	if not isinstance(old_key, dict) or not isinstance(new_key, dict):

1211	1211	signature = attr.ib()
1212	1212
1213	1213
1214		class SigningKeyEduUpdater(object):
	1214	class SigningKeyEduUpdater:
1215	1215	"""Handles incoming signing key updates from federation and updates the DB"""
1216	1216
1217	1217	def __init__(self, hs, e2e_keys_handler):

+1

-1

synapse/handlers/e2e_room_keys.py less more

28	28	logger = logging.getLogger(__name__)
29	29
30	30
31		class E2eRoomKeysHandler(object):
	31	class E2eRoomKeysHandler:
32	32	"""
33	33	Implements an optional realtime backup mechanism for encrypted E2E megolm room keys.
34	34	This gives a way for users to store and recover their megolm keys if they lose all

+26

-23

synapse/handlers/events.py less more

14	14
15	15	import logging
16	16	import random
	17	from typing import TYPE_CHECKING, Iterable, List, Optional
17	18
18	19	from synapse.api.constants import EventTypes, Membership
19	20	from synapse.api.errors import AuthError, SynapseError
20	21	from synapse.events import EventBase
21	22	from synapse.handlers.presence import format_user_presence_state
22	23	from synapse.logging.utils import log_function
23		from synapse.types import UserID
	24	from synapse.streams.config import PaginationConfig
	25	from synapse.types import JsonDict, UserID
24	26	from synapse.visibility import filter_events_for_client
25	27
26	28	from ._base import BaseHandler
	29
	30	if TYPE_CHECKING:
	31	from synapse.server import HomeServer
	32
27	33
28	34	logger = logging.getLogger(__name__)
29	35
30	36
31	37	class EventStreamHandler(BaseHandler):
32		def __init__(self, hs):
	38	def __init__(self, hs: "HomeServer"):
33	39	super(EventStreamHandler, self).__init__(hs)
34
35		# Count of active streams per user
36		self._streams_per_user = {}
37		# Grace timers per user to delay the "stopped" signal
38		self._stop_timer_per_user = {}
39	40
40	41	self.distributor = hs.get_distributor()
41	42	self.distributor.declare("started_user_eventstream")

51	52	@log_function
52	53	async def get_stream(
53	54	self,
54		auth_user_id,
55		pagin_config,
56		timeout=0,
57		as_client_event=True,
58		affect_presence=True,
59		room_id=None,
60		is_guest=False,
61		):
	55	auth_user_id: str,
	56	pagin_config: PaginationConfig,
	57	timeout: int = 0,
	58	as_client_event: bool = True,
	59	affect_presence: bool = True,
	60	room_id: Optional[str] = None,
	61	is_guest: bool = False,
	62	) -> JsonDict:
62	63	"""Fetches the events stream for a given user.
63	64	"""
64	65

97	98
98	99	# When the user joins a new room, or another user joins a currently
99	100	# joined room, we need to send down presence for those users.
100		to_add = []
	101	to_add = [] # type: List[JsonDict]
101	102	for event in events:
102	103	if not isinstance(event, EventBase):
103	104	continue

109	110	# Send down presence for everyone in the room.
110	111	users = await self.state.get_current_users_in_room(
111	112	event.room_id
112		)
	113	) # type: Iterable[str]
113	114	else:
114	115	users = [event.state_key]
115	116

143	144
144	145
145	146	class EventHandler(BaseHandler):
146		def __init__(self, hs):
	147	def __init__(self, hs: "HomeServer"):
147	148	super(EventHandler, self).__init__(hs)
148	149	self.storage = hs.get_storage()
149	150
150		async def get_event(self, user, room_id, event_id):
	151	async def get_event(
	152	self, user: UserID, room_id: Optional[str], event_id: str
	153	) -> Optional[EventBase]:
151	154	"""Retrieve a single specified event.
152	155
153	156	Args:
154		user (synapse.types.UserID): The user requesting the event
155		room_id (str\|None): The expected room id. We'll return None if the
	157	user: The user requesting the event
	158	room_id: The expected room id. We'll return None if the
156	159	event's room does not match.
157		event_id (str): The event ID to obtain.
	160	event_id: The event ID to obtain.
158	161	Returns:
159		dict: An event, or None if there is no event matching this ID.
	162	An event, or None if there is no event matching this ID.
160	163	Raises:
161	164	SynapseError if there was a problem retrieving this event, or
162	165	AuthError if the user does not have the rights to inspect this

+88

-35

synapse/handlers/federation.py less more

71	71	from synapse.replication.http.membership import ReplicationUserJoinedLeftRoomRestServlet
72	72	from synapse.state import StateResolutionStore, resolve_events_with_store
73	73	from synapse.storage.databases.main.events_worker import EventRedactBehaviour
74		from synapse.types import JsonDict, StateMap, UserID, get_domain_from_id
	74	from synapse.types import (
	75	JsonDict,
	76	MutableStateMap,
	77	StateMap,
	78	UserID,
	79	get_domain_from_id,
	80	)
75	81	from synapse.util.async_helpers import Linearizer, concurrently_execute
76	82	from synapse.util.distributor import user_joined_room
77	83	from synapse.util.retryutils import NotRetryingDestination

95	101
96	102	event = attr.ib(type=EventBase)
97	103	state = attr.ib(type=Optional[Sequence[EventBase]], default=None)
98		auth_events = attr.ib(type=Optional[StateMap[EventBase]], default=None)
	104	auth_events = attr.ib(type=Optional[MutableStateMap[EventBase]], default=None)
99	105
100	106
101	107	class FederationHandler(BaseHandler):

433	439	if not prevs - seen:
434	440	return
435	441
436		latest = await self.store.get_latest_event_ids_in_room(room_id)
	442	latest_list = await self.store.get_latest_event_ids_in_room(room_id)
437	443
438	444	# We add the prev events that we have seen to the latest
439	445	# list to ensure the remote server doesn't give them to us
440		latest = set(latest)
	446	latest = set(latest_list)
441	447	latest \|= seen
442	448
443	449	logger.info(

774	780	# keys across all devices.
775	781	current_keys = [
776	782	key
777		for device in cached_devices
	783	for device in cached_devices.values()
778	784	for key in device.get("keys", {}).get("keys", {}).values()
779	785	]
780	786

936	942
937	943	return events
938	944
939		async def maybe_backfill(self, room_id, current_depth):
	945	async def maybe_backfill(
	946	self, room_id: str, current_depth: int, limit: int
	947	) -> bool:
940	948	"""Checks the database to see if we should backfill before paginating,
941	949	and if so do.
	950
	951	Args:
	952	room_id
	953	current_depth: The depth from which we're paginating from. This is
	954	used to decide if we should backfill and what extremities to
	955	use.
	956	limit: The number of events that the pagination request will
	957	return. This is used as part of the heuristic to decide if we
	958	should back paginate.
942	959	"""
943	960	extremities = await self.store.get_oldest_events_with_depth_in_room(room_id)
944	961
945	962	if not extremities:
946	963	logger.debug("Not backfilling as no extremeties found.")
947		return
	964	return False
948	965
949	966	# We only want to paginate if we can actually see the events we'll get,
950	967	# as otherwise we'll just spend a lot of resources to get redacted

997	1014	sorted_extremeties_tuple = sorted(extremities.items(), key=lambda e: -int(e[1]))
998	1015	max_depth = sorted_extremeties_tuple[0][1]
999	1016
	1017	# If we're approaching an extremity we trigger a backfill, otherwise we
	1018	# no-op.
	1019	#
	1020	# We chose twice the limit here as then clients paginating backwards
	1021	# will send pagination requests that trigger backfill at least twice
	1022	# using the most recent extremity before it gets removed (see below). We
	1023	# chose more than one times the limit in case of failure, but choosing a
	1024	# much larger factor will result in triggering a backfill request much
	1025	# earlier than necessary.
	1026	if current_depth - 2 * limit > max_depth:
	1027	logger.debug(
	1028	"Not backfilling as we don't need to. %d < %d - 2 * %d",
	1029	max_depth,
	1030	current_depth,
	1031	limit,
	1032	)
	1033	return False
	1034
	1035	logger.debug(
	1036	"room_id: %s, backfill: current_depth: %s, max_depth: %s, extrems: %s",
	1037	room_id,
	1038	current_depth,
	1039	max_depth,
	1040	sorted_extremeties_tuple,
	1041	)
	1042
	1043	# We ignore extremities that have a greater depth than our current depth
	1044	# as:
	1045	# 1. we don't really care about getting events that have happened
	1046	# before our current position; and
	1047	# 2. we have likely previously tried and failed to backfill from that
	1048	# extremity, so to avoid getting "stuck" requesting the same
	1049	# backfill repeatedly we drop those extremities.
	1050	filtered_sorted_extremeties_tuple = [
	1051	t for t in sorted_extremeties_tuple if int(t[1]) <= current_depth
	1052	]
	1053
	1054	# However, we need to check that the filtered extremities are non-empty.
	1055	# If they are empty then either we can a) bail or b) still attempt to
	1056	# backill. We opt to try backfilling anyway just in case we do get
	1057	# relevant events.
	1058	if filtered_sorted_extremeties_tuple:
	1059	sorted_extremeties_tuple = filtered_sorted_extremeties_tuple
	1060
1000	1061	# We don't want to specify too many extremities as it causes the backfill
1001	1062	# request URI to be too long.
1002	1063	extremities = dict(sorted_extremeties_tuple[:5])
1003
1004		if current_depth > max_depth:
1005		logger.debug(
1006		"Not backfilling as we don't need to. %d < %d", max_depth, current_depth
1007		)
1008		return
1009	1064
1010	1065	# Now we need to decide which hosts to hit first.
1011	1066

1776	1831	"""Returns the state at the event. i.e. not including said event.
1777	1832	"""
1778	1833
1779		event = await self.store.get_event(
1780		event_id, allow_none=False, check_room_id=room_id
1781		)
	1834	event = await self.store.get_event(event_id, check_room_id=room_id)
1782	1835
1783	1836	state_groups = await self.state_store.get_state_groups(room_id, [event_id])
1784	1837

1804	1857	async def get_state_ids_for_pdu(self, room_id: str, event_id: str) -> List[str]:
1805	1858	"""Returns the state at the event. i.e. not including said event.
1806	1859	"""
1807		event = await self.store.get_event(
1808		event_id, allow_none=False, check_room_id=room_id
1809		)
	1860	event = await self.store.get_event(event_id, check_room_id=room_id)
1810	1861
1811	1862	state_groups = await self.state_store.get_state_groups_ids(room_id, [event_id])
1812	1863

1876	1927	else:
1877	1928	return None
1878	1929
1879		def get_min_depth_for_context(self, context):
1880		return self.store.get_min_depth(context)
	1930	async def get_min_depth_for_context(self, context):
	1931	return await self.store.get_min_depth(context)
1881	1932
1882	1933	async def _handle_new_event(
1883	1934	self, origin, event, state=None, auth_events=None, backfilled=False

2056	2107	origin: str,
2057	2108	event: EventBase,
2058	2109	state: Optional[Iterable[EventBase]],
2059		auth_events: Optional[StateMap[EventBase]],
	2110	auth_events: Optional[MutableStateMap[EventBase]],
2060	2111	backfilled: bool,
2061	2112	) -> EventContext:
2062	2113	context = await self.state_handler.compute_event_context(event, old_state=state)

2106	2157	if backfilled or event.internal_metadata.is_outlier():
2107	2158	return
2108	2159
2109		extrem_ids = await self.store.get_latest_event_ids_in_room(event.room_id)
2110		extrem_ids = set(extrem_ids)
	2160	extrem_ids_list = await self.store.get_latest_event_ids_in_room(event.room_id)
	2161	extrem_ids = set(extrem_ids_list)
2111	2162	prev_event_ids = set(event.prev_event_ids())
2112	2163
2113	2164	if extrem_ids == prev_event_ids:

2137	2188	)
2138	2189	state_sets = list(state_sets.values())
2139	2190	state_sets.append(state)
2140		current_state_ids = await self.state_handler.resolve_events(
	2191	current_states = await self.state_handler.resolve_events(
2141	2192	room_version, state_sets, event
2142	2193	)
2143		current_state_ids = {k: e.event_id for k, e in current_state_ids.items()}
	2194	current_state_ids = {
	2195	k: e.event_id for k, e in current_states.items()
	2196	} # type: StateMap[str]
2144	2197	else:
2145	2198	current_state_ids = await self.state_handler.get_current_state_ids(
2146	2199	event.room_id, latest_event_ids=extrem_ids

2152	2205
2153	2206	# Now check if event pass auth against said current state
2154	2207	auth_types = auth_types_for_event(event)
2155		current_state_ids = [e for k, e in current_state_ids.items() if k in auth_types]
2156
2157		current_auth_events = await self.store.get_events(current_state_ids)
	2208	current_state_ids_list = [
	2209	e for k, e in current_state_ids.items() if k in auth_types
	2210	]
	2211
	2212	auth_events_map = await self.store.get_events(current_state_ids_list)
2158	2213	current_auth_events = {
2159		(e.type, e.state_key): e for e in current_auth_events.values()
	2214	(e.type, e.state_key): e for e in auth_events_map.values()
2160	2215	}
2161	2216
2162	2217	try:

2172	2227	if not in_room:
2173	2228	raise AuthError(403, "Host not in room.")
2174	2229
2175		event = await self.store.get_event(
2176		event_id, allow_none=False, check_room_id=room_id
2177		)
	2230	event = await self.store.get_event(event_id, check_room_id=room_id)
2178	2231
2179	2232	# Just go through and process each event in `remote_auth_chain`. We
2180	2233	# don't want to fall into the trap of `missing` being wrong.

2226	2279	origin: str,
2227	2280	event: EventBase,
2228	2281	context: EventContext,
2229		auth_events: StateMap[EventBase],
	2282	auth_events: MutableStateMap[EventBase],
2230	2283	) -> EventContext:
2231	2284	"""
2232	2285

2277	2330	origin: str,
2278	2331	event: EventBase,
2279	2332	context: EventContext,
2280		auth_events: StateMap[EventBase],
	2333	auth_events: MutableStateMap[EventBase],
2281	2334	) -> EventContext:
2282	2335	"""Helper for do_auth. See there for docs.
2283	2336

+1

-1

synapse/handlers/groups_local.py less more

51	51	return f
52	52
53	53
54		class GroupsLocalWorkerHandler(object):
	54	class GroupsLocalWorkerHandler:
55	55	def __init__(self, hs):
56	56	self.hs = hs
57	57	self.store = hs.get_datastore()

+2

-3

synapse/handlers/identity.py less more

20	20	import urllib.parse
21	21	from typing import Awaitable, Callable, Dict, List, Optional, Tuple
22	22
23		from canonicaljson import json
24
25	23	from twisted.internet.error import TimeoutError
26	24
27	25	from synapse.api.errors import (

33	31	from synapse.config.emailconfig import ThreepidBehaviour
34	32	from synapse.http.client import SimpleHttpClient
35	33	from synapse.types import JsonDict, Requester
	34	from synapse.util import json_decoder
36	35	from synapse.util.hash import sha256_and_url_safe_base64
37	36	from synapse.util.stringutils import assert_valid_client_secret, random_string
38	37

176	175	except TimeoutError:
177	176	raise SynapseError(500, "Timed out contacting identity server")
178	177	except CodeMessageException as e:
179		data = json.loads(e.msg) # XXX WAT?
	178	data = json_decoder.decode(e.msg) # XXX WAT?
180	179	return data
181	180
182	181	logger.info("Got 404 when POSTing JSON %s, falling back to v1 URL", bind_url)

+48

-32

synapse/handlers/initial_sync.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
	16	from typing import TYPE_CHECKING
16	17
17	18	from twisted.internet import defer
18	19

21	22	from synapse.events.validator import EventValidator
22	23	from synapse.handlers.presence import format_user_presence_state
23	24	from synapse.logging.context import make_deferred_yieldable, run_in_background
	25	from synapse.storage.roommember import RoomsForUser
24	26	from synapse.streams.config import PaginationConfig
25		from synapse.types import StreamToken, UserID
	27	from synapse.types import JsonDict, Requester, StreamToken, UserID
26	28	from synapse.util import unwrapFirstError
27	29	from synapse.util.async_helpers import concurrently_execute
28	30	from synapse.util.caches.response_cache import ResponseCache

30	32
31	33	from ._base import BaseHandler
32	34
	35	if TYPE_CHECKING:
	36	from synapse.server import HomeServer
	37
	38
33	39	logger = logging.getLogger(__name__)
34	40
35	41
36	42	class InitialSyncHandler(BaseHandler):
37		def __init__(self, hs):
	43	def __init__(self, hs: "HomeServer"):
38	44	super(InitialSyncHandler, self).__init__(hs)
39	45	self.hs = hs
40	46	self.state = hs.get_state_handler()

47	53
48	54	def snapshot_all_rooms(
49	55	self,
50		user_id=None,
51		pagin_config=None,
52		as_client_event=True,
53		include_archived=False,
54		):
	56	user_id: str,
	57	pagin_config: PaginationConfig,
	58	as_client_event: bool = True,
	59	include_archived: bool = False,
	60	) -> JsonDict:
55	61	"""Retrieve a snapshot of all rooms the user is invited or has joined.
56	62
57	63	This snapshot may include messages for all rooms where the user is
58	64	joined, depending on the pagination config.
59	65
60	66	Args:
61		user_id (str): The ID of the user making the request.
62		pagin_config (synapse.api.streams.PaginationConfig): The pagination
63		config used to determine how many messages PER ROOM to return.
64		as_client_event (bool): True to get events in client-server format.
65		include_archived (bool): True to get rooms that the user has left
	67	user_id: The ID of the user making the request.
	68	pagin_config: The pagination config used to determine how many
	69	messages PER ROOM to return.
	70	as_client_event: True to get events in client-server format.
	71	include_archived: True to get rooms that the user has left
66	72	Returns:
67		A list of dicts with "room_id" and "membership" keys for all rooms
68		the user is currently invited or joined in on. Rooms where the user
69		is joined on, may return a "messages" key with messages, depending
70		on the specified PaginationConfig.
	73	A JsonDict with the same format as the response to `/intialSync`
	74	API
71	75	"""
72	76	key = (
73	77	user_id,

90	94
91	95	async def _snapshot_all_rooms(
92	96	self,
93		user_id=None,
94		pagin_config=None,
95		as_client_event=True,
96		include_archived=False,
97		):
	97	user_id: str,
	98	pagin_config: PaginationConfig,
	99	as_client_event: bool = True,
	100	include_archived: bool = False,
	101	) -> JsonDict:
98	102
99	103	memberships = [Membership.INVITE, Membership.JOIN]
100	104	if include_archived:

133	137	if limit is None:
134	138	limit = 10
135	139
136		async def handle_room(event):
	140	async def handle_room(event: RoomsForUser):
137	141	d = {
138	142	"room_id": event.room_id,
139	143	"membership": event.membership,

250	254
251	255	return ret
252	256
253		async def room_initial_sync(self, requester, room_id, pagin_config=None):
	257	async def room_initial_sync(
	258	self, requester: Requester, room_id: str, pagin_config: PaginationConfig
	259	) -> JsonDict:
254	260	"""Capture the a snapshot of a room. If user is currently a member of
255	261	the room this will be what is currently in the room. If the user left
256	262	the room this will be what was in the room when they left.
257	263
258	264	Args:
259		requester(Requester): The user to get a snapshot for.
260		room_id(str): The room to get a snapshot of.
261		pagin_config(synapse.streams.config.PaginationConfig):
262		The pagination config used to determine how many messages to
263		return.
	265	requester: The user to get a snapshot for.
	266	room_id: The room to get a snapshot of.
	267	pagin_config: The pagination config used to determine how many
	268	messages to return.
264	269	Raises:
265	270	AuthError if the user wasn't in the room.
266	271	Returns:

304	309	return result
305	310
306	311	async def _room_initial_sync_parted(
307		self, user_id, room_id, pagin_config, membership, member_event_id, is_peeking
308		):
	312	self,
	313	user_id: str,
	314	room_id: str,
	315	pagin_config: PaginationConfig,
	316	membership: Membership,
	317	member_event_id: str,
	318	is_peeking: bool,
	319	) -> JsonDict:
309	320	room_state = await self.state_store.get_state_for_events([member_event_id])
310	321
311	322	room_state = room_state[member_event_id]

349	360	}
350	361
351	362	async def _room_initial_sync_joined(
352		self, user_id, room_id, pagin_config, membership, is_peeking
353		):
	363	self,
	364	user_id: str,
	365	room_id: str,
	366	pagin_config: PaginationConfig,
	367	membership: Membership,
	368	is_peeking: bool,
	369	) -> JsonDict:
354	370	current_state = await self.state.get_current_state(room_id=room_id)
355	371
356	372	# TODO: These concurrently

+85

-45

synapse/handlers/message.py less more

14	14	# See the License for the specific language governing permissions and
15	15	# limitations under the License.
16	16	import logging
	17	import random
17	18	from typing import TYPE_CHECKING, Dict, List, Optional, Tuple
18	19
19		from canonicaljson import encode_canonical_json, json
	20	from canonicaljson import encode_canonical_json
20	21
21	22	from twisted.internet.interfaces import IDelayedCall
22	23

33	34	Codes,
34	35	ConsentNotGivenError,
35	36	NotFoundError,
	37	ShadowBanError,
36	38	SynapseError,
37	39	)
38	40	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, RoomVersions

46	48	from synapse.replication.http.send_event import ReplicationSendEventRestServlet
47	49	from synapse.storage.databases.main.events_worker import EventRedactBehaviour
48	50	from synapse.storage.state import StateFilter
49		from synapse.types import (
50		Collection,
51		Requester,
52		RoomAlias,
53		StreamToken,
54		UserID,
55		create_requester,
56		)
	51	from synapse.types import Requester, RoomAlias, StreamToken, UserID, create_requester
	52	from synapse.util import json_decoder
57	53	from synapse.util.async_helpers import Linearizer
58	54	from synapse.util.frozenutils import frozendict_json_encoder
59	55	from synapse.util.metrics import measure_func

67	63	logger = logging.getLogger(__name__)
68	64
69	65
70		class MessageHandler(object):
	66	class MessageHandler:
71	67	"""Contains some read only APIs to get state about a room
72	68	"""
73	69

91	87	)
92	88
93	89	async def get_room_data(
94		self,
95		user_id: str,
96		room_id: str,
97		event_type: str,
98		state_key: str,
99		is_guest: bool,
	90	self, user_id: str, room_id: str, event_type: str, state_key: str,
100	91	) -> dict:
101	92	""" Get data from a room.
102	93

105	96	room_id
106	97	event_type
107	98	state_key
108		is_guest
109	99	Returns:
110	100	The path data content.
111	101	Raises:
112		SynapseError if something went wrong.
	102	SynapseError or AuthError if the user is not in the room
113	103	"""
114	104	(
115	105	membership,

126	116	[membership_event_id], StateFilter.from_types([key])
127	117	)
128	118	data = room_state[membership_event_id].get(key)
	119	else:
	120	# check_user_in_room_or_world_readable, if it doesn't raise an AuthError, should
	121	# only ever return a Membership.JOIN/LEAVE object
	122	#
	123	# Safeguard in case it returned something else
	124	logger.error(
	125	"Attempted to retrieve data from a room for a user that has never been in it. "
	126	"This should not have happened."
	127	)
	128	raise SynapseError(403, "User not in room", errcode=Codes.FORBIDDEN)
129	129
130	130	return data
131	131

360	360	_DUMMY_EVENT_ROOM_EXCLUSION_EXPIRY = 7 * 24 * 60 * 60 * 1000
361	361
362	362
363		class EventCreationHandler(object):
	363	class EventCreationHandler:
364	364	def __init__(self, hs: "HomeServer"):
365	365	self.hs = hs
366	366	self.auth = hs.get_auth()

438	438	event_dict: dict,
439	439	token_id: Optional[str] = None,
440	440	txn_id: Optional[str] = None,
441		prev_event_ids: Optional[Collection[str]] = None,
	441	prev_event_ids: Optional[List[str]] = None,
442	442	require_consent: bool = True,
443	443	) -> Tuple[EventBase, EventContext]:
444	444	"""

643	643	event: EventBase,
644	644	context: EventContext,
645	645	ratelimit: bool = True,
	646	ignore_shadow_ban: bool = False,
646	647	) -> int:
647	648	"""
648	649	Persists and notifies local clients and federation of an event.
649	650
650	651	Args:
651		requester
652		event the event to send.
653		context: the context of the event.
	652	requester: The requester sending the event.
	653	event: The event to send.
	654	context: The context of the event.
654	655	ratelimit: Whether to rate limit this send.
	656	ignore_shadow_ban: True if shadow-banned users should be allowed to
	657	send this event.
655	658
656	659	Return:
657	660	The stream_id of the persisted event.
	661
	662	Raises:
	663	ShadowBanError if the requester has been shadow-banned.
658	664	"""
659	665	if event.type == EventTypes.Member:
660	666	raise SynapseError(
661	667	500, "Tried to send member event through non-member codepath"
662	668	)
663	669
	670	if not ignore_shadow_ban and requester.shadow_banned:
	671	# We randomly sleep a bit just to annoy the requester.
	672	await self.clock.sleep(random.randint(1, 10))
	673	raise ShadowBanError()
	674
664	675	user = UserID.from_string(event.sender)
665	676
666	677	assert self.hs.is_mine(user), "User must be our own: %s" % (user,)
667	678
668	679	if event.is_state():
669		prev_state = await self.deduplicate_state_event(event, context)
670		if prev_state is not None:
	680	prev_event = await self.deduplicate_state_event(event, context)
	681	if prev_event is not None:
671	682	logger.info(
672	683	"Not bothering to persist state event %s duplicated by %s",
673	684	event.event_id,
674		prev_state.event_id,
675		)
676		return prev_state
	685	prev_event.event_id,
	686	)
	687	return await self.store.get_stream_id_for_event(prev_event.event_id)
677	688
678	689	return await self.handle_new_client_event(
679	690	requester=requester, event=event, context=context, ratelimit=ratelimit

681	692
682	693	async def deduplicate_state_event(
683	694	self, event: EventBase, context: EventContext
684		) -> None:
	695	) -> Optional[EventBase]:
685	696	"""
686	697	Checks whether event is in the latest resolved state in context.
687	698
688		If so, returns the version of the event in context.
689		Otherwise, returns None.
	699	Args:
	700	event: The event to check for duplication.
	701	context: The event context.
	702
	703	Returns:
	704	The previous verion of the event is returned, if it is found in the
	705	event context. Otherwise, None is returned.
690	706	"""
691	707	prev_state_ids = await context.get_prev_state_ids()
692	708	prev_event_id = prev_state_ids.get((event.type, event.state_key))
693	709	if not prev_event_id:
694		return
	710	return None
695	711	prev_event = await self.store.get_event(prev_event_id, allow_none=True)
696	712	if not prev_event:
697		return
	713	return None
698	714
699	715	if prev_event and event.user_id == prev_event.user_id:
700	716	prev_content = encode_canonical_json(prev_event.content)
701	717	next_content = encode_canonical_json(event.content)
702	718	if prev_content == next_content:
703	719	return prev_event
704		return
	720	return None
705	721
706	722	async def create_and_send_nonmember_event(
707	723	self,

709	725	event_dict: dict,
710	726	ratelimit: bool = True,
711	727	txn_id: Optional[str] = None,
	728	ignore_shadow_ban: bool = False,
712	729	) -> Tuple[EventBase, int]:
713	730	"""
714	731	Creates an event, then sends it.
715	732
716	733	See self.create_event and self.send_nonmember_event.
717		"""
	734
	735	Args:
	736	requester: The requester sending the event.
	737	event_dict: An entire event.
	738	ratelimit: Whether to rate limit this send.
	739	txn_id: The transaction ID.
	740	ignore_shadow_ban: True if shadow-banned users should be allowed to
	741	send this event.
	742
	743	Raises:
	744	ShadowBanError if the requester has been shadow-banned.
	745	"""
	746	if not ignore_shadow_ban and requester.shadow_banned:
	747	# We randomly sleep a bit just to annoy the requester.
	748	await self.clock.sleep(random.randint(1, 10))
	749	raise ShadowBanError()
718	750
719	751	# We limit the number of concurrent event sends in a room so that we
720	752	# don't fork the DAG too much. If we don't limit then we can end up in

733	765	raise SynapseError(403, spam_error, Codes.FORBIDDEN)
734	766
735	767	stream_id = await self.send_nonmember_event(
736		requester, event, context, ratelimit=ratelimit
	768	requester,
	769	event,
	770	context,
	771	ratelimit=ratelimit,
	772	ignore_shadow_ban=ignore_shadow_ban,
737	773	)
738	774	return event, stream_id
739	775

742	778	self,
743	779	builder: EventBuilder,
744	780	requester: Optional[Requester] = None,
745		prev_event_ids: Optional[Collection[str]] = None,
	781	prev_event_ids: Optional[List[str]] = None,
746	782	) -> Tuple[EventBase, EventContext]:
747	783	"""Create a new event for a local client
748	784

858	894	# Ensure that we can round trip before trying to persist in db
859	895	try:
860	896	dump = frozendict_json_encoder.encode(event.content)
861		json.loads(dump)
	897	json_decoder.decode(dump)
862	898	except Exception:
863	899	logger.exception("Failed to encode content: %r", event.content)
864	900	raise

890	926	except Exception:
891	927	# Ensure that we actually remove the entries in the push actions
892	928	# staging area, if we calculated them.
893		run_in_background(
894		self.store.remove_push_actions_from_staging, event.event_id
895		)
	929	await self.store.remove_push_actions_from_staging(event.event_id)
896	930	raise
897	931
898	932	async def _validate_canonical_alias(

956	990	allow_none=True,
957	991	)
958	992
959		is_admin_redaction = (
	993	is_admin_redaction = bool(
960	994	original_event and event.sender != original_event.sender
961	995	)
962	996

1076	1110	auth_events_ids = self.auth.compute_auth_events(
1077	1111	event, prev_state_ids, for_verification=True
1078	1112	)
1079		auth_events = await self.store.get_events(auth_events_ids)
1080		auth_events = {(e.type, e.state_key): e for e in auth_events.values()}
	1113	auth_events_map = await self.store.get_events(auth_events_ids)
	1114	auth_events = {(e.type, e.state_key): e for e in auth_events_map.values()}
1081	1115
1082	1116	room_version = await self.store.get_room_version_id(event.room_id)
1083	1117	room_version_obj = KNOWN_ROOM_VERSIONS[room_version]

1175	1209
1176	1210	event.internal_metadata.proactively_send = False
1177	1211
	1212	# Since this is a dummy-event it is OK if it is sent by a
	1213	# shadow-banned user.
1178	1214	await self.send_nonmember_event(
1179		requester, event, context, ratelimit=False
	1215	requester,
	1216	event,
	1217	context,
	1218	ratelimit=False,
	1219	ignore_shadow_ban=True,
1180	1220	)
1181	1221	dummy_event_sent = True
1182	1222	break

+25

-10

synapse/handlers/oidc_handler.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14		import json
15	14	import logging
16	15	from typing import TYPE_CHECKING, Dict, Generic, List, Optional, Tuple, TypeVar
17	16	from urllib.parse import urlencode

37	36	from synapse.http.server import respond_with_html
38	37	from synapse.http.site import SynapseRequest
39	38	from synapse.logging.context import make_deferred_yieldable
40		from synapse.push.mailer import load_jinja2_templates
41	39	from synapse.types import UserID, map_username_to_mxid_localpart
	40	from synapse.util import json_decoder
42	41
43	42	if TYPE_CHECKING:
44	43	from synapse.server import HomeServer

93	92	"""
94	93
95	94	def __init__(self, hs: "HomeServer"):
	95	self.hs = hs
96	96	self._callback_url = hs.config.oidc_callback_url # type: str
97	97	self._scopes = hs.config.oidc_scopes # type: List[str]
98	98	self._client_auth = ClientAuth(

122	122	self._hostname = hs.hostname # type: str
123	123	self._server_name = hs.config.server_name # type: str
124	124	self._macaroon_secret_key = hs.config.macaroon_secret_key
125		self._error_template = load_jinja2_templates(
126		hs.config.sso_template_dir, ["sso_error.html"]
127		)[0]
	125	self._error_template = hs.config.sso_error_template
128	126
129	127	# identifier for the external_ids table
130	128	self._auth_provider_id = "oidc"

369	367	# and check for an error field. If not, we respond with a generic
370	368	# error message.
371	369	try:
372		resp = json.loads(resp_body.decode("utf-8"))
	370	resp = json_decoder.decode(resp_body.decode("utf-8"))
373	371	error = resp["error"]
374	372	description = resp.get("error_description", error)
375	373	except (ValueError, KeyError):

386	384
387	385	# Since it is a not a 5xx code, body should be a valid JSON. It will
388	386	# raise if not.
389		resp = json.loads(resp_body.decode("utf-8"))
	387	resp = json_decoder.decode(resp_body.decode("utf-8"))
390	388
391	389	if "error" in resp:
392	390	error = resp["error"]

691	689	self._render_error(request, "invalid_token", str(e))
692	690	return
693	691
	692	# Pull out the user-agent and IP from the request.
	693	user_agent = request.requestHeaders.getRawHeaders(b"User-Agent", default=[b""])[
	694	0
	695	].decode("ascii", "surrogateescape")
	696	ip_address = self.hs.get_ip_from_request(request)
	697
694	698	# Call the mapper to register/login the user
695	699	try:
696		user_id = await self._map_userinfo_to_user(userinfo, token)
	700	user_id = await self._map_userinfo_to_user(
	701	userinfo, token, user_agent, ip_address
	702	)
697	703	except MappingException as e:
698	704	logger.exception("Could not map user")
699	705	self._render_error(request, "mapping_error", str(e))

830	836	now = self._clock.time_msec()
831	837	return now < expiry
832	838
833		async def _map_userinfo_to_user(self, userinfo: UserInfo, token: Token) -> str:
	839	async def _map_userinfo_to_user(
	840	self, userinfo: UserInfo, token: Token, user_agent: str, ip_address: str
	841	) -> str:
834	842	"""Maps a UserInfo object to a mxid.
835	843
836	844	UserInfo should have a claim that uniquely identifies users. This claim

845	853	Args:
846	854	userinfo: an object representing the user
847	855	token: a dict with the tokens obtained from the provider
	856	user_agent: The user agent of the client making the request.
	857	ip_address: The IP address of the client making the request.
848	858
849	859	Raises:
850	860	MappingException: if there was an error while mapping some properties

858	868	raise MappingException(
859	869	"Failed to extract subject from OIDC response: %s" % (e,)
860	870	)
	871	# Some OIDC providers use integer IDs, but Synapse expects external IDs
	872	# to be strings.
	873	remote_user_id = str(remote_user_id)
861	874
862	875	logger.info(
863	876	"Looking for existing mapping for user %s:%s",

901	914	# It's the first time this user is logging in and the mapped mxid was
902	915	# not taken, register the user
903	916	registered_user_id = await self._registration_handler.register_user(
904		localpart=localpart, default_display_name=attributes["display_name"],
	917	localpart=localpart,
	918	default_display_name=attributes["display_name"],
	919	user_agent_ips=(user_agent, ip_address),
905	920	)
906	921
907	922	await self._datastore.record_user_external_id(

+90

-57

synapse/handlers/pagination.py less more

13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
15	15	import logging
	16	from typing import TYPE_CHECKING, Any, Dict, Optional, Set
16	17
17	18	from twisted.python.failure import Failure
18	19
19	20	from synapse.api.constants import EventTypes, Membership
20	21	from synapse.api.errors import SynapseError
	22	from synapse.api.filtering import Filter
21	23	from synapse.logging.context import run_in_background
22	24	from synapse.metrics.background_process_metrics import run_as_background_process
23	25	from synapse.storage.state import StateFilter
24		from synapse.types import RoomStreamToken
	26	from synapse.streams.config import PaginationConfig
	27	from synapse.types import Requester, RoomStreamToken
25	28	from synapse.util.async_helpers import ReadWriteLock
26	29	from synapse.util.stringutils import random_string
27	30	from synapse.visibility import filter_events_for_client
28	31
	32	if TYPE_CHECKING:
	33	from synapse.server import HomeServer
	34
	35
29	36	logger = logging.getLogger(__name__)
30	37
31	38
32		class PurgeStatus(object):
	39	class PurgeStatus:
33	40	"""Object tracking the status of a purge request
34	41
35	42	This class contains information on the progress of a purge request, for

57	64	return {"status": PurgeStatus.STATUS_TEXT[self.status]}
58	65
59	66
60		class PaginationHandler(object):
	67	class PaginationHandler:
61	68	"""Handles pagination and purge history requests.
62	69
63	70	These are in the same handler due to the fact we need to block clients
64	71	paginating during a purge.
65	72	"""
66	73
67		def __init__(self, hs):
	74	def __init__(self, hs: "HomeServer"):
68	75	self.hs = hs
69	76	self.auth = hs.get_auth()
70	77	self.store = hs.get_datastore()

74	81	self._server_name = hs.hostname
75	82
76	83	self.pagination_lock = ReadWriteLock()
77		self._purges_in_progress_by_room = set()
	84	self._purges_in_progress_by_room = set() # type: Set[str]
78	85	# map from purge id to PurgeStatus
79		self._purges_by_id = {}
	86	self._purges_by_id = {} # type: Dict[str, PurgeStatus]
80	87	self._event_serializer = hs.get_event_client_serializer()
81	88
82	89	self._retention_default_max_lifetime = hs.config.retention_default_max_lifetime
	90
	91	self._retention_allowed_lifetime_min = hs.config.retention_allowed_lifetime_min
	92	self._retention_allowed_lifetime_max = hs.config.retention_allowed_lifetime_max
83	93
84	94	if hs.config.retention_enabled:
85	95	# Run the purge jobs described in the configuration file.

95	105	job["longest_max_lifetime"],
96	106	)
97	107
98		async def purge_history_for_rooms_in_range(self, min_ms, max_ms):
	108	async def purge_history_for_rooms_in_range(
	109	self, min_ms: Optional[int], max_ms: Optional[int]
	110	):
99	111	"""Purge outdated events from rooms within the given retention range.
100	112
101	113	If a default retention policy is defined in the server's configuration and its

103	115	retention policy.
104	116
105	117	Args:
106		min_ms (int\|None): Duration in milliseconds that define the lower limit of
	118	min_ms: Duration in milliseconds that define the lower limit of
107	119	the range to handle (exclusive). If None, it means that the range has no
108	120	lower limit.
109		max_ms (int\|None): Duration in milliseconds that define the upper limit of
	121	max_ms: Duration in milliseconds that define the upper limit of
110	122	the range to handle (inclusive). If None, it means that the range has no
111	123	upper limit.
112	124	"""
113		# We want the storage layer to to include rooms with no retention policy in its
	125	# We want the storage layer to include rooms with no retention policy in its
114	126	# return value only if a default retention policy is defined in the server's
115	127	# configuration and that policy's 'max_lifetime' is either lower (or equal) than
116	128	# max_ms or higher than min_ms (or both).

151	163	)
152	164	continue
153	165
154		max_lifetime = retention_policy["max_lifetime"]
155
156		if max_lifetime is None:
157		# If max_lifetime is None, it means that include_null equals True,
158		# therefore we can safely assume that there is a default policy defined
159		# in the server's configuration.
160		max_lifetime = self._retention_default_max_lifetime
	166	# If max_lifetime is None, it means that the room has no retention policy.
	167	# Given we only retrieve such rooms when there's a default retention policy
	168	# defined in the server's configuration, we can safely assume that's the
	169	# case and use it for this room.
	170	max_lifetime = (
	171	retention_policy["max_lifetime"] or self._retention_default_max_lifetime
	172	)
	173
	174	# Cap the effective max_lifetime to be within the range allowed in the
	175	# config.
	176	# We do this in two steps:
	177	# 1. Make sure it's higher or equal to the minimum allowed value, and if
	178	# it's not replace it with that value. This is because the server
	179	# operator can be required to not delete information before a given
	180	# time, e.g. to comply with freedom of information laws.
	181	# 2. Make sure the resulting value is lower or equal to the maximum allowed
	182	# value, and if it's not replace it with that value. This is because the
	183	# server operator can be required to delete any data after a specific
	184	# amount of time.
	185	if self._retention_allowed_lifetime_min is not None:
	186	max_lifetime = max(self._retention_allowed_lifetime_min, max_lifetime)
	187
	188	if self._retention_allowed_lifetime_max is not None:
	189	max_lifetime = min(max_lifetime, self._retention_allowed_lifetime_max)
	190
	191	logger.debug("[purge] max_lifetime for room %s: %s", room_id, max_lifetime)
161	192
162	193	# Figure out what token we should start purging at.
163	194	ts = self.clock.time_msec() - max_lifetime

194	225	"_purge_history", self._purge_history, purge_id, room_id, token, True,
195	226	)
196	227
197		def start_purge_history(self, room_id, token, delete_local_events=False):
	228	def start_purge_history(
	229	self, room_id: str, token: str, delete_local_events: bool = False
	230	) -> str:
198	231	"""Start off a history purge on a room.
199	232
200	233	Args:
201		room_id (str): The room to purge from
202
203		token (str): topological token to delete events before
204		delete_local_events (bool): True to delete local events as well as
	234	room_id: The room to purge from
	235	token: topological token to delete events before
	236	delete_local_events: True to delete local events as well as
205	237	remote ones
206	238
207	239	Returns:
208		str: unique ID for this purge transaction.
	240	unique ID for this purge transaction.
209	241	"""
210	242	if room_id in self._purges_in_progress_by_room:
211	243	raise SynapseError(

224	256	)
225	257	return purge_id
226	258
227		async def _purge_history(self, purge_id, room_id, token, delete_local_events):
	259	async def _purge_history(
	260	self, purge_id: str, room_id: str, token: str, delete_local_events: bool
	261	) -> None:
228	262	"""Carry out a history purge on a room.
229	263
230	264	Args:
231		purge_id (str): The id for this purge
232		room_id (str): The room to purge from
233		token (str): topological token to delete events before
234		delete_local_events (bool): True to delete local events as well as
235		remote ones
	265	purge_id: The id for this purge
	266	room_id: The room to purge from
	267	token: topological token to delete events before
	268	delete_local_events: True to delete local events as well as remote ones
236	269	"""
237	270	self._purges_in_progress_by_room.add(room_id)
238	271	try:

257	290
258	291	self.hs.get_reactor().callLater(24 * 3600, clear_purge)
259	292
260		def get_purge_status(self, purge_id):
	293	def get_purge_status(self, purge_id: str) -> Optional[PurgeStatus]:
261	294	"""Get the current status of an active purge
262	295
263	296	Args:
264		purge_id (str): purge_id returned by start_purge_history
265
266		Returns:
267		PurgeStatus\|None
	297	purge_id: purge_id returned by start_purge_history
268	298	"""
269	299	return self._purges_by_id.get(purge_id)
270	300
271		async def purge_room(self, room_id):
	301	async def purge_room(self, room_id: str) -> None:
272	302	"""Purge the given room from the database"""
273		with (await self.pagination_lock.write(room_id)):
	303	with await self.pagination_lock.write(room_id):
274	304	# check we know about the room
275	305	await self.store.get_room_version_id(room_id)
276	306

284	314
285	315	async def get_messages(
286	316	self,
287		requester,
288		room_id=None,
289		pagin_config=None,
290		as_client_event=True,
291		event_filter=None,
292		):
	317	requester: Requester,
	318	room_id: str,
	319	pagin_config: PaginationConfig,
	320	as_client_event: bool = True,
	321	event_filter: Optional[Filter] = None,
	322	) -> Dict[str, Any]:
293	323	"""Get messages in a room.
294	324
295	325	Args:
296		requester (Requester): The user requesting messages.
297		room_id (str): The room they want messages from.
298		pagin_config (synapse.api.streams.PaginationConfig): The pagination
299		config rules to apply, if any.
300		as_client_event (bool): True to get events in client-server format.
301		event_filter (Filter): Filter to apply to results or None
	326	requester: The user requesting messages.
	327	room_id: The room they want messages from.
	328	pagin_config: The pagination config rules to apply, if any.
	329	as_client_event: True to get events in client-server format.
	330	event_filter: Filter to apply to results or None
302	331	Returns:
303		dict: Pagination API results
	332	Pagination API results
304	333	"""
305	334	user_id = requester.user.to_string()
306	335

320	349
321	350	source_config = pagin_config.get_source_config("room")
322	351
323		with (await self.pagination_lock.read(room_id)):
	352	with await self.pagination_lock.read(room_id):
324	353	(
325	354	membership,
326	355	member_event_id,

332	361	# if we're going backwards, we might need to backfill. This
333	362	# requires that we have a topo token.
334	363	if room_token.topological:
335		max_topo = room_token.topological
	364	curr_topo = room_token.topological
336	365	else:
337		max_topo = await self.store.get_max_topological_token(
	366	curr_topo = await self.store.get_current_topological_token(
338	367	room_id, room_token.stream
339	368	)
340	369

342	371	# If they have left the room then clamp the token to be before
343	372	# they left the room, to save the effort of loading from the
344	373	# database.
	374
	375	# This is only None if the room is world_readable, in which
	376	# case "JOIN" would have been returned.
	377	assert member_event_id
	378
345	379	leave_token = await self.store.get_topological_token_for_event(
346	380	member_event_id
347	381	)
348		leave_token = RoomStreamToken.parse(leave_token)
349		if leave_token.topological < max_topo:
	382	if RoomStreamToken.parse(leave_token).topological < curr_topo:
350	383	source_config.from_key = str(leave_token)
351	384
352	385	await self.hs.get_handlers().federation_handler.maybe_backfill(
353		room_id, max_topo
	386	room_id, curr_topo, limit=source_config.limit,
354	387	)
355	388
356	389	events, next_key = await self.store.paginate_room_events(

393	426	)
394	427
395	428	if state_ids:
396		state = await self.store.get_events(list(state_ids.values()))
397		state = state.values()
	429	state_dict = await self.store.get_events(list(state_ids.values()))
	430	state = state_dict.values()
398	431
399	432	time_now = self.clock.time_msec()
400	433

+1

-1

synapse/handlers/password_policy.py less more

21	21	logger = logging.getLogger(__name__)
22	22
23	23
24		class PasswordPolicyHandler(object):
	24	class PasswordPolicyHandler:
25	25	def __init__(self, hs):
26	26	self.policy = hs.config.password_policy
27	27	self.enabled = hs.config.password_policy_enabled

+5

-5

synapse/handlers/presence.py less more

32	32	import synapse.metrics
33	33	from synapse.api.constants import EventTypes, Membership, PresenceState
34	34	from synapse.api.errors import SynapseError
	35	from synapse.api.presence import UserPresenceState
35	36	from synapse.logging.context import run_in_background
36	37	from synapse.logging.utils import log_function
37	38	from synapse.metrics import LaterGauge
38	39	from synapse.metrics.background_process_metrics import run_as_background_process
39	40	from synapse.state import StateHandler
40	41	from synapse.storage.databases.main import DataStore
41		from synapse.storage.presence import UserPresenceState
42		from synapse.types import JsonDict, UserID, get_domain_from_id
	42	from synapse.types import Collection, JsonDict, UserID, get_domain_from_id
43	43	from synapse.util.async_helpers import Linearizer
44	44	from synapse.util.caches.descriptors import cached
45	45	from synapse.util.metrics import Measure

1009	1009	return content
1010	1010
1011	1011
1012		class PresenceEventSource(object):
	1012	class PresenceEventSource:
1013	1013	def __init__(self, hs):
1014	1014	# We can't call get_presence_handler here because there's a cycle:
1015	1015	#

1317	1317
1318	1318	async def get_interested_remotes(
1319	1319	store: DataStore, states: List[UserPresenceState], state_handler: StateHandler
1320		) -> List[Tuple[List[str], List[UserPresenceState]]]:
	1320	) -> List[Tuple[Collection[str], List[UserPresenceState]]]:
1321	1321	"""Given a list of presence states figure out which remote servers
1322	1322	should be sent which.
1323	1323

1333	1333	each tuple the list of UserPresenceState should be sent to each
1334	1334	destination
1335	1335	"""
1336		hosts_and_states = []
	1336	hosts_and_states = [] # type: List[Tuple[Collection[str], List[UserPresenceState]]]
1337	1337
1338	1338	# First we look up the rooms each user is in (as well as any explicit
1339	1339	# subscriptions), then for each distinct room we look up the remote

+21

-2

synapse/handlers/profile.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
	16	import random
16	17
17	18	from synapse.api.errors import (
18	19	AuthError,

159	160	Codes.FORBIDDEN,
160	161	)
161	162
	163	if not isinstance(new_displayname, str):
	164	raise SynapseError(400, "Invalid displayname")
	165
162	166	if len(new_displayname) > MAX_DISPLAYNAME_LEN:
163	167	raise SynapseError(
164	168	400, "Displayname is too long (max %i)" % (MAX_DISPLAYNAME_LEN,)

212	216	async def set_avatar_url(
213	217	self, target_user, requester, new_avatar_url, by_admin=False
214	218	):
215		"""target_user is the user whose avatar_url is to be changed;
216		auth_user is the user attempting to make this change."""
	219	"""Set a new avatar URL for a user.
	220
	221	Args:
	222	target_user (UserID): the user whose avatar URL is to be changed.
	223	requester (Requester): The user attempting to make this change.
	224	new_avatar_url (str): The avatar URL to give this user.
	225	by_admin (bool): Whether this change was made by an administrator.
	226	"""
217	227	if not self.hs.is_mine(target_user):
218	228	raise SynapseError(400, "User is not hosted on this homeserver")
219	229

227	237	400, "Changing avatar is disabled on this server", Codes.FORBIDDEN
228	238	)
229	239
	240	if not isinstance(new_avatar_url, str):
	241	raise SynapseError(400, "Invalid displayname")
	242
230	243	if len(new_avatar_url) > MAX_AVATAR_URL_LEN:
231	244	raise SynapseError(
232	245	400, "Avatar URL is too long (max %i)" % (MAX_AVATAR_URL_LEN,)

276	289	return
277	290
278	291	await self.ratelimit(requester)
	292
	293	# Do not actually update the room state for shadow-banned users.
	294	if requester.shadow_banned:
	295	# We randomly sleep a bit just to annoy the requester.
	296	await self.clock.sleep(random.randint(1, 10))
	297	return
279	298
280	299	room_ids = await self.store.get_rooms_for_user(target_user.to_string())
281	300

+1

-1

synapse/handlers/receipts.py less more

122	122	await self.federation.send_read_receipt(receipt)
123	123
124	124
125		class ReceiptEventSource(object):
	125	class ReceiptEventSource:
126	126	def __init__(self, hs):
127	127	self.store = hs.get_datastore()
128	128

+33

-1

synapse/handlers/register.py less more

25	25	ReplicationPostRegisterActionsServlet,
26	26	ReplicationRegisterServlet,
27	27	)
	28	from synapse.spam_checker_api import RegistrationBehaviour
28	29	from synapse.storage.state import StateFilter
29	30	from synapse.types import RoomAlias, UserID, create_requester
30	31

51	52	self.macaroon_gen = hs.get_macaroon_generator()
52	53	self._server_notices_mxid = hs.config.server_notices_mxid
53	54
	55	self.spam_checker = hs.get_spam_checker()
	56
54	57	if hs.config.worker_app:
55	58	self._register_client = ReplicationRegisterServlet.make_client(hs)
56	59	self._register_device_client = RegisterDeviceReplicationServlet.make_client(

123	126	try:
124	127	int(localpart)
125	128	raise SynapseError(
126		400, "Numeric user IDs are reserved for guest users."
	129	400,
	130	"Numeric user IDs are reserved for guest users.",
	131	errcode=Codes.INVALID_USERNAME,
127	132	)
128	133	except ValueError:
129	134	pass

141	146	address=None,
142	147	bind_emails=[],
143	148	by_admin=False,
	149	user_agent_ips=None,
144	150	):
145	151	"""Registers a new client on the server.
146	152

158	164	bind_emails (List[str]): list of emails to bind to this account.
159	165	by_admin (bool): True if this registration is being made via the
160	166	admin api, otherwise False.
	167	user_agent_ips (List[(str, str)]): Tuples of IP addresses and user-agents used
	168	during the registration process.
161	169	Returns:
162	170	str: user_id
163	171	Raises:
164	172	SynapseError if there was a problem registering.
165	173	"""
166	174	self.check_registration_ratelimit(address)
	175
	176	result = self.spam_checker.check_registration_for_spam(
	177	threepid, localpart, user_agent_ips or [],
	178	)
	179
	180	if result == RegistrationBehaviour.DENY:
	181	logger.info(
	182	"Blocked registration of %r", localpart,
	183	)
	184	# We return a 429 to make it not obvious that they've been
	185	# denied.
	186	raise SynapseError(429, "Rate limited")
	187
	188	shadow_banned = result == RegistrationBehaviour.SHADOW_BAN
	189	if shadow_banned:
	190	logger.info(
	191	"Shadow banning registration of %r", localpart,
	192	)
167	193
168	194	# do not check_auth_blocking if the call is coming through the Admin API
169	195	if not by_admin:

193	219	admin=admin,
194	220	user_type=user_type,
195	221	address=address,
	222	shadow_banned=shadow_banned,
196	223	)
197	224
198	225	if self.hs.config.user_directory_search_all_users:

223	250	make_guest=make_guest,
224	251	create_profile_with_displayname=default_display_name,
225	252	address=address,
	253	shadow_banned=shadow_banned,
226	254	)
227	255
228	256	# Successfully registered

528	556	admin=False,
529	557	user_type=None,
530	558	address=None,
	559	shadow_banned=False,
531	560	):
532	561	"""Register user in the datastore.
533	562

545	574	user_type (str\|None): type of user. One of the values from
546	575	api.constants.UserTypes, or None for a normal user.
547	576	address (str\|None): the IP address used to perform the registration.
	577	shadow_banned (bool): Whether to shadow-ban the user
548	578
549	579	Returns:
550	580	Awaitable

560	590	admin=admin,
561	591	user_type=user_type,
562	592	address=address,
	593	shadow_banned=shadow_banned,
563	594	)
564	595	else:
565	596	return self.store.register_user(

571	602	create_profile_with_displayname=create_profile_with_displayname,
572	603	admin=admin,
573	604	user_type=user_type,
	605	shadow_banned=shadow_banned,
574	606	)
575	607
576	608	async def register_device(

+107

-51

synapse/handlers/room.py less more

19	19	import itertools
20	20	import logging
21	21	import math
	22	import random
22	23	import string
23	24	from collections import OrderedDict
24		from typing import Awaitable, Optional, Tuple
	25	from typing import TYPE_CHECKING, Any, Awaitable, Dict, List, Optional, Tuple
25	26
26	27	from synapse.api.constants import (
27	28	EventTypes,

31	32	RoomEncryptionAlgorithms,
32	33	)
33	34	from synapse.api.errors import AuthError, Codes, NotFoundError, StoreError, SynapseError
	35	from synapse.api.filtering import Filter
34	36	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, RoomVersion
	37	from synapse.events import EventBase
35	38	from synapse.events.utils import copy_power_levels_contents
36	39	from synapse.http.endpoint import parse_and_validate_server_name
37	40	from synapse.storage.state import StateFilter
38	41	from synapse.types import (
	42	JsonDict,
	43	MutableStateMap,
39	44	Requester,
40	45	RoomAlias,
41	46	RoomID,

46	51	create_requester,
47	52	)
48	53	from synapse.util import stringutils
49		from synapse.util.async_helpers import Linearizer, maybe_awaitable
	54	from synapse.util.async_helpers import Linearizer
50	55	from synapse.util.caches.response_cache import ResponseCache
51	56	from synapse.visibility import filter_events_for_client
52	57
53	58	from ._base import BaseHandler
54	59
	60	if TYPE_CHECKING:
	61	from synapse.server import HomeServer
	62
55	63	logger = logging.getLogger(__name__)
56	64
57	65	id_server_scheme = "https://"

60	68
61	69
62	70	class RoomCreationHandler(BaseHandler):
63		def __init__(self, hs):
	71	def __init__(self, hs: "HomeServer"):
64	72	super(RoomCreationHandler, self).__init__(hs)
65	73
66	74	self.spam_checker = hs.get_spam_checker()

91	99	"guest_can_join": False,
92	100	"power_level_content_override": {},
93	101	},
94		}
	102	} # type: Dict[str, Dict[str, Any]]
95	103
96	104	# Modify presets to selectively enable encryption by default per homeserver config
97	105	for preset_name, preset_config in self._presets_dict.items():

128	136
129	137	Returns:
130	138	the new room id
	139
	140	Raises:
	141	ShadowBanError if the requester is shadow-banned.
131	142	"""
132	143	await self.ratelimit(requester)
133	144

163	174	async def _upgrade_room(
164	175	self, requester: Requester, old_room_id: str, new_version: RoomVersion
165	176	):
	177	"""
	178	Args:
	179	requester: the user requesting the upgrade
	180	old_room_id: the id of the room to be replaced
	181	new_versions: the version to upgrade the room to
	182
	183	Raises:
	184	ShadowBanError if the requester is shadow-banned.
	185	"""
166	186	user_id = requester.user.to_string()
167	187
168	188	# start by allocating a new room id

214	234
215	235	old_room_state = await tombstone_context.get_current_state_ids()
216	236
	237	# We know the tombstone event isn't an outlier so it has current state.
	238	assert old_room_state is not None
	239
217	240	# update any aliases
218	241	await self._move_aliases_to_new_room(
219	242	requester, old_room_id, new_room_id, old_room_state

246	269	old_room_id: the id of the room to be replaced
247	270	new_room_id: the id of the replacement room
248	271	old_room_state: the state map for the old room
	272
	273	Raises:
	274	ShadowBanError if the requester is shadow-banned.
249	275	"""
250	276	old_room_pl_event_id = old_room_state.get((EventTypes.PowerLevels, ""))
251	277

424	450	old_room_member_state_events = await self.store.get_events(
425	451	old_room_member_state_ids.values()
426	452	)
427		for k, old_event in old_room_member_state_events.items():
	453	for old_event in old_room_member_state_events.values():
428	454	# Only transfer ban events
429	455	if (
430	456	"membership" in old_event.content

527	553	logger.error("Unable to send updated alias events in new room: %s", e)
528	554
529	555	async def create_room(
530		self, requester, config, ratelimit=True, creator_join_profile=None
	556	self,
	557	requester: Requester,
	558	config: JsonDict,
	559	ratelimit: bool = True,
	560	creator_join_profile: Optional[JsonDict] = None,
531	561	) -> Tuple[dict, int]:
532	562	""" Creates a new room.
533	563
534	564	Args:
535		requester (synapse.types.Requester):
	565	requester:
536	566	The user who requested the room creation.
537		config (dict) : A dict of configuration options.
538		ratelimit (bool): set to False to disable the rate limiter
539
540		creator_join_profile (dict\|None):
	567	config : A dict of configuration options.
	568	ratelimit: set to False to disable the rate limiter
	569
	570	creator_join_profile:
541	571	Set to override the displayname and avatar for the creating
542	572	user in this room. If unset, displayname and avatar will be
543	573	derived from the user's profile. If set, should contain the

600	630	Codes.UNSUPPORTED_ROOM_VERSION,
601	631	)
602	632
	633	room_alias = None
603	634	if "room_alias_name" in config:
604	635	for wchar in string.whitespace:
605	636	if wchar in config["room_alias_name"]:

610	641
611	642	if mapping:
612	643	raise SynapseError(400, "Room alias already taken", Codes.ROOM_IN_USE)
613		else:
614		room_alias = None
615
	644
	645	invite_3pid_list = config.get("invite_3pid", [])
616	646	invite_list = config.get("invite", [])
617	647	for i in invite_list:
618	648	try:

620	650	parse_and_validate_server_name(uid.domain)
621	651	except Exception:
622	652	raise SynapseError(400, "Invalid user_id: %s" % (i,))
	653
	654	if (invite_list or invite_3pid_list) and requester.shadow_banned:
	655	# We randomly sleep a bit just to annoy the requester.
	656	await self.clock.sleep(random.randint(1, 10))
	657
	658	# Allow the request to go through, but remove any associated invites.
	659	invite_3pid_list = []
	660	invite_list = []
623	661
624	662	await self.event_creation_handler.assert_accepted_privacy_policy(requester)
625	663

634	672	"Not a valid power_level_content_override: 'users' did not contain %s"
635	673	% (user_id,),
636	674	)
637
638		invite_3pid_list = config.get("invite_3pid", [])
639	675
640	676	visibility = config.get("visibility", None)
641	677	is_public = visibility == "public"

731	767	if is_direct:
732	768	content["is_direct"] = is_direct
733	769
	770	# Note that update_membership with an action of "invite" can raise a
	771	# ShadowBanError, but this was handled above by emptying invite_list.
734	772	_, last_stream_id = await self.room_member_handler.update_membership(
735	773	requester,
736	774	UserID.from_string(invitee),

745	783	id_access_token = invite_3pid.get("id_access_token") # optional
746	784	address = invite_3pid["address"]
747	785	medium = invite_3pid["medium"]
	786	# Note that do_3pid_invite can raise a ShadowBanError, but this was
	787	# handled above by emptying invite_3pid_list.
748	788	last_stream_id = await self.hs.get_room_member_handler().do_3pid_invite(
749	789	room_id,
750	790	requester.user,

770	810
771	811	async def _send_events_for_new_room(
772	812	self,
773		creator, # A Requester object.
774		room_id,
775		preset_config,
776		invite_list,
777		initial_state,
778		creation_content,
779		room_alias=None,
780		power_level_content_override=None, # Doesn't apply when initial state has power level state event content
781		creator_join_profile=None,
	813	creator: Requester,
	814	room_id: str,
	815	preset_config: str,
	816	invite_list: List[str],
	817	initial_state: MutableStateMap,
	818	creation_content: JsonDict,
	819	room_alias: Optional[RoomAlias] = None,
	820	power_level_content_override: Optional[JsonDict] = None,
	821	creator_join_profile: Optional[JsonDict] = None,
782	822	) -> int:
783	823	"""Sends the initial events into a new room.
	824
	825	`power_level_content_override` doesn't apply when initial state has
	826	power level state event content.
784	827
785	828	Returns:
786	829	The stream_id of the last event persisted.
787	830	"""
788	831
789		def create(etype, content, **kwargs):
	832	creator_id = creator.user.to_string()
	833
	834	event_keys = {"room_id": room_id, "sender": creator_id, "state_key": ""}
	835
	836	def create(etype: str, content: JsonDict, **kwargs) -> JsonDict:
790	837	e = {"type": etype, "content": content}
791	838
792	839	e.update(event_keys)

794	841
795	842	return e
796	843
797		async def send(etype, content, **kwargs) -> int:
	844	async def send(etype: str, content: JsonDict, **kwargs) -> int:
798	845	event = create(etype, content, **kwargs)
799	846	logger.debug("Sending %s in new room", etype)
	847	# Allow these events to be sent even if the user is shadow-banned to
	848	# allow the room creation to complete.
800	849	(
801	850	_,
802	851	last_stream_id,
803	852	) = await self.event_creation_handler.create_and_send_nonmember_event(
804		creator, event, ratelimit=False
	853	creator, event, ratelimit=False, ignore_shadow_ban=True,
805	854	)
806	855	return last_stream_id
807	856
808	857	config = self._presets_dict[preset_config]
809
810		creator_id = creator.user.to_string()
811
812		event_keys = {"room_id": room_id, "sender": creator_id, "state_key": ""}
813	858
814	859	creation_content.update({"creator": creator_id})
815	860	await send(etype=EventTypes.Create, content=creation_content)

851	896	"kick": 50,
852	897	"redact": 50,
853	898	"invite": 50,
854		}
	899	} # type: JsonDict
855	900
856	901	if config["original_invitees_have_ops"]:
857	902	for invitee in invite_list:

905	950	return last_sent_stream_id
906	951
907	952	async def _generate_room_id(
908		self, creator_id: str, is_public: str, room_version: RoomVersion,
	953	self, creator_id: str, is_public: bool, room_version: RoomVersion,
909	954	):
910	955	# autogen room IDs and try to create it. We may clash, so just
911	956	# try a few times till one goes through, giving up eventually.

928	973	raise StoreError(500, "Couldn't generate a room ID.")
929	974
930	975
931		class RoomContextHandler(object):
932		def __init__(self, hs):
	976	class RoomContextHandler:
	977	def __init__(self, hs: "HomeServer"):
933	978	self.hs = hs
934	979	self.store = hs.get_datastore()
935	980	self.storage = hs.get_storage()
936	981	self.state_store = self.storage.state
937	982
938		async def get_event_context(self, user, room_id, event_id, limit, event_filter):
	983	async def get_event_context(
	984	self,
	985	user: UserID,
	986	room_id: str,
	987	event_id: str,
	988	limit: int,
	989	event_filter: Optional[Filter],
	990	) -> Optional[JsonDict]:
939	991	"""Retrieves events, pagination tokens and state around a given event
940	992	in a room.
941	993
942	994	Args:
943		user (UserID)
944		room_id (str)
945		event_id (str)
946		limit (int): The maximum number of events to return in total
	995	user
	996	room_id
	997	event_id
	998	limit: The maximum number of events to return in total
947	999	(excluding state).
948		event_filter (Filter\|None): the filter to apply to the events returned
	1000	event_filter: the filter to apply to the events returned
949	1001	(excluding the target event_id)
950	1002
951	1003	Returns:

1031	1083	return results
1032	1084
1033	1085
1034		class RoomEventSource(object):
1035		def __init__(self, hs):
	1086	class RoomEventSource:
	1087	def __init__(self, hs: "HomeServer"):
1036	1088	self.store = hs.get_datastore()
1037	1089
1038	1090	async def get_new_events(
1039		self, user, from_key, limit, room_ids, is_guest, explicit_room_id=None
1040		):
	1091	self,
	1092	user: UserID,
	1093	from_key: str,
	1094	limit: int,
	1095	room_ids: List[str],
	1096	is_guest: bool,
	1097	explicit_room_id: Optional[str] = None,
	1098	) -> Tuple[List[EventBase], str]:
1041	1099	# We just ignore the key for now.
1042	1100
1043	1101	to_key = self.get_current_key()

1087	1145	return self.store.get_room_events_max_id(room_id)
1088	1146
1089	1147
1090		class RoomShutdownHandler(object):
	1148	class RoomShutdownHandler:
1091	1149
1092	1150	DEFAULT_MESSAGE = (
1093	1151	"Sharing illegal content on this server is not permitted and rooms in"

1095	1153	)
1096	1154	DEFAULT_ROOM_NAME = "Content Violation Notification"
1097	1155
1098		def __init__(self, hs):
	1156	def __init__(self, hs: "HomeServer"):
1099	1157	self.hs = hs
1100	1158	self.room_member_handler = hs.get_room_member_handler()
1101	1159	self._room_creation_handler = hs.get_room_creation_handler()

1271	1329	ratelimit=False,
1272	1330	)
1273	1331
1274		aliases_for_room = await maybe_awaitable(
1275		self.store.get_aliases_for_room(room_id)
1276		)
	1332	aliases_for_room = await self.store.get_aliases_for_room(room_id)
1277	1333
1278	1334	await self.store.update_aliases_for_room(
1279	1335	room_id, new_room_id, requester_user_id

+68

-14

synapse/handlers/room_member.py less more

14	14
15	15	import abc
16	16	import logging
	17	import random
17	18	from http import HTTPStatus
18		from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Tuple, Union
	19	from typing import TYPE_CHECKING, Iterable, List, Optional, Tuple, Union
19	20
20	21	from unpaddedbase64 import encode_base64
21	22
22	23	from synapse import types
23	24	from synapse.api.constants import MAX_DEPTH, EventTypes, Membership
24		from synapse.api.errors import AuthError, Codes, LimitExceededError, SynapseError
	25	from synapse.api.errors import (
	26	AuthError,
	27	Codes,
	28	LimitExceededError,
	29	ShadowBanError,
	30	SynapseError,
	31	)
25	32	from synapse.api.ratelimiting import Ratelimiter
26	33	from synapse.api.room_versions import EventFormatVersions
27	34	from synapse.crypto.event_signing import compute_event_reference_hash

30	37	from synapse.events.snapshot import EventContext
31	38	from synapse.events.validator import EventValidator
32	39	from synapse.storage.roommember import RoomsForUser
33		from synapse.types import Collection, JsonDict, Requester, RoomAlias, RoomID, UserID
	40	from synapse.types import JsonDict, Requester, RoomAlias, RoomID, StateMap, UserID
34	41	from synapse.util.async_helpers import Linearizer
35	42	from synapse.util.distributor import user_joined_room, user_left_room
36	43

43	50	logger = logging.getLogger(__name__)
44	51
45	52
46		class RoomMemberHandler(object):
	53	class RoomMemberHandler:
47	54	# TODO(paul): This handler currently contains a messy conflation of
48	55	# low-level API that works on UserID objects and so on, and REST-level
49	56	# API that takes ID strings and returns pagination chunks. These concerns

168	175	target: UserID,
169	176	room_id: str,
170	177	membership: str,
171		prev_event_ids: Collection[str],
	178	prev_event_ids: List[str],
172	179	txn_id: Optional[str] = None,
173	180	ratelimit: bool = True,
174	181	content: Optional[dict] = None,

300	307	content: Optional[dict] = None,
301	308	require_consent: bool = True,
302	309	) -> Tuple[str, int]:
	310	"""Update a user's membership in a room.
	311
	312	Params:
	313	requester: The user who is performing the update.
	314	target: The user whose membership is being updated.
	315	room_id: The room ID whose membership is being updated.
	316	action: The membership change, see synapse.api.constants.Membership.
	317	txn_id: The transaction ID, if given.
	318	remote_room_hosts: Remote servers to send the update to.
	319	third_party_signed: Information from a 3PID invite.
	320	ratelimit: Whether to rate limit the request.
	321	content: The content of the created event.
	322	require_consent: Whether consent is required.
	323
	324	Returns:
	325	A tuple of the new event ID and stream ID.
	326
	327	Raises:
	328	ShadowBanError if a shadow-banned requester attempts to send an invite.
	329	"""
	330	if action == Membership.INVITE and requester.shadow_banned:
	331	# We randomly sleep a bit just to annoy the requester.
	332	await self.clock.sleep(random.randint(1, 10))
	333	raise ShadowBanError()
	334
303	335	key = (room_id,)
304	336
305	337	with (await self.member_linearizer.queue(key)):

339	371	# later on.
340	372	content = dict(content)
341	373
342		if not self.allow_per_room_profiles:
	374	if not self.allow_per_room_profiles or requester.shadow_banned:
343	375	# Strip profile data, knowing that new profile data will be added to the
344	376	# event's content in event_creation_handler.create_event() using the target's
345	377	# global profile.

709	741	if prev_member_event.membership == Membership.JOIN:
710	742	await self._user_left_room(target_user, room_id)
711	743
712		async def _can_guest_join(
713		self, current_state_ids: Dict[Tuple[str, str], str]
714		) -> bool:
	744	async def _can_guest_join(self, current_state_ids: StateMap[str]) -> bool:
715	745	"""
716	746	Returns whether a guest can join a room based on its current state.
717	747	"""

721	751
722	752	guest_access = await self.store.get_event(guest_access_id)
723	753
724		return (
	754	return bool(
725	755	guest_access
726	756	and guest_access.content
727	757	and "guest_access" in guest_access.content

778	808	txn_id: Optional[str],
779	809	id_access_token: Optional[str] = None,
780	810	) -> int:
	811	"""Invite a 3PID to a room.
	812
	813	Args:
	814	room_id: The room to invite the 3PID to.
	815	inviter: The user sending the invite.
	816	medium: The 3PID's medium.
	817	address: The 3PID's address.
	818	id_server: The identity server to use.
	819	requester: The user making the request.
	820	txn_id: The transaction ID this is part of, or None if this is not
	821	part of a transaction.
	822	id_access_token: The optional identity server access token.
	823
	824	Returns:
	825	The new stream ID.
	826
	827	Raises:
	828	ShadowBanError if the requester has been shadow-banned.
	829	"""
781	830	if self.config.block_non_admin_invites:
782	831	is_requester_admin = await self.auth.is_server_admin(requester.user)
783	832	if not is_requester_admin:

785	834	403, "Invites have been disabled on this server", Codes.FORBIDDEN
786	835	)
787	836
	837	if requester.shadow_banned:
	838	# We randomly sleep a bit just to annoy the requester.
	839	await self.clock.sleep(random.randint(1, 10))
	840	raise ShadowBanError()
	841
788	842	# We need to rate limit before we send out any 3PID invites, so we
789	843	# can't just rely on the standard ratelimiting of events.
790	844	await self.base_handler.ratelimit(requester)

809	863	)
810	864
811	865	if invitee:
	866	# Note that update_membership with an action of "invite" can raise
	867	# a ShadowBanError, but this was done above already.
812	868	_, stream_id = await self.update_membership(
813	869	requester, UserID.from_string(invitee), room_id, "invite", txn_id=txn_id
814	870	)

914	970	)
915	971	return stream_id
916	972
917		async def _is_host_in_room(
918		self, current_state_ids: Dict[Tuple[str, str], str]
919		) -> bool:
	973	async def _is_host_in_room(self, current_state_ids: StateMap[str]) -> bool:
920	974	# Have we just created the room, and is this about to be the very
921	975	# first member event?
922	976	create_event_id = current_state_ids.get(("m.room.create", ""))

1047	1101	return event_id, stream_id
1048	1102
1049	1103	# The room is too large. Leave.
1050		requester = types.create_requester(user, None, False, None)
	1104	requester = types.create_requester(user, None, False, False, None)
1051	1105	await self.update_membership(
1052	1106	requester=requester, target=user, room_id=room_id, action="leave"
1053	1107	)

+18

-4

synapse/handlers/saml_handler.py less more

53	53
54	54	class SamlHandler:
55	55	def __init__(self, hs: "synapse.server.HomeServer"):
	56	self.hs = hs
56	57	self._saml_client = Saml2Client(hs.config.saml2_sp_config)
57	58	self._auth = hs.get_auth()
58	59	self._auth_handler = hs.get_auth_handler()

132	133	# the dict.
133	134	self.expire_sessions()
134	135
	136	# Pull out the user-agent and IP from the request.
	137	user_agent = request.requestHeaders.getRawHeaders(b"User-Agent", default=[b""])[
	138	0
	139	].decode("ascii", "surrogateescape")
	140	ip_address = self.hs.get_ip_from_request(request)
	141
135	142	user_id, current_session = await self._map_saml_response_to_user(
136		resp_bytes, relay_state
	143	resp_bytes, relay_state, user_agent, ip_address
137	144	)
138	145
139	146	# Complete the interactive auth session or the login.

146	153	await self._auth_handler.complete_sso_login(user_id, request, relay_state)
147	154
148	155	async def _map_saml_response_to_user(
149		self, resp_bytes: str, client_redirect_url: str
	156	self,
	157	resp_bytes: str,
	158	client_redirect_url: str,
	159	user_agent: str,
	160	ip_address: str,
150	161	) -> Tuple[str, Optional[Saml2SessionData]]:
151	162	"""
152	163	Given a sample response, retrieve the cached session and user for it.

154	165	Args:
155	166	resp_bytes: The SAML response.
156	167	client_redirect_url: The redirect URL passed in by the client.
	168	user_agent: The user agent of the client making the request.
	169	ip_address: The IP address of the client making the request.
157	170
158	171	Returns:
159	172	Tuple of the user ID and SAML session associated with this response.

290	303	localpart=localpart,
291	304	default_display_name=displayname,
292	305	bind_emails=emails,
	306	user_agent_ips=(user_agent, ip_address),
293	307	)
294	308
295	309	await self._datastore.record_user_external_id(

345	359
346	360
347	361	@attr.s
348		class SamlConfig(object):
	362	class SamlConfig:
349	363	mxid_source_attribute = attr.ib()
350	364	mxid_mapper = attr.ib()
351	365
352	366
353		class DefaultSamlMappingProvider(object):
	367	class DefaultSamlMappingProvider:
354	368	__version__ = "0.0.1"
355	369
356	370	def __init__(self, parsed_config: SamlConfig, module_api: ModuleApi):

+1

-1

synapse/handlers/state_deltas.py less more

17	17	logger = logging.getLogger(__name__)
18	18
19	19
20		class StateDeltasHandler(object):
	20	class StateDeltasHandler:
21	21	def __init__(self, hs):
22	22	self.store = hs.get_datastore()
23	23

+31

-25

synapse/handlers/sync.py less more

15	15
16	16	import itertools
17	17	import logging
18		from typing import Any, Dict, FrozenSet, List, Optional, Set, Tuple
	18	from typing import TYPE_CHECKING, Any, Dict, FrozenSet, List, Optional, Set, Tuple
19	19
20	20	import attr
21	21	from prometheus_client import Counter

30	30	from synapse.types import (
31	31	Collection,
32	32	JsonDict,
	33	MutableStateMap,
33	34	RoomStreamToken,
34	35	StateMap,
35	36	StreamToken,

42	43	from synapse.util.metrics import Measure, measure_func
43	44	from synapse.visibility import filter_events_for_client
44	45
	46	if TYPE_CHECKING:
	47	from synapse.server import HomeServer
	48
45	49	logger = logging.getLogger(__name__)
46	50
47	51	# Debug logger for https://github.com/matrix-org/synapse/issues/4422

93	97	__bool__ = __nonzero__ # python3
94	98
95	99
96		@attr.s(slots=True, frozen=True)
	100	# We can't freeze this class, because we need to update it after it's instantiated to
	101	# update its unread count. This is because we calculate the unread count for a room only
	102	# if there are updates for it, which we check after the instance has been created.
	103	# This should not be a big deal because we update the notification counts afterwards as
	104	# well anyway.
	105	@attr.s(slots=True)
97	106	class JoinedSyncResult:
98	107	room_id = attr.ib(type=str)
99	108	timeline = attr.ib(type=TimelineBatch)

102	111	account_data = attr.ib(type=List[JsonDict])
103	112	unread_notifications = attr.ib(type=JsonDict)
104	113	summary = attr.ib(type=Optional[JsonDict])
	114	unread_count = attr.ib(type=int)
105	115
106	116	def __nonzero__(self) -> bool:
107	117	"""Make the result appear empty if there are no updates. This is used

235	245	__bool__ = __nonzero__ # python3
236	246
237	247
238		class SyncHandler(object):
239		def __init__(self, hs):
	248	class SyncHandler:
	249	def __init__(self, hs: "HomeServer"):
240	250	self.hs_config = hs.config
241	251	self.store = hs.get_datastore()
242	252	self.notifier = hs.get_notifier()

587	597	room_id: str,
588	598	sync_config: SyncConfig,
589	599	batch: TimelineBatch,
590		state: StateMap[EventBase],
	600	state: MutableStateMap[EventBase],
591	601	now_token: StreamToken,
592	602	) -> Optional[JsonDict]:
593	603	""" Works out a room summary block for this room, summarising the number

709	719	]
710	720
711	721	missing_hero_state = await self.store.get_events(missing_hero_event_ids)
712		missing_hero_state = missing_hero_state.values()
713
714		for s in missing_hero_state:
	722
	723	for s in missing_hero_state.values():
715	724	cache.set(s.state_key, s.event_id)
716	725	state[(EventTypes.Member, s.state_key)] = s
717	726

735	744	since_token: Optional[StreamToken],
736	745	now_token: StreamToken,
737	746	full_state: bool,
738		) -> StateMap[EventBase]:
	747	) -> MutableStateMap[EventBase]:
739	748	""" Works out the difference in state between the start of the timeline
740	749	and the previous sync.
741	750

929	938
930	939	async def unread_notifs_for_room_id(
931	940	self, room_id: str, sync_config: SyncConfig
932		) -> Optional[Dict[str, str]]:
	941	) -> Dict[str, int]:
933	942	with Measure(self.clock, "unread_notifs_for_room_id"):
934	943	last_unread_event_id = await self.store.get_last_receipt_event_id_for_user(
935	944	user_id=sync_config.user.to_string(),

937	946	receipt_type="m.read",
938	947	)
939	948
940		if last_unread_event_id:
941		notifs = await self.store.get_unread_event_push_actions_by_room_for_user(
942		room_id, sync_config.user.to_string(), last_unread_event_id
943		)
944		return notifs
945
946		# There is no new information in this period, so your notification
947		# count is whatever it was last time.
948		return None
	949	notifs = await self.store.get_unread_event_push_actions_by_room_for_user(
	950	room_id, sync_config.user.to_string(), last_unread_event_id
	951	)
	952	return notifs
949	953
950	954	async def generate_sync_result(
951	955	self,

1768	1772	ignored_users: Set[str],
1769	1773	room_builder: "RoomSyncResultBuilder",
1770	1774	ephemeral: List[JsonDict],
1771		tags: Optional[List[JsonDict]],
	1775	tags: Optional[Dict[str, Dict[str, Any]]],
1772	1776	account_data: Dict[str, JsonDict],
1773	1777	always_include: bool = False,
1774	1778	):

1884	1888	)
1885	1889
1886	1890	if room_builder.rtype == "joined":
1887		unread_notifications = {} # type: Dict[str, str]
	1891	unread_notifications = {} # type: Dict[str, int]
1888	1892	room_sync = JoinedSyncResult(
1889	1893	room_id=room_id,
1890	1894	timeline=batch,

1893	1897	account_data=account_data_events,
1894	1898	unread_notifications=unread_notifications,
1895	1899	summary=summary,
	1900	unread_count=0,
1896	1901	)
1897	1902
1898	1903	if room_sync or always_include:
1899	1904	notifs = await self.unread_notifs_for_room_id(room_id, sync_config)
1900	1905
1901		if notifs is not None:
1902		unread_notifications["notification_count"] = notifs["notify_count"]
1903		unread_notifications["highlight_count"] = notifs["highlight_count"]
	1906	unread_notifications["notification_count"] = notifs["notify_count"]
	1907	unread_notifications["highlight_count"] = notifs["highlight_count"]
	1908
	1909	room_sync.unread_count = notifs["unread_count"]
1904	1910
1905	1911	sync_result_builder.joined.append(room_sync)
1906	1912

2068	2074
2069	2075
2070	2076	@attr.s
2071		class RoomSyncResultBuilder(object):
	2077	class RoomSyncResultBuilder:
2072	2078	"""Stores information needed to create either a `JoinedSyncResult` or
2073	2079	`ArchivedSyncResult`.
2074	2080

+17

-6

synapse/handlers/typing.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
	16	import random
16	17	from collections import namedtuple
17	18	from typing import TYPE_CHECKING, List, Set, Tuple
18	19
19		from synapse.api.errors import AuthError, SynapseError
	20	from synapse.api.errors import AuthError, ShadowBanError, SynapseError
20	21	from synapse.metrics.background_process_metrics import run_as_background_process
21	22	from synapse.replication.tcp.streams import TypingStream
22	23	from synapse.types import UserID, get_domain_from_id

226	227	self._stopped_typing(member)
227	228	return
228	229
229		async def started_typing(self, target_user, auth_user, room_id, timeout):
	230	async def started_typing(self, target_user, requester, room_id, timeout):
230	231	target_user_id = target_user.to_string()
231		auth_user_id = auth_user.to_string()
	232	auth_user_id = requester.user.to_string()
232	233
233	234	if not self.is_mine_id(target_user_id):
234	235	raise SynapseError(400, "User is not hosted on this homeserver")
235	236
236	237	if target_user_id != auth_user_id:
237	238	raise AuthError(400, "Cannot set another user's typing state")
	239
	240	if requester.shadow_banned:
	241	# We randomly sleep a bit just to annoy the requester.
	242	await self.clock.sleep(random.randint(1, 10))
	243	raise ShadowBanError()
238	244
239	245	await self.auth.check_user_in_room(room_id, target_user_id)
240	246

255	261
256	262	self._push_update(member=member, typing=True)
257	263
258		async def stopped_typing(self, target_user, auth_user, room_id):
	264	async def stopped_typing(self, target_user, requester, room_id):
259	265	target_user_id = target_user.to_string()
260		auth_user_id = auth_user.to_string()
	266	auth_user_id = requester.user.to_string()
261	267
262	268	if not self.is_mine_id(target_user_id):
263	269	raise SynapseError(400, "User is not hosted on this homeserver")
264	270
265	271	if target_user_id != auth_user_id:
266	272	raise AuthError(400, "Cannot set another user's typing state")
	273
	274	if requester.shadow_banned:
	275	# We randomly sleep a bit just to annoy the requester.
	276	await self.clock.sleep(random.randint(1, 10))
	277	raise ShadowBanError()
267	278
268	279	await self.auth.check_user_in_room(room_id, target_user_id)
269	280

400	411	raise Exception("Typing writer instance got typing info over replication")
401	412
402	413
403		class TypingNotificationEventSource(object):
	414	class TypingNotificationEventSource:
404	415	def __init__(self, hs):
405	416	self.hs = hs
406	417	self.clock = hs.get_clock()

+2

-3

synapse/handlers/ui_auth/checkers.py less more

15	15	import logging
16	16	from typing import Any
17	17
18		from canonicaljson import json
19
20	18	from twisted.web.client import PartialDownloadError
21	19
22	20	from synapse.api.constants import LoginType
23	21	from synapse.api.errors import Codes, LoginError, SynapseError
24	22	from synapse.config.emailconfig import ThreepidBehaviour
	23	from synapse.util import json_decoder
25	24
26	25	logger = logging.getLogger(__name__)
27	26

116	115	except PartialDownloadError as pde:
117	116	# Twisted is silly
118	117	data = pde.response
119		resp_body = json.loads(data.decode("utf-8"))
	118	resp_body = json_decoder.decode(data.decode("utf-8"))
120	119
121	120	if "success" in resp_body:
122	121	# Note that we do NOT check the hostname here: we explicitly

+7

-1

synapse/handlers/user_directory.py less more

233	233	async def _handle_room_publicity_change(
234	234	self, room_id, prev_event_id, event_id, typ
235	235	):
236		"""Handle a room having potentially changed from/to world_readable/publically
	236	"""Handle a room having potentially changed from/to world_readable/publicly
237	237	joinable.
238	238
239	239	Args:

387	387
388	388	prev_name = prev_event.content.get("displayname")
389	389	new_name = event.content.get("displayname")
	390	# If the new name is an unexpected form, do not update the directory.
	391	if not isinstance(new_name, str):
	392	new_name = prev_name
390	393
391	394	prev_avatar = prev_event.content.get("avatar_url")
392	395	new_avatar = event.content.get("avatar_url")
	396	# If the new avatar is an unexpected form, do not update the directory.
	397	if not isinstance(new_avatar, str):
	398	new_avatar = prev_avatar
393	399
394	400	if prev_name != new_name or prev_avatar != new_avatar:
395	401	await self.store.update_profile_in_user_dir(user_id, new_name, new_avatar)

+10

-9

synapse/http/client.py less more

18	18	from io import BytesIO
19	19
20	20	import treq
21		from canonicaljson import encode_canonical_json, json
	21	from canonicaljson import encode_canonical_json
22	22	from netaddr import IPAddress
23	23	from prometheus_client import Counter
24	24	from zope.interface import implementer, provider

46	46	from synapse.http.proxyagent import ProxyAgent
47	47	from synapse.logging.context import make_deferred_yieldable
48	48	from synapse.logging.opentracing import set_tag, start_active_span, tags
	49	from synapse.util import json_decoder
49	50	from synapse.util.async_helpers import timeout_deferred
50	51
51	52	logger = logging.getLogger(__name__)

84	85	return _scheduler
85	86
86	87
87		class IPBlacklistingResolver(object):
	88	class IPBlacklistingResolver:
88	89	"""
89	90	A proxy for reactor.nameResolver which only produces non-blacklisted IP
90	91	addresses, preventing DNS rebinding attacks on URL preview.

131	132	r.resolutionComplete()
132	133
133	134	@provider(IResolutionReceiver)
134		class EndpointReceiver(object):
	135	class EndpointReceiver:
135	136	@staticmethod
136	137	def resolutionBegan(resolutionInProgress):
137	138	pass

190	191	)
191	192
192	193
193		class SimpleHttpClient(object):
	194	class SimpleHttpClient:
194	195	"""
195	196	A simple, no-frills HTTP client with methods that wrap up common ways of
196	197	using HTTP in Matrix

242	243	)
243	244
244	245	@implementer(IReactorPluggableNameResolver)
245		class Reactor(object):
	246	class Reactor:
246	247	def __getattr__(_self, attr):
247	248	if attr == "nameResolver":
248	249	return nameResolver

390	391	body = await make_deferred_yieldable(readBody(response))
391	392
392	393	if 200 <= response.code < 300:
393		return json.loads(body.decode("utf-8"))
	394	return json_decoder.decode(body.decode("utf-8"))
394	395	else:
395	396	raise HttpResponseException(
396	397	response.code, response.phrase.decode("ascii", errors="replace"), body

432	433	body = await make_deferred_yieldable(readBody(response))
433	434
434	435	if 200 <= response.code < 300:
435		return json.loads(body.decode("utf-8"))
	436	return json_decoder.decode(body.decode("utf-8"))
436	437	else:
437	438	raise HttpResponseException(
438	439	response.code, response.phrase.decode("ascii", errors="replace"), body

462	463	actual_headers.update(headers)
463	464
464	465	body = await self.get_raw(uri, args, headers=headers)
465		return json.loads(body.decode("utf-8"))
	466	return json_decoder.decode(body.decode("utf-8"))
466	467
467	468	async def put_json(self, uri, json_body, args={}, headers=None):
468	469	""" Puts some json to the given URI.

505	506	body = await make_deferred_yieldable(readBody(response))
506	507
507	508	if 200 <= response.code < 300:
508		return json.loads(body.decode("utf-8"))
	509	return json_decoder.decode(body.decode("utf-8"))
509	510	else:
510	511	raise HttpResponseException(
511	512	response.code, response.phrase.decode("ascii", errors="replace"), body

+1

-1

synapse/http/connectproxyclient.py less more

30	30
31	31
32	32	@implementer(IStreamClientEndpoint)
33		class HTTPConnectProxyEndpoint(object):
	33	class HTTPConnectProxyEndpoint:
34	34	"""An Endpoint implementation which will send a CONNECT request to an http proxy
35	35
36	36	Wraps an existing HostnameEndpoint for the proxy.

+5

-5

synapse/http/federation/matrix_federation_agent.py less more

35	35
36	36
37	37	@implementer(IAgent)
38		class MatrixFederationAgent(object):
	38	class MatrixFederationAgent:
39	39	"""An Agent-like thing which provides a `request` method which correctly
40	40	handles resolving matrix server names when using matrix://. Handles standard
41	41	https URIs as normal.

133	133	and not _is_ip_literal(parsed_uri.hostname)
134	134	and not parsed_uri.port
135	135	):
136		well_known_result = yield self._well_known_resolver.get_well_known(
137		parsed_uri.hostname
	136	well_known_result = yield defer.ensureDeferred(
	137	self._well_known_resolver.get_well_known(parsed_uri.hostname)
138	138	)
139	139	delegated_server = well_known_result.delegated_server
140	140

174	174
175	175
176	176	@implementer(IAgentEndpointFactory)
177		class MatrixHostnameEndpointFactory(object):
	177	class MatrixHostnameEndpointFactory:
178	178	"""Factory for MatrixHostnameEndpoint for parsing to an Agent.
179	179	"""
180	180

197	197
198	198
199	199	@implementer(IStreamClientEndpoint)
200		class MatrixHostnameEndpoint(object):
	200	class MatrixHostnameEndpoint:
201	201	"""An endpoint that resolves matrix:// URLs using Matrix server name
202	202	resolution (i.e. via SRV). Does not check for well-known delegation.
203	203

+2

-2

synapse/http/federation/srv_resolver.py less more

32	32
33	33
34	34	@attr.s(slots=True, frozen=True)
35		class Server(object):
	35	class Server:
36	36	"""
37	37	Our record of an individual server which can be tried to reach a destination.
38	38

95	95	return results
96	96
97	97
98		class SrvResolver(object):
	98	class SrvResolver:
99	99	"""Interface to the dns client to do SRV lookups, with result caching.
100	100
101	101	The default resolver in twisted.names doesn't do any caching (it has a CacheResolver,

+37

-33

synapse/http/federation/well_known_resolver.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		import json
16	15	import logging
17	16	import random
18	17	import time
	18	from typing import Callable, Dict, Optional, Tuple
19	19
20	20	import attr
21	21

23	23	from twisted.web.client import RedirectAgent, readBody
24	24	from twisted.web.http import stringToDatetime
25	25	from twisted.web.http_headers import Headers
	26	from twisted.web.iweb import IResponse
26	27
27	28	from synapse.logging.context import make_deferred_yieldable
28		from synapse.util import Clock
	29	from synapse.util import Clock, json_decoder
29	30	from synapse.util.caches.ttlcache import TTLCache
30	31	from synapse.util.metrics import Measure
31	32

69	70
70	71
71	72	@attr.s(slots=True, frozen=True)
72		class WellKnownLookupResult(object):
	73	class WellKnownLookupResult:
73	74	delegated_server = attr.ib()
74	75
75	76
76		class WellKnownResolver(object):
	77	class WellKnownResolver:
77	78	"""Handles well-known lookups for matrix servers.
78	79	"""
79	80

99	100	self._well_known_agent = RedirectAgent(agent)
100	101	self.user_agent = user_agent
101	102
102		@defer.inlineCallbacks
103		def get_well_known(self, server_name):
	103	async def get_well_known(self, server_name: bytes) -> WellKnownLookupResult:
104	104	"""Attempt to fetch and parse a .well-known file for the given server
105	105
106	106	Args:
107		server_name (bytes): name of the server, from the requested url
	107	server_name: name of the server, from the requested url
108	108
109	109	Returns:
110		Deferred[WellKnownLookupResult]: The result of the lookup
	110	The result of the lookup
111	111	"""
112	112	try:
113	113	prev_result, expiry, ttl = self._well_known_cache.get_with_expiry(

124	124	# requests for the same server in parallel?
125	125	try:
126	126	with Measure(self._clock, "get_well_known"):
127		result, cache_period = yield self._fetch_well_known(server_name)
	127	result, cache_period = await self._fetch_well_known(
	128	server_name
	129	) # type: Tuple[Optional[bytes], float]
128	130
129	131	except _FetchWellKnownFailure as e:
130	132	if prev_result and e.temporary:

153	155
154	156	return WellKnownLookupResult(delegated_server=result)
155	157
156		@defer.inlineCallbacks
157		def _fetch_well_known(self, server_name):
	158	async def _fetch_well_known(self, server_name: bytes) -> Tuple[bytes, float]:
158	159	"""Actually fetch and parse a .well-known, without checking the cache
159	160
160	161	Args:
161		server_name (bytes): name of the server, from the requested url
	162	server_name: name of the server, from the requested url
162	163
163	164	Raises:
164	165	_FetchWellKnownFailure if we fail to lookup a result
165	166
166	167	Returns:
167		Deferred[Tuple[bytes,int]]: The lookup result and cache period.
	168	The lookup result and cache period.
168	169	"""
169	170
170	171	had_valid_well_known = self._had_valid_well_known_cache.get(server_name, False)

172	173	# We do this in two steps to differentiate between possibly transient
173	174	# errors (e.g. can't connect to host, 503 response) and more permenant
174	175	# errors (such as getting a 404 response).
175		response, body = yield self._make_well_known_request(
	176	response, body = await self._make_well_known_request(
176	177	server_name, retry=had_valid_well_known
177	178	)
178	179

180	181	if response.code != 200:
181	182	raise Exception("Non-200 response %s" % (response.code,))
182	183
183		parsed_body = json.loads(body.decode("utf-8"))
	184	parsed_body = json_decoder.decode(body.decode("utf-8"))
184	185	logger.info("Response from .well-known: %s", parsed_body)
185	186
186	187	result = parsed_body["m.server"].encode("ascii")

215	216
216	217	return result, cache_period
217	218
218		@defer.inlineCallbacks
219		def _make_well_known_request(self, server_name, retry):
	219	async def _make_well_known_request(
	220	self, server_name: bytes, retry: bool
	221	) -> Tuple[IResponse, bytes]:
220	222	"""Make the well known request.
221	223
222	224	This will retry the request if requested and it fails (with unable
223	225	to connect or receives a 5xx error).
224	226
225	227	Args:
226		server_name (bytes)
227		retry (bool): Whether to retry the request if it fails.
	228	server_name: name of the server, from the requested url
	229	retry: Whether to retry the request if it fails.
228	230
229	231	Returns:
230		Deferred[tuple[IResponse, bytes]] Returns the response object and
231		body. Response may be a non-200 response.
	232	Returns the response object and body. Response may be a non-200 response.
232	233	"""
233	234	uri = b"https://%s/.well-known/matrix/server" % (server_name,)
234	235	uri_str = uri.decode("ascii")

243	244
244	245	logger.info("Fetching %s", uri_str)
245	246	try:
246		response = yield make_deferred_yieldable(
	247	response = await make_deferred_yieldable(
247	248	self._well_known_agent.request(
248	249	b"GET", uri, headers=Headers(headers)
249	250	)
250	251	)
251		body = yield make_deferred_yieldable(readBody(response))
	252	body = await make_deferred_yieldable(readBody(response))
252	253
253	254	if 500 <= response.code < 600:
254	255	raise Exception("Non-200 response %s" % (response.code,))

265	266	logger.info("Error fetching %s: %s. Retrying", uri_str, e)
266	267
267	268	# Sleep briefly in the hopes that they come back up
268		yield self._clock.sleep(0.5)
269
270
271		def _cache_period_from_headers(headers, time_now=time.time):
	269	await self._clock.sleep(0.5)
	270
	271
	272	def _cache_period_from_headers(
	273	headers: Headers, time_now: Callable[[], float] = time.time
	274	) -> Optional[float]:
272	275	cache_controls = _parse_cache_control(headers)
273	276
274	277	if b"no-store" in cache_controls:
275	278	return 0
276	279
277	280	if b"max-age" in cache_controls:
278		try:
279		max_age = int(cache_controls[b"max-age"])
280		return max_age
281		except ValueError:
282		pass
	281	max_age = cache_controls[b"max-age"]
	282	if max_age:
	283	try:
	284	return int(max_age)
	285	except ValueError:
	286	pass
283	287
284	288	expires = headers.getRawHeaders(b"expires")
285	289	if expires is not None:

295	299	return None
296	300
297	301
298		def _parse_cache_control(headers):
	302	def _parse_cache_control(headers: Headers) -> Dict[bytes, Optional[bytes]]:
299	303	cache_controls = {}
300	304	for hdr in headers.getRawHeaders(b"cache-control", []):
301	305	for directive in hdr.split(b","):

+7

-4

synapse/http/matrixfederationclient.py less more

53	53	start_active_span,
54	54	tags,
55	55	)
	56	from synapse.util import json_decoder
56	57	from synapse.util.async_helpers import timeout_deferred
57	58	from synapse.util.metrics import Measure
58	59

75	76
76	77
77	78	@attr.s(frozen=True)
78		class MatrixFederationRequest(object):
	79	class MatrixFederationRequest:
79	80	method = attr.ib()
80	81	"""HTTP method
81	82	:type: str

163	164	try:
164	165	check_content_type_is_json(response.headers)
165	166
166		d = treq.json_content(response)
	167	# Use the custom JSON decoder (partially re-implements treq.json_content).
	168	d = treq.text_content(response, encoding="utf-8")
	169	d.addCallback(json_decoder.decode)
167	170	d = timeout_deferred(d, timeout=timeout_sec, reactor=reactor)
168	171
169	172	body = await make_deferred_yieldable(d)

202	205	return body
203	206
204	207
205		class MatrixFederationHttpClient(object):
	208	class MatrixFederationHttpClient:
206	209	"""HTTP client used to talk to other homeservers over the federation
207	210	protocol. Send client certificates and signs requests.
208	211

225	228	)
226	229
227	230	@implementer(IReactorPluggableNameResolver)
228		class Reactor(object):
	231	class Reactor:
229	232	def __getattr__(_self, attr):
230	233	if attr == "nameResolver":
231	234	return nameResolver

+1

-1

synapse/http/request_metrics.py less more

144	144	)
145	145
146	146
147		class RequestMetrics(object):
	147	class RequestMetrics:
148	148	def start(self, time_sec, name, method):
149	149	self.start = time_sec
150	150	self.start_context = current_context()

+103

-11

synapse/http/server.py less more

21	21	import urllib
22	22	from http import HTTPStatus
23	23	from io import BytesIO
24		from typing import Any, Callable, Dict, Tuple, Union
	24	from typing import Any, Callable, Dict, Iterator, List, Tuple, Union
25	25
26	26	import jinja2
27		from canonicaljson import encode_canonical_json, encode_pretty_printed_json
28
29		from twisted.internet import defer
	27	from canonicaljson import iterencode_canonical_json, iterencode_pretty_printed_json
	28	from zope.interface import implementer
	29
	30	from twisted.internet import defer, interfaces
30	31	from twisted.python import failure
31	32	from twisted.web import resource
32	33	from twisted.web.server import NOT_DONE_YET, Request

172	173	return preserve_fn(wrapped_async_request_handler)
173	174
174	175
175		class HttpServer(object):
	176	class HttpServer:
176	177	""" Interface for registering callbacks on a HTTP server
177	178	"""
178	179

496	497
497	498	class RootOptionsRedirectResource(OptionsResource, RootRedirect):
498	499	pass
	500
	501
	502	@implementer(interfaces.IPushProducer)
	503	class _ByteProducer:
	504	"""
	505	Iteratively write bytes to the request.
	506	"""
	507
	508	# The minimum number of bytes for each chunk. Note that the last chunk will
	509	# usually be smaller than this.
	510	min_chunk_size = 1024
	511
	512	def __init__(
	513	self, request: Request, iterator: Iterator[bytes],
	514	):
	515	self._request = request
	516	self._iterator = iterator
	517	self._paused = False
	518
	519	# Register the producer and start producing data.
	520	self._request.registerProducer(self, True)
	521	self.resumeProducing()
	522
	523	def _send_data(self, data: List[bytes]) -> None:
	524	"""
	525	Send a list of bytes as a chunk of a response.
	526	"""
	527	if not data:
	528	return
	529	self._request.write(b"".join(data))
	530
	531	def pauseProducing(self) -> None:
	532	self._paused = True
	533
	534	def resumeProducing(self) -> None:
	535	# We've stopped producing in the meantime (note that this might be
	536	# re-entrant after calling write).
	537	if not self._request:
	538	return
	539
	540	self._paused = False
	541
	542	# Write until there's backpressure telling us to stop.
	543	while not self._paused:
	544	# Get the next chunk and write it to the request.
	545	#
	546	# The output of the JSON encoder is buffered and coalesced until
	547	# min_chunk_size is reached. This is because JSON encoders produce
	548	# very small output per iteration and the Request object converts
	549	# each call to write() to a separate chunk. Without this there would
	550	# be an explosion in bytes written (e.g. b"{" becoming "1\r\n{\r\n").
	551	#
	552	# Note that buffer stores a list of bytes (instead of appending to
	553	# bytes) to hopefully avoid many allocations.
	554	buffer = []
	555	buffered_bytes = 0
	556	while buffered_bytes < self.min_chunk_size:
	557	try:
	558	data = next(self._iterator)
	559	buffer.append(data)
	560	buffered_bytes += len(data)
	561	except StopIteration:
	562	# The entire JSON object has been serialized, write any
	563	# remaining data, finalize the producer and the request, and
	564	# clean-up any references.
	565	self._send_data(buffer)
	566	self._request.unregisterProducer()
	567	self._request.finish()
	568	self.stopProducing()
	569	return
	570
	571	self._send_data(buffer)
	572
	573	def stopProducing(self) -> None:
	574	# Clear a circular reference.
	575	self._request = None
	576
	577
	578	def _encode_json_bytes(json_object: Any) -> Iterator[bytes]:
	579	"""
	580	Encode an object into JSON. Returns an iterator of bytes.
	581	"""
	582	for chunk in json_encoder.iterencode(json_object):
	583	yield chunk.encode("utf-8")
499	584
500	585
501	586	def respond_with_json(

532	617	return None
533	618
534	619	if pretty_print:
535		json_bytes = encode_pretty_printed_json(json_object) + b"\n"
	620	encoder = iterencode_pretty_printed_json
536	621	else:
537	622	if canonical_json or synapse.events.USE_FROZEN_DICTS:
538		# canonicaljson already encodes to bytes
539		json_bytes = encode_canonical_json(json_object)
	623	encoder = iterencode_canonical_json
540	624	else:
541		json_bytes = json_encoder.encode(json_object).encode("utf-8")
542
543		return respond_with_json_bytes(request, code, json_bytes, send_cors=send_cors)
	625	encoder = _encode_json_bytes
	626
	627	request.setResponseCode(code)
	628	request.setHeader(b"Content-Type", b"application/json")
	629	request.setHeader(b"Cache-Control", b"no-cache, no-store, must-revalidate")
	630
	631	if send_cors:
	632	set_cors_headers(request)
	633
	634	_ByteProducer(request, encoder(json_object))
	635	return NOT_DONE_YET
544	636
545	637
546	638	def respond_with_json_bytes(

+3

-4

synapse/http/servlet.py less more

16	16
17	17	import logging
18	18
19		from canonicaljson import json
20
21	19	from synapse.api.errors import Codes, SynapseError
	20	from synapse.util import json_decoder
22	21
23	22	logger = logging.getLogger(__name__)
24	23

214	213	return None
215	214
216	215	try:
217		content = json.loads(content_bytes.decode("utf-8"))
	216	content = json_decoder.decode(content_bytes.decode("utf-8"))
218	217	except Exception as e:
219	218	logger.warning("Unable to parse JSON: %s", e)
220	219	raise SynapseError(400, "Content not JSON.", errcode=Codes.NOT_JSON)

256	255	raise SynapseError(400, "Missing params: %r" % absent, Codes.MISSING_PARAM)
257	256
258	257
259		class RestServlet(object):
	258	class RestServlet:
260	259
261	260	""" A Synapse REST Servlet.
262	261

+3

-3

synapse/logging/_structured.py less more

54	54
55	55	@attr.s
56	56	@implementer(ILogObserver)
57		class LogContextObserver(object):
	57	class LogContextObserver:
58	58	"""
59	59	An ILogObserver which adds Synapse-specific log context information.
60	60

168	168
169	169
170	170	@attr.s
171		class DrainConfiguration(object):
	171	class DrainConfiguration:
172	172	name = attr.ib()
173	173	type = attr.ib()
174	174	location = attr.ib()

176	176
177	177
178	178	@attr.s
179		class NetworkJSONTerseOptions(object):
	179	class NetworkJSONTerseOptions:
180	180	maximum_buffer = attr.ib(type=int)
181	181
182	182

+2

-2

synapse/logging/_terse_json.py less more

151	151
152	152	@attr.s
153	153	@implementer(IPushProducer)
154		class LogProducer(object):
	154	class LogProducer:
155	155	"""
156	156	An IPushProducer that writes logs from its buffer to its transport when it
157	157	is resumed.

189	189
190	190	@attr.s
191	191	@implementer(ILogObserver)
192		class TerseJSONToTCPLogObserver(object):
	192	class TerseJSONToTCPLogObserver:
193	193	"""
194	194	An IObserver that writes JSON logs to a TCP target.
195	195

+3

-3

synapse/logging/context.py less more

73	73	get_thread_id = threading.get_ident
74	74
75	75
76		class ContextResourceUsage(object):
	76	class ContextResourceUsage:
77	77	"""Object for tracking the resources used by a log context
78	78
79	79	Attributes:

178	178	LoggingContextOrSentinel = Union["LoggingContext", "_Sentinel"]
179	179
180	180
181		class _Sentinel(object):
	181	class _Sentinel:
182	182	"""Sentinel to represent the root context"""
183	183
184	184	__slots__ = ["previous_context", "finished", "request", "scope", "tag"]

225	225	SENTINEL_CONTEXT = _Sentinel()
226	226
227	227
228		class LoggingContext(object):
	228	class LoggingContext:
229	229	"""Additional context for log formatting. Contexts are scoped within a
230	230	"with" block.
231	231

+7

-5

synapse/logging/opentracing.py less more

171	171	from typing import TYPE_CHECKING, Dict, Optional, Type
172	172
173	173	import attr
174		from canonicaljson import json
175	174
176	175	from twisted.internet import defer
177	176
178	177	from synapse.config import ConfigError
	178	from synapse.util import json_decoder, json_encoder
179	179
180	180	if TYPE_CHECKING:
181	181	from synapse.http.site import SynapseRequest

184	184	# Helper class
185	185
186	186
187		class _DummyTagNames(object):
	187	class _DummyTagNames:
188	188	"""wrapper of opentracings tags. We need to have them if we
189	189	want to reference them without opentracing around. Clearly they
190	190	should never actually show up in a trace. `set_tags` overwrites

498	498	if opentracing is None:
499	499	return _noop_context_manager()
500	500
501		carrier = json.loads(edu_content.get("context", "{}")).get("opentracing", {})
	501	carrier = json_decoder.decode(edu_content.get("context", "{}")).get(
	502	"opentracing", {}
	503	)
502	504	context = opentracing.tracer.extract(opentracing.Format.TEXT_MAP, carrier)
503	505	_references = [
504	506	opentracing.child_of(span_context_from_string(x))

689	691	opentracing.tracer.inject(
690	692	opentracing.tracer.active_span, opentracing.Format.TEXT_MAP, carrier
691	693	)
692		return json.dumps(carrier)
	694	return json_encoder.encode(carrier)
693	695
694	696
695	697	@only_if_tracing

698	700	Returns:
699	701	The active span context decoded from a string.
700	702	"""
701		carrier = json.loads(carrier)
	703	carrier = json_decoder.decode(carrier)
702	704	return opentracing.tracer.extract(opentracing.Format.TEXT_MAP, carrier)
703	705
704	706

+8

-8

synapse/metrics/__init__.py less more

50	50	HAVE_PROC_SELF_STAT = os.path.exists("/proc/self/stat")
51	51
52	52
53		class RegistryProxy(object):
	53	class RegistryProxy:
54	54	@staticmethod
55	55	def collect():
56	56	for metric in REGISTRY.collect():

59	59
60	60
61	61	@attr.s(hash=True)
62		class LaterGauge(object):
	62	class LaterGauge:
63	63
64	64	name = attr.ib(type=str)
65	65	desc = attr.ib(type=str)

99	99	all_gauges[self.name] = self
100	100
101	101
102		class InFlightGauge(object):
	102	class InFlightGauge:
103	103	"""Tracks number of things (e.g. requests, Measure blocks, etc) in flight
104	104	at any given time.
105	105

205	205
206	206
207	207	@attr.s(hash=True)
208		class BucketCollector(object):
	208	class BucketCollector:
209	209	"""
210	210	Like a Histogram, but allows buckets to be point-in-time instead of
211	211	incrementally added to.

268	268	#
269	269
270	270
271		class CPUMetrics(object):
	271	class CPUMetrics:
272	272	def __init__(self):
273	273	ticks_per_sec = 100
274	274	try:

328	328	)
329	329
330	330
331		class GCCounts(object):
	331	class GCCounts:
332	332	def collect(self):
333	333	cm = GaugeMetricFamily("python_gc_counts", "GC object counts", labels=["gen"])
334	334	for n, m in enumerate(gc.get_count()):

346	346	#
347	347
348	348
349		class PyPyGCStats(object):
	349	class PyPyGCStats:
350	350	def collect(self):
351	351
352	352	# @stats is a pretty-printer object with __str__() returning a nice table,

481	481	last_ticked = time.time()
482	482
483	483
484		class ReactorLastSeenMetric(object):
	484	class ReactorLastSeenMetric:
485	485	def collect(self):
486	486	cm = GaugeMetricFamily(
487	487	"python_twisted_reactor_last_seen",

+3

-3

synapse/metrics/background_process_metrics.py less more

104	104	_bg_metrics_lock = threading.Lock()
105	105
106	106
107		class _Collector(object):
	107	class _Collector:
108	108	"""A custom metrics collector for the background process metrics.
109	109
110	110	Ensures that all of the metrics are up-to-date with any in-flight processes

139	139	REGISTRY.register(_Collector())
140	140
141	141
142		class _BackgroundProcess(object):
	142	class _BackgroundProcess:
143	143	def __init__(self, desc, ctx):
144	144	self.desc = desc
145	145	self._context = ctx

174	174	It returns a Deferred which completes when the function completes, but it doesn't
175	175	follow the synapse logcontext rules, which makes it appropriate for passing to
176	176	clock.looping_call and friends (or for firing-and-forgetting in the middle of a
177		normal synapse inlineCallbacks function).
	177	normal synapse async function).
178	178
179	179	Args:
180	180	desc: a description for this background process type

+8

-4

synapse/module_api/__init__.py less more

30	30	logger = logging.getLogger(__name__)
31	31
32	32
33		class ModuleApi(object):
	33	class ModuleApi:
34	34	"""A proxy object that gets passed to various plugin modules so they
35	35	can register new users etc if necessary.
36	36	"""

166	166	external_id: id on that system
167	167	user_id: complete mxid that it is mapped to
168	168	"""
169		return self._store.record_user_external_id(
170		auth_provider_id, remote_user_id, registered_user_id
	169	return defer.ensureDeferred(
	170	self._store.record_user_external_id(
	171	auth_provider_id, remote_user_id, registered_user_id
	172	)
171	173	)
172	174
173	175	def generate_short_term_login_token(

222	224	Returns:
223	225	Deferred[object]: result of func
224	226	"""
225		return self._store.db_pool.runInteraction(desc, func, args, *kwargs)
	227	return defer.ensureDeferred(
	228	self._store.db_pool.runInteraction(desc, func, args, *kwargs)
	229	)
226	230
227	231	def complete_sso_login(
228	232	self, registered_user_id: str, request: SynapseRequest, client_redirect_url: str

+3

-3

synapse/notifier.py less more

67	67	return n
68	68
69	69
70		class _NotificationListener(object):
	70	class _NotificationListener:
71	71	""" This represents a single client connection to the events stream.
72	72	The events stream handler will have yielded to the deferred, so to
73	73	notify the handler it is sufficient to resolve the deferred.

79	79	self.deferred = deferred
80	80
81	81
82		class _NotifierUserStream(object):
	82	class _NotifierUserStream:
83	83	"""This represents a user connected to the event stream.
84	84	It tracks the most recent stream token for that user.
85	85	At a given point a user may have a number of streams listening for

167	167	__bool__ = __nonzero__ # python3
168	168
169	169
170		class Notifier(object):
	170	class Notifier:
171	171	""" This class is responsible for notifying any listeners when there are
172	172	new events available for it.
173	173

+1

-1

synapse/push/action_generator.py less more

21	21	logger = logging.getLogger(__name__)
22	22
23	23
24		class ActionGenerator(object):
	24	class ActionGenerator:
25	25	def __init__(self, hs):
26	26	self.hs = hs
27	27	self.clock = hs.get_clock()

+73

-32

synapse/push/bulk_push_rule_evaluator.py less more

18	18
19	19	from prometheus_client import Counter
20	20
21		from synapse.api.constants import EventTypes, Membership
	21	from synapse.api.constants import EventTypes, Membership, RelationTypes
22	22	from synapse.event_auth import get_user_power_level
	23	from synapse.events import EventBase
	24	from synapse.events.snapshot import EventContext
23	25	from synapse.state import POWER_KEY
24	26	from synapse.util.async_helpers import Linearizer
25	27	from synapse.util.caches import register_cache

50	52	)
51	53
52	54
53		class BulkPushRuleEvaluator(object):
	55	STATE_EVENT_TYPES_TO_MARK_UNREAD = {
	56	EventTypes.Topic,
	57	EventTypes.Name,
	58	EventTypes.RoomAvatar,
	59	EventTypes.Tombstone,
	60	}
	61
	62
	63	def _should_count_as_unread(event: EventBase, context: EventContext) -> bool:
	64	# Exclude rejected and soft-failed events.
	65	if context.rejected or event.internal_metadata.is_soft_failed():
	66	return False
	67
	68	# Exclude notices.
	69	if (
	70	not event.is_state()
	71	and event.type == EventTypes.Message
	72	and event.content.get("msgtype") == "m.notice"
	73	):
	74	return False
	75
	76	# Exclude edits.
	77	relates_to = event.content.get("m.relates_to", {})
	78	if relates_to.get("rel_type") == RelationTypes.REPLACE:
	79	return False
	80
	81	# Mark events that have a non-empty string body as unread.
	82	body = event.content.get("body")
	83	if isinstance(body, str) and body:
	84	return True
	85
	86	# Mark some state events as unread.
	87	if event.is_state() and event.type in STATE_EVENT_TYPES_TO_MARK_UNREAD:
	88	return True
	89
	90	# Mark encrypted events as unread.
	91	if not event.is_state() and event.type == EventTypes.Encrypted:
	92	return True
	93
	94	return False
	95
	96
	97	class BulkPushRuleEvaluator:
54	98	"""Calculates the outcome of push rules for an event for all users in the
55	99	room at once.
56	100	"""

132	176	return pl_event.content if pl_event else {}, sender_level
133	177
134	178	async def action_for_event_by_user(self, event, context) -> None:
135		"""Given an event and context, evaluate the push rules and insert the
136		results into the event_push_actions_staging table.
137		"""
	179	"""Given an event and context, evaluate the push rules, check if the message
	180	should increment the unread count, and insert the results into the
	181	event_push_actions_staging table.
	182	"""
	183	count_as_unread = _should_count_as_unread(event, context)
	184
138	185	rules_by_user = await self._get_rules_for_event(event, context)
139	186	actions_by_user = {}
140	187

170	217	# that user, as they might not be already joined.
171	218	if event.type == EventTypes.Member and event.state_key == uid:
172	219	display_name = event.content.get("displayname", None)
	220
	221	if count_as_unread:
	222	# Add an element for the current user if the event needs to be marked as
	223	# unread, so that add_push_actions_to_staging iterates over it.
	224	# If the event shouldn't be marked as unread but should notify the
	225	# current user, it'll be added to the dict later.
	226	actions_by_user[uid] = []
173	227
174	228	for rule in rules:
175	229	if "enabled" in rule and not rule["enabled"]:

188	242	# Mark in the DB staging area the push actions for users who should be
189	243	# notified for this event. (This will then get handled when we persist
190	244	# the event)
191		await self.store.add_push_actions_to_staging(event.event_id, actions_by_user)
	245	await self.store.add_push_actions_to_staging(
	246	event.event_id, actions_by_user, count_as_unread,
	247	)
192	248
193	249
194	250	def _condition_checker(evaluator, conditions, uid, display_name, cache):

211	267	return True
212	268
213	269
214		class RulesForRoom(object):
	270	class RulesForRoom:
215	271	"""Caches push rules for users in a room.
216	272
217	273	This efficiently handles users joining/leaving the room by not invalidating

368	424	Args:
369	425	ret_rules_by_user (dict): Partiallly filled dict of push rules. Gets
370	426	updated with any new rules.
371		member_event_ids (list): List of event ids for membership events that
372		have happened since the last time we filled rules_by_user
	427	member_event_ids (dict): Dict of user id to event id for membership events
	428	that have happened since the last time we filled rules_by_user
373	429	state_group: The state group we are currently computing push rules
374	430	for. Used when updating the cache.
375	431	"""

389	445	if logger.isEnabledFor(logging.DEBUG):
390	446	logger.debug("Found members %r: %r", self.room_id, members.values())
391	447
392		interested_in_user_ids = {
	448	user_ids = {
393	449	user_id
394	450	for user_id, membership in members.values()
395	451	if membership == Membership.JOIN
396	452	}
397	453
398		logger.debug("Joined: %r", interested_in_user_ids)
399
400		if_users_with_pushers = await self.store.get_if_users_have_pushers(
401		interested_in_user_ids, on_invalidate=self.invalidate_all_cb
402		)
403
404		user_ids = {
405		uid for uid, have_pusher in if_users_with_pushers.items() if have_pusher
406		}
407
408		logger.debug("With pushers: %r", user_ids)
409
410		users_with_receipts = await self.store.get_users_with_read_receipts_in_room(
411		self.room_id, on_invalidate=self.invalidate_all_cb
412		)
413
414		logger.debug("With receipts: %r", users_with_receipts)
415
416		# any users with pushers must be ours: they have pushers
417		for uid in users_with_receipts:
418		if uid in interested_in_user_ids:
419		user_ids.add(uid)
	454	logger.debug("Joined: %r", user_ids)
	455
	456	# Previously we only considered users with pushers or read receipts in that
	457	# room. We can't do this anymore because we use push actions to calculate unread
	458	# counts, which don't rely on the user having pushers or sent a read receipt into
	459	# the room. Therefore we just need to filter for local users here.
	460	user_ids = list(filter(self.is_mine_id, user_ids))
420	461
421	462	rules_by_user = await self.store.bulk_get_push_rules(
422	463	user_ids, on_invalidate=self.invalidate_all_cb

+1

-1

synapse/push/emailpusher.py less more

44	44	INCLUDE_ALL_UNREAD_NOTIFS = False
45	45
46	46
47		class EmailPusher(object):
	47	class EmailPusher:
48	48	"""
49	49	A pusher that sends email notifications about events (approximately)
50	50	when they happen.

+1

-1

synapse/push/httppusher.py less more

48	48	)
49	49
50	50
51		class HttpPusher(object):
	51	class HttpPusher:
52	52	INITIAL_BACKOFF_SEC = 1 # in seconds because that's what Twisted takes
53	53	MAX_BACKOFF_SEC = 60 * 60
54	54

+2

-72

synapse/push/mailer.py less more

15	15	import email.mime.multipart
16	16	import email.utils
17	17	import logging
18		import time
19		import urllib
	18	import urllib.parse
20	19	from email.mime.multipart import MIMEMultipart
21	20	from email.mime.text import MIMEText
22	21	from typing import Iterable, List, TypeVar

92	91	# ALLOWED_SCHEMES = ["http", "https", "ftp", "mailto"]
93	92
94	93
95		class Mailer(object):
	94	class Mailer:
96	95	def __init__(self, hs, app_name, template_html, template_text):
97	96	self.hs = hs
98	97	self.template_html = template_html

639	638	for c in s:
640	639	tot += ord(c)
641	640	return tot
642
643
644		def format_ts_filter(value, format):
645		return time.strftime(format, time.localtime(value / 1000))
646
647
648		def load_jinja2_templates(
649		template_dir,
650		template_filenames,
651		apply_format_ts_filter=False,
652		apply_mxc_to_http_filter=False,
653		public_baseurl=None,
654		):
655		"""Loads and returns one or more jinja2 templates and applies optional filters
656
657		Args:
658		template_dir (str): The directory where templates are stored
659		template_filenames (list[str]): A list of template filenames
660		apply_format_ts_filter (bool): Whether to apply a template filter that formats
661		timestamps
662		apply_mxc_to_http_filter (bool): Whether to apply a template filter that converts
663		mxc urls to http urls
664		public_baseurl (str\|None): The public baseurl of the server. Required for
665		apply_mxc_to_http_filter to be enabled
666
667		Returns:
668		A list of jinja2 templates corresponding to the given list of filenames,
669		with order preserved
670		"""
671		logger.info(
672		"loading email templates %s from '%s'", template_filenames, template_dir
673		)
674		loader = jinja2.FileSystemLoader(template_dir)
675		env = jinja2.Environment(loader=loader)
676
677		if apply_format_ts_filter:
678		env.filters["format_ts"] = format_ts_filter
679
680		if apply_mxc_to_http_filter and public_baseurl:
681		env.filters["mxc_to_http"] = _create_mxc_to_http_filter(public_baseurl)
682
683		templates = []
684		for template_filename in template_filenames:
685		template = env.get_template(template_filename)
686		templates.append(template)
687
688		return templates
689
690
691		def _create_mxc_to_http_filter(public_baseurl):
692		def mxc_to_http_filter(value, width, height, resize_method="crop"):
693		if value[0:6] != "mxc://":
694		return ""
695
696		serverAndMediaId = value[6:]
697		fragment = None
698		if "#" in serverAndMediaId:
699		(serverAndMediaId, fragment) = serverAndMediaId.split("#", 1)
700		fragment = "#" + fragment
701
702		params = {"width": width, "height": height, "method": resize_method}
703		return "%s_matrix/media/v1/thumbnail/%s?%s%s" % (
704		public_baseurl,
705		serverAndMediaId,
706		urllib.parse.urlencode(params),
707		fragment or "",
708		)
709
710		return mxc_to_http_filter

+1

-1

synapse/push/push_rule_evaluator.py less more

104	104	return tweaks
105	105
106	106
107		class PushRuleEvaluatorForEvent(object):
	107	class PushRuleEvaluatorForEvent:
108	108	def __init__(
109	109	self,
110	110	event: EventBase,

+8

-25

synapse/push/pusher.py less more

14	14
15	15	import logging
16	16
	17	from synapse.push.emailpusher import EmailPusher
	18	from synapse.push.mailer import Mailer
	19
17	20	from .httppusher import HttpPusher
18	21
19	22	logger = logging.getLogger(__name__)
20	23
21		# We try importing this if we can (it will fail if we don't
22		# have the optional email dependencies installed). We don't
23		# yet have the config to know if we need the email pusher,
24		# but importing this after daemonizing seems to fail
25		# (even though a simple test of importing from a daemonized
26		# process works fine)
27		try:
28		from synapse.push.emailpusher import EmailPusher
29		from synapse.push.mailer import Mailer, load_jinja2_templates
30		except Exception:
31		pass
32	24
33
34		class PusherFactory(object):
	25	class PusherFactory:
35	26	def __init__(self, hs):
36	27	self.hs = hs
37	28	self.config = hs.config

42	33	if hs.config.email_enable_notifs:
43	34	self.mailers = {} # app_name -> Mailer
44	35
45		self.notif_template_html, self.notif_template_text = load_jinja2_templates(
46		self.config.email_template_dir,
47		[
48		self.config.email_notif_template_html,
49		self.config.email_notif_template_text,
50		],
51		apply_format_ts_filter=True,
52		apply_mxc_to_http_filter=True,
53		public_baseurl=self.config.public_baseurl,
54		)
	36	self._notif_template_html = hs.config.email_notif_template_html
	37	self._notif_template_text = hs.config.email_notif_template_text
55	38
56	39	self.pusher_types["email"] = self._create_email_pusher
57	40

72	55	mailer = Mailer(
73	56	hs=self.hs,
74	57	app_name=app_name,
75		template_html=self.notif_template_html,
76		template_text=self.notif_template_text,
	58	template_html=self._notif_template_html,
	59	template_text=self._notif_template_text,
77	60	)
78	61	self.mailers[app_name] = mailer
79	62	return EmailPusher(self.hs, pusherdict, mailer)

+4

-4

synapse/python_dependencies.py less more

42	42	"jsonschema>=2.5.1",
43	43	"frozendict>=1",
44	44	"unpaddedbase64>=1.1.0",
45		"canonicaljson>=1.2.0",
	45	"canonicaljson>=1.3.0",
46	46	# we use the type definitions added in signedjson 1.1.
47	47	"signedjson>=1.1.0",
48	48	"pynacl>=1.2.1",

65	65	"msgpack>=0.5.2",
66	66	"phonenumbers>=8.2.0",
67	67	"prometheus_client>=0.0.18,<0.9.0",
68		# we use attr.validators.deep_iterable, which arrived in 19.1.0
	68	# we use attr.validators.deep_iterable, which arrived in 19.1.0 (Note:
	69	# Fedora 31 only has 19.1, so if we want to upgrade we should wait until 33
	70	# is out in November.)
69	71	"attrs>=19.1.0",
70	72	"netaddr>=0.7.18",
71	73	"Jinja2>=2.9",

77	79	"matrix-synapse-ldap3": ["matrix-synapse-ldap3>=0.1"],
78	80	# we use execute_batch, which arrived in psycopg 2.7.
79	81	"postgres": ["psycopg2>=2.7"],
80		# ConsentResource uses select_autoescape, which arrived in jinja 2.9
81		"resources.consent": ["Jinja2>=2.9"],
82	82	# ACME support is required to provision TLS certificates from authorities
83	83	# that use the protocol, such as Let's Encrypt.
84	84	"acme": [

+1

-1

synapse/replication/http/_base.py less more

32	32	logger = logging.getLogger(__name__)
33	33
34	34
35		class ReplicationEndpoint(object):
	35	class ReplicationEndpoint:
36	36	"""Helper base class for defining new replication HTTP endpoints.
37	37
38	38	This creates an endpoint under `/_synapse/replication/:NAME/:PATH_ARGS..`

+4

-0

synapse/replication/http/register.py less more

43	43	admin,
44	44	user_type,
45	45	address,
	46	shadow_banned,
46	47	):
47	48	"""
48	49	Args:

59	60	user_type (str\|None): type of user. One of the values from
60	61	api.constants.UserTypes, or None for a normal user.
61	62	address (str\|None): the IP address used to perform the regitration.
	63	shadow_banned (bool): Whether to shadow-ban the user
62	64	"""
63	65	return {
64	66	"password_hash": password_hash,

69	71	"admin": admin,
70	72	"user_type": user_type,
71	73	"address": address,
	74	"shadow_banned": shadow_banned,
72	75	}
73	76
74	77	async def _handle_request(self, request, user_id):

86	89	admin=content["admin"],
87	90	user_type=content["user_type"],
88	91	address=content["address"],
	92	shadow_banned=content["shadow_banned"],
89	93	)
90	94
91	95	return 200, {}

+11

-3

synapse/replication/slave/storage/_slaved_id_tracker.py less more

15	15	from synapse.storage.util.id_generators import _load_current_id
16	16
17	17
18		class SlavedIdTracker(object):
	18	class SlavedIdTracker:
19	19	def __init__(self, db_conn, table, column, extra_tables=[], step=1):
20	20	self.step = step
21	21	self._current = _load_current_id(db_conn, table, column, step)
22	22	for table, column in extra_tables:
23		self.advance(_load_current_id(db_conn, table, column))
	23	self.advance(None, _load_current_id(db_conn, table, column))
24	24
25		def advance(self, new_id):
	25	def advance(self, instance_name, new_id):
26	26	self._current = (max if self.step > 0 else min)(self._current, new_id)
27	27
28	28	def get_current_token(self):

32	32	int
33	33	"""
34	34	return self._current
	35
	36	def get_current_token_for_writer(self, instance_name: str) -> int:
	37	"""Returns the position of the given writer.
	38
	39	For streams with single writers this is equivalent to
	40	`get_current_token`.
	41	"""
	42	return self.get_current_token()

+2

-2

synapse/replication/slave/storage/account_data.py less more

40	40
41	41	def process_replication_rows(self, stream_name, instance_name, token, rows):
42	42	if stream_name == TagAccountDataStream.NAME:
43		self._account_data_id_gen.advance(token)
	43	self._account_data_id_gen.advance(instance_name, token)
44	44	for row in rows:
45	45	self.get_tags_for_user.invalidate((row.user_id,))
46	46	self._account_data_stream_cache.entity_has_changed(row.user_id, token)
47	47	elif stream_name == AccountDataStream.NAME:
48		self._account_data_id_gen.advance(token)
	48	self._account_data_id_gen.advance(instance_name, token)
49	49	for row in rows:
50	50	if not row.room_id:
51	51	self.get_global_account_data_by_type_for_user.invalidate(

+1

-1

synapse/replication/slave/storage/deviceinbox.py less more

45	45
46	46	def process_replication_rows(self, stream_name, instance_name, token, rows):
47	47	if stream_name == ToDeviceStream.NAME:
48		self._device_inbox_id_gen.advance(token)
	48	self._device_inbox_id_gen.advance(instance_name, token)
49	49	for row in rows:
50	50	if row.entity.startswith("@"):
51	51	self._device_inbox_stream_cache.entity_has_changed(

+5

-2

synapse/replication/slave/storage/devices.py less more

47	47	"DeviceListFederationStreamChangeCache", device_list_max
48	48	)
49	49
	50	def get_device_stream_token(self) -> int:
	51	return self._device_list_id_gen.get_current_token()
	52
50	53	def process_replication_rows(self, stream_name, instance_name, token, rows):
51	54	if stream_name == DeviceListsStream.NAME:
52		self._device_list_id_gen.advance(token)
	55	self._device_list_id_gen.advance(instance_name, token)
53	56	self._invalidate_caches_for_devices(token, rows)
54	57	elif stream_name == UserSignatureStream.NAME:
55		self._device_list_id_gen.advance(token)
	58	self._device_list_id_gen.advance(instance_name, token)
56	59	for row in rows:
57	60	self._user_signature_stream_cache.entity_has_changed(row.user_id, token)
58	61	return super().process_replication_rows(stream_name, instance_name, token, rows)

+1

-1

synapse/replication/slave/storage/groups.py less more

39	39
40	40	def process_replication_rows(self, stream_name, instance_name, token, rows):
41	41	if stream_name == GroupServerStream.NAME:
42		self._group_updates_id_gen.advance(token)
	42	self._group_updates_id_gen.advance(instance_name, token)
43	43	for row in rows:
44	44	self._group_updates_stream_cache.entity_has_changed(row.user_id, token)
45	45

+1

-1

synapse/replication/slave/storage/presence.py less more

43	43
44	44	def process_replication_rows(self, stream_name, instance_name, token, rows):
45	45	if stream_name == PresenceStream.NAME:
46		self._presence_id_gen.advance(token)
	46	self._presence_id_gen.advance(instance_name, token)
47	47	for row in rows:
48	48	self.presence_stream_cache.entity_has_changed(row.user_id, token)
49	49	self._get_presence_for_user.invalidate((row.user_id,))

+5

-7

synapse/replication/slave/storage/push_rule.py less more

13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
15	15
	16	from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
16	17	from synapse.replication.tcp.streams import PushRulesStream
17	18	from synapse.storage.databases.main.push_rule import PushRulesWorkerStore
18	19

20	21
21	22
22	23	class SlavedPushRuleStore(SlavedEventStore, PushRulesWorkerStore):
23		def get_push_rules_stream_token(self):
24		return (
25		self._push_rules_stream_id_gen.get_current_token(),
26		self._stream_id_gen.get_current_token(),
27		)
28
29	24	def get_max_push_rules_stream_id(self):
30	25	return self._push_rules_stream_id_gen.get_current_token()
31	26
32	27	def process_replication_rows(self, stream_name, instance_name, token, rows):
	28	# We assert this for the benefit of mypy
	29	assert isinstance(self._push_rules_stream_id_gen, SlavedIdTracker)
	30
33	31	if stream_name == PushRulesStream.NAME:
34		self._push_rules_stream_id_gen.advance(token)
	32	self._push_rules_stream_id_gen.advance(instance_name, token)
35	33	for row in rows:
36	34	self.get_push_rules_for_user.invalidate((row.user_id,))
37	35	self.get_push_rules_enabled_for_user.invalidate((row.user_id,))

+1

-1

synapse/replication/slave/storage/pushers.py less more

33	33
34	34	def process_replication_rows(self, stream_name, instance_name, token, rows):
35	35	if stream_name == PushersStream.NAME:
36		self._pushers_id_gen.advance(token)
	36	self._pushers_id_gen.advance(instance_name, token)
37	37	return super().process_replication_rows(stream_name, instance_name, token, rows)

+1

-1

synapse/replication/slave/storage/receipts.py less more

45	45
46	46	def process_replication_rows(self, stream_name, instance_name, token, rows):
47	47	if stream_name == ReceiptsStream.NAME:
48		self._receipts_id_gen.advance(token)
	48	self._receipts_id_gen.advance(instance_name, token)
49	49	for row in rows:
50	50	self.invalidate_caches_for_receipt(
51	51	row.room_id, row.receipt_type, row.user_id

+1

-1

synapse/replication/slave/storage/room.py less more

32	32
33	33	def process_replication_rows(self, stream_name, instance_name, token, rows):
34	34	if stream_name == PublicRoomsStream.NAME:
35		self._public_room_id_gen.advance(token)
	35	self._public_room_id_gen.advance(instance_name, token)
36	36
37	37	return super().process_replication_rows(stream_name, instance_name, token, rows)

+2

-4

synapse/replication/tcp/client.py less more

13	13	# limitations under the License.
14	14	"""A replication client for use by synapse workers.
15	15	"""
16		import heapq
17	16	import logging
18	17	from typing import TYPE_CHECKING, Dict, List, Tuple
19	18

218	217
219	218	waiting_list = self._streams_to_waiters.setdefault(stream_name, [])
220	219
221		# We insert into the list using heapq as it is more efficient than
222		# pushing then resorting each time.
223		heapq.heappush(waiting_list, (position, deferred))
	220	waiting_list.append((position, deferred))
	221	waiting_list.sort(key=lambda t: t[0])
224	222
225	223	# We measure here to get in flight counts and average waiting time.
226	224	with Measure(self._clock, "repl.wait_for_stream_position"):

+5

-7

synapse/replication/tcp/commands.py less more

20	20	import logging
21	21	from typing import Tuple, Type
22	22
23		from canonicaljson import json
24
25		from synapse.util import json_encoder as _json_encoder
	23	from synapse.util import json_decoder, json_encoder
26	24
27	25	logger = logging.getLogger(__name__)
28	26

124	122	stream_name,
125	123	instance_name,
126	124	None if token == "batch" else int(token),
127		json.loads(row_json),
	125	json_decoder.decode(row_json),
128	126	)
129	127
130	128	def to_line(self):

133	131	self.stream_name,
134	132	self.instance_name,
135	133	str(self.token) if self.token is not None else "batch",
136		_json_encoder.encode(self.row),
	134	json_encoder.encode(self.row),
137	135	)
138	136	)
139	137

358	356	def from_line(cls, line):
359	357	user_id, jsn = line.split(" ", 1)
360	358
361		access_token, ip, user_agent, device_id, last_seen = json.loads(jsn)
	359	access_token, ip, user_agent, device_id, last_seen = json_decoder.decode(jsn)
362	360
363	361	return cls(user_id, access_token, ip, user_agent, device_id, last_seen)
364	362

366	364	return (
367	365	self.user_id
368	366	+ " "
369		+ _json_encoder.encode(
	367	+ json_encoder.encode(
370	368	(
371	369	self.access_token,
372	370	self.ip,

+1

-1

synapse/replication/tcp/protocol.py less more

112	112	PING_TIMEOUT_MS = PING_TIME * PING_TIMEOUT_MULTIPLIER
113	113
114	114
115		class ConnectionStates(object):
	115	class ConnectionStates:
116	116	CONNECTING = "connecting"
117	117	ESTABLISHED = "established"
118	118	PAUSED = "paused"

+1

-1

synapse/replication/tcp/resource.py less more

57	57	)
58	58
59	59
60		class ReplicationStreamer(object):
	60	class ReplicationStreamer:
61	61	"""Handles replication connections.
62	62
63	63	This needs to be poked when new replication data may be available. When new

+3

-3

synapse/replication/tcp/streams/_base.py less more

78	78	UpdateFunction = Callable[[str, Token, Token, int], Awaitable[StreamUpdateResult]]
79	79
80	80
81		class Stream(object):
	81	class Stream:
82	82	"""Base class for the streams.
83	83
84	84	Provides a `get_updates()` function that returns new updates since the last

351	351	)
352	352
353	353	def _current_token(self, instance_name: str) -> int:
354		push_rules_token, _ = self.store.get_push_rules_stream_token()
	354	push_rules_token = self.store.get_max_push_rules_stream_id()
355	355	return push_rules_token
356	356
357	357

404	404	store = hs.get_datastore()
405	405	super().__init__(
406	406	hs.get_instance_name(),
407		store.get_cache_stream_token,
	407	store.get_cache_stream_token_for_writer,
408	408	store.get_all_updated_caches,
409	409	)
410	410

+2

-2

synapse/replication/tcp/streams/events.py less more

48	48
49	49
50	50	@attr.s(slots=True, frozen=True)
51		class EventsStreamRow(object):
	51	class EventsStreamRow:
52	52	"""A parsed row from the events replication stream"""
53	53
54	54	type = attr.ib() # str: the TypeId of one of the *EventsStreamRows
55	55	data = attr.ib() # BaseEventsStreamRow
56	56
57	57
58		class BaseEventsStreamRow(object):
	58	class BaseEventsStreamRow:
59	59	"""Base class for rows to be sent in the events stream.
60	60
61	61	Specifies how to identify, serialize and deserialize the different types.

+4

-0

synapse/rest/__init__.py less more

49	49	room_keys,
50	50	room_upgrade_rest_servlet,
51	51	sendtodevice,
	52	shared_rooms,
52	53	sync,
53	54	tags,
54	55	thirdparty,

124	125	synapse.rest.admin.register_servlets_for_client_rest_resource(
125	126	hs, client_resource
126	127	)
	128
	129	# unstable
	130	shared_rooms.register_servlets(hs, client_resource)

+3

-0

synapse/rest/admin/rooms.py less more

315	315	join_rules_event = room_state.get((EventTypes.JoinRules, ""))
316	316	if join_rules_event:
317	317	if not (join_rules_event.content.get("join_rule") == JoinRules.PUBLIC):
	318	# update_membership with an action of "invite" can raise a
	319	# ShadowBanError. This is not handled since it is assumed that
	320	# an admin isn't going to call this API with a shadow-banned user.
318	321	await self.room_member_handler.update_membership(
319	322	requester=requester,
320	323	target=fake_requester.user,

+3

-1

synapse/rest/admin/users.py less more

72	72	The parameters `from` and `limit` are required only for pagination.
73	73	By default, a `limit` of 100 is used.
74	74	The parameter `user_id` can be used to filter by user id.
	75	The parameter `name` can be used to filter by user id or display name.
75	76	The parameter `guests` can be used to exclude guest users.
76	77	The parameter `deactivated` can be used to include deactivated users.
77	78	"""

88	89	start = parse_integer(request, "from", default=0)
89	90	limit = parse_integer(request, "limit", default=100)
90	91	user_id = parse_string(request, "user_id", default=None)
	92	name = parse_string(request, "name", default=None)
91	93	guests = parse_boolean(request, "guests", default=True)
92	94	deactivated = parse_boolean(request, "deactivated", default=False)
93	95
94	96	users, total = await self.store.get_users_paginate(
95		start, limit, user_id, guests, deactivated
	97	start, limit, user_id, name, guests, deactivated
96	98	)
97	99	ret = {"users": users, "total": total}
98	100	if len(users) >= limit:

+1

-1

synapse/rest/client/transactions.py less more

24	24	CLEANUP_PERIOD_MS = 1000 * 60 * 30 # 30 mins
25	25
26	26
27		class HttpTransactionCache(object):
	27	class HttpTransactionCache:
28	28	def __init__(self, hs):
29	29	self.hs = hs
30	30	self.auth = self.hs.get_auth()

+6

-54

synapse/rest/client/v1/login.py less more

17	17
18	18	from synapse.api.errors import Codes, LoginError, SynapseError
19	19	from synapse.api.ratelimiting import Ratelimiter
	20	from synapse.handlers.auth import (
	21	convert_client_dict_legacy_fields_to_identifier,
	22	login_id_phone_to_thirdparty,
	23	)
20	24	from synapse.http.server import finish_request
21	25	from synapse.http.servlet import (
22	26	RestServlet,

27	31	from synapse.rest.client.v2_alpha._base import client_patterns
28	32	from synapse.rest.well_known import WellKnownBuilder
29	33	from synapse.types import JsonDict, UserID
30		from synapse.util.msisdn import phone_number_to_msisdn
31	34	from synapse.util.threepids import canonicalise_email
32	35
33	36	logger = logging.getLogger(__name__)
34
35
36		def login_submission_legacy_convert(submission):
37		"""
38		If the input login submission is an old style object
39		(ie. with top-level user / medium / address) convert it
40		to a typed object.
41		"""
42		if "user" in submission:
43		submission["identifier"] = {"type": "m.id.user", "user": submission["user"]}
44		del submission["user"]
45
46		if "medium" in submission and "address" in submission:
47		submission["identifier"] = {
48		"type": "m.id.thirdparty",
49		"medium": submission["medium"],
50		"address": submission["address"],
51		}
52		del submission["medium"]
53		del submission["address"]
54
55
56		def login_id_thirdparty_from_phone(identifier):
57		"""
58		Convert a phone login identifier type to a generic threepid identifier
59		Args:
60		identifier(dict): Login identifier dict of type 'm.id.phone'
61
62		Returns: Login identifier dict of type 'm.id.threepid'
63		"""
64		if "country" not in identifier or (
65		# The specification requires a "phone" field, while Synapse used to require a "number"
66		# field. Accept both for backwards compatibility.
67		"phone" not in identifier
68		and "number" not in identifier
69		):
70		raise SynapseError(400, "Invalid phone-type identifier")
71
72		# Accept both "phone" and "number" as valid keys in m.id.phone
73		phone_number = identifier.get("phone", identifier["number"])
74
75		msisdn = phone_number_to_msisdn(identifier["country"], phone_number)
76
77		return {"type": "m.id.thirdparty", "medium": "msisdn", "address": msisdn}
78	37
79	38
80	39	class LoginRestServlet(RestServlet):

193	152	login_submission.get("address"),
194	153	login_submission.get("user"),
195	154	)
196		login_submission_legacy_convert(login_submission)
197
198		if "identifier" not in login_submission:
199		raise SynapseError(400, "Missing param: identifier")
200
201		identifier = login_submission["identifier"]
202		if "type" not in identifier:
203		raise SynapseError(400, "Login identifier has no type")
	155	identifier = convert_client_dict_legacy_fields_to_identifier(login_submission)
204	156
205	157	# convert phone type identifiers to generic threepids
206	158	if identifier["type"] == "m.id.phone":
207		identifier = login_id_thirdparty_from_phone(identifier)
	159	identifier = login_id_phone_to_thirdparty(identifier)
208	160
209	161	# convert threepid identifiers to user IDs
210	162	if identifier["type"] == "m.id.thirdparty":

+6

-5

synapse/rest/client/v1/push_rule.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14
15	14
16	15	from synapse.api.errors import (
17	16	NotFoundError,

159	158	return 200, {}
160	159
161	160	def notify_user(self, user_id):
162		stream_id, _ = self.store.get_push_rules_stream_token()
	161	stream_id = self.store.get_max_push_rules_stream_id()
163	162	self.notifier.on_new_event("push_rules_key", stream_id, users=[user_id])
164	163
165		def set_rule_attr(self, user_id, spec, val):
	164	async def set_rule_attr(self, user_id, spec, val):
166	165	if spec["attr"] == "enabled":
167	166	if isinstance(val, dict) and "enabled" in val:
168	167	val = val["enabled"]

172	171	# bools directly, so let's not break them.
173	172	raise SynapseError(400, "Value for 'enabled' must be boolean")
174	173	namespaced_rule_id = _namespaced_rule_id_from_spec(spec)
175		return self.store.set_push_rule_enabled(user_id, namespaced_rule_id, val)
	174	return await self.store.set_push_rule_enabled(
	175	user_id, namespaced_rule_id, val
	176	)
176	177	elif spec["attr"] == "actions":
177	178	actions = val.get("actions")
178	179	_check_actions(actions)

187	188
188	189	if namespaced_rule_id not in rule_ids:
189	190	raise SynapseError(404, "Unknown rule %r" % (namespaced_rule_id,))
190		return self.store.set_push_rule_actions(
	191	return await self.store.set_push_rule_actions(
191	192	user_id, namespaced_rule_id, actions, is_default_rule
192	193	)
193	194	else:

+128

-95

synapse/rest/client/v1/room.py less more

20	20	from typing import List, Optional
21	21	from urllib import parse as urlparse
22	22
23		from canonicaljson import json
24
25	23	from synapse.api.constants import EventTypes, Membership
26	24	from synapse.api.errors import (
27	25	AuthError,
28	26	Codes,
29	27	HttpResponseException,
30	28	InvalidClientCredentialsError,
	29	ShadowBanError,
31	30	SynapseError,
32	31	)
33	32	from synapse.api.filtering import Filter

45	44	from synapse.storage.state import StateFilter
46	45	from synapse.streams.config import PaginationConfig
47	46	from synapse.types import RoomAlias, RoomID, StreamToken, ThirdPartyInstanceID, UserID
	47	from synapse.util import json_decoder
	48	from synapse.util.stringutils import random_string
48	49
49	50	MYPY = False
50	51	if MYPY:

169	170	room_id=room_id,
170	171	event_type=event_type,
171	172	state_key=state_key,
172		is_guest=requester.is_guest,
173	173	)
174	174
175	175	if not data:

199	199	if state_key is not None:
200	200	event_dict["state_key"] = state_key
201	201
202		if event_type == EventTypes.Member:
203		membership = content.get("membership", None)
204		event_id, _ = await self.room_member_handler.update_membership(
205		requester,
206		target=UserID.from_string(state_key),
207		room_id=room_id,
208		action=membership,
209		content=content,
210		)
211		else:
	202	try:
	203	if event_type == EventTypes.Member:
	204	membership = content.get("membership", None)
	205	event_id, _ = await self.room_member_handler.update_membership(
	206	requester,
	207	target=UserID.from_string(state_key),
	208	room_id=room_id,
	209	action=membership,
	210	content=content,
	211	)
	212	else:
	213	(
	214	event,
	215	_,
	216	) = await self.event_creation_handler.create_and_send_nonmember_event(
	217	requester, event_dict, txn_id=txn_id
	218	)
	219	event_id = event.event_id
	220	except ShadowBanError:
	221	event_id = "$" + random_string(43)
	222
	223	set_tag("event_id", event_id)
	224	ret = {"event_id": event_id}
	225	return 200, ret
	226
	227
	228	# TODO: Needs unit testing for generic events + feedback
	229	class RoomSendEventRestServlet(TransactionRestServlet):
	230	def __init__(self, hs):
	231	super(RoomSendEventRestServlet, self).__init__(hs)
	232	self.event_creation_handler = hs.get_event_creation_handler()
	233	self.auth = hs.get_auth()
	234
	235	def register(self, http_server):
	236	# /rooms/$roomid/send/$event_type[/$txn_id]
	237	PATTERNS = "/rooms/(?P<room_id>[^/])/send/(?P<event_type>[^/])"
	238	register_txn_path(self, PATTERNS, http_server, with_get=True)
	239
	240	async def on_POST(self, request, room_id, event_type, txn_id=None):
	241	requester = await self.auth.get_user_by_req(request, allow_guest=True)
	242	content = parse_json_object_from_request(request)
	243
	244	event_dict = {
	245	"type": event_type,
	246	"content": content,
	247	"room_id": room_id,
	248	"sender": requester.user.to_string(),
	249	}
	250
	251	if b"ts" in request.args and requester.app_service:
	252	event_dict["origin_server_ts"] = parse_integer(request, "ts", 0)
	253
	254	try:
212	255	(
213	256	event,
214	257	_,

216	259	requester, event_dict, txn_id=txn_id
217	260	)
218	261	event_id = event.event_id
	262	except ShadowBanError:
	263	event_id = "$" + random_string(43)
219	264
220	265	set_tag("event_id", event_id)
221		ret = {"event_id": event_id}
222		return 200, ret
223
224
225		# TODO: Needs unit testing for generic events + feedback
226		class RoomSendEventRestServlet(TransactionRestServlet):
227		def __init__(self, hs):
228		super(RoomSendEventRestServlet, self).__init__(hs)
229		self.event_creation_handler = hs.get_event_creation_handler()
230		self.auth = hs.get_auth()
231
232		def register(self, http_server):
233		# /rooms/$roomid/send/$event_type[/$txn_id]
234		PATTERNS = "/rooms/(?P<room_id>[^/])/send/(?P<event_type>[^/])"
235		register_txn_path(self, PATTERNS, http_server, with_get=True)
236
237		async def on_POST(self, request, room_id, event_type, txn_id=None):
238		requester = await self.auth.get_user_by_req(request, allow_guest=True)
239		content = parse_json_object_from_request(request)
240
241		event_dict = {
242		"type": event_type,
243		"content": content,
244		"room_id": room_id,
245		"sender": requester.user.to_string(),
246		}
247
248		if b"ts" in request.args and requester.app_service:
249		event_dict["origin_server_ts"] = parse_integer(request, "ts", 0)
250
251		event, _ = await self.event_creation_handler.create_and_send_nonmember_event(
252		requester, event_dict, txn_id=txn_id
253		)
254
255		set_tag("event_id", event.event_id)
256		return 200, {"event_id": event.event_id}
	266	return 200, {"event_id": event_id}
257	267
258	268	def on_GET(self, request, room_id, event_type, txn_id):
259	269	return 200, "Not implemented"

518	528	filter_str = parse_string(request, b"filter", encoding="utf-8")
519	529	if filter_str:
520	530	filter_json = urlparse.unquote(filter_str)
521		event_filter = Filter(json.loads(filter_json)) # type: Optional[Filter]
	531	event_filter = Filter(
	532	json_decoder.decode(filter_json)
	533	) # type: Optional[Filter]
522	534	if (
523	535	event_filter
524	536	and event_filter.filter_json.get("event_format", "client")

630	642	filter_str = parse_string(request, b"filter", encoding="utf-8")
631	643	if filter_str:
632	644	filter_json = urlparse.unquote(filter_str)
633		event_filter = Filter(json.loads(filter_json)) # type: Optional[Filter]
	645	event_filter = Filter(
	646	json_decoder.decode(filter_json)
	647	) # type: Optional[Filter]
634	648	else:
635	649	event_filter = None
636	650

715	729	content = {}
716	730
717	731	if membership_action == "invite" and self._has_3pid_invite_keys(content):
718		await self.room_member_handler.do_3pid_invite(
719		room_id,
720		requester.user,
721		content["medium"],
722		content["address"],
723		content["id_server"],
724		requester,
725		txn_id,
726		content.get("id_access_token"),
727		)
	732	try:
	733	await self.room_member_handler.do_3pid_invite(
	734	room_id,
	735	requester.user,
	736	content["medium"],
	737	content["address"],
	738	content["id_server"],
	739	requester,
	740	txn_id,
	741	content.get("id_access_token"),
	742	)
	743	except ShadowBanError:
	744	# Pretend the request succeeded.
	745	pass
728	746	return 200, {}
729	747
730	748	target = requester.user

736	754	if "reason" in content:
737	755	event_content = {"reason": content["reason"]}
738	756
739		await self.room_member_handler.update_membership(
740		requester=requester,
741		target=target,
742		room_id=room_id,
743		action=membership_action,
744		txn_id=txn_id,
745		third_party_signed=content.get("third_party_signed", None),
746		content=event_content,
747		)
	757	try:
	758	await self.room_member_handler.update_membership(
	759	requester=requester,
	760	target=target,
	761	room_id=room_id,
	762	action=membership_action,
	763	txn_id=txn_id,
	764	third_party_signed=content.get("third_party_signed", None),
	765	content=event_content,
	766	)
	767	except ShadowBanError:
	768	# Pretend the request succeeded.
	769	pass
748	770
749	771	return_value = {}
750	772

782	804	requester = await self.auth.get_user_by_req(request)
783	805	content = parse_json_object_from_request(request)
784	806
785		event, _ = await self.event_creation_handler.create_and_send_nonmember_event(
786		requester,
787		{
788		"type": EventTypes.Redaction,
789		"content": content,
790		"room_id": room_id,
791		"sender": requester.user.to_string(),
792		"redacts": event_id,
793		},
794		txn_id=txn_id,
795		)
796
797		set_tag("event_id", event.event_id)
798		return 200, {"event_id": event.event_id}
	807	try:
	808	(
	809	event,
	810	_,
	811	) = await self.event_creation_handler.create_and_send_nonmember_event(
	812	requester,
	813	{
	814	"type": EventTypes.Redaction,
	815	"content": content,
	816	"room_id": room_id,
	817	"sender": requester.user.to_string(),
	818	"redacts": event_id,
	819	},
	820	txn_id=txn_id,
	821	)
	822	event_id = event.event_id
	823	except ShadowBanError:
	824	event_id = "$" + random_string(43)
	825
	826	set_tag("event_id", event_id)
	827	return 200, {"event_id": event_id}
799	828
800	829	def on_PUT(self, request, room_id, event_id, txn_id):
801	830	set_tag("txn_id", txn_id)

838	867	# Limit timeout to stop people from setting silly typing timeouts.
839	868	timeout = min(content.get("timeout", 30000), 120000)
840	869
841		if content["typing"]:
842		await self.typing_handler.started_typing(
843		target_user=target_user,
844		auth_user=requester.user,
845		room_id=room_id,
846		timeout=timeout,
847		)
848		else:
849		await self.typing_handler.stopped_typing(
850		target_user=target_user, auth_user=requester.user, room_id=room_id
851		)
	870	try:
	871	if content["typing"]:
	872	await self.typing_handler.started_typing(
	873	target_user=target_user,
	874	requester=requester,
	875	room_id=room_id,
	876	timeout=timeout,
	877	)
	878	else:
	879	await self.typing_handler.stopped_typing(
	880	target_user=target_user, requester=requester, room_id=room_id
	881	)
	882	except ShadowBanError:
	883	# Pretend this worked without error.
	884	pass
852	885
853	886	return 200, {}
854	887

+22

-32

synapse/rest/client/v2_alpha/account.py less more

14	14	# See the License for the specific language governing permissions and
15	15	# limitations under the License.
16	16	import logging
	17	import random
17	18	from http import HTTPStatus
18	19
19	20	from synapse.api.constants import LoginType

31	32	parse_json_object_from_request,
32	33	parse_string,
33	34	)
34		from synapse.push.mailer import Mailer, load_jinja2_templates
	35	from synapse.push.mailer import Mailer
35	36	from synapse.util.msisdn import phone_number_to_msisdn
36	37	from synapse.util.stringutils import assert_valid_client_secret, random_string
37	38	from synapse.util.threepids import canonicalise_email, check_3pid_allowed

52	53	self.identity_handler = hs.get_handlers().identity_handler
53	54
54	55	if self.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
55		template_html, template_text = load_jinja2_templates(
56		self.config.email_template_dir,
57		[
58		self.config.email_password_reset_template_html,
59		self.config.email_password_reset_template_text,
60		],
61		apply_format_ts_filter=True,
62		apply_mxc_to_http_filter=True,
63		public_baseurl=self.config.public_baseurl,
64		)
65	56	self.mailer = Mailer(
66	57	hs=self.hs,
67	58	app_name=self.config.email_app_name,
68		template_html=template_html,
69		template_text=template_text,
	59	template_html=self.config.email_password_reset_template_html,
	60	template_text=self.config.email_password_reset_template_text,
70	61	)
71	62
72	63	async def on_POST(self, request):

118	109	if self.config.request_token_inhibit_3pid_errors:
119	110	# Make the client think the operation succeeded. See the rationale in the
120	111	# comments for request_token_inhibit_3pid_errors.
	112	# Also wait for some random amount of time between 100ms and 1s to make it
	113	# look like we did something.
	114	await self.hs.clock.sleep(random.randint(1, 10) / 10)
121	115	return 200, {"sid": random_string(16)}
122	116
123	117	raise SynapseError(400, "Email not found", Codes.THREEPID_NOT_FOUND)

168	162	self.clock = hs.get_clock()
169	163	self.store = hs.get_datastore()
170	164	if self.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
171		(self.failure_email_template,) = load_jinja2_templates(
172		self.config.email_template_dir,
173		[self.config.email_password_reset_template_failure_html],
	165	self._failure_email_template = (
	166	self.config.email_password_reset_template_failure_html
174	167	)
175	168
176	169	async def on_GET(self, request, medium):

213	206	return None
214	207
215	208	# Otherwise show the success template
216		html = self.config.email_password_reset_template_success_html
	209	html = self.config.email_password_reset_template_success_html_content
217	210	status_code = 200
218	211	except ThreepidValidationError as e:
219	212	status_code = e.code
220	213
221	214	# Show a failure page with a reason
222	215	template_vars = {"failure_reason": e.msg}
223		html = self.failure_email_template.render(**template_vars)
	216	html = self._failure_email_template.render(**template_vars)
224	217
225	218	respond_with_html(request, status_code, html)
226	219

410	403	self.store = self.hs.get_datastore()
411	404
412	405	if self.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
413		template_html, template_text = load_jinja2_templates(
414		self.config.email_template_dir,
415		[
416		self.config.email_add_threepid_template_html,
417		self.config.email_add_threepid_template_text,
418		],
419		public_baseurl=self.config.public_baseurl,
420		)
421	406	self.mailer = Mailer(
422	407	hs=self.hs,
423	408	app_name=self.config.email_app_name,
424		template_html=template_html,
425		template_text=template_text,
	409	template_html=self.config.email_add_threepid_template_html,
	410	template_text=self.config.email_add_threepid_template_text,
426	411	)
427	412
428	413	async def on_POST(self, request):

466	451	if self.config.request_token_inhibit_3pid_errors:
467	452	# Make the client think the operation succeeded. See the rationale in the
468	453	# comments for request_token_inhibit_3pid_errors.
	454	# Also wait for some random amount of time between 100ms and 1s to make it
	455	# look like we did something.
	456	await self.hs.clock.sleep(random.randint(1, 10) / 10)
469	457	return 200, {"sid": random_string(16)}
470	458
471	459	raise SynapseError(400, "Email is already in use", Codes.THREEPID_IN_USE)

534	522	if self.hs.config.request_token_inhibit_3pid_errors:
535	523	# Make the client think the operation succeeded. See the rationale in the
536	524	# comments for request_token_inhibit_3pid_errors.
	525	# Also wait for some random amount of time between 100ms and 1s to make it
	526	# look like we did something.
	527	await self.hs.clock.sleep(random.randint(1, 10) / 10)
537	528	return 200, {"sid": random_string(16)}
538	529
539	530	raise SynapseError(400, "MSISDN is already in use", Codes.THREEPID_IN_USE)

577	568	self.clock = hs.get_clock()
578	569	self.store = hs.get_datastore()
579	570	if self.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
580		(self.failure_email_template,) = load_jinja2_templates(
581		self.config.email_template_dir,
582		[self.config.email_add_threepid_template_failure_html],
	571	self._failure_email_template = (
	572	self.config.email_add_threepid_template_failure_html
583	573	)
584	574
585	575	async def on_GET(self, request):

630	620
631	621	# Show a failure page with a reason
632	622	template_vars = {"failure_reason": e.msg}
633		html = self.failure_email_template.render(**template_vars)
	623	html = self._failure_email_template.render(**template_vars)
634	624
635	625	respond_with_html(request, status_code, html)
636	626

+4

-0

synapse/rest/client/v2_alpha/groups.py less more

15	15
16	16	import logging
17	17
	18	from synapse.api.errors import SynapseError
18	19	from synapse.http.servlet import RestServlet, parse_json_object_from_request
19	20	from synapse.types import GroupID
20	21

323	324	async def on_GET(self, request, group_id):
324	325	requester = await self.auth.get_user_by_req(request, allow_guest=True)
325	326	requester_user_id = requester.user.to_string()
	327
	328	if not GroupID.is_valid(group_id):
	329	raise SynapseError(400, "%s was not legal group ID" % (group_id,))
326	330
327	331	result = await self.groups_handler.get_rooms_in_group(
328	332	group_id, requester_user_id

+19

-26

synapse/rest/client/v2_alpha/register.py less more

15	15
16	16	import hmac
17	17	import logging
	18	import random
18	19	from typing import List, Union
19	20
20	21	import synapse

43	44	parse_json_object_from_request,
44	45	parse_string,
45	46	)
46		from synapse.push.mailer import load_jinja2_templates
	47	from synapse.push.mailer import Mailer
47	48	from synapse.util.msisdn import phone_number_to_msisdn
48	49	from synapse.util.ratelimitutils import FederationRateLimiter
49	50	from synapse.util.stringutils import assert_valid_client_secret, random_string

80	81	self.config = hs.config
81	82
82	83	if self.hs.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
83		from synapse.push.mailer import Mailer, load_jinja2_templates
84
85		template_html, template_text = load_jinja2_templates(
86		self.config.email_template_dir,
87		[
88		self.config.email_registration_template_html,
89		self.config.email_registration_template_text,
90		],
91		apply_format_ts_filter=True,
92		apply_mxc_to_http_filter=True,
93		public_baseurl=self.config.public_baseurl,
94		)
95	84	self.mailer = Mailer(
96	85	hs=self.hs,
97	86	app_name=self.config.email_app_name,
98		template_html=template_html,
99		template_text=template_text,
	87	template_html=self.config.email_registration_template_html,
	88	template_text=self.config.email_registration_template_text,
100	89	)
101	90
102	91	async def on_POST(self, request):

142	131	if self.hs.config.request_token_inhibit_3pid_errors:
143	132	# Make the client think the operation succeeded. See the rationale in the
144	133	# comments for request_token_inhibit_3pid_errors.
	134	# Also wait for some random amount of time between 100ms and 1s to make it
	135	# look like we did something.
	136	await self.hs.clock.sleep(random.randint(1, 10) / 10)
145	137	return 200, {"sid": random_string(16)}
146	138
147	139	raise SynapseError(400, "Email is already in use", Codes.THREEPID_IN_USE)

214	206	if self.hs.config.request_token_inhibit_3pid_errors:
215	207	# Make the client think the operation succeeded. See the rationale in the
216	208	# comments for request_token_inhibit_3pid_errors.
	209	# Also wait for some random amount of time between 100ms and 1s to make it
	210	# look like we did something.
	211	await self.hs.clock.sleep(random.randint(1, 10) / 10)
217	212	return 200, {"sid": random_string(16)}
218	213
219	214	raise SynapseError(

261	256	self.store = hs.get_datastore()
262	257
263	258	if self.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
264		(self.failure_email_template,) = load_jinja2_templates(
265		self.config.email_template_dir,
266		[self.config.email_registration_template_failure_html],
267		)
268
269		if self.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
270		(self.failure_email_template,) = load_jinja2_templates(
271		self.config.email_template_dir,
272		[self.config.email_registration_template_failure_html],
	259	self._failure_email_template = (
	260	self.config.email_registration_template_failure_html
273	261	)
274	262
275	263	async def on_GET(self, request, medium):

317	305
318	306	# Show a failure page with a reason
319	307	template_vars = {"failure_reason": e.msg}
320		html = self.failure_email_template.render(**template_vars)
	308	html = self._failure_email_template.render(**template_vars)
321	309
322	310	respond_with_html(request, status_code, html)
323	311

609	597	Codes.THREEPID_IN_USE,
610	598	)
611	599
	600	entries = await self.store.get_user_agents_ips_to_ui_auth_session(
	601	session_id
	602	)
	603
612	604	registered_user_id = await self.registration_handler.register_user(
613	605	localpart=desired_username,
614	606	password_hash=password_hash,
615	607	guest_access_token=guest_access_token,
616	608	threepid=threepid,
617	609	address=client_addr,
	610	user_agent_ips=entries,
618	611	)
619	612	# Necessary due to auth checks prior to the threepid being
620	613	# written to the db

664	657	(object) params: registration parameters, from which we pull
665	658	device_id, initial_device_name and inhibit_login
666	659	Returns:
667		(object) dictionary for response from /register
	660	dictionary for response from /register
668	661	"""
669	662	result = {"user_id": user_id, "home_server": self.hs.hostname}
670	663	if not params.get("inhibit_login", False):

+14

-6

synapse/rest/client/v2_alpha/relations.py less more

21	21	import logging
22	22
23	23	from synapse.api.constants import EventTypes, RelationTypes
24		from synapse.api.errors import SynapseError
	24	from synapse.api.errors import ShadowBanError, SynapseError
25	25	from synapse.http.servlet import (
26	26	RestServlet,
27	27	parse_integer,

34	34	PaginationChunk,
35	35	RelationPaginationToken,
36	36	)
	37	from synapse.util.stringutils import random_string
37	38
38	39	from ._base import client_patterns
39	40

110	111	"sender": requester.user.to_string(),
111	112	}
112	113
113		event, _ = await self.event_creation_handler.create_and_send_nonmember_event(
114		requester, event_dict=event_dict, txn_id=txn_id
115		)
116
117		return 200, {"event_id": event.event_id}
	114	try:
	115	(
	116	event,
	117	_,
	118	) = await self.event_creation_handler.create_and_send_nonmember_event(
	119	requester, event_dict=event_dict, txn_id=txn_id
	120	)
	121	event_id = event.event_id
	122	except ShadowBanError:
	123	event_id = "$" + random_string(43)
	124
	125	return 200, {"event_id": event_id}
118	126
119	127
120	128	class RelationPaginationServlet(RestServlet):

+9

-5

synapse/rest/client/v2_alpha/room_upgrade_rest_servlet.py less more

14	14
15	15	import logging
16	16
17		from synapse.api.errors import Codes, SynapseError
	17	from synapse.api.errors import Codes, ShadowBanError, SynapseError
18	18	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
19	19	from synapse.http.servlet import (
20	20	RestServlet,
21	21	assert_params_in_dict,
22	22	parse_json_object_from_request,
23	23	)
	24	from synapse.util import stringutils
24	25
25	26	from ._base import client_patterns
26	27

61	62
62	63	content = parse_json_object_from_request(request)
63	64	assert_params_in_dict(content, ("new_version",))
64		new_version = content["new_version"]
65	65
66	66	new_version = KNOWN_ROOM_VERSIONS.get(content["new_version"])
67	67	if new_version is None:

71	71	Codes.UNSUPPORTED_ROOM_VERSION,
72	72	)
73	73
74		new_room_id = await self._room_creation_handler.upgrade_room(
75		requester, room_id, new_version
76		)
	74	try:
	75	new_room_id = await self._room_creation_handler.upgrade_room(
	76	requester, room_id, new_version
	77	)
	78	except ShadowBanError:
	79	# Generate a random room ID.
	80	new_room_id = stringutils.random_string(18)
77	81
78	82	ret = {"replacement_room": new_room_id}
79	83

+68

-0

synapse/rest/client/v2_alpha/shared_rooms.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 Half-Shot
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	import logging
	15
	16	from synapse.api.errors import Codes, SynapseError
	17	from synapse.http.servlet import RestServlet
	18	from synapse.types import UserID
	19
	20	from ._base import client_patterns
	21
	22	logger = logging.getLogger(__name__)
	23
	24
	25	class UserSharedRoomsServlet(RestServlet):
	26	"""
	27	GET /uk.half-shot.msc2666/user/shared_rooms/{user_id} HTTP/1.1
	28	"""
	29
	30	PATTERNS = client_patterns(
	31	"/uk.half-shot.msc2666/user/shared_rooms/(?P<user_id>[^/]*)",
	32	releases=(), # This is an unstable feature
	33	)
	34
	35	def __init__(self, hs):
	36	super(UserSharedRoomsServlet, self).__init__()
	37	self.auth = hs.get_auth()
	38	self.store = hs.get_datastore()
	39	self.user_directory_active = hs.config.update_user_directory
	40
	41	async def on_GET(self, request, user_id):
	42
	43	if not self.user_directory_active:
	44	raise SynapseError(
	45	code=400,
	46	msg="The user directory is disabled on this server. Cannot determine shared rooms.",
	47	errcode=Codes.FORBIDDEN,
	48	)
	49
	50	UserID.from_string(user_id)
	51
	52	requester = await self.auth.get_user_by_req(request)
	53	if user_id == requester.user.to_string():
	54	raise SynapseError(
	55	code=400,
	56	msg="You cannot request a list of shared rooms with yourself",
	57	errcode=Codes.FORBIDDEN,
	58	)
	59	rooms = await self.store.get_shared_rooms_for_users(
	60	requester.user.to_string(), user_id
	61	)
	62
	63	return 200, {"joined": list(rooms)}
	64
	65
	66	def register_servlets(hs, http_server):
	67	UserSharedRoomsServlet(hs).register(http_server)

+3

-3

synapse/rest/client/v2_alpha/sync.py less more

15	15	import itertools
16	16	import logging
17	17
18		from canonicaljson import json
19
20	18	from synapse.api.constants import PresenceState
21	19	from synapse.api.errors import Codes, StoreError, SynapseError
22	20	from synapse.api.filtering import DEFAULT_FILTER_COLLECTION, FilterCollection

28	26	from synapse.handlers.sync import SyncConfig
29	27	from synapse.http.servlet import RestServlet, parse_boolean, parse_integer, parse_string
30	28	from synapse.types import StreamToken
	29	from synapse.util import json_decoder
31	30
32	31	from ._base import client_patterns, set_timeline_upper_limit
33	32

124	123	filter_collection = DEFAULT_FILTER_COLLECTION
125	124	elif filter_id.startswith("{"):
126	125	try:
127		filter_object = json.loads(filter_id)
	126	filter_object = json_decoder.decode(filter_id)
128	127	set_timeline_upper_limit(
129	128	filter_object, self.hs.config.filter_timeline_limit
130	129	)

425	424	result["ephemeral"] = {"events": ephemeral_events}
426	425	result["unread_notifications"] = room.unread_notifications
427	426	result["summary"] = room.summary
	427	result["org.matrix.msc2654.unread_count"] = room.unread_count
428	428
429	429	return result
430	430

+21

-0

synapse/rest/client/versions.py less more

18	18	import logging
19	19	import re
20	20
	21	from synapse.api.constants import RoomCreationPreset
21	22	from synapse.http.servlet import RestServlet
22	23
23	24	logger = logging.getLogger(__name__)

29	30	def __init__(self, hs):
30	31	super(VersionsRestServlet, self).__init__()
31	32	self.config = hs.config
	33
	34	# Calculate these once since they shouldn't change after start-up.
	35	self.e2ee_forced_public = (
	36	RoomCreationPreset.PUBLIC_CHAT
	37	in self.config.encryption_enabled_by_default_for_room_presets
	38	)
	39	self.e2ee_forced_private = (
	40	RoomCreationPreset.PRIVATE_CHAT
	41	in self.config.encryption_enabled_by_default_for_room_presets
	42	)
	43	self.e2ee_forced_trusted_private = (
	44	RoomCreationPreset.TRUSTED_PRIVATE_CHAT
	45	in self.config.encryption_enabled_by_default_for_room_presets
	46	)
32	47
33	48	def on_GET(self, request):
34	49	return (

59	74	"org.matrix.e2e_cross_signing": True,
60	75	# Implements additional endpoints as described in MSC2432
61	76	"org.matrix.msc2432": True,
	77	# Implements additional endpoints as described in MSC2666
	78	"uk.half-shot.msc2666": True,
	79	# Whether new rooms will be set to encrypted or not (based on presets).
	80	"io.element.e2ee_forced.public": self.e2ee_forced_public,
	81	"io.element.e2ee_forced.private": self.e2ee_forced_private,
	82	"io.element.e2ee_forced.trusted_private": self.e2ee_forced_trusted_private,
62	83	},
63	84	},
64	85	)

+7

-5

synapse/rest/key/v2/remote_key_resource.py less more

14	14	import logging
15	15	from typing import Dict, Set
16	16
17		from canonicaljson import encode_canonical_json, json
18	17	from signedjson.sign import sign_json
19	18
20	19	from synapse.api.errors import Codes, SynapseError
21	20	from synapse.crypto.keyring import ServerKeyFetcher
22		from synapse.http.server import DirectServeJsonResource, respond_with_json_bytes
	21	from synapse.http.server import DirectServeJsonResource, respond_with_json
23	22	from synapse.http.servlet import parse_integer, parse_json_object_from_request
	23	from synapse.util import json_decoder
24	24
25	25	logger = logging.getLogger(__name__)
26	26
27	27
28	28	class RemoteKey(DirectServeJsonResource):
29		"""HTTP resource for retreiving the TLS certificate and NACL signature
	29	"""HTTP resource for retrieving the TLS certificate and NACL signature
30	30	verification keys for a collection of servers. Checks that the reported
31	31	X.509 TLS certificate matches the one used in the HTTPS connection. Checks
32	32	that the NACL signature for the remote server is valid. Returns a dict of

208	208	# Cast to bytes since postgresql returns a memoryview.
209	209	json_results.add(bytes(result["key_json"]))
210	210
	211	# If there is a cache miss, request the missing keys, then recurse (and
	212	# ensure the result is sent).
211	213	if cache_misses and query_remote_on_cache_miss:
212	214	await self.fetcher.get_keys(cache_misses)
213	215	await self.query_keys(request, query, query_remote_on_cache_miss=False)
214	216	else:
215	217	signed_keys = []
216	218	for key_json in json_results:
217		key_json = json.loads(key_json.decode("utf-8"))
	219	key_json = json_decoder.decode(key_json.decode("utf-8"))
218	220	for signing_key in self.config.key_server_signing_keys:
219	221	key_json = sign_json(key_json, self.config.server_name, signing_key)
220	222

222	224
223	225	results = {"server_keys": signed_keys}
224	226
225		respond_with_json_bytes(request, 200, encode_canonical_json(results))
	227	respond_with_json(request, 200, results, canonical_json=True)

+2

-2

synapse/rest/media/v1/_base.py less more

234	234	finish_request(request)
235	235
236	236
237		class Responder(object):
	237	class Responder:
238	238	"""Represents a response that can be streamed to the requester.
239	239
240	240	Responder is a context manager which must be used, so that any resources

259	259	pass
260	260
261	261
262		class FileInfo(object):
	262	class FileInfo:
263	263	"""Details about a requested/uploaded file.
264	264
265	265	Attributes:

+1

-1

synapse/rest/media/v1/filepath.py less more

32	32	return _wrapped
33	33
34	34
35		class MediaFilePaths(object):
	35	class MediaFilePaths:
36	36	"""Describes where files are stored on disk.
37	37
38	38	Most of the functions have a `*_rel` variant which returns a file path that

+1

-1

synapse/rest/media/v1/media_repository.py less more

61	61	UPDATE_RECENTLY_ACCESSED_TS = 60 * 1000
62	62
63	63
64		class MediaRepository(object):
	64	class MediaRepository:
65	65	def __init__(self, hs):
66	66	self.hs = hs
67	67	self.auth = hs.get_auth()

+1

-1

synapse/rest/media/v1/media_storage.py less more

33	33	logger = logging.getLogger(__name__)
34	34
35	35
36		class MediaStorage(object):
	36	class MediaStorage:
37	37	"""Responsible for storing/fetching files from local sources.
38	38
39	39	Args:

+1

-1

synapse/rest/media/v1/thumbnailer.py less more

30	30	}
31	31
32	32
33		class Thumbnailer(object):
	33	class Thumbnailer:
34	34
35	35	FORMATS = {"image/jpeg": "JPEG", "image/png": "PNG"}
36	36

+3

-3

synapse/rest/well_known.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		import json
16	15	import logging
17	16
18	17	from twisted.web.resource import Resource
19	18
20	19	from synapse.http.server import set_cors_headers
	20	from synapse.util import json_encoder
21	21
22	22	logger = logging.getLogger(__name__)
23	23
24	24
25		class WellKnownBuilder(object):
	25	class WellKnownBuilder:
26	26	"""Utility to construct the well-known response
27	27
28	28	Args:

66	66
67	67	logger.debug("returning: %s", r)
68	68	request.setHeader(b"Content-Type", b"application/json")
69		return json.dumps(r).encode("utf-8")
	69	return json_encoder.encode(r).encode("utf-8")

+1

-1

synapse/secrets.py less more

36	36	import binascii
37	37	import os
38	38
39		class Secrets(object):
	39	class Secrets:
40	40	def token_bytes(self, nbytes=32):
41	41	return os.urandom(nbytes)
42	42

+5

-0

synapse/server.py less more

113	113	from synapse.types import DomainSpecificString
114	114	from synapse.util import Clock
115	115	from synapse.util.distributor import Distributor
	116	from synapse.util.ratelimitutils import FederationRateLimiter
116	117	from synapse.util.stringutils import random_string
117	118
118	119	logger = logging.getLogger(__name__)

641	642	def get_replication_streams(self) -> Dict[str, Stream]:
642	643	return {stream.NAME: stream(self) for stream in STREAMS_MAP.values()}
643	644
	645	@cache_in_self
	646	def get_federation_ratelimiter(self) -> FederationRateLimiter:
	647	return FederationRateLimiter(self.clock, config=self.config.rc_federation)
	648
644	649	async def remove_pusher(self, app_id: str, push_key: str, user_id: str):
645	650	return await self.get_pusherpool().remove_pusher(app_id, push_key, user_id)
646	651

+1

-1

synapse/server_notices/consent_server_notices.py less more

22	22	logger = logging.getLogger(__name__)
23	23
24	24
25		class ConsentServerNotices(object):
	25	class ConsentServerNotices:
26	26	"""Keeps track of whether we need to send users server_notices about
27	27	privacy policy consent, and sends one if we do.
28	28	"""

+1

-1

synapse/server_notices/resource_limits_server_notices.py less more

26	26	logger = logging.getLogger(__name__)
27	27
28	28
29		class ResourceLimitsServerNotices(object):
	29	class ResourceLimitsServerNotices:
30	30	""" Keeps track of whether the server has reached it's resource limit and
31	31	ensures that the client is kept up to date.
32	32	"""

+1

-1

synapse/server_notices/server_notices_manager.py less more

24	24	SERVER_NOTICE_ROOM_TAG = "m.server_notice"
25	25
26	26
27		class ServerNoticesManager(object):
	27	class ServerNoticesManager:
28	28	def __init__(self, hs):
29	29	"""
30	30

+1

-1

synapse/server_notices/server_notices_sender.py less more

19	19	)
20	20
21	21
22		class ServerNoticesSender(object):
	22	class ServerNoticesSender:
23	23	"""A centralised place which sends server notices automatically when
24	24	Certain Events take place
25	25	"""

+1

-1

synapse/server_notices/worker_server_notices_sender.py less more

13	13	# limitations under the License.
14	14
15	15
16		class WorkerServerNoticesSender(object):
	16	class WorkerServerNoticesSender:
17	17	"""Stub impl of ServerNoticesSender which does nothing"""
18	18
19	19	def __init__(self, hs):

+17

-4

synapse/spam_checker_api/__init__.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
	15	from enum import Enum
15	16
16	17	from twisted.internet import defer
17	18

24	25	logger = logging.getLogger(__name__)
25	26
26	27
27		class SpamCheckerApi(object):
	28	class RegistrationBehaviour(Enum):
	29	"""
	30	Enum to define whether a registration request should allowed, denied, or shadow-banned.
	31	"""
	32
	33	ALLOW = "allow"
	34	SHADOW_BAN = "shadow_ban"
	35	DENY = "deny"
	36
	37
	38	class SpamCheckerApi:
28	39	"""A proxy object that gets passed to spam checkers so they can get
29	40	access to rooms and other relevant information.
30	41	"""

47	58	twisted.internet.defer.Deferred[list(synapse.events.FrozenEvent)]:
48	59	The filtered state events in the room.
49	60	"""
50		state_ids = yield self._store.get_filtered_current_state_ids(
51		room_id=room_id, state_filter=StateFilter.from_types(types)
	61	state_ids = yield defer.ensureDeferred(
	62	self._store.get_filtered_current_state_ids(
	63	room_id=room_id, state_filter=StateFilter.from_types(types)
	64	)
52	65	)
53		state = yield self._store.get_events(state_ids.values())
	66	state = yield defer.ensureDeferred(self._store.get_events(state_ids.values()))
54	67	return state.values()

+157

-97

synapse/state/__init__.py less more

15	15
16	16	import logging
17	17	from collections import namedtuple
18		from typing import Awaitable, Dict, Iterable, List, Optional, Set
	18	from typing import (
	19	Awaitable,
	20	Dict,
	21	Iterable,
	22	List,
	23	Optional,
	24	Sequence,
	25	Set,
	26	Union,
	27	cast,
	28	overload,
	29	)
19	30
20	31	import attr
21	32	from frozendict import frozendict
22	33	from prometheus_client import Histogram
	34	from typing_extensions import Literal
23	35
24	36	from synapse.api.constants import EventTypes
25	37	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, StateResolutionVersions

29	41	from synapse.state import v1, v2
30	42	from synapse.storage.databases.main.events_worker import EventRedactBehaviour
31	43	from synapse.storage.roommember import ProfileInfo
32		from synapse.types import StateMap
	44	from synapse.types import Collection, MutableStateMap, StateMap
33	45	from synapse.util import Clock
34	46	from synapse.util.async_helpers import Linearizer
35	47	from synapse.util.caches.expiringcache import ExpiringCache

64	76	return s
65	77
66	78
67		class _StateCacheEntry(object):
	79	class _StateCacheEntry:
68	80	__slots__ = ["state", "state_group", "state_id", "prev_group", "delta_ids"]
69	81
70		def __init__(self, state, state_group, prev_group=None, delta_ids=None):
71		# dict[(str, str), str] map from (type, state_key) to event_id
	82	def __init__(
	83	self,
	84	state: StateMap[str],
	85	state_group: Optional[int],
	86	prev_group: Optional[int] = None,
	87	delta_ids: Optional[StateMap[str]] = None,
	88	):
	89	# A map from (type, state_key) to event_id.
72	90	self.state = frozendict(state)
73	91
74	92	# the ID of a state group if one and only one is involved.

94	112	return len(self.state)
95	113
96	114
97		class StateHandler(object):
	115	class StateHandler:
98	116	"""Fetches bits of state from the stores, and does state resolution
99	117	where necessary
100	118	"""

106	124	self.hs = hs
107	125	self._state_resolution_handler = hs.get_state_resolution_handler()
108	126
	127	@overload
109	128	async def get_current_state(
110		self, room_id, event_type=None, state_key="", latest_event_ids=None
111		):
112		""" Retrieves the current state for the room. This is done by
	129	self,
	130	room_id: str,
	131	event_type: Literal[None] = None,
	132	state_key: str = "",
	133	latest_event_ids: Optional[List[str]] = None,
	134	) -> StateMap[EventBase]:
	135	...
	136
	137	@overload
	138	async def get_current_state(
	139	self,
	140	room_id: str,
	141	event_type: str,
	142	state_key: str = "",
	143	latest_event_ids: Optional[List[str]] = None,
	144	) -> Optional[EventBase]:
	145	...
	146
	147	async def get_current_state(
	148	self,
	149	room_id: str,
	150	event_type: Optional[str] = None,
	151	state_key: str = "",
	152	latest_event_ids: Optional[List[str]] = None,
	153	) -> Union[Optional[EventBase], StateMap[EventBase]]:
	154	"""Retrieves the current state for the room. This is done by
113	155	calling `get_latest_events_in_room` to get the leading edges of the
114	156	event graph and then resolving any of the state conflicts.
115	157
116	158	This is equivalent to getting the state of an event that were to send
117	159	next before receiving any new events.
118	160
119		If `event_type` is specified, then the method returns only the one
120		event (or None) with that `event_type` and `state_key`.
121
122		Returns:
123		map from (type, state_key) to event
	161	Returns:
	162	If `event_type` is specified, then the method returns only the one
	163	event (or None) with that `event_type` and `state_key`.
	164
	165	Otherwise, a map from (type, state_key) to event.
124	166	"""
125	167	if not latest_event_ids:
126	168	latest_event_ids = await self.store.get_latest_event_ids_in_room(room_id)
	169	assert latest_event_ids is not None
127	170
128	171	logger.debug("calling resolve_state_groups from get_current_state")
129	172	ret = await self.resolve_state_groups_for_events(room_id, latest_event_ids)

139	182	state_map = await self.store.get_events(
140	183	list(state.values()), get_prev_content=False
141	184	)
142		state = {
	185	return {
143	186	key: state_map[e_id] for key, e_id in state.items() if e_id in state_map
144	187	}
145	188
146		return state
147
148		async def get_current_state_ids(self, room_id, latest_event_ids=None):
	189	async def get_current_state_ids(
	190	self, room_id: str, latest_event_ids: Optional[Iterable[str]] = None
	191	) -> StateMap[str]:
149	192	"""Get the current state, or the state at a set of events, for a room
150	193
151	194	Args:
152		room_id (str):
153
154		latest_event_ids (iterable[str]\|None): if given, the forward
155		extremities to resolve. If None, we look them up from the
156		database (via a cache)
157
158		Returns:
159		Deferred[dict[(str, str), str)]]: the state dict, mapping from
160		(event_type, state_key) -> event_id
	195	room_id:
	196	latest_event_ids: if given, the forward extremities to resolve. If
	197	None, we look them up from the database (via a cache).
	198
	199	Returns:
	200	the state dict, mapping from (event_type, state_key) -> event_id
161	201	"""
162	202	if not latest_event_ids:
163	203	latest_event_ids = await self.store.get_latest_event_ids_in_room(room_id)
	204	assert latest_event_ids is not None
164	205
165	206	logger.debug("calling resolve_state_groups from get_current_state_ids")
166	207	ret = await self.resolve_state_groups_for_events(room_id, latest_event_ids)
167		state = ret.state
168
169		return state
	208	return ret.state
170	209
171	210	async def get_current_users_in_room(
172	211	self, room_id: str, latest_event_ids: Optional[List[str]] = None

182	221	"""
183	222	if not latest_event_ids:
184	223	latest_event_ids = await self.store.get_latest_event_ids_in_room(room_id)
	224	assert latest_event_ids is not None
	225
185	226	logger.debug("calling resolve_state_groups from get_current_users_in_room")
186	227	entry = await self.resolve_state_groups_for_events(room_id, latest_event_ids)
187		joined_users = await self.store.get_joined_users_from_state(room_id, entry)
188		return joined_users
189
190		async def get_current_hosts_in_room(self, room_id):
	228	return await self.store.get_joined_users_from_state(room_id, entry)
	229
	230	async def get_current_hosts_in_room(self, room_id: str) -> Set[str]:
191	231	event_ids = await self.store.get_latest_event_ids_in_room(room_id)
192	232	return await self.get_hosts_in_room_at_events(room_id, event_ids)
193	233
194		async def get_hosts_in_room_at_events(self, room_id, event_ids):
	234	async def get_hosts_in_room_at_events(
	235	self, room_id: str, event_ids: List[str]
	236	) -> Set[str]:
195	237	"""Get the hosts that were in a room at the given event ids
196	238
197	239	Args:
198		room_id (str):
199		event_ids (list[str]):
200
201		Returns:
202		Deferred[list[str]]: the hosts in the room at the given events
	240	room_id:
	241	event_ids:
	242
	243	Returns:
	244	The hosts in the room at the given events
203	245	"""
204	246	entry = await self.resolve_state_groups_for_events(room_id, event_ids)
205		joined_hosts = await self.store.get_joined_hosts(room_id, entry)
206		return joined_hosts
	247	return await self.store.get_joined_hosts(room_id, entry)
207	248
208	249	async def compute_event_context(
209	250	self, event: EventBase, old_state: Optional[Iterable[EventBase]] = None
210		):
	251	) -> EventContext:
211	252	"""Build an EventContext structure for the event.
212	253
213	254	This works out what the current state should be for the event, and

220	261	when receiving an event from federation where we don't have the
221	262	prev events for, e.g. when backfilling.
222	263	Returns:
223		synapse.events.snapshot.EventContext:
	264	The event context.
224	265	"""
225	266
226	267	if event.internal_metadata.is_outlier():

261	302	# if we're given the state before the event, then we use that
262	303	state_ids_before_event = {
263	304	(s.type, s.state_key): s.event_id for s in old_state
264		}
	305	} # type: StateMap[str]
265	306	state_group_before_event = None
266	307	state_group_before_event_prev_group = None
267	308	deltas_to_state_group_before_event = None

345	386	)
346	387
347	388	@measure_func()
348		async def resolve_state_groups_for_events(self, room_id, event_ids):
	389	async def resolve_state_groups_for_events(
	390	self, room_id: str, event_ids: Iterable[str]
	391	) -> _StateCacheEntry:
349	392	""" Given a list of event_ids this method fetches the state at each
350	393	event, resolves conflicts between them and returns them.
351	394
352	395	Args:
353		room_id (str)
354		event_ids (list[str])
355		explicit_room_version (str\|None): If set uses the the given room
356		version to choose the resolution algorithm. If None, then
357		checks the database for room version.
358
359		Returns:
360		Deferred[_StateCacheEntry]: resolved state
	396	room_id
	397	event_ids
	398
	399	Returns:
	400	The resolved state
361	401	"""
362	402	logger.debug("resolve_state_groups event_ids %s", event_ids)
363	403

393	433	)
394	434	return result
395	435
396		async def resolve_events(self, room_version, state_sets, event):
	436	async def resolve_events(
	437	self,
	438	room_version: str,
	439	state_sets: Collection[Iterable[EventBase]],
	440	event: EventBase,
	441	) -> StateMap[EventBase]:
397	442	logger.info(
398	443	"Resolving state for %s with %d groups", event.room_id, len(state_sets)
399	444	)

413	458	state_res_store=StateResolutionStore(self.store),
414	459	)
415	460
416		new_state = {key: state_map[ev_id] for key, ev_id in new_state.items()}
417
418		return new_state
419
420
421		class StateResolutionHandler(object):
	461	return {key: state_map[ev_id] for key, ev_id in new_state.items()}
	462
	463
	464	class StateResolutionHandler:
422	465	"""Responsible for doing state conflict resolution.
423	466
424	467	Note that the storage layer depends on this handler, so all functions must

443	486
444	487	@log_function
445	488	async def resolve_state_groups(
446		self, room_id, room_version, state_groups_ids, event_map, state_res_store
	489	self,
	490	room_id: str,
	491	room_version: str,
	492	state_groups_ids: Dict[int, StateMap[str]],
	493	event_map: Optional[Dict[str, EventBase]],
	494	state_res_store: "StateResolutionStore",
447	495	):
448	496	"""Resolves conflicts between a set of state groups
449	497

451	499	not be called for a single state group
452	500
453	501	Args:
454		room_id (str): room we are resolving for (used for logging and sanity checks)
455		room_version (str): version of the room
456		state_groups_ids (dict[int, dict[(str, str), str]]):
457		map from state group id to the state in that state group
	502	room_id: room we are resolving for (used for logging and sanity checks)
	503	room_version: version of the room
	504	state_groups_ids:
	505	A map from state group id to the state in that state group
458	506	(where 'state' is a map from state key to event id)
459	507
460		event_map(dict[str,FrozenEvent]\|None):
	508	event_map:
461	509	a dict from event_id to event, for any events that we happen to
462	510	have in flight (eg, those currently being persisted). This will be
463	511	used as a starting point fof finding the state we need; any missing

465	513
466	514	If None, all events will be fetched via state_res_store.
467	515
468		state_res_store (StateResolutionStore)
469
470		Returns:
471		_StateCacheEntry: resolved state
	516	state_res_store
	517
	518	Returns:
	519	The resolved state
472	520	"""
473	521	logger.debug("resolve_state_groups state_groups %s", state_groups_ids.keys())
474	522

492	540	#
493	541	# XXX: is this actually worthwhile, or should we just let
494	542	# resolve_events_with_store do it?
495		new_state = {}
	543	new_state = {} # type: MutableStateMap[str]
496	544	conflicted_state = False
497	545	for st in state_groups_ids.values():
498	546	for key, e_id in st.items():

506	554	if conflicted_state:
507	555	logger.info("Resolving conflicted state for %r", room_id)
508	556	with Measure(self.clock, "state._resolve_events"):
509		new_state = await resolve_events_with_store(
510		self.clock,
511		room_id,
512		room_version,
513		list(state_groups_ids.values()),
514		event_map=event_map,
515		state_res_store=state_res_store,
	557	# resolve_events_with_store returns a StateMap, but we can
	558	# treat it as a MutableStateMap as it is above. It isn't
	559	# actually mutated anymore (and is frozen in
	560	# _make_state_cache_entry below).
	561	new_state = cast(
	562	MutableStateMap,
	563	await resolve_events_with_store(
	564	self.clock,
	565	room_id,
	566	room_version,
	567	list(state_groups_ids.values()),
	568	event_map=event_map,
	569	state_res_store=state_res_store,
	570	),
516	571	)
517	572
518	573	# if the new state matches any of the input state groups, we can

529	584	return cache
530	585
531	586
532		def _make_state_cache_entry(new_state, state_groups_ids):
	587	def _make_state_cache_entry(
	588	new_state: StateMap[str], state_groups_ids: Dict[int, StateMap[str]]
	589	) -> _StateCacheEntry:
533	590	"""Given a resolved state, and a set of input state groups, pick one to base
534	591	a new state group on (if any), and return an appropriately-constructed
535	592	_StateCacheEntry.
536	593
537	594	Args:
538		new_state (dict[(str, str), str]): resolved state map (mapping from
539		(type, state_key) to event_id)
540
541		state_groups_ids (dict[int, dict[(str, str), str]]):
542		map from state group id to the state in that state group
543		(where 'state' is a map from state key to event id)
	595	new_state: resolved state map (mapping from (type, state_key) to event_id)
	596
	597	state_groups_ids:
	598	map from state group id to the state in that state group (where
	599	'state' is a map from state key to event id)
544	600
545	601	Returns:
546		_StateCacheEntry
	602	The cache entry.
547	603	"""
548	604	# if the new state matches any of the input state groups, we can
549	605	# use that state group again. Otherwise we will generate a state_id

584	640	clock: Clock,
585	641	room_id: str,
586	642	room_version: str,
587		state_sets: List[StateMap[str]],
	643	state_sets: Sequence[StateMap[str]],
588	644	event_map: Optional[Dict[str, EventBase]],
589	645	state_res_store: "StateResolutionStore",
590	646	) -> Awaitable[StateMap[str]]:

622	678
623	679
624	680	@attr.s
625		class StateResolutionStore(object):
	681	class StateResolutionStore:
626	682	"""Interface that allows state resolution algorithms to access the database
627	683	in well defined way.
628	684

632	688
633	689	store = attr.ib()
634	690
635		def get_events(self, event_ids, allow_rejected=False):
	691	def get_events(
	692	self, event_ids: Iterable[str], allow_rejected: bool = False
	693	) -> Awaitable[Dict[str, EventBase]]:
636	694	"""Get events from the database
637	695
638	696	Args:
639		event_ids (list): The event_ids of the events to fetch
640		allow_rejected (bool): If True return rejected events.
641
642		Returns:
643		Deferred[dict[str, FrozenEvent]]: Dict from event_id to event.
	697	event_ids: The event_ids of the events to fetch
	698	allow_rejected: If True return rejected events.
	699
	700	Returns:
	701	An awaitable which resolves to a dict from event_id to event.
644	702	"""
645	703
646	704	return self.store.get_events(

650	708	allow_rejected=allow_rejected,
651	709	)
652	710
653		def get_auth_chain_difference(self, state_sets: List[Set[str]]):
	711	def get_auth_chain_difference(
	712	self, state_sets: List[Set[str]]
	713	) -> Awaitable[Set[str]]:
654	714	"""Given sets of state events figure out the auth chain difference (as
655	715	per state res v2 algorithm).
656	716

659	719	chain.
660	720
661	721	Returns:
662		Deferred[Set[str]]: Set of event IDs.
	722	An awaitable that resolves to a set of event IDs.
663	723	"""
664	724
665	725	return self.store.get_auth_chain_difference(state_sets)

+62

-31

synapse/state/v1.py less more

14	14
15	15	import hashlib
16	16	import logging
17		from typing import Awaitable, Callable, Dict, List, Optional
	17	from typing import (
	18	Awaitable,
	19	Callable,
	20	Dict,
	21	Iterable,
	22	List,
	23	Optional,
	24	Sequence,
	25	Set,
	26	Tuple,
	27	)
18	28
19	29	from synapse import event_auth
20	30	from synapse.api.constants import EventTypes
21	31	from synapse.api.errors import AuthError
22	32	from synapse.api.room_versions import RoomVersions
23	33	from synapse.events import EventBase
24		from synapse.types import StateMap
	34	from synapse.types import MutableStateMap, StateMap
25	35
26	36	logger = logging.getLogger(__name__)
27	37

31	41
32	42	async def resolve_events_with_store(
33	43	room_id: str,
34		state_sets: List[StateMap[str]],
	44	state_sets: Sequence[StateMap[str]],
35	45	event_map: Optional[Dict[str, EventBase]],
36		state_map_factory: Callable[[List[str]], Awaitable],
37		):
	46	state_map_factory: Callable[[Iterable[str]], Awaitable[Dict[str, EventBase]]],
	47	) -> StateMap[str]:
38	48	"""
39	49	Args:
40	50	room_id: the room we are working in

55	65	an Awaitable that resolves to a dict of event_id to event.
56	66
57	67	Returns:
58		Deferred[dict[(str, str), str]]:
59		a map from (type, state_key) to event_id.
	68	A map from (type, state_key) to event_id.
60	69	"""
61	70	if len(state_sets) == 1:
62	71	return state_sets[0]

74	83	"Asking for %d/%d conflicted events", len(needed_events), needed_event_count
75	84	)
76	85
77		# dict[str, FrozenEvent]: a map from state event id to event. Only includes
78		# the state events which are in conflict (and those in event_map)
	86	# A map from state event id to event. Only includes the state events which
	87	# are in conflict (and those in event_map).
79	88	state_map = await state_map_factory(needed_events)
80	89	if event_map is not None:
81	90	state_map.update(event_map)

90	99
91	100	# get the ids of the auth events which allow us to authenticate the
92	101	# conflicted state, picking only from the unconflicting state.
93		#
94		# dict[(str, str), str]: a map from state key to event id
95	102	auth_events = _create_auth_events_from_maps(
96	103	unconflicted_state, conflicted_state, state_map
97	104	)

121	128	)
122	129
123	130
124		def _seperate(state_sets):
	131	def _seperate(
	132	state_sets: Iterable[StateMap[str]],
	133	) -> Tuple[MutableStateMap[str], MutableStateMap[Set[str]]]:
125	134	"""Takes the state_sets and figures out which keys are conflicted and
126	135	which aren't. i.e., which have multiple different event_ids associated
127	136	with them in different state sets.
128	137
129	138	Args:
130		state_sets(iterable[dict[(str, str), str]]):
	139	state_sets:
131	140	List of dicts of (type, state_key) -> event_id, which are the
132	141	different state groups to resolve.
133	142
134	143	Returns:
135		(dict[(str, str), str], dict[(str, str), set[str]]):
136		A tuple of (unconflicted_state, conflicted_state), where:
137
138		unconflicted_state is a dict mapping (type, state_key)->event_id
139		for unconflicted state keys.
140
141		conflicted_state is a dict mapping (type, state_key) to a set of
142		event ids for conflicted state keys.
	144	A tuple of (unconflicted_state, conflicted_state), where:
	145
	146	unconflicted_state is a dict mapping (type, state_key)->event_id
	147	for unconflicted state keys.
	148
	149	conflicted_state is a dict mapping (type, state_key) to a set of
	150	event ids for conflicted state keys.
143	151	"""
144	152	state_set_iterator = iter(state_sets)
145	153	unconflicted_state = dict(next(state_set_iterator))
146		conflicted_state = {}
	154	conflicted_state = {} # type: MutableStateMap[Set[str]]
147	155
148	156	for state_set in state_set_iterator:
149	157	for key, value in state_set.items():

170	178	return unconflicted_state, conflicted_state
171	179
172	180
173		def _create_auth_events_from_maps(unconflicted_state, conflicted_state, state_map):
	181	def _create_auth_events_from_maps(
	182	unconflicted_state: StateMap[str],
	183	conflicted_state: StateMap[Set[str]],
	184	state_map: Dict[str, EventBase],
	185	) -> StateMap[str]:
	186	"""
	187
	188	Args:
	189	unconflicted_state: The unconflicted state map.
	190	conflicted_state: The conflicted state map.
	191	state_map:
	192
	193	Returns:
	194	A map from state key to event id.
	195	"""
174	196	auth_events = {}
175	197	for event_ids in conflicted_state.values():
176	198	for event_id in event_ids:

178	200	keys = event_auth.auth_types_for_event(state_map[event_id])
179	201	for key in keys:
180	202	if key not in auth_events:
181		event_id = unconflicted_state.get(key, None)
182		if event_id:
183		auth_events[key] = event_id
	203	auth_event_id = unconflicted_state.get(key, None)
	204	if auth_event_id:
	205	auth_events[key] = auth_event_id
184	206	return auth_events
185	207
186	208
187	209	def _resolve_with_state(
188		unconflicted_state_ids, conflicted_state_ids, auth_event_ids, state_map
	210	unconflicted_state_ids: MutableStateMap[str],
	211	conflicted_state_ids: StateMap[Set[str]],
	212	auth_event_ids: StateMap[str],
	213	state_map: Dict[str, EventBase],
189	214	):
190	215	conflicted_state = {}
191	216	for key, event_ids in conflicted_state_ids.items():

214	239	return new_state
215	240
216	241
217		def _resolve_state_events(conflicted_state, auth_events):
	242	def _resolve_state_events(
	243	conflicted_state: StateMap[List[EventBase]], auth_events: MutableStateMap[EventBase]
	244	) -> StateMap[EventBase]:
218	245	""" This is where we actually decide which of the conflicted state to
219	246	use.
220	247

254	281	return resolved_state
255	282
256	283
257		def _resolve_auth_events(events, auth_events):
	284	def _resolve_auth_events(
	285	events: List[EventBase], auth_events: StateMap[EventBase]
	286	) -> EventBase:
258	287	reverse = list(reversed(_ordered_events(events)))
259	288
260	289	auth_keys = {

288	317	return event
289	318
290	319
291		def _resolve_normal_events(events, auth_events):
	320	def _resolve_normal_events(
	321	events: List[EventBase], auth_events: StateMap[EventBase]
	322	) -> EventBase:
292	323	for event in _ordered_events(events):
293	324	try:
294	325	# The signatures have already been checked at this point

308	339	return event
309	340
310	341
311		def _ordered_events(events):
	342	def _ordered_events(events: Iterable[EventBase]) -> List[EventBase]:
312	343	def key_func(e):
313	344	# we have to use utf-8 rather than ascii here because it turns out we allow
314	345	# people to send us events with non-ascii event IDs :/

+190

-111

synapse/state/v2.py less more

15	15	import heapq
16	16	import itertools
17	17	import logging
18		from typing import Dict, List, Optional
	18	from typing import (
	19	Any,
	20	Callable,
	21	Dict,
	22	Generator,
	23	Iterable,
	24	List,
	25	Optional,
	26	Sequence,
	27	Set,
	28	Tuple,
	29	overload,
	30	)
	31
	32	from typing_extensions import Literal
19	33
20	34	import synapse.state
21	35	from synapse import event_auth

23	37	from synapse.api.errors import AuthError
24	38	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
25	39	from synapse.events import EventBase
26		from synapse.types import StateMap
	40	from synapse.types import MutableStateMap, StateMap
27	41	from synapse.util import Clock
28	42
29	43	logger = logging.getLogger(__name__)

39	53	clock: Clock,
40	54	room_id: str,
41	55	room_version: str,
42		state_sets: List[StateMap[str]],
	56	state_sets: Sequence[StateMap[str]],
43	57	event_map: Optional[Dict[str, EventBase]],
44	58	state_res_store: "synapse.state.StateResolutionStore",
45		):
	59	) -> StateMap[str]:
46	60	"""Resolves the state using the v2 state resolution algorithm
47	61
48	62	Args:

62	76	state_res_store:
63	77
64	78	Returns:
65		Deferred[dict[(str, str), str]]:
66		a map from (type, state_key) to event_id.
	79	A map from (type, state_key) to event_id.
67	80	"""
68	81
69	82	logger.debug("Computing conflicted state")

170	183	return resolved_state
171	184
172	185
173		async def _get_power_level_for_sender(room_id, event_id, event_map, state_res_store):
	186	async def _get_power_level_for_sender(
	187	room_id: str,
	188	event_id: str,
	189	event_map: Dict[str, EventBase],
	190	state_res_store: "synapse.state.StateResolutionStore",
	191	) -> int:
174	192	"""Return the power level of the sender of the given event according to
175	193	their auth events.
176	194
177	195	Args:
178		room_id (str)
179		event_id (str)
180		event_map (dict[str,FrozenEvent])
181		state_res_store (StateResolutionStore)
182
183		Returns:
184		Deferred[int]
	196	room_id
	197	event_id
	198	event_map
	199	state_res_store
	200
	201	Returns:
	202	The power level.
185	203	"""
186	204	event = await _get_event(room_id, event_id, event_map, state_res_store)
187	205

216	234	return int(level)
217	235
218	236
219		async def _get_auth_chain_difference(state_sets, event_map, state_res_store):
	237	async def _get_auth_chain_difference(
	238	state_sets: Sequence[StateMap[str]],
	239	event_map: Dict[str, EventBase],
	240	state_res_store: "synapse.state.StateResolutionStore",
	241	) -> Set[str]:
220	242	"""Compare the auth chains of each state set and return the set of events
221	243	that only appear in some but not all of the auth chains.
222	244
223	245	Args:
224		state_sets (list)
225		event_map (dict[str,FrozenEvent])
226		state_res_store (StateResolutionStore)
227
228		Returns:
229		Deferred[set[str]]: Set of event IDs
	246	state_sets
	247	event_map
	248	state_res_store
	249
	250	Returns:
	251	Set of event IDs
230	252	"""
231	253
232	254	difference = await state_res_store.get_auth_chain_difference(

236	258	return difference
237	259
238	260
239		def _seperate(state_sets):
	261	def _seperate(
	262	state_sets: Iterable[StateMap[str]],
	263	) -> Tuple[StateMap[str], StateMap[Set[str]]]:
240	264	"""Return the unconflicted and conflicted state. This is different than in
241	265	the original algorithm, as this defines a key to be conflicted if one of
242	266	the state sets doesn't have that key.
243	267
244	268	Args:
245		state_sets (list)
246
247		Returns:
248		tuple[dict, dict]: A tuple of unconflicted and conflicted state. The
249		conflicted state dict is a map from type/state_key to set of event IDs
	269	state_sets
	270
	271	Returns:
	272	A tuple of unconflicted and conflicted state. The conflicted state dict
	273	is a map from type/state_key to set of event IDs
250	274	"""
251	275	unconflicted_state = {}
252	276	conflicted_state = {}

259	283	event_ids.discard(None)
260	284	conflicted_state[key] = event_ids
261	285
262		return unconflicted_state, conflicted_state
263
264
265		def _is_power_event(event):
	286	# mypy doesn't understand that discarding None above means that conflicted
	287	# state is StateMap[Set[str]], not StateMap[Set[Optional[Str]]].
	288	return unconflicted_state, conflicted_state # type: ignore
	289
	290
	291	def _is_power_event(event: EventBase) -> bool:
266	292	"""Return whether or not the event is a "power event", as defined by the
267	293	v2 state resolution algorithm
268	294
269	295	Args:
270		event (FrozenEvent)
271
272		Returns:
273		boolean
	296	event
	297
	298	Returns:
	299	True if the event is a power event.
274	300	"""
275	301	if (event.type, event.state_key) in (
276	302	(EventTypes.PowerLevels, ""),

287	313
288	314
289	315	async def _add_event_and_auth_chain_to_graph(
290		graph, room_id, event_id, event_map, state_res_store, auth_diff
291		):
	316	graph: Dict[str, Set[str]],
	317	room_id: str,
	318	event_id: str,
	319	event_map: Dict[str, EventBase],
	320	state_res_store: "synapse.state.StateResolutionStore",
	321	auth_diff: Set[str],
	322	) -> None:
292	323	"""Helper function for _reverse_topological_power_sort that add the event
293	324	and its auth chain (that is in the auth diff) to the graph
294	325
295	326	Args:
296		graph (dict[str, set[str]]): A map from event ID to the events auth
297		event IDs
298		room_id (str): the room we are working in
299		event_id (str): Event to add to the graph
300		event_map (dict[str,FrozenEvent])
301		state_res_store (StateResolutionStore)
302		auth_diff (set[str]): Set of event IDs that are in the auth difference.
	327	graph: A map from event ID to the events auth event IDs
	328	room_id: the room we are working in
	329	event_id: Event to add to the graph
	330	event_map
	331	state_res_store
	332	auth_diff: Set of event IDs that are in the auth difference.
303	333	"""
304	334
305	335	state = [event_id]

317	347
318	348
319	349	async def _reverse_topological_power_sort(
320		clock, room_id, event_ids, event_map, state_res_store, auth_diff
321		):
	350	clock: Clock,
	351	room_id: str,
	352	event_ids: Iterable[str],
	353	event_map: Dict[str, EventBase],
	354	state_res_store: "synapse.state.StateResolutionStore",
	355	auth_diff: Set[str],
	356	) -> List[str]:
322	357	"""Returns a list of the event_ids sorted by reverse topological ordering,
323	358	and then by power level and origin_server_ts
324	359
325	360	Args:
326		clock (Clock)
327		room_id (str): the room we are working in
328		event_ids (list[str]): The events to sort
329		event_map (dict[str,FrozenEvent])
330		state_res_store (StateResolutionStore)
331		auth_diff (set[str]): Set of event IDs that are in the auth difference.
332
333		Returns:
334		Deferred[list[str]]: The sorted list
335		"""
336
337		graph = {}
	361	clock
	362	room_id: the room we are working in
	363	event_ids: The events to sort
	364	event_map
	365	state_res_store
	366	auth_diff: Set of event IDs that are in the auth difference.
	367
	368	Returns:
	369	The sorted list
	370	"""
	371
	372	graph = {} # type: Dict[str, Set[str]]
338	373	for idx, event_id in enumerate(event_ids, start=1):
339	374	await _add_event_and_auth_chain_to_graph(
340	375	graph, room_id, event_id, event_map, state_res_store, auth_diff

371	406
372	407
373	408	async def _iterative_auth_checks(
374		clock, room_id, room_version, event_ids, base_state, event_map, state_res_store
375		):
	409	clock: Clock,
	410	room_id: str,
	411	room_version: str,
	412	event_ids: List[str],
	413	base_state: StateMap[str],
	414	event_map: Dict[str, EventBase],
	415	state_res_store: "synapse.state.StateResolutionStore",
	416	) -> MutableStateMap[str]:
376	417	"""Sequentially apply auth checks to each event in given list, updating the
377	418	state as it goes along.
378	419
379	420	Args:
380		clock (Clock)
381		room_id (str)
382		room_version (str)
383		event_ids (list[str]): Ordered list of events to apply auth checks to
384		base_state (StateMap[str]): The set of state to start with
385		event_map (dict[str,FrozenEvent])
386		state_res_store (StateResolutionStore)
387
388		Returns:
389		Deferred[StateMap[str]]: Returns the final updated state
390		"""
391		resolved_state = base_state.copy()
	421	clock
	422	room_id
	423	room_version
	424	event_ids: Ordered list of events to apply auth checks to
	425	base_state: The set of state to start with
	426	event_map
	427	state_res_store
	428
	429	Returns:
	430	Returns the final updated state
	431	"""
	432	resolved_state = dict(base_state)
392	433	room_version_obj = KNOWN_ROOM_VERSIONS[room_version]
393	434
394	435	for idx, event_id in enumerate(event_ids, start=1):

438	479
439	480
440	481	async def _mainline_sort(
441		clock, room_id, event_ids, resolved_power_event_id, event_map, state_res_store
442		):
	482	clock: Clock,
	483	room_id: str,
	484	event_ids: List[str],
	485	resolved_power_event_id: Optional[str],
	486	event_map: Dict[str, EventBase],
	487	state_res_store: "synapse.state.StateResolutionStore",
	488	) -> List[str]:
443	489	"""Returns a sorted list of event_ids sorted by mainline ordering based on
444	490	the given event resolved_power_event_id
445	491
446	492	Args:
447		clock (Clock)
448		room_id (str): room we're working in
449		event_ids (list[str]): Events to sort
450		resolved_power_event_id (str): The final resolved power level event ID
451		event_map (dict[str,FrozenEvent])
452		state_res_store (StateResolutionStore)
453
454		Returns:
455		Deferred[list[str]]: The sorted list
	493	clock
	494	room_id: room we're working in
	495	event_ids: Events to sort
	496	resolved_power_event_id: The final resolved power level event ID
	497	event_map
	498	state_res_store
	499
	500	Returns:
	501	The sorted list
456	502	"""
457	503	if not event_ids:
458	504	# It's possible for there to be no event IDs here to sort, so we can

504	550
505	551
506	552	async def _get_mainline_depth_for_event(
507		event, mainline_map, event_map, state_res_store
508		):
	553	event: EventBase,
	554	mainline_map: Dict[str, int],
	555	event_map: Dict[str, EventBase],
	556	state_res_store: "synapse.state.StateResolutionStore",
	557	) -> int:
509	558	"""Get the mainline depths for the given event based on the mainline map
510	559
511	560	Args:
512		event (FrozenEvent)
513		mainline_map (dict[str, int]): Map from event_id to mainline depth for
514		events in the mainline.
515		event_map (dict[str,FrozenEvent])
516		state_res_store (StateResolutionStore)
517
518		Returns:
519		Deferred[int]
	561	event
	562	mainline_map: Map from event_id to mainline depth for events in the mainline.
	563	event_map
	564	state_res_store
	565
	566	Returns:
	567	The mainline depth
520	568	"""
521	569
522	570	room_id = event.room_id
	571	tmp_event = event # type: Optional[EventBase]
523	572
524	573	# We do an iterative search, replacing `event with the power level in its
525	574	# auth events (if any)
526		while event:
	575	while tmp_event:
527	576	depth = mainline_map.get(event.event_id)
528	577	if depth is not None:
529	578	return depth
530	579
531		auth_events = event.auth_event_ids()
532		event = None
	580	auth_events = tmp_event.auth_event_ids()
	581	tmp_event = None
533	582
534	583	for aid in auth_events:
535	584	aev = await _get_event(
536	585	room_id, aid, event_map, state_res_store, allow_none=True
537	586	)
538	587	if aev and (aev.type, aev.state_key) == (EventTypes.PowerLevels, ""):
539		event = aev
	588	tmp_event = aev
540	589	break
541	590
542	591	# Didn't find a power level auth event, so we just return 0
543	592	return 0
544	593
545	594
546		async def _get_event(room_id, event_id, event_map, state_res_store, allow_none=False):
	595	@overload
	596	async def _get_event(
	597	room_id: str,
	598	event_id: str,
	599	event_map: Dict[str, EventBase],
	600	state_res_store: "synapse.state.StateResolutionStore",
	601	allow_none: Literal[False] = False,
	602	) -> EventBase:
	603	...
	604
	605
	606	@overload
	607	async def _get_event(
	608	room_id: str,
	609	event_id: str,
	610	event_map: Dict[str, EventBase],
	611	state_res_store: "synapse.state.StateResolutionStore",
	612	allow_none: Literal[True],
	613	) -> Optional[EventBase]:
	614	...
	615
	616
	617	async def _get_event(
	618	room_id: str,
	619	event_id: str,
	620	event_map: Dict[str, EventBase],
	621	state_res_store: "synapse.state.StateResolutionStore",
	622	allow_none: bool = False,
	623	) -> Optional[EventBase]:
547	624	"""Helper function to look up event in event_map, falling back to looking
548	625	it up in the store
549	626
550	627	Args:
551		room_id (str)
552		event_id (str)
553		event_map (dict[str,FrozenEvent])
554		state_res_store (StateResolutionStore)
555		allow_none (bool): if the event is not found, return None rather than raising
	628	room_id
	629	event_id
	630	event_map
	631	state_res_store
	632	allow_none: if the event is not found, return None rather than raising
556	633	an exception
557	634
558	635	Returns:
559		Deferred[Optional[FrozenEvent]]
	636	The event, or none if the event does not exist (and allow_none is True).
560	637	"""
561	638	if event_id not in event_map:
562	639	events = await state_res_store.get_events([event_id], allow_rejected=True)

576	653	return event
577	654
578	655
579		def lexicographical_topological_sort(graph, key):
	656	def lexicographical_topological_sort(
	657	graph: Dict[str, Set[str]], key: Callable[[str], Any]
	658	) -> Generator[str, None, None]:
580	659	"""Performs a lexicographic reverse topological sort on the graph.
581	660
582	661	This returns a reverse topological sort (i.e. if node A references B then B

586	665	NOTE: `graph` is modified during the sort.
587	666
588	667	Args:
589		graph (dict[str, set[str]]): A representation of the graph where each
590		node is a key in the dict and its value are the nodes edges.
591		key (func): A function that takes a node and returns a value that is
592		comparable and used to order nodes
	668	graph: A representation of the graph where each node is a key in the
	669	dict and its value are the nodes edges.
	670	key: A function that takes a node and returns a value that is comparable
	671	and used to order nodes
593	672
594	673	Yields:
595		str: The next node in the topological sort
	674	The next node in the topological sort
596	675	"""
597	676
598	677	# Note, this is basically Kahn's algorithm except we look at nodes with no
599	678	# outgoing edges, c.f.
600	679	# https://en.wikipedia.org/wiki/Topological_sorting#Kahn's_algorithm
601	680	outdegree_map = graph
602		reverse_graph = {}
	681	reverse_graph = {} # type: Dict[str, Set[str]]
603	682
604	683	# Lists of nodes with zero out degree. Is actually a tuple of
605	684	# `(key(node), node)` so that sorting does the right thing

+1

-1

synapse/storage/__init__.py less more

36	36	__all__ = ["DataStores", "DataStore"]
37	37
38	38
39		class Storage(object):
	39	class Storage:
40	40	"""The high level interfaces for talking to various storage layers.
41	41	"""
42	42

+3

-4

synapse/storage/_base.py less more

18	18	from abc import ABCMeta
19	19	from typing import Any, Optional
20	20
21		from canonicaljson import json
22
23	21	from synapse.storage.database import LoggingTransaction # noqa: F401
24	22	from synapse.storage.database import make_in_list_sql_clause # noqa: F401
25	23	from synapse.storage.database import DatabasePool
26	24	from synapse.types import Collection, get_domain_from_id
	25	from synapse.util import json_decoder
27	26
28	27	logger = logging.getLogger(__name__)
29	28

98	97	if isinstance(db_content, memoryview):
99	98	db_content = db_content.tobytes()
100	99
101		# Decode it to a Unicode string before feeding it to json.loads, since
	100	# Decode it to a Unicode string before feeding it to the JSON decoder, since
102	101	# Python 3.5 does not support deserializing bytes.
103	102	if isinstance(db_content, (bytes, bytearray)):
104	103	db_content = db_content.decode("utf8")
105	104
106	105	try:
107		return json.loads(db_content)
	106	return json_decoder.decode(db_content)
108	107	except Exception:
109	108	logging.warning("Tried to decode '%r' as JSON and failed", db_content)
110	109	raise

+16

-20

synapse/storage/background_updates.py less more

15	15	import logging
16	16	from typing import Optional
17	17
18		from canonicaljson import json
19
20		from twisted.internet import defer
21
22	18	from synapse.metrics.background_process_metrics import run_as_background_process
	19	from synapse.util import json_encoder
23	20
24	21	from . import engines
25	22
26	23	logger = logging.getLogger(__name__)
27	24
28	25
29		class BackgroundUpdatePerformance(object):
	26	class BackgroundUpdatePerformance:
30	27	"""Tracks the how long a background update is taking to update its items"""
31	28
32	29	def __init__(self, name):

73	70	return float(self.total_item_count) / float(self.total_duration_ms)
74	71
75	72
76		class BackgroundUpdater(object):
	73	class BackgroundUpdater:
77	74	""" Background updates are updates to the database that run in the
78	75	background. Each update processes a batch of data at once. We attempt to
79	76	limit the impact of each update by monitoring how long each batch takes to

307	304	update_name (str): Name of update
308	305	"""
309	306
310		@defer.inlineCallbacks
311		def noop_update(progress, batch_size):
312		yield self._end_background_update(update_name)
	307	async def noop_update(progress, batch_size):
	308	await self._end_background_update(update_name)
313	309	return 1
314	310
315	311	self.register_background_update_handler(update_name, noop_update)

408	404	else:
409	405	runner = create_index_sqlite
410	406
411		@defer.inlineCallbacks
412		def updater(progress, batch_size):
	407	async def updater(progress, batch_size):
413	408	if runner is not None:
414	409	logger.info("Adding index %s to %s", index_name, table)
415		yield self.db_pool.runWithConnection(runner)
416		yield self._end_background_update(update_name)
	410	await self.db_pool.runWithConnection(runner)
	411	await self._end_background_update(update_name)
417	412	return 1
418	413
419	414	self.register_background_update_handler(update_name, updater)
420	415
421		def _end_background_update(self, update_name):
	416	async def _end_background_update(self, update_name: str) -> None:
422	417	"""Removes a completed background update task from the queue.
423	418
424	419	Args:
425		update_name(str): The name of the completed task to remove
	420	update_name:: The name of the completed task to remove
	421
426	422	Returns:
427		A deferred that completes once the task is removed.
	423	None, completes once the task is removed.
428	424	"""
429	425	if update_name != self._current_background_update:
430	426	raise Exception(

432	428	% update_name
433	429	)
434	430	self._current_background_update = None
435		return self.db_pool.simple_delete_one(
	431	await self.db_pool.simple_delete_one(
436	432	"background_updates", keyvalues={"update_name": update_name}
437	433	)
438	434
439		def _background_update_progress(self, update_name: str, progress: dict):
	435	async def _background_update_progress(self, update_name: str, progress: dict):
440	436	"""Update the progress of a background update
441	437
442	438	Args:

444	440	progress: The progress of the update.
445	441	"""
446	442
447		return self.db_pool.runInteraction(
	443	return await self.db_pool.runInteraction(
448	444	"background_update_progress",
449	445	self._background_update_progress_txn,
450	446	update_name,

460	456	progress(dict): The progress of the update.
461	457	"""
462	458
463		progress_json = json.dumps(progress)
	459	progress_json = json_encoder.encode(progress)
464	460
465	461	self.db_pool.simple_update_one_txn(
466	462	txn,

+546

-313

synapse/storage/database.py less more

27	27	Optional,
28	28	Tuple,
29	29	TypeVar,
	30	cast,
	31	overload,
30	32	)
31	33
32	34	from prometheus_client import Histogram
	35	from typing_extensions import Literal
33	36
34	37	from twisted.enterprise import adbapi
35		from twisted.internet import defer
36	38
37	39	from synapse.api.errors import StoreError
38	40	from synapse.config.database import DatabaseConnectionConfig

124	126	method.
125	127
126	128	Args:
127		txn: The database transcation object to wrap.
	129	txn: The database transaction object to wrap.
128	130	name: The name of this transactions for logging.
129	131	database_engine
130	132	after_callbacks: A list that callbacks will be appended to

159	161	self.after_callbacks = after_callbacks
160	162	self.exception_callbacks = exception_callbacks
161	163
162		def call_after(self, callback: "Callable[..., None]", args, *kwargs):
	164	def call_after(self, callback: "Callable[..., None]", args: Any, *kwargs: Any):
163	165	"""Call the given callback on the main twisted thread after the
164	166	transaction has finished. Used to invalidate the caches on the
165	167	correct thread.

170	172	assert self.after_callbacks is not None
171	173	self.after_callbacks.append((callback, args, kwargs))
172	174
173		def call_on_exception(self, callback: "Callable[..., None]", args, *kwargs):
	175	def call_on_exception(
	176	self, callback: "Callable[..., None]", args: Any, *kwargs: Any
	177	):
174	178	# if self.exception_callbacks is None, that means that whatever constructed the
175	179	# LoggingTransaction isn't expecting there to be any callbacks; assert that
176	180	# is not the case.

194	198	def description(self) -> Any:
195	199	return self.txn.description
196	200
197		def execute_batch(self, sql, args):
	201	def execute_batch(self, sql: str, args: Iterable[Iterable[Any]]) -> None:
198	202	if isinstance(self.database_engine, PostgresEngine):
199	203	from psycopg2.extras import execute_batch # type: ignore
200	204

203	207	for val in args:
204	208	self.execute(sql, val)
205	209
206		def execute(self, sql: str, *args: Any):
	210	def execute(self, sql: str, *args: Any) -> None:
207	211	self._do_execute(self.txn.execute, sql, *args)
208	212
209		def executemany(self, sql: str, *args: Any):
	213	def executemany(self, sql: str, *args: Any) -> None:
210	214	self._do_execute(self.txn.executemany, sql, *args)
211	215
212	216	def _make_sql_one_line(self, sql: str) -> str:
213	217	"Strip newlines out of SQL so that the loggers in the DB are on one line"
214	218	return " ".join(line.strip() for line in sql.splitlines() if line.strip())
215	219
216		def _do_execute(self, func, sql, *args):
	220	def _do_execute(self, func, sql: str, *args: Any) -> None:
217	221	sql = self._make_sql_one_line(sql)
218	222
219	223	# TODO(paul): Maybe use 'info' and 'debug' for values?

239	243	sql_logger.debug("[SQL time] {%s} %f sec", self.name, secs)
240	244	sql_query_timer.labels(sql.split()[0]).observe(secs)
241	245
242		def close(self):
	246	def close(self) -> None:
243	247	self.txn.close()
244	248
245	249
246		class PerformanceCounters(object):
	250	class PerformanceCounters:
247	251	def __init__(self):
248	252	self.current_counters = {}
249	253	self.previous_counters = {}
250	254
251		def update(self, key, duration_secs):
	255	def update(self, key: str, duration_secs: float) -> None:
252	256	count, cum_time = self.current_counters.get(key, (0, 0))
253	257	count += 1
254	258	cum_time += duration_secs
255	259	self.current_counters[key] = (count, cum_time)
256	260
257		def interval(self, interval_duration_secs, limit=3):
	261	def interval(self, interval_duration_secs: float, limit: int = 3) -> str:
258	262	counters = []
259	263	for name, (count, cum_time) in self.current_counters.items():
260	264	prev_count, prev_time = self.previous_counters.get(name, (0, 0))

278	282	return top_n_counters
279	283
280	284
281		class DatabasePool(object):
	285	R = TypeVar("R")
	286
	287
	288	class DatabasePool:
282	289	"""Wraps a single physical database and connection pool.
283	290
284	291	A single database may be used by multiple data stores.

326	333	self._check_safe_to_upsert,
327	334	)
328	335
329		def is_running(self):
	336	def is_running(self) -> bool:
330	337	"""Is the database pool currently running
331	338	"""
332	339	return self._db_pool.running
333	340
334		@defer.inlineCallbacks
335		def _check_safe_to_upsert(self):
	341	async def _check_safe_to_upsert(self) -> None:
336	342	"""
337	343	Is it safe to use native UPSERT?
338	344

341	347
342	348	If the background updates have not completed, wait 15 sec and check again.
343	349	"""
344		updates = yield self.simple_select_list(
	350	updates = await self.simple_select_list(
345	351	"background_updates",
346	352	keyvalues=None,
347	353	retcols=["update_name"],

363	369	self._check_safe_to_upsert,
364	370	)
365	371
366		def start_profiling(self):
	372	def start_profiling(self) -> None:
367	373	self._previous_loop_ts = monotonic_time()
368	374
369	375	def loop():

387	393	self._clock.looping_call(loop, 10000)
388	394
389	395	def new_transaction(
390		self, conn, desc, after_callbacks, exception_callbacks, func, args, *kwargs
391		):
	396	self,
	397	conn: Connection,
	398	desc: str,
	399	after_callbacks: List[_CallbackListEntry],
	400	exception_callbacks: List[_CallbackListEntry],
	401	func: "Callable[..., R]",
	402	*args: Any,
	403	**kwargs: Any
	404	) -> R:
392	405	start = monotonic_time()
393	406	txn_id = self._TXN_ID
394	407

493	506	self._txn_perf_counters.update(desc, duration)
494	507	sql_txn_timer.labels(desc).observe(duration)
495	508
496		@defer.inlineCallbacks
497		def runInteraction(self, desc: str, func: Callable, args: Any, *kwargs: Any):
	509	async def runInteraction(
	510	self, desc: str, func: "Callable[..., R]", args: Any, *kwargs: Any
	511	) -> R:
498	512	"""Starts a transaction on the database and runs a given function
499	513
500	514	Arguments:

507	521	kwargs: named args to pass to `func`
508	522
509	523	Returns:
510		Deferred: The result of func
	524	The result of func
511	525	"""
512	526	after_callbacks = [] # type: List[_CallbackListEntry]
513	527	exception_callbacks = [] # type: List[_CallbackListEntry]

516	530	logger.warning("Starting db txn '%s' from sentinel context", desc)
517	531
518	532	try:
519		result = yield self.runWithConnection(
	533	result = await self.runWithConnection(
520	534	self.new_transaction,
521	535	desc,
522	536	after_callbacks,

533	547	after_callback(after_args, *after_kwargs)
534	548	raise
535	549
536		return result
537
538		@defer.inlineCallbacks
539		def runWithConnection(self, func: Callable, args: Any, *kwargs: Any):
	550	return cast(R, result)
	551
	552	async def runWithConnection(
	553	self, func: "Callable[..., R]", args: Any, *kwargs: Any
	554	) -> R:
540	555	"""Wraps the .runWithConnection() method on the underlying db_pool.
541	556
542	557	Arguments:

547	562	kwargs: named args to pass to `func`
548	563
549	564	Returns:
550		Deferred: The result of func
	565	The result of func
551	566	"""
552	567	parent_context = current_context() # type: Optional[LoggingContextOrSentinel]
553	568	if not parent_context:

570	585
571	586	return func(conn, args, *kwargs)
572	587
573		result = yield make_deferred_yieldable(
	588	return await make_deferred_yieldable(
574	589	self._db_pool.runWithConnection(inner_func, args, *kwargs)
575	590	)
576	591
577		return result
578
579	592	@staticmethod
580		def cursor_to_dict(cursor):
	593	def cursor_to_dict(cursor: Cursor) -> List[Dict[str, Any]]:
581	594	"""Converts a SQL cursor into an list of dicts.
582	595
583	596	Args:
584		cursor : The DBAPI cursor which has executed a query.
	597	cursor: The DBAPI cursor which has executed a query.
585	598	Returns:
586	599	A list of dicts where the key is the column header.
587	600	"""

589	602	results = [dict(zip(col_headers, row)) for row in cursor]
590	603	return results
591	604
592		def execute(self, desc, decoder, query, *args):
	605	@overload
	606	async def execute(
	607	self, desc: str, decoder: Literal[None], query: str, *args: Any
	608	) -> List[Tuple[Any, ...]]:
	609	...
	610
	611	@overload
	612	async def execute(
	613	self, desc: str, decoder: Callable[[Cursor], R], query: str, *args: Any
	614	) -> R:
	615	...
	616
	617	async def execute(
	618	self,
	619	desc: str,
	620	decoder: Optional[Callable[[Cursor], R]],
	621	query: str,
	622	*args: Any
	623	) -> R:
593	624	"""Runs a single query for a result set.
594	625
595	626	Args:
	627	desc: description of the transaction, for logging and metrics
596	628	decoder - The function which can resolve the cursor results to
597	629	something meaningful.
598	630	query - The query string to execute

608	640	else:
609	641	return txn.fetchall()
610	642
611		return self.runInteraction(desc, interaction)
	643	return await self.runInteraction(desc, interaction)
612	644
613	645	# "Simple" SQL API methods that operate on a single table with no JOINs,
614	646	# no complex WHERE clauses, just a dict of values for columns.
615	647
616		@defer.inlineCallbacks
617		def simple_insert(self, table, values, or_ignore=False, desc="simple_insert"):
	648	async def simple_insert(
	649	self,
	650	table: str,
	651	values: Dict[str, Any],
	652	or_ignore: bool = False,
	653	desc: str = "simple_insert",
	654	) -> bool:
618	655	"""Executes an INSERT query on the named table.
619	656
620	657	Args:
621		table : string giving the table name
622		values : dict of new column names and values for them
623		or_ignore : bool stating whether an exception should be raised
	658	table: string giving the table name
	659	values: dict of new column names and values for them
	660	or_ignore: bool stating whether an exception should be raised
624	661	when a conflicting row already exists. If True, False will be
625	662	returned by the function instead
626		desc : string giving a description of the transaction
	663	desc: description of the transaction, for logging and metrics
627	664
628	665	Returns:
629		bool: Whether the row was inserted or not. Only useful when
630		`or_ignore` is True
	666	Whether the row was inserted or not. Only useful when `or_ignore` is True
631	667	"""
632	668	try:
633		yield self.runInteraction(desc, self.simple_insert_txn, table, values)
	669	await self.runInteraction(desc, self.simple_insert_txn, table, values)
634	670	except self.engine.module.IntegrityError:
635	671	# We have to do or_ignore flag at this layer, since we can't reuse
636	672	# a cursor after we receive an error from the db.

640	676	return True
641	677
642	678	@staticmethod
643		def simple_insert_txn(txn, table, values):
	679	def simple_insert_txn(
	680	txn: LoggingTransaction, table: str, values: Dict[str, Any]
	681	) -> None:
644	682	keys, vals = zip(*values.items())
645	683
646	684	sql = "INSERT INTO %s (%s) VALUES(%s)" % (

651	689
652	690	txn.execute(sql, vals)
653	691
654		def simple_insert_many(self, table, values, desc):
655		return self.runInteraction(desc, self.simple_insert_many_txn, table, values)
	692	async def simple_insert_many(
	693	self, table: str, values: List[Dict[str, Any]], desc: str
	694	) -> None:
	695	"""Executes an INSERT query on the named table.
	696
	697	Args:
	698	table: string giving the table name
	699	values: dict of new column names and values for them
	700	desc: description of the transaction, for logging and metrics
	701	"""
	702	await self.runInteraction(desc, self.simple_insert_many_txn, table, values)
656	703
657	704	@staticmethod
658		def simple_insert_many_txn(txn, table, values):
	705	def simple_insert_many_txn(
	706	txn: LoggingTransaction, table: str, values: List[Dict[str, Any]]
	707	) -> None:
	708	"""Executes an INSERT query on the named table.
	709
	710	Args:
	711	txn: The transaction to use.
	712	table: string giving the table name
	713	values: dict of new column names and values for them
	714	"""
659	715	if not values:
660	716	return
661	717

683	739
684	740	txn.executemany(sql, vals)
685	741
686		@defer.inlineCallbacks
687		def simple_upsert(
688		self,
689		table,
690		keyvalues,
691		values,
692		insertion_values={},
693		desc="simple_upsert",
694		lock=True,
695		):
	742	async def simple_upsert(
	743	self,
	744	table: str,
	745	keyvalues: Dict[str, Any],
	746	values: Dict[str, Any],
	747	insertion_values: Dict[str, Any] = {},
	748	desc: str = "simple_upsert",
	749	lock: bool = True,
	750	) -> Optional[bool]:
696	751	"""
697	752
698	753	`lock` should generally be set to True (the default), but can be set

706	761	this table.
707	762
708	763	Args:
709		table (str): The table to upsert into
710		keyvalues (dict): The unique key columns and their new values
711		values (dict): The nonunique columns and their new values
712		insertion_values (dict): additional key/values to use only when
713		inserting
714		lock (bool): True to lock the table when doing the upsert.
	764	table: The table to upsert into
	765	keyvalues: The unique key columns and their new values
	766	values: The nonunique columns and their new values
	767	insertion_values: additional key/values to use only when inserting
	768	desc: description of the transaction, for logging and metrics
	769	lock: True to lock the table when doing the upsert.
715	770	Returns:
716		Deferred(None or bool): Native upserts always return None. Emulated
717		upserts return True if a new entry was created, False if an existing
718		one was updated.
	771	Native upserts always return None. Emulated upserts return True if a
	772	new entry was created, False if an existing one was updated.
719	773	"""
720	774	attempts = 0
721	775	while True:
722	776	try:
723		result = yield self.runInteraction(
	777	return await self.runInteraction(
724	778	desc,
725	779	self.simple_upsert_txn,
726	780	table,

729	783	insertion_values,
730	784	lock=lock,
731	785	)
732		return result
733	786	except self.engine.module.IntegrityError as e:
734	787	attempts += 1
735	788	if attempts >= 5:

743	796	)
744	797
745	798	def simple_upsert_txn(
746		self, txn, table, keyvalues, values, insertion_values={}, lock=True
747		):
	799	self,
	800	txn: LoggingTransaction,
	801	table: str,
	802	keyvalues: Dict[str, Any],
	803	values: Dict[str, Any],
	804	insertion_values: Dict[str, Any] = {},
	805	lock: bool = True,
	806	) -> Optional[bool]:
748	807	"""
749	808	Pick the UPSERT method which works best on the platform. Either the
750	809	native one (Pg9.5+, recent SQLites), or fall back to an emulated method.
751	810
752	811	Args:
753	812	txn: The transaction to use.
754		table (str): The table to upsert into
755		keyvalues (dict): The unique key tables and their new values
756		values (dict): The nonunique columns and their new values
757		insertion_values (dict): additional key/values to use only when
758		inserting
759		lock (bool): True to lock the table when doing the upsert.
	813	table: The table to upsert into
	814	keyvalues: The unique key tables and their new values
	815	values: The nonunique columns and their new values
	816	insertion_values: additional key/values to use only when inserting
	817	lock: True to lock the table when doing the upsert.
760	818	Returns:
761		None or bool: Native upserts always return None. Emulated
762		upserts return True if a new entry was created, False if an existing
763		one was updated.
	819	Native upserts always return None. Emulated upserts return True if a
	820	new entry was created, False if an existing one was updated.
764	821	"""
765	822	if self.engine.can_native_upsert and table not in self._unsafe_to_upsert_tables:
766		return self.simple_upsert_txn_native_upsert(
	823	self.simple_upsert_txn_native_upsert(
767	824	txn, table, keyvalues, values, insertion_values=insertion_values
768	825	)
	826	return None
769	827	else:
770	828	return self.simple_upsert_txn_emulated(
771	829	txn,

777	835	)
778	836
779	837	def simple_upsert_txn_emulated(
780		self, txn, table, keyvalues, values, insertion_values={}, lock=True
781		):
782		"""
783		Args:
784		table (str): The table to upsert into
785		keyvalues (dict): The unique key tables and their new values
786		values (dict): The nonunique columns and their new values
787		insertion_values (dict): additional key/values to use only when
788		inserting
789		lock (bool): True to lock the table when doing the upsert.
	838	self,
	839	txn: LoggingTransaction,
	840	table: str,
	841	keyvalues: Dict[str, Any],
	842	values: Dict[str, Any],
	843	insertion_values: Dict[str, Any] = {},
	844	lock: bool = True,
	845	) -> bool:
	846	"""
	847	Args:
	848	table: The table to upsert into
	849	keyvalues: The unique key tables and their new values
	850	values: The nonunique columns and their new values
	851	insertion_values: additional key/values to use only when inserting
	852	lock: True to lock the table when doing the upsert.
790	853	Returns:
791		bool: Return True if a new entry was created, False if an existing
	854	Returns True if a new entry was created, False if an existing
792	855	one was updated.
793	856	"""
794	857	# We need to lock the table :(, unless we're really careful

846	909	return True
847	910
848	911	def simple_upsert_txn_native_upsert(
849		self, txn, table, keyvalues, values, insertion_values={}
850		):
	912	self,
	913	txn: LoggingTransaction,
	914	table: str,
	915	keyvalues: Dict[str, Any],
	916	values: Dict[str, Any],
	917	insertion_values: Dict[str, Any] = {},
	918	) -> None:
851	919	"""
852	920	Use the native UPSERT functionality in recent PostgreSQL versions.
853	921
854	922	Args:
855		table (str): The table to upsert into
856		keyvalues (dict): The unique key tables and their new values
857		values (dict): The nonunique columns and their new values
858		insertion_values (dict): additional key/values to use only when
859		inserting
860		Returns:
861		None
	923	table: The table to upsert into
	924	keyvalues: The unique key tables and their new values
	925	values: The nonunique columns and their new values
	926	insertion_values: additional key/values to use only when inserting
862	927	"""
863	928	allvalues = {} # type: Dict[str, Any]
864	929	allvalues.update(keyvalues)

988	1053
989	1054	return txn.execute_batch(sql, args)
990	1055
991		def simple_select_one(
992		self, table, keyvalues, retcols, allow_none=False, desc="simple_select_one"
993		):
	1056	@overload
	1057	async def simple_select_one(
	1058	self,
	1059	table: str,
	1060	keyvalues: Dict[str, Any],
	1061	retcols: Iterable[str],
	1062	allow_none: Literal[False] = False,
	1063	desc: str = "simple_select_one",
	1064	) -> Dict[str, Any]:
	1065	...
	1066
	1067	@overload
	1068	async def simple_select_one(
	1069	self,
	1070	table: str,
	1071	keyvalues: Dict[str, Any],
	1072	retcols: Iterable[str],
	1073	allow_none: Literal[True] = True,
	1074	desc: str = "simple_select_one",
	1075	) -> Optional[Dict[str, Any]]:
	1076	...
	1077
	1078	async def simple_select_one(
	1079	self,
	1080	table: str,
	1081	keyvalues: Dict[str, Any],
	1082	retcols: Iterable[str],
	1083	allow_none: bool = False,
	1084	desc: str = "simple_select_one",
	1085	) -> Optional[Dict[str, Any]]:
994	1086	"""Executes a SELECT query on the named table, which is expected to
995	1087	return a single row, returning multiple columns from it.
996	1088
997	1089	Args:
998		table : string giving the table name
999		keyvalues : dict of column names and values to select the row with
1000		retcols : list of strings giving the names of the columns to return
1001
1002		allow_none : If true, return None instead of failing if the SELECT
1003		statement returns no rows
1004		"""
1005		return self.runInteraction(
	1090	table: string giving the table name
	1091	keyvalues: dict of column names and values to select the row with
	1092	retcols: list of strings giving the names of the columns to return
	1093	allow_none: If true, return None instead of failing if the SELECT
	1094	statement returns no rows
	1095	desc: description of the transaction, for logging and metrics
	1096	"""
	1097	return await self.runInteraction(
1006	1098	desc, self.simple_select_one_txn, table, keyvalues, retcols, allow_none
1007	1099	)
1008	1100
1009		def simple_select_one_onecol(
1010		self,
1011		table,
1012		keyvalues,
1013		retcol,
1014		allow_none=False,
1015		desc="simple_select_one_onecol",
1016		):
	1101	@overload
	1102	async def simple_select_one_onecol(
	1103	self,
	1104	table: str,
	1105	keyvalues: Dict[str, Any],
	1106	retcol: str,
	1107	allow_none: Literal[False] = False,
	1108	desc: str = "simple_select_one_onecol",
	1109	) -> Any:
	1110	...
	1111
	1112	@overload
	1113	async def simple_select_one_onecol(
	1114	self,
	1115	table: str,
	1116	keyvalues: Dict[str, Any],
	1117	retcol: str,
	1118	allow_none: Literal[True] = True,
	1119	desc: str = "simple_select_one_onecol",
	1120	) -> Optional[Any]:
	1121	...
	1122
	1123	async def simple_select_one_onecol(
	1124	self,
	1125	table: str,
	1126	keyvalues: Dict[str, Any],
	1127	retcol: str,
	1128	allow_none: bool = False,
	1129	desc: str = "simple_select_one_onecol",
	1130	) -> Optional[Any]:
1017	1131	"""Executes a SELECT query on the named table, which is expected to
1018	1132	return a single row, returning a single column from it.
1019	1133
1020	1134	Args:
1021		table : string giving the table name
1022		keyvalues : dict of column names and values to select the row with
1023		retcol : string giving the name of the column to return
1024		"""
1025		return self.runInteraction(
	1135	table: string giving the table name
	1136	keyvalues: dict of column names and values to select the row with
	1137	retcol: string giving the name of the column to return
	1138	allow_none: If true, return None instead of failing if the SELECT
	1139	statement returns no rows
	1140	desc: description of the transaction, for logging and metrics
	1141	"""
	1142	return await self.runInteraction(
1026	1143	desc,
1027	1144	self.simple_select_one_onecol_txn,
1028	1145	table,

1031	1148	allow_none=allow_none,
1032	1149	)
1033	1150
	1151	@overload
1034	1152	@classmethod
1035	1153	def simple_select_one_onecol_txn(
1036		cls, txn, table, keyvalues, retcol, allow_none=False
1037		):
	1154	cls,
	1155	txn: LoggingTransaction,
	1156	table: str,
	1157	keyvalues: Dict[str, Any],
	1158	retcol: str,
	1159	allow_none: Literal[False] = False,
	1160	) -> Any:
	1161	...
	1162
	1163	@overload
	1164	@classmethod
	1165	def simple_select_one_onecol_txn(
	1166	cls,
	1167	txn: LoggingTransaction,
	1168	table: str,
	1169	keyvalues: Dict[str, Any],
	1170	retcol: str,
	1171	allow_none: Literal[True] = True,
	1172	) -> Optional[Any]:
	1173	...
	1174
	1175	@classmethod
	1176	def simple_select_one_onecol_txn(
	1177	cls,
	1178	txn: LoggingTransaction,
	1179	table: str,
	1180	keyvalues: Dict[str, Any],
	1181	retcol: str,
	1182	allow_none: bool = False,
	1183	) -> Optional[Any]:
1038	1184	ret = cls.simple_select_onecol_txn(
1039	1185	txn, table=table, keyvalues=keyvalues, retcol=retcol
1040	1186	)

1048	1194	raise StoreError(404, "No row found")
1049	1195
1050	1196	@staticmethod
1051		def simple_select_onecol_txn(txn, table, keyvalues, retcol):
	1197	def simple_select_onecol_txn(
	1198	txn: LoggingTransaction, table: str, keyvalues: Dict[str, Any], retcol: str,
	1199	) -> List[Any]:
1052	1200	sql = ("SELECT %(retcol)s FROM %(table)s") % {"retcol": retcol, "table": table}
1053	1201
1054	1202	if keyvalues:

1059	1207
1060	1208	return [r[0] for r in txn]
1061	1209
1062		def simple_select_onecol(
1063		self, table, keyvalues, retcol, desc="simple_select_onecol"
1064		):
	1210	async def simple_select_onecol(
	1211	self,
	1212	table: str,
	1213	keyvalues: Optional[Dict[str, Any]],
	1214	retcol: str,
	1215	desc: str = "simple_select_onecol",
	1216	) -> List[Any]:
1065	1217	"""Executes a SELECT query on the named table, which returns a list
1066	1218	comprising of the values of the named column from the selected rows.
1067	1219
1068	1220	Args:
1069		table (str): table name
1070		keyvalues (dict\|None): column names and values to select the rows with
1071		retcol (str): column whos value we wish to retrieve.
	1221	table: table name
	1222	keyvalues: column names and values to select the rows with
	1223	retcol: column whos value we wish to retrieve.
	1224	desc: description of the transaction, for logging and metrics
1072	1225
1073	1226	Returns:
1074		Deferred: Results in a list
1075		"""
1076		return self.runInteraction(
	1227	Results in a list
	1228	"""
	1229	return await self.runInteraction(
1077	1230	desc, self.simple_select_onecol_txn, table, keyvalues, retcol
1078	1231	)
1079	1232
1080		def simple_select_list(self, table, keyvalues, retcols, desc="simple_select_list"):
	1233	async def simple_select_list(
	1234	self,
	1235	table: str,
	1236	keyvalues: Optional[Dict[str, Any]],
	1237	retcols: Iterable[str],
	1238	desc: str = "simple_select_list",
	1239	) -> List[Dict[str, Any]]:
1081	1240	"""Executes a SELECT query on the named table, which may return zero or
1082	1241	more rows, returning the result as a list of dicts.
1083	1242
1084	1243	Args:
1085		table (str): the table name
1086		keyvalues (dict[str, Any] \| None):
	1244	table: the table name
	1245	keyvalues:
1087	1246	column names and values to select the rows with, or None to not
1088	1247	apply a WHERE clause.
1089		retcols (iterable[str]): the names of the columns to return
	1248	retcols: the names of the columns to return
	1249	desc: description of the transaction, for logging and metrics
	1250
1090	1251	Returns:
1091		defer.Deferred: resolves to list[dict[str, Any]]
1092		"""
1093		return self.runInteraction(
	1252	A list of dictionaries.
	1253	"""
	1254	return await self.runInteraction(
1094	1255	desc, self.simple_select_list_txn, table, keyvalues, retcols
1095	1256	)
1096	1257
1097	1258	@classmethod
1098		def simple_select_list_txn(cls, txn, table, keyvalues, retcols):
	1259	def simple_select_list_txn(
	1260	cls,
	1261	txn: LoggingTransaction,
	1262	table: str,
	1263	keyvalues: Optional[Dict[str, Any]],
	1264	retcols: Iterable[str],
	1265	) -> List[Dict[str, Any]]:
1099	1266	"""Executes a SELECT query on the named table, which may return zero or
1100	1267	more rows, returning the result as a list of dicts.
1101	1268
1102	1269	Args:
1103		txn : Transaction object
1104		table (str): the table name
1105		keyvalues (dict[str, T] \| None):
	1270	txn: Transaction object
	1271	table: the table name
	1272	keyvalues:
1106	1273	column names and values to select the rows with, or None to not
1107	1274	apply a WHERE clause.
1108		retcols (iterable[str]): the names of the columns to return
	1275	retcols: the names of the columns to return
1109	1276	"""
1110	1277	if keyvalues:
1111	1278	sql = "SELECT %s FROM %s WHERE %s" % (

1120	1287
1121	1288	return cls.cursor_to_dict(txn)
1122	1289
1123		@defer.inlineCallbacks
1124		def simple_select_many_batch(
1125		self,
1126		table,
1127		column,
1128		iterable,
1129		retcols,
1130		keyvalues={},
1131		desc="simple_select_many_batch",
1132		batch_size=100,
1133		):
	1290	async def simple_select_many_batch(
	1291	self,
	1292	table: str,
	1293	column: str,
	1294	iterable: Iterable[Any],
	1295	retcols: Iterable[str],
	1296	keyvalues: Dict[str, Any] = {},
	1297	desc: str = "simple_select_many_batch",
	1298	batch_size: int = 100,
	1299	) -> List[Any]:
1134	1300	"""Executes a SELECT query on the named table, which may return zero or
1135	1301	more rows, returning the result as a list of dicts.
1136	1302
1137		Filters rows by if value of `column` is in `iterable`.
1138
1139		Args:
1140		table : string giving the table name
1141		column : column name to test for inclusion against `iterable`
1142		iterable : list
1143		keyvalues : dict of column names and values to select the rows with
1144		retcols : list of strings giving the names of the columns to return
	1303	Filters rows by whether the value of `column` is in `iterable`.
	1304
	1305	Args:
	1306	table: string giving the table name
	1307	column: column name to test for inclusion against `iterable`
	1308	iterable: list
	1309	retcols: list of strings giving the names of the columns to return
	1310	keyvalues: dict of column names and values to select the rows with
	1311	desc: description of the transaction, for logging and metrics
	1312	batch_size: the number of rows for each select query
1145	1313	"""
1146	1314	results = [] # type: List[Dict[str, Any]]
1147	1315

1155	1323	it_list[i : i + batch_size] for i in range(0, len(it_list), batch_size)
1156	1324	]
1157	1325	for chunk in chunks:
1158		rows = yield self.runInteraction(
	1326	rows = await self.runInteraction(
1159	1327	desc,
1160	1328	self.simple_select_many_txn,
1161	1329	table,

1170	1338	return results
1171	1339
1172	1340	@classmethod
1173		def simple_select_many_txn(cls, txn, table, column, iterable, keyvalues, retcols):
	1341	def simple_select_many_txn(
	1342	cls,
	1343	txn: LoggingTransaction,
	1344	table: str,
	1345	column: str,
	1346	iterable: Iterable[Any],
	1347	keyvalues: Dict[str, Any],
	1348	retcols: Iterable[str],
	1349	) -> List[Dict[str, Any]]:
1174	1350	"""Executes a SELECT query on the named table, which may return zero or
1175	1351	more rows, returning the result as a list of dicts.
1176	1352
1177		Filters rows by if value of `column` is in `iterable`.
1178
1179		Args:
1180		txn : Transaction object
1181		table : string giving the table name
1182		column : column name to test for inclusion against `iterable`
1183		iterable : list
1184		keyvalues : dict of column names and values to select the rows with
1185		retcols : list of strings giving the names of the columns to return
	1353	Filters rows by whether the value of `column` is in `iterable`.
	1354
	1355	Args:
	1356	txn: Transaction object
	1357	table: string giving the table name
	1358	column: column name to test for inclusion against `iterable`
	1359	iterable: list
	1360	keyvalues: dict of column names and values to select the rows with
	1361	retcols: list of strings giving the names of the columns to return
1186	1362	"""
1187	1363	if not iterable:
1188	1364	return []

1203	1379	txn.execute(sql, values)
1204	1380	return cls.cursor_to_dict(txn)
1205	1381
1206		def simple_update(self, table, keyvalues, updatevalues, desc):
1207		return self.runInteraction(
	1382	async def simple_update(
	1383	self,
	1384	table: str,
	1385	keyvalues: Dict[str, Any],
	1386	updatevalues: Dict[str, Any],
	1387	desc: str,
	1388	) -> int:
	1389	return await self.runInteraction(
1208	1390	desc, self.simple_update_txn, table, keyvalues, updatevalues
1209	1391	)
1210	1392
1211	1393	@staticmethod
1212		def simple_update_txn(txn, table, keyvalues, updatevalues):
	1394	def simple_update_txn(
	1395	txn: LoggingTransaction,
	1396	table: str,
	1397	keyvalues: Dict[str, Any],
	1398	updatevalues: Dict[str, Any],
	1399	) -> int:
1213	1400	if keyvalues:
1214	1401	where = "WHERE %s" % " AND ".join("%s = ?" % k for k in keyvalues.keys())
1215	1402	else:

1225	1412
1226	1413	return txn.rowcount
1227	1414
1228		def simple_update_one(
1229		self, table, keyvalues, updatevalues, desc="simple_update_one"
1230		):
	1415	async def simple_update_one(
	1416	self,
	1417	table: str,
	1418	keyvalues: Dict[str, Any],
	1419	updatevalues: Dict[str, Any],
	1420	desc: str = "simple_update_one",
	1421	) -> None:
1231	1422	"""Executes an UPDATE query on the named table, setting new values for
1232	1423	columns in a row matching the key values.
1233	1424
1234	1425	Args:
1235		table : string giving the table name
1236		keyvalues : dict of column names and values to select the row with
1237		updatevalues : dict giving column names and values to update
1238		retcols : optional list of column names to return
1239
1240		If present, retcols gives a list of column names on which to perform
1241		a SELECT statement before performing the UPDATE statement. The values
1242		of these will be returned in a dict.
1243
1244		These are performed within the same transaction, allowing an atomic
1245		get-and-set. This can be used to implement compare-and-set by putting
1246		the update column in the 'keyvalues' dict as well.
1247		"""
1248		return self.runInteraction(
	1426	table: string giving the table name
	1427	keyvalues: dict of column names and values to select the row with
	1428	updatevalues: dict giving column names and values to update
	1429	desc: description of the transaction, for logging and metrics
	1430	"""
	1431	await self.runInteraction(
1249	1432	desc, self.simple_update_one_txn, table, keyvalues, updatevalues
1250	1433	)
1251	1434
1252	1435	@classmethod
1253		def simple_update_one_txn(cls, txn, table, keyvalues, updatevalues):
	1436	def simple_update_one_txn(
	1437	cls,
	1438	txn: LoggingTransaction,
	1439	table: str,
	1440	keyvalues: Dict[str, Any],
	1441	updatevalues: Dict[str, Any],
	1442	) -> None:
1254	1443	rowcount = cls.simple_update_txn(txn, table, keyvalues, updatevalues)
1255	1444
1256	1445	if rowcount == 0:

1258	1447	if rowcount > 1:
1259	1448	raise StoreError(500, "More than one row matched (%s)" % (table,))
1260	1449
	1450	# Ideally we could use the overload decorator here to specify that the
	1451	# return type is only optional if allow_none is True, but this does not work
	1452	# when you call a static method from an instance.
	1453	# See https://github.com/python/mypy/issues/7781
1261	1454	@staticmethod
1262		def simple_select_one_txn(txn, table, keyvalues, retcols, allow_none=False):
	1455	def simple_select_one_txn(
	1456	txn: LoggingTransaction,
	1457	table: str,
	1458	keyvalues: Dict[str, Any],
	1459	retcols: Iterable[str],
	1460	allow_none: bool = False,
	1461	) -> Optional[Dict[str, Any]]:
1263	1462	select_sql = "SELECT %s FROM %s WHERE %s" % (
1264	1463	", ".join(retcols),
1265	1464	table,

1278	1477
1279	1478	return dict(zip(retcols, row))
1280	1479
1281		def simple_delete_one(self, table, keyvalues, desc="simple_delete_one"):
	1480	async def simple_delete_one(
	1481	self, table: str, keyvalues: Dict[str, Any], desc: str = "simple_delete_one"
	1482	) -> None:
1282	1483	"""Executes a DELETE query on the named table, expecting to delete a
1283	1484	single row.
1284	1485
1285	1486	Args:
1286		table : string giving the table name
1287		keyvalues : dict of column names and values to select the row with
1288		"""
1289		return self.runInteraction(desc, self.simple_delete_one_txn, table, keyvalues)
	1487	table: string giving the table name
	1488	keyvalues: dict of column names and values to select the row with
	1489	desc: description of the transaction, for logging and metrics
	1490	"""
	1491	await self.runInteraction(desc, self.simple_delete_one_txn, table, keyvalues)
1290	1492
1291	1493	@staticmethod
1292		def simple_delete_one_txn(txn, table, keyvalues):
	1494	def simple_delete_one_txn(
	1495	txn: LoggingTransaction, table: str, keyvalues: Dict[str, Any]
	1496	) -> None:
1293	1497	"""Executes a DELETE query on the named table, expecting to delete a
1294	1498	single row.
1295	1499
1296	1500	Args:
1297		table : string giving the table name
1298		keyvalues : dict of column names and values to select the row with
	1501	table: string giving the table name
	1502	keyvalues: dict of column names and values to select the row with
1299	1503	"""
1300	1504	sql = "DELETE FROM %s WHERE %s" % (
1301	1505	table,

1308	1512	if txn.rowcount > 1:
1309	1513	raise StoreError(500, "More than one row matched (%s)" % (table,))
1310	1514
1311		def simple_delete(self, table, keyvalues, desc):
1312		return self.runInteraction(desc, self.simple_delete_txn, table, keyvalues)
	1515	async def simple_delete(
	1516	self, table: str, keyvalues: Dict[str, Any], desc: str
	1517	) -> int:
	1518	"""Executes a DELETE query on the named table.
	1519
	1520	Filters rows by the key-value pairs.
	1521
	1522	Args:
	1523	table: string giving the table name
	1524	keyvalues: dict of column names and values to select the row with
	1525	desc: description of the transaction, for logging and metrics
	1526
	1527	Returns:
	1528	The number of deleted rows.
	1529	"""
	1530	return await self.runInteraction(desc, self.simple_delete_txn, table, keyvalues)
1313	1531
1314	1532	@staticmethod
1315		def simple_delete_txn(txn, table, keyvalues):
	1533	def simple_delete_txn(
	1534	txn: LoggingTransaction, table: str, keyvalues: Dict[str, Any]
	1535	) -> int:
	1536	"""Executes a DELETE query on the named table.
	1537
	1538	Filters rows by the key-value pairs.
	1539
	1540	Args:
	1541	table: string giving the table name
	1542	keyvalues: dict of column names and values to select the row with
	1543
	1544	Returns:
	1545	The number of deleted rows.
	1546	"""
1316	1547	sql = "DELETE FROM %s WHERE %s" % (
1317	1548	table,
1318	1549	" AND ".join("%s = ?" % (k,) for k in keyvalues),

1321	1552	txn.execute(sql, list(keyvalues.values()))
1322	1553	return txn.rowcount
1323	1554
1324		def simple_delete_many(self, table, column, iterable, keyvalues, desc):
1325		return self.runInteraction(
	1555	async def simple_delete_many(
	1556	self,
	1557	table: str,
	1558	column: str,
	1559	iterable: Iterable[Any],
	1560	keyvalues: Dict[str, Any],
	1561	desc: str,
	1562	) -> int:
	1563	"""Executes a DELETE query on the named table.
	1564
	1565	Filters rows by if value of `column` is in `iterable`.
	1566
	1567	Args:
	1568	table: string giving the table name
	1569	column: column name to test for inclusion against `iterable`
	1570	iterable: list
	1571	keyvalues: dict of column names and values to select the rows with
	1572	desc: description of the transaction, for logging and metrics
	1573
	1574	Returns:
	1575	Number rows deleted
	1576	"""
	1577	return await self.runInteraction(
1326	1578	desc, self.simple_delete_many_txn, table, column, iterable, keyvalues
1327	1579	)
1328	1580
1329	1581	@staticmethod
1330		def simple_delete_many_txn(txn, table, column, iterable, keyvalues):
	1582	def simple_delete_many_txn(
	1583	txn: LoggingTransaction,
	1584	table: str,
	1585	column: str,
	1586	iterable: Iterable[Any],
	1587	keyvalues: Dict[str, Any],
	1588	) -> int:
1331	1589	"""Executes a DELETE query on the named table.
1332	1590
1333	1591	Filters rows by if value of `column` is in `iterable`.
1334	1592
1335	1593	Args:
1336		txn : Transaction object
1337		table : string giving the table name
1338		column : column name to test for inclusion against `iterable`
1339		iterable : list
1340		keyvalues : dict of column names and values to select the rows with
	1594	txn: Transaction object
	1595	table: string giving the table name
	1596	column: column name to test for inclusion against `iterable`
	1597	iterable: list
	1598	keyvalues: dict of column names and values to select the rows with
1341	1599
1342	1600	Returns:
1343		int: Number rows deleted
	1601	Number rows deleted
1344	1602	"""
1345	1603	if not iterable:
1346	1604	return 0

1361	1619	return txn.rowcount
1362	1620
1363	1621	def get_cache_dict(
1364		self, db_conn, table, entity_column, stream_column, max_value, limit=100000
1365		):
	1622	self,
	1623	db_conn: Connection,
	1624	table: str,
	1625	entity_column: str,
	1626	stream_column: str,
	1627	max_value: int,
	1628	limit: int = 100000,
	1629	) -> Tuple[Dict[Any, int], int]:
1366	1630	# Fetch a mapping of room_id -> max stream position for "recent" rooms.
1367	1631	# It doesn't really matter how many we get, the StreamChangeCache will
1368	1632	# do the right thing to ensure it respects the max size of cache.

1393	1657
1394	1658	return cache, min_val
1395	1659
1396		def simple_select_list_paginate(
1397		self,
1398		table,
1399		orderby,
1400		start,
1401		limit,
1402		retcols,
1403		filters=None,
1404		keyvalues=None,
1405		order_direction="ASC",
1406		desc="simple_select_list_paginate",
1407		):
	1660	@classmethod
	1661	def simple_select_list_paginate_txn(
	1662	cls,
	1663	txn: LoggingTransaction,
	1664	table: str,
	1665	orderby: str,
	1666	start: int,
	1667	limit: int,
	1668	retcols: Iterable[str],
	1669	filters: Optional[Dict[str, Any]] = None,
	1670	keyvalues: Optional[Dict[str, Any]] = None,
	1671	order_direction: str = "ASC",
	1672	) -> List[Dict[str, Any]]:
1408	1673	"""
1409	1674	Executes a SELECT query on the named table with start and limit,
1410	1675	of row numbers, which may return zero or number of rows from start to limit,
1411	1676	returning the result as a list of dicts.
1412	1677
1413		Args:
1414		table (str): the table name
1415		filters (dict[str, T] \| None):
1416		column names and values to filter the rows with, or None to not
1417		apply a WHERE ? LIKE ? clause.
1418		keyvalues (dict[str, T] \| None):
1419		column names and values to select the rows with, or None to not
1420		apply a WHERE clause.
1421		orderby (str): Column to order the results by.
1422		start (int): Index to begin the query at.
1423		limit (int): Number of results to return.
1424		retcols (iterable[str]): the names of the columns to return
1425		order_direction (str): Whether the results should be ordered "ASC" or "DESC".
1426		Returns:
1427		defer.Deferred: resolves to list[dict[str, Any]]
1428		"""
1429		return self.runInteraction(
1430		desc,
1431		self.simple_select_list_paginate_txn,
1432		table,
1433		orderby,
1434		start,
1435		limit,
1436		retcols,
1437		filters=filters,
1438		keyvalues=keyvalues,
1439		order_direction=order_direction,
1440		)
1441
1442		@classmethod
1443		def simple_select_list_paginate_txn(
1444		cls,
1445		txn,
1446		table,
1447		orderby,
1448		start,
1449		limit,
1450		retcols,
1451		filters=None,
1452		keyvalues=None,
1453		order_direction="ASC",
1454		):
1455		"""
1456		Executes a SELECT query on the named table with start and limit,
1457		of row numbers, which may return zero or number of rows from start to limit,
1458		returning the result as a list of dicts.
1459
1460	1678	Use `filters` to search attributes using SQL wildcards and/or `keyvalues` to
1461	1679	select attributes with exact matches. All constraints are joined together
1462	1680	using 'AND'.
1463	1681
1464	1682	Args:
1465		txn : Transaction object
1466		table (str): the table name
1467		orderby (str): Column to order the results by.
1468		start (int): Index to begin the query at.
1469		limit (int): Number of results to return.
1470		retcols (iterable[str]): the names of the columns to return
1471		filters (dict[str, T] \| None):
	1683	txn: Transaction object
	1684	table: the table name
	1685	orderby: Column to order the results by.
	1686	start: Index to begin the query at.
	1687	limit: Number of results to return.
	1688	retcols: the names of the columns to return
	1689	filters:
1472	1690	column names and values to filter the rows with, or None to not
1473	1691	apply a WHERE ? LIKE ? clause.
1474		keyvalues (dict[str, T] \| None):
	1692	keyvalues:
1475	1693	column names and values to select the rows with, or None to not
1476	1694	apply a WHERE clause.
1477		order_direction (str): Whether the results should be ordered "ASC" or "DESC".
	1695	order_direction: Whether the results should be ordered "ASC" or "DESC".
	1696
1478	1697	Returns:
1479		defer.Deferred: resolves to list[dict[str, Any]]
	1698	The result as a list of dictionaries.
1480	1699	"""
1481	1700	if order_direction not in ["ASC", "DESC"]:
1482	1701	raise ValueError("order_direction must be one of 'ASC' or 'DESC'.")

1502	1721
1503	1722	return cls.cursor_to_dict(txn)
1504	1723
1505		def simple_search_list(self, table, term, col, retcols, desc="simple_search_list"):
	1724	async def simple_search_list(
	1725	self,
	1726	table: str,
	1727	term: Optional[str],
	1728	col: str,
	1729	retcols: Iterable[str],
	1730	desc="simple_search_list",
	1731	) -> Optional[List[Dict[str, Any]]]:
1506	1732	"""Executes a SELECT query on the named table, which may return zero or
1507	1733	more rows, returning the result as a list of dicts.
1508	1734
1509	1735	Args:
1510		table (str): the table name
1511		term (str \| None):
1512		term for searching the table matched to a column.
1513		col (str): column to query term should be matched to
1514		retcols (iterable[str]): the names of the columns to return
	1736	table: the table name
	1737	term: term for searching the table matched to a column.
	1738	col: column to query term should be matched to
	1739	retcols: the names of the columns to return
	1740
1515	1741	Returns:
1516		defer.Deferred: resolves to list[dict[str, Any]] or None
1517		"""
1518
1519		return self.runInteraction(
	1742	A list of dictionaries or None.
	1743	"""
	1744
	1745	return await self.runInteraction(
1520	1746	desc, self.simple_search_list_txn, table, term, col, retcols
1521	1747	)
1522	1748
1523	1749	@classmethod
1524		def simple_search_list_txn(cls, txn, table, term, col, retcols):
	1750	def simple_search_list_txn(
	1751	cls,
	1752	txn: LoggingTransaction,
	1753	table: str,
	1754	term: Optional[str],
	1755	col: str,
	1756	retcols: Iterable[str],
	1757	) -> Optional[List[Dict[str, Any]]]:
1525	1758	"""Executes a SELECT query on the named table, which may return zero or
1526	1759	more rows, returning the result as a list of dicts.
1527	1760
1528	1761	Args:
1529		txn : Transaction object
1530		table (str): the table name
1531		term (str \| None):
1532		term for searching the table matched to a column.
1533		col (str): column to query term should be matched to
1534		retcols (iterable[str]): the names of the columns to return
	1762	txn: Transaction object
	1763	table: the table name
	1764	term: term for searching the table matched to a column.
	1765	col: column to query term should be matched to
	1766	retcols: the names of the columns to return
	1767
1535	1768	Returns:
1536		defer.Deferred: resolves to list[dict[str, Any]] or None
	1769	None if no term is given, otherwise a list of dictionaries.
1537	1770	"""
1538	1771	if term:
1539	1772	sql = "SELECT %s FROM %s WHERE %s LIKE ?" % (", ".join(retcols), table, col)
1540	1773	termvalues = ["%%" + term + "%%"]
1541	1774	txn.execute(sql, termvalues)
1542	1775	else:
1543		return 0
	1776	return None
1544	1777
1545	1778	return cls.cursor_to_dict(txn)
1546	1779
1547	1780
1548	1781	def make_in_list_sql_clause(
1549		database_engine, column: str, iterable: Iterable
	1782	database_engine: BaseDatabaseEngine, column: str, iterable: Iterable
1550	1783	) -> Tuple[str, list]:
1551	1784	"""Returns an SQL clause that checks the given column is in the iterable.
1552	1785

+26

-8

synapse/storage/databases/__init__.py less more

23	23	logger = logging.getLogger(__name__)
24	24
25	25
26		class Databases(object):
	26	class Databases:
27	27	"""The various databases.
28	28
29	29	These are low level interfaces to physical databases.

46	46	engine = create_engine(database_config.config)
47	47
48	48	with make_conn(database_config, engine) as db_conn:
49		logger.info("Preparing database %r...", db_name)
	49	logger.info("[database config %r]: Checking database server", db_name)
	50	engine.check_database(db_conn)
50	51
51		engine.check_database(db_conn)
	52	logger.info(
	53	"[database config %r]: Preparing for databases %r",
	54	db_name,
	55	database_config.databases,
	56	)
52	57	prepare_database(
53	58	db_conn, engine, hs.config, databases=database_config.databases,
54	59	)

56	61	database = DatabasePool(hs, database_config, engine)
57	62
58	63	if "main" in database_config.databases:
59		logger.info("Starting 'main' data store")
	64	logger.info(
	65	"[database config %r]: Starting 'main' database", db_name
	66	)
60	67
61	68	# Sanity check we don't try and configure the main store on
62	69	# multiple databases.

71	78	persist_events = PersistEventsStore(hs, database, main)
72	79
73	80	if "state" in database_config.databases:
74		logger.info("Starting 'state' data store")
	81	logger.info(
	82	"[database config %r]: Starting 'state' database", db_name
	83	)
75	84
76	85	# Sanity check we don't try and configure the state store on
77	86	# multiple databases.

84	93
85	94	self.databases.append(database)
86	95
87		logger.info("Database %r prepared", db_name)
	96	logger.info("[database config %r]: prepared", db_name)
	97
	98	# Closing the context manager doesn't close the connection.
	99	# psycopg will close the connection when the object gets GCed, but only
	100	# if the PID is the same as when the connection was opened [1], and
	101	# it may not be if we fork in the meantime.
	102	#
	103	# [1]: https://github.com/psycopg/psycopg2/blob/2_8_5/psycopg/connection_type.c#L1378
	104
	105	db_conn.close()
88	106
89	107	# Sanity check that we have actually configured all the required stores.
90	108	if not main:
91		raise Exception("No 'main' data store configured")
	109	raise Exception("No 'main' database configured")
92	110
93	111	if not state:
94		raise Exception("No 'main' data store configured")
	112	raise Exception("No 'state' database configured")
95	113
96	114	# We use local variables here to ensure that the databases do not have
97	115	# optional types.

+72

-52

synapse/storage/databases/main/__init__.py less more

17	17	import calendar
18	18	import logging
19	19	import time
	20	from typing import Any, Dict, List, Optional, Tuple
20	21
21	22	from synapse.api.constants import PresenceState
22	23	from synapse.config.homeserver import HomeServerConfig

27	28	MultiWriterIdGenerator,
28	29	StreamIdGenerator,
29	30	)
	31	from synapse.types import get_domain_from_id
30	32	from synapse.util.caches.stream_change_cache import StreamChangeCache
31	33
32	34	from .account_data import AccountDataStore

262	264	# Used in _generate_user_daily_visits to keep track of progress
263	265	self._last_user_visit_update = self._get_start_of_day()
264	266
	267	def get_device_stream_token(self) -> int:
	268	return self._device_list_id_gen.get_current_token()
	269
265	270	def take_presence_startup_info(self):
266	271	active_on_startup = self._presence_on_startup
267	272	self._presence_on_startup = None

289	294
290	295	return [UserPresenceState(**row) for row in rows]
291	296
292		def count_daily_users(self):
	297	async def count_daily_users(self) -> int:
293	298	"""
294	299	Counts the number of users who used this homeserver in the last 24 hours.
295	300	"""
296	301	yesterday = int(self._clock.time_msec()) - (1000 * 60 * 60 * 24)
297		return self.db_pool.runInteraction(
	302	return await self.db_pool.runInteraction(
298	303	"count_daily_users", self._count_users, yesterday
299	304	)
300	305
301		def count_monthly_users(self):
	306	async def count_monthly_users(self) -> int:
302	307	"""
303	308	Counts the number of users who used this homeserver in the last 30 days.
304	309	Note this method is intended for phonehome metrics only and is different

306	311	amongst other things, includes a 3 day grace period before a user counts.
307	312	"""
308	313	thirty_days_ago = int(self._clock.time_msec()) - (1000 * 60 * 60 * 24 * 30)
309		return self.db_pool.runInteraction(
	314	return await self.db_pool.runInteraction(
310	315	"count_monthly_users", self._count_users, thirty_days_ago
311	316	)
312	317

325	330	(count,) = txn.fetchone()
326	331	return count
327	332
328		def count_r30_users(self):
	333	async def count_r30_users(self) -> Dict[str, int]:
329	334	"""
330	335	Counts the number of 30 day retained users, defined as:-
331	336	* Users who have created their accounts more than 30 days ago
332	337	* Where last seen at most 30 days ago
333	338	* Where account creation and last_seen are > 30 days apart
334	339
335		Returns counts globaly for a given user as well as breaking
336		by platform
	340	Returns:
	341	A mapping of counts globally as well as broken out by platform.
337	342	"""
338	343
339	344	def _count_r30_users(txn):

406	411
407	412	return results
408	413
409		return self.db_pool.runInteraction("count_r30_users", _count_r30_users)
	414	return await self.db_pool.runInteraction("count_r30_users", _count_r30_users)
410	415
411	416	def _get_start_of_day(self):
412	417	"""

416	421	today_start = calendar.timegm((now.tm_year, now.tm_mon, now.tm_mday, 0, 0, 0))
417	422	return today_start * 1000
418	423
419		def generate_user_daily_visits(self):
	424	async def generate_user_daily_visits(self) -> None:
420	425	"""
421	426	Generates daily visit data for use in cohort/ retention analysis
422	427	"""

471	476	# frequently
472	477	self._last_user_visit_update = now
473	478
474		return self.db_pool.runInteraction(
	479	await self.db_pool.runInteraction(
475	480	"generate_user_daily_visits", _generate_user_daily_visits
476	481	)
477	482
478		def get_users(self):
	483	async def get_users(self) -> List[Dict[str, Any]]:
479	484	"""Function to retrieve a list of users in users table.
480	485
481		Args:
482	486	Returns:
483		defer.Deferred: resolves to list[dict[str, Any]]
484		"""
485		return self.db_pool.simple_select_list(
	487	A list of dictionaries representing users.
	488	"""
	489	return await self.db_pool.simple_select_list(
486	490	table="users",
487	491	keyvalues={},
488	492	retcols=[

496	500	desc="get_users",
497	501	)
498	502
499		def get_users_paginate(
500		self, start, limit, name=None, guests=True, deactivated=False
501		):
	503	async def get_users_paginate(
	504	self,
	505	start: int,
	506	limit: int,
	507	user_id: Optional[str] = None,
	508	name: Optional[str] = None,
	509	guests: bool = True,
	510	deactivated: bool = False,
	511	) -> Tuple[List[Dict[str, Any]], int]:
502	512	"""Function to retrieve a paginated list of users from
503	513	users list. This will return a json list of users and the
504	514	total number of users matching the filter criteria.
505	515
506	516	Args:
507		start (int): start number to begin the query from
508		limit (int): number of rows to retrieve
509		name (string): filter for user names
510		guests (bool): whether to in include guest users
511		deactivated (bool): whether to include deactivated users
	517	start: start number to begin the query from
	518	limit: number of rows to retrieve
	519	user_id: search for user_id. ignored if name is not None
	520	name: search for local part of user_id or display name
	521	guests: whether to in include guest users
	522	deactivated: whether to include deactivated users
512	523	Returns:
513		defer.Deferred: resolves to list[dict[str, Any]], int
	524	A tuple of a list of mappings from user to information and a count of total users.
514	525	"""
515	526
516	527	def get_users_paginate_txn(txn):
517	528	filters = []
518		args = []
	529	args = [self.hs.config.server_name]
519	530
520	531	if name:
	532	filters.append("(name LIKE ? OR displayname LIKE ?)")
	533	args.extend(["@%" + name + "%:%", "%" + name + "%"])
	534	elif user_id:
521	535	filters.append("name LIKE ?")
522		args.append("%" + name + "%")
	536	args.extend(["%" + user_id + "%"])
523	537
524	538	if not guests:
525	539	filters.append("is_guest = 0")

529	543
530	544	where_clause = "WHERE " + " AND ".join(filters) if len(filters) > 0 else ""
531	545
532		sql = "SELECT COUNT(*) as total_users FROM users %s" % (where_clause)
533		txn.execute(sql, args)
534		count = txn.fetchone()[0]
535
536		args = [self.hs.config.server_name] + args + [limit, start]
537		sql = """
538		SELECT name, user_type, is_guest, admin, deactivated, displayname, avatar_url
	546	sql_base = """
539	547	FROM users as u
540	548	LEFT JOIN profiles AS p ON u.name = '@' \|\| p.user_id \|\| ':' \|\| ?
541	549	{}
542		ORDER BY u.name LIMIT ? OFFSET ?
543	550	""".format(
544	551	where_clause
545	552	)
	553	sql = "SELECT COUNT(*) as total_users " + sql_base
	554	txn.execute(sql, args)
	555	count = txn.fetchone()[0]
	556
	557	sql = (
	558	"SELECT name, user_type, is_guest, admin, deactivated, displayname, avatar_url "
	559	+ sql_base
	560	+ " ORDER BY u.name LIMIT ? OFFSET ?"
	561	)
	562	args += [limit, start]
546	563	txn.execute(sql, args)
547	564	users = self.db_pool.cursor_to_dict(txn)
548	565	return users, count
549	566
550		return self.db_pool.runInteraction(
	567	return await self.db_pool.runInteraction(
551	568	"get_users_paginate_txn", get_users_paginate_txn
552	569	)
553	570
554		def search_users(self, term):
	571	async def search_users(self, term: str) -> Optional[List[Dict[str, Any]]]:
555	572	"""Function to search users list for one or more users with
556	573	the matched term.
557	574
558	575	Args:
559		term (str): search term
560		col (str): column to query term should be matched to
	576	term: search term
	577
561	578	Returns:
562		defer.Deferred: resolves to list[dict[str, Any]]
563		"""
564		return self.db_pool.simple_search_list(
	579	A list of dictionaries or None.
	580	"""
	581	return await self.db_pool.simple_search_list(
565	582	table="users",
566	583	term=term,
567	584	col="name",

574	591	"""Called before upgrading an existing database to check that it is broadly sane
575	592	compared with the configuration.
576	593	"""
577		domain = config.server_name
578
579		sql = database_engine.convert_param_style(
580		"SELECT COUNT(*) FROM users WHERE name NOT LIKE ?"
581		)
582		pat = "%:" + domain
583		cur.execute(sql, (pat,))
584		num_not_matching = cur.fetchall()[0][0]
585		if num_not_matching == 0:
	594	logger.info("Checking database for consistency with configuration...")
	595
	596	# if there are any users in the database, check that the username matches our
	597	# configured server name.
	598
	599	cur.execute("SELECT name FROM users LIMIT 1")
	600	rows = cur.fetchall()
	601	if not rows:
	602	return
	603
	604	user_domain = get_domain_from_id(rows[0][0])
	605	if user_domain == config.server_name:
586	606	return
587	607
588	608	raise Exception(
589	609	"Found users in database not native to %s!\n"
590		"You cannot changed a synapse server_name after it's been configured"
591		% (domain,)
	610	"You cannot change a synapse server_name after it's been configured"
	611	% (config.server_name,)
592	612	)
593	613
594	614

+38

-33

synapse/storage/databases/main/account_data.py less more

15	15
16	16	import abc
17	17	import logging
18		from typing import List, Optional, Tuple
19
20		from twisted.internet import defer
	18	from typing import Dict, List, Optional, Tuple
21	19
22	20	from synapse.storage._base import SQLBaseStore, db_to_json
23	21	from synapse.storage.database import DatabasePool

57	55	raise NotImplementedError()
58	56
59	57	@cached()
60		def get_account_data_for_user(self, user_id):
	58	async def get_account_data_for_user(
	59	self, user_id: str
	60	) -> Tuple[Dict[str, JsonDict], Dict[str, Dict[str, JsonDict]]]:
61	61	"""Get all the client account_data for a user.
62	62
63	63	Args:
64		user_id(str): The user to get the account_data for.
65		Returns:
66		A deferred pair of a dict of global account_data and a dict
67		mapping from room_id string to per room account_data dicts.
	64	user_id: The user to get the account_data for.
	65	Returns:
	66	A 2-tuple of a dict of global account_data and a dict mapping from
	67	room_id string to per room account_data dicts.
68	68	"""
69	69
70	70	def get_account_data_for_user_txn(txn):

93	93
94	94	return global_account_data, by_room
95	95
96		return self.db_pool.runInteraction(
	96	return await self.db_pool.runInteraction(
97	97	"get_account_data_for_user", get_account_data_for_user_txn
98	98	)
99	99

119	119	return None
120	120
121	121	@cached(num_args=2)
122		def get_account_data_for_room(self, user_id, room_id):
	122	async def get_account_data_for_room(
	123	self, user_id: str, room_id: str
	124	) -> Dict[str, JsonDict]:
123	125	"""Get all the client account_data for a user for a room.
124	126
125	127	Args:
126		user_id(str): The user to get the account_data for.
127		room_id(str): The room to get the account_data for.
128		Returns:
129		A deferred dict of the room account_data
	128	user_id: The user to get the account_data for.
	129	room_id: The room to get the account_data for.
	130	Returns:
	131	A dict of the room account_data
130	132	"""
131	133
132	134	def get_account_data_for_room_txn(txn):

141	143	row["account_data_type"]: db_to_json(row["content"]) for row in rows
142	144	}
143	145
144		return self.db_pool.runInteraction(
	146	return await self.db_pool.runInteraction(
145	147	"get_account_data_for_room", get_account_data_for_room_txn
146	148	)
147	149
148	150	@cached(num_args=3, max_entries=5000)
149		def get_account_data_for_room_and_type(self, user_id, room_id, account_data_type):
	151	async def get_account_data_for_room_and_type(
	152	self, user_id: str, room_id: str, account_data_type: str
	153	) -> Optional[JsonDict]:
150	154	"""Get the client account_data of given type for a user for a room.
151	155
152	156	Args:
153		user_id(str): The user to get the account_data for.
154		room_id(str): The room to get the account_data for.
155		account_data_type (str): The account data type to get.
156		Returns:
157		A deferred of the room account_data for that type, or None if
158		there isn't any set.
	157	user_id: The user to get the account_data for.
	158	room_id: The room to get the account_data for.
	159	account_data_type: The account data type to get.
	160	Returns:
	161	The room account_data for that type, or None if there isn't any set.
159	162	"""
160	163
161	164	def get_account_data_for_room_and_type_txn(txn):

173	176
174	177	return db_to_json(content_json) if content_json else None
175	178
176		return self.db_pool.runInteraction(
	179	return await self.db_pool.runInteraction(
177	180	"get_account_data_for_room_and_type", get_account_data_for_room_and_type_txn
178	181	)
179	182

237	240	"get_updated_room_account_data", get_updated_room_account_data_txn
238	241	)
239	242
240		def get_updated_account_data_for_user(self, user_id, stream_id):
	243	async def get_updated_account_data_for_user(
	244	self, user_id: str, stream_id: int
	245	) -> Tuple[Dict[str, JsonDict], Dict[str, Dict[str, JsonDict]]]:
241	246	"""Get all the client account_data for a that's changed for a user
242	247
243	248	Args:
244		user_id(str): The user to get the account_data for.
245		stream_id(int): The point in the stream since which to get updates
	249	user_id: The user to get the account_data for.
	250	stream_id: The point in the stream since which to get updates
246	251	Returns:
247	252	A deferred pair of a dict of global account_data and a dict
248	253	mapping from room_id string to per room account_data dicts.

276	281	user_id, int(stream_id)
277	282	)
278	283	if not changed:
279		return defer.succeed(({}, {}))
280
281		return self.db_pool.runInteraction(
	284	return ({}, {})
	285
	286	return await self.db_pool.runInteraction(
282	287	"get_updated_account_data_for_user", get_updated_account_data_for_user_txn
283	288	)
284	289

335	340	"""
336	341	content_json = json_encoder.encode(content)
337	342
338		with self._account_data_id_gen.get_next() as next_id:
	343	with await self._account_data_id_gen.get_next() as next_id:
339	344	# no need to lock here as room_account_data has a unique constraint
340	345	# on (user_id, room_id, account_data_type) so simple_upsert will
341	346	# retry if there is a conflict.

383	388	"""
384	389	content_json = json_encoder.encode(content)
385	390
386		with self._account_data_id_gen.get_next() as next_id:
	391	with await self._account_data_id_gen.get_next() as next_id:
387	392	# no need to lock here as account_data has a unique constraint on
388	393	# (user_id, account_data_type) so simple_upsert will retry if
389	394	# there is a conflict.

415	420
416	421	return self._account_data_id_gen.get_current_token()
417	422
418		def _update_max_stream_id(self, next_id: int):
	423	async def _update_max_stream_id(self, next_id: int) -> None:
419	424	"""Update the max stream_id
420	425
421	426	Args:

434	439	)
435	440	txn.execute(update_max_id_sql, (next_id, next_id))
436	441
437		return self.db_pool.runInteraction("update_account_data_max_stream_id", _update)
	442	await self.db_pool.runInteraction("update_account_data_max_stream_id", _update)

+11

-17

synapse/storage/databases/main/appservice.py less more

15	15	import logging
16	16	import re
17	17
18		from canonicaljson import json
19
20	18	from synapse.appservice import AppServiceTransaction
21	19	from synapse.config.appservice import load_appservices
22	20	from synapse.storage._base import SQLBaseStore, db_to_json
23	21	from synapse.storage.database import DatabasePool
24	22	from synapse.storage.databases.main.events_worker import EventsWorkerStore
	23	from synapse.util import json_encoder
25	24
26	25	logger = logging.getLogger(__name__)
27	26

161	160	return result.get("state")
162	161	return None
163	162
164		def set_appservice_state(self, service, state):
	163	async def set_appservice_state(self, service, state) -> None:
165	164	"""Set the application service state.
166	165
167	166	Args:
168	167	service(ApplicationService): The service whose state to set.
169	168	state(ApplicationServiceState): The connectivity state to apply.
170		Returns:
171		A Deferred which resolves when the state was set successfully.
172		"""
173		return self.db_pool.simple_upsert(
	169	"""
	170	await self.db_pool.simple_upsert(
174	171	"application_services_state", {"as_id": service.id}, {"state": state}
175	172	)
176	173
177		def create_appservice_txn(self, service, events):
	174	async def create_appservice_txn(self, service, events):
178	175	"""Atomically creates a new transaction for this application service
179	176	with the given list of events.
180	177

203	200	new_txn_id = max(highest_txn_id, last_txn_id) + 1
204	201
205	202	# Insert new txn into txn table
206		event_ids = json.dumps([e.event_id for e in events])
	203	event_ids = json_encoder.encode([e.event_id for e in events])
207	204	txn.execute(
208	205	"INSERT INTO application_services_txns(as_id, txn_id, event_ids) "
209	206	"VALUES(?,?,?)",

211	208	)
212	209	return AppServiceTransaction(service=service, id=new_txn_id, events=events)
213	210
214		return self.db_pool.runInteraction(
	211	return await self.db_pool.runInteraction(
215	212	"create_appservice_txn", _create_appservice_txn
216	213	)
217	214
218		def complete_appservice_txn(self, txn_id, service):
	215	async def complete_appservice_txn(self, txn_id, service) -> None:
219	216	"""Completes an application service transaction.
220	217
221	218	Args:
222	219	txn_id(str): The transaction ID being completed.
223	220	service(ApplicationService): The application service which was sent
224	221	this transaction.
225		Returns:
226		A Deferred which resolves if this transaction was stored
227		successfully.
228	222	"""
229	223	txn_id = int(txn_id)
230	224

260	254	{"txn_id": txn_id, "as_id": service.id},
261	255	)
262	256
263		return self.db_pool.runInteraction(
	257	await self.db_pool.runInteraction(
264	258	"complete_appservice_txn", _complete_appservice_txn
265	259	)
266	260

314	308	else:
315	309	return int(last_txn_id[0]) # select 'last_txn' col
316	310
317		def set_appservice_last_pos(self, pos):
	311	async def set_appservice_last_pos(self, pos) -> None:
318	312	def set_appservice_last_pos_txn(txn):
319	313	txn.execute(
320	314	"UPDATE appservice_stream_position SET stream_ordering = ?", (pos,)
321	315	)
322	316
323		return self.db_pool.runInteraction(
	317	await self.db_pool.runInteraction(
324	318	"set_appservice_last_pos", set_appservice_last_pos_txn
325	319	)
326	320

+2

-2

synapse/storage/databases/main/cache.py less more

298	298	},
299	299	)
300	300
301		def get_cache_stream_token(self, instance_name):
	301	def get_cache_stream_token_for_writer(self, instance_name: str) -> int:
302	302	if self._cache_id_gen:
303		return self._cache_id_gen.get_current_token(instance_name)
	303	return self._cache_id_gen.get_current_token_for_writer(instance_name)
304	304	else:
305	305	return 0

+2

-2

synapse/storage/databases/main/client_ips.py less more

395	395	self._batch_row_update[key] = (user_agent, device_id, now)
396	396
397	397	@wrap_as_background_process("update_client_ips")
398		def _update_client_ips_batch(self):
	398	async def _update_client_ips_batch(self) -> None:
399	399
400	400	# If the DB pool has already terminated, don't try updating
401	401	if not self.db_pool.is_running():

404	404	to_update = self._batch_row_update
405	405	self._batch_row_update = {}
406	406
407		return self.db_pool.runInteraction(
	407	await self.db_pool.runInteraction(
408	408	"_update_client_ips_batch", self._update_client_ips_batch_txn, to_update
409	409	)
410	410

+8

-8

synapse/storage/databases/main/deviceinbox.py less more

189	189	)
190	190
191	191	@trace
192		def delete_device_msgs_for_remote(self, destination, up_to_stream_id):
	192	async def delete_device_msgs_for_remote(
	193	self, destination: str, up_to_stream_id: int
	194	) -> None:
193	195	"""Used to delete messages when the remote destination acknowledges
194	196	their receipt.
195	197
196	198	Args:
197		destination(str): The destination server_name
198		up_to_stream_id(int): Where to delete messages up to.
199		Returns:
200		A deferred that resolves when the messages have been deleted.
	199	destination: The destination server_name
	200	up_to_stream_id: Where to delete messages up to.
201	201	"""
202	202
203	203	def delete_messages_for_remote_destination_txn(txn):

208	208	)
209	209	txn.execute(sql, (destination, up_to_stream_id))
210	210
211		return self.db_pool.runInteraction(
	211	await self.db_pool.runInteraction(
212	212	"delete_device_msgs_for_remote", delete_messages_for_remote_destination_txn
213	213	)
214	214

361	361	rows.append((destination, stream_id, now_ms, edu_json))
362	362	txn.executemany(sql, rows)
363	363
364		with self._device_inbox_id_gen.get_next() as stream_id:
	364	with await self._device_inbox_id_gen.get_next() as stream_id:
365	365	now_ms = self.clock.time_msec()
366	366	await self.db_pool.runInteraction(
367	367	"add_messages_to_device_inbox", add_messages_txn, now_ms, stream_id

410	410	txn, stream_id, local_messages_by_user_then_device
411	411	)
412	412
413		with self._device_inbox_id_gen.get_next() as stream_id:
	413	with await self._device_inbox_id_gen.get_next() as stream_id:
414	414	now_ms = self.clock.time_msec()
415	415	await self.db_pool.runInteraction(
416	416	"add_messages_from_remote_to_device_inbox",

+49

-100

synapse/storage/databases/main/devices.py less more

13	13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14	14	# See the License for the specific language governing permissions and
15	15	# limitations under the License.
	16	import abc
16	17	import logging
17		from typing import Dict, Iterable, List, Optional, Set, Tuple
	18	from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
18	19
19	20	from synapse.api.errors import Codes, StoreError
20	21	from synapse.logging.opentracing import (

46	47
47	48
48	49	class DeviceWorkerStore(SQLBaseStore):
49		def get_device(self, user_id: str, device_id: str):
	50	async def get_device(self, user_id: str, device_id: str) -> Dict[str, Any]:
50	51	"""Retrieve a device. Only returns devices that are not marked as
51	52	hidden.
52	53

54	55	user_id: The ID of the user which owns the device
55	56	device_id: The ID of the device to retrieve
56	57	Returns:
57		defer.Deferred for a dict containing the device information
	58	A dict containing the device information
58	59	Raises:
59	60	StoreError: if the device is not found
60	61	"""
61		return self.db_pool.simple_select_one(
	62	return await self.db_pool.simple_select_one(
62	63	table="devices",
63	64	keyvalues={"user_id": user_id, "device_id": device_id, "hidden": False},
64	65	retcols=("user_id", "device_id", "display_name"),

100	101	update included in the response), and the list of updates, where
101	102	each update is a pair of EDU type and EDU contents.
102	103	"""
103		now_stream_id = self._device_list_id_gen.get_current_token()
	104	now_stream_id = self.get_device_stream_token()
104	105
105	106	has_changed = self._device_list_federation_stream_cache.has_entity_changed(
106	107	destination, int(from_stream_id)

253	254	List of objects representing an device update EDU
254	255	"""
255	256	devices = (
256		await self.db_pool.runInteraction(
257		"_get_e2e_device_keys_txn",
258		self._get_e2e_device_keys_txn,
	257	await self.get_e2e_device_keys_and_signatures(
259	258	query_map.keys(),
260	259	include_all_devices=True,
261	260	include_deleted_devices=True,

291	290	prev_id = stream_id
292	291
293	292	if device is not None:
294		key_json = device.get("key_json", None)
295		if key_json:
296		result["keys"] = db_to_json(key_json)
297
298		if "signatures" in device:
299		for sig_user_id, sigs in device["signatures"].items():
300		result["keys"].setdefault("signatures", {}).setdefault(
301		sig_user_id, {}
302		).update(sigs)
303
304		device_display_name = device.get("device_display_name", None)
	293	keys = device.keys
	294	if keys:
	295	result["keys"] = keys
	296
	297	device_display_name = device.display_name
305	298	if device_display_name:
306	299	result["device_display_name"] = device_display_name
307	300	else:

311	304
312	305	return results
313	306
314		def _get_last_device_update_for_remote_user(
	307	async def _get_last_device_update_for_remote_user(
315	308	self, destination: str, user_id: str, from_stream_id: int
316		):
	309	) -> int:
317	310	def f(txn):
318	311	prev_sent_id_sql = """
319	312	SELECT coalesce(max(stream_id), 0) as stream_id

324	317	rows = txn.fetchall()
325	318	return rows[0][0]
326	319
327		return self.db_pool.runInteraction("get_last_device_update_for_remote_user", f)
328
329		def mark_as_sent_devices_by_remote(self, destination: str, stream_id: int):
	320	return await self.db_pool.runInteraction(
	321	"get_last_device_update_for_remote_user", f
	322	)
	323
	324	async def mark_as_sent_devices_by_remote(
	325	self, destination: str, stream_id: int
	326	) -> None:
330	327	"""Mark that updates have successfully been sent to the destination.
331	328	"""
332		return self.db_pool.runInteraction(
	329	await self.db_pool.runInteraction(
333	330	"mark_as_sent_devices_by_remote",
334	331	self._mark_as_sent_devices_by_remote_txn,
335	332	destination,

379	376	THe new stream ID.
380	377	"""
381	378
382		with self._device_list_id_gen.get_next() as stream_id:
	379	with await self._device_list_id_gen.get_next() as stream_id:
383	380	await self.db_pool.runInteraction(
384	381	"add_user_sig_change_to_streams",
385	382	self._add_user_signature_change_txn,

411	408	},
412	409	)
413	410
	411	@abc.abstractmethod
414	412	def get_device_stream_token(self) -> int:
415		return self._device_list_id_gen.get_current_token()
	413	"""Get the current stream id from the _device_list_id_gen"""
	414	...
416	415
417	416	@trace
418	417	async def get_user_devices_from_cache(

479	478	return {
480	479	device["device_id"]: db_to_json(device["content"]) for device in devices
481	480	}
482
483		def get_devices_with_keys_by_user(self, user_id: str):
484		"""Get all devices (with any device keys) for a user
485
486		Returns:
487		Deferred which resolves to (stream_id, devices)
488		"""
489		return self.db_pool.runInteraction(
490		"get_devices_with_keys_by_user",
491		self._get_devices_with_keys_by_user_txn,
492		user_id,
493		)
494
495		def _get_devices_with_keys_by_user_txn(
496		self, txn: LoggingTransaction, user_id: str
497		) -> Tuple[int, List[JsonDict]]:
498		now_stream_id = self._device_list_id_gen.get_current_token()
499
500		devices = self._get_e2e_device_keys_txn(
501		txn, [(user_id, None)], include_all_devices=True
502		)
503
504		if devices:
505		user_devices = devices[user_id]
506		results = []
507		for device_id, device in user_devices.items():
508		result = {"device_id": device_id}
509
510		key_json = device.get("key_json", None)
511		if key_json:
512		result["keys"] = db_to_json(key_json)
513
514		if "signatures" in device:
515		for sig_user_id, sigs in device["signatures"].items():
516		result["keys"].setdefault("signatures", {}).setdefault(
517		sig_user_id, {}
518		).update(sigs)
519
520		device_display_name = device.get("device_display_name", None)
521		if device_display_name:
522		result["device_display_name"] = device_display_name
523
524		results.append(result)
525
526		return now_stream_id, results
527
528		return now_stream_id, []
529	481
530	482	async def get_users_whose_devices_changed(
531	483	self, from_key: str, user_ids: Iterable[str]

655	607	)
656	608
657	609	@cached(max_entries=10000)
658		def get_device_list_last_stream_id_for_remote(self, user_id: str):
	610	async def get_device_list_last_stream_id_for_remote(
	611	self, user_id: str
	612	) -> Optional[Any]:
659	613	"""Get the last stream_id we got for a user. May be None if we haven't
660	614	got any information for them.
661	615	"""
662		return self.db_pool.simple_select_one_onecol(
	616	return await self.db_pool.simple_select_one_onecol(
663	617	table="device_lists_remote_extremeties",
664	618	keyvalues={"user_id": user_id},
665	619	retcol="stream_id",

670	624	@cachedList(
671	625	cached_method_name="get_device_list_last_stream_id_for_remote",
672	626	list_name="user_ids",
673		inlineCallbacks=True,
674	627	)
675		def get_device_list_last_stream_id_for_remotes(self, user_ids: str):
676		rows = yield self.db_pool.simple_select_many_batch(
	628	async def get_device_list_last_stream_id_for_remotes(self, user_ids: str):
	629	rows = await self.db_pool.simple_select_many_batch(
677	630	table="device_lists_remote_extremeties",
678	631	column="user_id",
679	632	iterable=user_ids,

714	667
715	668	return {row["user_id"] for row in rows}
716	669
717		def mark_remote_user_device_cache_as_stale(self, user_id: str):
	670	async def mark_remote_user_device_cache_as_stale(self, user_id: str) -> None:
718	671	"""Records that the server has reason to believe the cache of the devices
719	672	for the remote users is out of date.
720	673	"""
721		return self.db_pool.simple_upsert(
	674	await self.db_pool.simple_upsert(
722	675	table="device_lists_remote_resync",
723	676	keyvalues={"user_id": user_id},
724	677	values={},

726	679	desc="make_remote_user_device_cache_as_stale",
727	680	)
728	681
729		def mark_remote_user_device_list_as_unsubscribed(self, user_id: str):
	682	async def mark_remote_user_device_list_as_unsubscribed(self, user_id: str) -> None:
730	683	"""Mark that we no longer track device lists for remote user.
731	684	"""
732	685

740	693	txn, self.get_device_list_last_stream_id_for_remote, (user_id,)
741	694	)
742	695
743		return self.db_pool.runInteraction(
	696	await self.db_pool.runInteraction(
744	697	"mark_remote_user_device_list_as_unsubscribed",
745	698	_mark_remote_user_device_list_as_unsubscribed_txn,
746	699	)

1001	954	desc="update_device",
1002	955	)
1003	956
1004		def update_remote_device_list_cache_entry(
	957	async def update_remote_device_list_cache_entry(
1005	958	self, user_id: str, device_id: str, content: JsonDict, stream_id: int
1006		):
	959	) -> None:
1007	960	"""Updates a single device in the cache of a remote user's devicelist.
1008	961
1009	962	Note: assumes that we are the only thread that can be updating this user's

1014	967	device_id: ID of decivice being updated
1015	968	content: new data on this device
1016	969	stream_id: the version of the device list
1017
1018		Returns:
1019		Deferred[None]
1020		"""
1021		return self.db_pool.runInteraction(
	970	"""
	971	await self.db_pool.runInteraction(
1022	972	"update_remote_device_list_cache_entry",
1023	973	self._update_remote_device_list_cache_entry_txn,
1024	974	user_id,

1070	1020	lock=False,
1071	1021	)
1072	1022
1073		def update_remote_device_list_cache(
	1023	async def update_remote_device_list_cache(
1074	1024	self, user_id: str, devices: List[dict], stream_id: int
1075		):
	1025	) -> None:
1076	1026	"""Replace the entire cache of the remote user's devices.
1077	1027
1078	1028	Note: assumes that we are the only thread that can be updating this user's

1082	1032	user_id: User to update device list for
1083	1033	devices: list of device objects supplied over federation
1084	1034	stream_id: the version of the device list
1085
1086		Returns:
1087		Deferred[None]
1088		"""
1089		return self.db_pool.runInteraction(
	1035	"""
	1036	await self.db_pool.runInteraction(
1090	1037	"update_remote_device_list_cache",
1091	1038	self._update_remote_device_list_cache_txn,
1092	1039	user_id,

1096	1043
1097	1044	def _update_remote_device_list_cache_txn(
1098	1045	self, txn: LoggingTransaction, user_id: str, devices: List[dict], stream_id: int
1099		):
	1046	) -> None:
1100	1047	self.db_pool.simple_delete_txn(
1101	1048	txn, table="device_lists_remote_cache", keyvalues={"user_id": user_id}
1102	1049	)

1146	1093	if not device_ids:
1147	1094	return
1148	1095
1149		with self._device_list_id_gen.get_next_mult(len(device_ids)) as stream_ids:
	1096	with await self._device_list_id_gen.get_next_mult(
	1097	len(device_ids)
	1098	) as stream_ids:
1150	1099	await self.db_pool.runInteraction(
1151	1100	"add_device_change_to_stream",
1152	1101	self._add_device_change_to_stream_txn,

1159	1108	return stream_ids[-1]
1160	1109
1161	1110	context = get_active_span_text_map()
1162		with self._device_list_id_gen.get_next_mult(
	1111	with await self._device_list_id_gen.get_next_mult(
1163	1112	len(hosts) * len(device_ids)
1164	1113	) as stream_ids:
1165	1114	await self.db_pool.runInteraction(

+8

-8

synapse/storage/databases/main/directory.py less more

13	13	# limitations under the License.
14	14
15	15	from collections import namedtuple
16		from typing import Iterable, Optional
	16	from typing import Iterable, List, Optional
17	17
18	18	from synapse.api.errors import SynapseError
19	19	from synapse.storage._base import SQLBaseStore

58	58
59	59	return RoomAliasMapping(room_id, room_alias.to_string(), servers)
60	60
61		def get_room_alias_creator(self, room_alias):
62		return self.db_pool.simple_select_one_onecol(
	61	async def get_room_alias_creator(self, room_alias: str) -> str:
	62	return await self.db_pool.simple_select_one_onecol(
63	63	table="room_aliases",
64	64	keyvalues={"room_alias": room_alias},
65	65	retcol="creator",

67	67	)
68	68
69	69	@cached(max_entries=5000)
70		def get_aliases_for_room(self, room_id):
71		return self.db_pool.simple_select_onecol(
	70	async def get_aliases_for_room(self, room_id: str) -> List[str]:
	71	return await self.db_pool.simple_select_onecol(
72	72	"room_aliases",
73	73	{"room_id": room_id},
74	74	"room_alias",

158	158
159	159	return room_id
160	160
161		def update_aliases_for_room(
	161	async def update_aliases_for_room(
162	162	self, old_room_id: str, new_room_id: str, creator: Optional[str] = None,
163		):
	163	) -> None:
164	164	"""Repoint all of the aliases for a given room, to a different room.
165	165
166	166	Args:

188	188	txn, self.get_aliases_for_room, (new_room_id,)
189	189	)
190	190
191		return self.db_pool.runInteraction(
	191	await self.db_pool.runInteraction(
192	192	"_update_aliases_for_room_txn", _update_aliases_for_room_txn
193	193	)

+40

-32

synapse/storage/databases/main/e2e_room_keys.py less more

13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
15	15
	16	from typing import Optional
	17
16	18	from synapse.api.errors import StoreError
17	19	from synapse.logging.opentracing import log_kv, trace
18	20	from synapse.storage._base import SQLBaseStore, db_to_json

148	150
149	151	return sessions
150	152
151		def get_e2e_room_keys_multi(self, user_id, version, room_keys):
	153	async def get_e2e_room_keys_multi(self, user_id, version, room_keys):
152	154	"""Get multiple room keys at a time. The difference between this function and
153	155	get_e2e_room_keys is that this function can be used to retrieve
154	156	multiple specific keys at a time, whereas get_e2e_room_keys is used for

163	165	that we want to query
164	166
165	167	Returns:
166		Deferred[dict[str, dict[str, dict]]]: a map of room IDs to session IDs to room key
167		"""
168
169		return self.db_pool.runInteraction(
	168	dict[str, dict[str, dict]]: a map of room IDs to session IDs to room key
	169	"""
	170
	171	return await self.db_pool.runInteraction(
170	172	"get_e2e_room_keys_multi",
171	173	self._get_e2e_room_keys_multi_txn,
172	174	user_id,

222	224
223	225	return ret
224	226
225		def count_e2e_room_keys(self, user_id, version):
	227	async def count_e2e_room_keys(self, user_id: str, version: str) -> int:
226	228	"""Get the number of keys in a backup version.
227	229
228	230	Args:
229		user_id (str): the user whose backup we're querying
230		version (str): the version ID of the backup we're querying about
231		"""
232
233		return self.db_pool.simple_select_one_onecol(
	231	user_id: the user whose backup we're querying
	232	version: the version ID of the backup we're querying about
	233	"""
	234
	235	return await self.db_pool.simple_select_one_onecol(
234	236	table="e2e_room_keys",
235	237	keyvalues={"user_id": user_id, "version": version},
236	238	retcol="COUNT(*)",

280	282	raise StoreError(404, "No current backup version")
281	283	return row[0]
282	284
283		def get_e2e_room_keys_version_info(self, user_id, version=None):
	285	async def get_e2e_room_keys_version_info(self, user_id, version=None):
284	286	"""Get info metadata about a version of our room_keys backup.
285	287
286	288	Args:

290	292	Raises:
291	293	StoreError: with code 404 if there are no e2e_room_keys_versions present
292	294	Returns:
293		A deferred dict giving the info metadata for this backup version, with
	295	A dict giving the info metadata for this backup version, with
294	296	fields including:
295	297	version(str)
296	298	algorithm(str)

321	323	result["etag"] = 0
322	324	return result
323	325
324		return self.db_pool.runInteraction(
	326	return await self.db_pool.runInteraction(
325	327	"get_e2e_room_keys_version_info", _get_e2e_room_keys_version_info_txn
326	328	)
327	329
328	330	@trace
329		def create_e2e_room_keys_version(self, user_id, info):
	331	async def create_e2e_room_keys_version(self, user_id: str, info: dict) -> str:
330	332	"""Atomically creates a new version of this user's e2e_room_keys store
331	333	with the given version info.
332	334

335	337	info(dict): the info about the backup version to be created
336	338
337	339	Returns:
338		A deferred string for the newly created version ID
	340	The newly created version ID
339	341	"""
340	342
341	343	def _create_e2e_room_keys_version_txn(txn):

362	364
363	365	return new_version
364	366
365		return self.db_pool.runInteraction(
	367	return await self.db_pool.runInteraction(
366	368	"create_e2e_room_keys_version_txn", _create_e2e_room_keys_version_txn
367	369	)
368	370
369	371	@trace
370		def update_e2e_room_keys_version(
371		self, user_id, version, info=None, version_etag=None
372		):
	372	async def update_e2e_room_keys_version(
	373	self,
	374	user_id: str,
	375	version: str,
	376	info: Optional[dict] = None,
	377	version_etag: Optional[int] = None,
	378	) -> None:
373	379	"""Update a given backup version
374	380
375	381	Args:
376		user_id(str): the user whose backup version we're updating
377		version(str): the version ID of the backup version we're updating
378		info (dict): the new backup version info to store. If None, then
379		the backup version info is not updated
380		version_etag (Optional[int]): etag of the keys in the backup. If
381		None, then the etag is not updated
	382	user_id: the user whose backup version we're updating
	383	version: the version ID of the backup version we're updating
	384	info: the new backup version info to store. If None, then the backup
	385	version info is not updated.
	386	version_etag: etag of the keys in the backup. If None, then the etag
	387	is not updated.
382	388	"""
383	389	updatevalues = {}
384	390

388	394	updatevalues["etag"] = version_etag
389	395
390	396	if updatevalues:
391		return self.db_pool.simple_update(
	397	await self.db_pool.simple_update(
392	398	table="e2e_room_keys_versions",
393	399	keyvalues={"user_id": user_id, "version": version},
394	400	updatevalues=updatevalues,

396	402	)
397	403
398	404	@trace
399		def delete_e2e_room_keys_version(self, user_id, version=None):
	405	async def delete_e2e_room_keys_version(
	406	self, user_id: str, version: Optional[str] = None
	407	) -> None:
400	408	"""Delete a given backup version of the user's room keys.
401	409	Doesn't delete their actual key data.
402	410
403	411	Args:
404		user_id(str): the user whose backup version we're deleting
405		version(str): Optional. the version ID of the backup version we're deleting
	412	user_id: the user whose backup version we're deleting
	413	version: Optional. the version ID of the backup version we're deleting
406	414	If missing, we delete the current backup version info.
407	415	Raises:
408	416	StoreError: with code 404 if there are no e2e_room_keys_versions present,

423	431	keyvalues={"user_id": user_id, "version": this_version},
424	432	)
425	433
426		return self.db_pool.simple_update_one_txn(
	434	self.db_pool.simple_update_one_txn(
427	435	txn,
428	436	table="e2e_room_keys_versions",
429	437	keyvalues={"user_id": user_id, "version": this_version},
430	438	updatevalues={"deleted": 1},
431	439	)
432	440
433		return self.db_pool.runInteraction(
	441	await self.db_pool.runInteraction(
434	442	"delete_e2e_room_keys_version", _delete_e2e_room_keys_version_txn
435	443	)

+238

-122

synapse/storage/databases/main/end_to_end_keys.py less more

13	13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14	14	# See the License for the specific language governing permissions and
15	15	# limitations under the License.
16		from typing import Dict, Iterable, List, Optional, Tuple
17
	16	import abc
	17	from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Tuple
	18
	19	import attr
18	20	from canonicaljson import encode_canonical_json
19	21
20	22	from twisted.enterprise.adbapi import Connection

22	24	from synapse.logging.opentracing import log_kv, set_tag, trace
23	25	from synapse.storage._base import SQLBaseStore, db_to_json
24	26	from synapse.storage.database import make_in_list_sql_clause
	27	from synapse.storage.types import Cursor
	28	from synapse.types import JsonDict
25	29	from synapse.util import json_encoder
26	30	from synapse.util.caches.descriptors import cached, cachedList
27	31	from synapse.util.iterutils import batch_iter
28	32
	33	if TYPE_CHECKING:
	34	from synapse.handlers.e2e_keys import SignatureListItem
	35
	36
	37	@attr.s
	38	class DeviceKeyLookupResult:
	39	"""The type returned by get_e2e_device_keys_and_signatures"""
	40
	41	display_name = attr.ib(type=Optional[str])
	42
	43	# the key data from e2e_device_keys_json. Typically includes fields like
	44	# "algorithm", "keys" (including the curve25519 identity key and the ed25519 signing
	45	# key) and "signatures" (a map from (user id) to (key id/device_id) to signature.)
	46	keys = attr.ib(type=Optional[JsonDict])
	47
29	48
30	49	class EndToEndKeyWorkerStore(SQLBaseStore):
	50	async def get_e2e_device_keys_for_federation_query(
	51	self, user_id: str
	52	) -> Tuple[int, List[JsonDict]]:
	53	"""Get all devices (with any device keys) for a user
	54
	55	Returns:
	56	(stream_id, devices)
	57	"""
	58	now_stream_id = self.get_device_stream_token()
	59
	60	devices = await self.get_e2e_device_keys_and_signatures([(user_id, None)])
	61
	62	if devices:
	63	user_devices = devices[user_id]
	64	results = []
	65	for device_id, device in user_devices.items():
	66	result = {"device_id": device_id}
	67
	68	keys = device.keys
	69	if keys:
	70	result["keys"] = keys
	71
	72	device_display_name = device.display_name
	73	if device_display_name:
	74	result["device_display_name"] = device_display_name
	75
	76	results.append(result)
	77
	78	return now_stream_id, results
	79
	80	return now_stream_id, []
	81
31	82	@trace
32		async def get_e2e_device_keys(
33		self, query_list, include_all_devices=False, include_deleted_devices=False
34		):
35		"""Fetch a list of device keys.
	83	async def get_e2e_device_keys_for_cs_api(
	84	self, query_list: List[Tuple[str, Optional[str]]]
	85	) -> Dict[str, Dict[str, JsonDict]]:
	86	"""Fetch a list of device keys, formatted suitably for the C/S API.
36	87	Args:
37	88	query_list(list): List of pairs of user_ids and device_ids.
38		include_all_devices (bool): whether to include entries for devices
39		that don't have device keys
40		include_deleted_devices (bool): whether to include null entries for
41		devices which no longer exist (but were in the query_list).
42		This option only takes effect if include_all_devices is true.
43	89	Returns:
44	90	Dict mapping from user-id to dict mapping from device_id to
45	91	key data. The key data will be a dict in the same format as the

49	95	if not query_list:
50	96	return {}
51	97
52		results = await self.db_pool.runInteraction(
53		"get_e2e_device_keys",
54		self._get_e2e_device_keys_txn,
55		query_list,
56		include_all_devices,
57		include_deleted_devices,
58		)
	98	results = await self.get_e2e_device_keys_and_signatures(query_list)
59	99
60	100	# Build the result structure, un-jsonify the results, and add the
61	101	# "unsigned" section

63	103	for user_id, device_keys in results.items():
64	104	rv[user_id] = {}
65	105	for device_id, device_info in device_keys.items():
66		r = db_to_json(device_info.pop("key_json"))
	106	r = device_info.keys
67	107	r["unsigned"] = {}
68		display_name = device_info["device_display_name"]
	108	display_name = device_info.display_name
69	109	if display_name is not None:
70	110	r["unsigned"]["device_display_name"] = display_name
71		if "signatures" in device_info:
72		for sig_user_id, sigs in device_info["signatures"].items():
73		r.setdefault("signatures", {}).setdefault(
74		sig_user_id, {}
75		).update(sigs)
76	111	rv[user_id][device_id] = r
77	112
78	113	return rv
79	114
80	115	@trace
	116	async def get_e2e_device_keys_and_signatures(
	117	self,
	118	query_list: List[Tuple[str, Optional[str]]],
	119	include_all_devices: bool = False,
	120	include_deleted_devices: bool = False,
	121	) -> Dict[str, Dict[str, Optional[DeviceKeyLookupResult]]]:
	122	"""Fetch a list of device keys
	123
	124	Any cross-signatures made on the keys by the owner of the device are also
	125	included.
	126
	127	The cross-signatures are added to the `signatures` field within the `keys`
	128	object in the response.
	129
	130	Args:
	131	query_list: List of pairs of user_ids and device_ids. Device id can be None
	132	to indicate "all devices for this user"
	133
	134	include_all_devices: whether to return devices without device keys
	135
	136	include_deleted_devices: whether to include null entries for
	137	devices which no longer exist (but were in the query_list).
	138	This option only takes effect if include_all_devices is true.
	139
	140	Returns:
	141	Dict mapping from user-id to dict mapping from device_id to
	142	key data.
	143	"""
	144	set_tag("include_all_devices", include_all_devices)
	145	set_tag("include_deleted_devices", include_deleted_devices)
	146
	147	result = await self.db_pool.runInteraction(
	148	"get_e2e_device_keys",
	149	self._get_e2e_device_keys_txn,
	150	query_list,
	151	include_all_devices,
	152	include_deleted_devices,
	153	)
	154
	155	# get the (user_id, device_id) tuples to look up cross-signatures for
	156	signature_query = (
	157	(user_id, device_id)
	158	for user_id, dev in result.items()
	159	for device_id, d in dev.items()
	160	if d is not None and d.keys is not None
	161	)
	162
	163	for batch in batch_iter(signature_query, 50):
	164	cross_sigs_result = await self.db_pool.runInteraction(
	165	"get_e2e_cross_signing_signatures",
	166	self._get_e2e_cross_signing_signatures_for_devices_txn,
	167	batch,
	168	)
	169
	170	# add each cross-signing signature to the correct device in the result dict.
	171	for (user_id, key_id, device_id, signature) in cross_sigs_result:
	172	target_device_result = result[user_id][device_id]
	173	target_device_signatures = target_device_result.keys.setdefault(
	174	"signatures", {}
	175	)
	176	signing_user_signatures = target_device_signatures.setdefault(
	177	user_id, {}
	178	)
	179	signing_user_signatures[key_id] = signature
	180
	181	log_kv(result)
	182	return result
	183
81	184	def _get_e2e_device_keys_txn(
82	185	self, txn, query_list, include_all_devices=False, include_deleted_devices=False
83		):
84		set_tag("include_all_devices", include_all_devices)
85		set_tag("include_deleted_devices", include_deleted_devices)
86
	186	) -> Dict[str, Dict[str, Optional[DeviceKeyLookupResult]]]:
	187	"""Get information on devices from the database
	188
	189	The results include the device's keys and self-signatures, but not any
	190	cross-signing signatures which have been added subsequently (for which, see
	191	get_e2e_device_keys_and_signatures)
	192	"""
87	193	query_clauses = []
88	194	query_params = []
89		signature_query_clauses = []
90		signature_query_params = []
91	195
92	196	if include_all_devices is False:
93	197	include_deleted_devices = False

98	202	for (user_id, device_id) in query_list:
99	203	query_clause = "user_id = ?"
100	204	query_params.append(user_id)
101		signature_query_clause = "target_user_id = ?"
102		signature_query_params.append(user_id)
103	205
104	206	if device_id is not None:
105	207	query_clause += " AND device_id = ?"
106	208	query_params.append(device_id)
107		signature_query_clause += " AND target_device_id = ?"
108		signature_query_params.append(device_id)
109
110		signature_query_clause += " AND user_id = ?"
111		signature_query_params.append(user_id)
112	209
113	210	query_clauses.append(query_clause)
114		signature_query_clauses.append(signature_query_clause)
115	211
116	212	sql = (
117	213	"SELECT user_id, device_id, "
118		" d.display_name AS device_display_name, "
	214	" d.display_name, "
119	215	" k.key_json"
120	216	" FROM devices d"
121	217	" %s JOIN e2e_device_keys_json k USING (user_id, device_id)"

126	222	)
127	223
128	224	txn.execute(sql, query_params)
129		rows = self.db_pool.cursor_to_dict(txn)
130
131		result = {}
132		for row in rows:
	225
	226	result = {} # type: Dict[str, Dict[str, Optional[DeviceKeyLookupResult]]]
	227	for (user_id, device_id, display_name, key_json) in txn:
133	228	if include_deleted_devices:
134		deleted_devices.remove((row["user_id"], row["device_id"]))
135		result.setdefault(row["user_id"], {})[row["device_id"]] = row
	229	deleted_devices.remove((user_id, device_id))
	230	result.setdefault(user_id, {})[device_id] = DeviceKeyLookupResult(
	231	display_name, db_to_json(key_json) if key_json else None
	232	)
136	233
137	234	if include_deleted_devices:
138	235	for user_id, device_id in deleted_devices:
139	236	result.setdefault(user_id, {})[device_id] = None
140	237
141		# get signatures on the device
142		signature_sql = ("SELECT * FROM e2e_cross_signing_signatures WHERE %s") % (
	238	return result
	239
	240	def _get_e2e_cross_signing_signatures_for_devices_txn(
	241	self, txn: Cursor, device_query: Iterable[Tuple[str, str]]
	242	) -> List[Tuple[str, str, str, str]]:
	243	"""Get cross-signing signatures for a given list of devices
	244
	245	Returns signatures made by the owners of the devices.
	246
	247	Returns: a list of results; each entry in the list is a tuple of
	248	(user_id, key_id, target_device_id, signature).
	249	"""
	250	signature_query_clauses = []
	251	signature_query_params = []
	252
	253	for (user_id, device_id) in device_query:
	254	signature_query_clauses.append(
	255	"target_user_id = ? AND target_device_id = ? AND user_id = ?"
	256	)
	257	signature_query_params.extend([user_id, device_id, user_id])
	258
	259	signature_sql = """
	260	SELECT user_id, key_id, target_device_id, signature
	261	FROM e2e_cross_signing_signatures WHERE %s
	262	""" % (
143	263	" OR ".join("(" + q + ")" for q in signature_query_clauses)
144	264	)
145	265
146	266	txn.execute(signature_sql, signature_query_params)
147		rows = self.db_pool.cursor_to_dict(txn)
148
149		# add each cross-signing signature to the correct device in the result dict.
150		for row in rows:
151		signing_user_id = row["user_id"]
152		signing_key_id = row["key_id"]
153		target_user_id = row["target_user_id"]
154		target_device_id = row["target_device_id"]
155		signature = row["signature"]
156
157		target_user_result = result.get(target_user_id)
158		if not target_user_result:
159		continue
160
161		target_device_result = target_user_result.get(target_device_id)
162		if not target_device_result:
163		# note that target_device_result will be None for deleted devices.
164		continue
165
166		target_device_signatures = target_device_result.setdefault("signatures", {})
167		signing_user_signatures = target_device_signatures.setdefault(
168		signing_user_id, {}
169		)
170		signing_user_signatures[signing_key_id] = signature
171
172		log_kv(result)
173		return result
	267	return txn.fetchall()
174	268
175	269	async def get_e2e_one_time_keys(
176	270	self, user_id: str, device_id: str, key_ids: List[str]

248	342	)
249	343
250	344	@cached(max_entries=10000)
251		def count_e2e_one_time_keys(self, user_id, device_id):
	345	async def count_e2e_one_time_keys(
	346	self, user_id: str, device_id: str
	347	) -> Dict[str, int]:
252	348	""" Count the number of one time keys the server has for a device
253	349	Returns:
254		Dict mapping from algorithm to number of keys for that algorithm.
	350	A mapping from algorithm to number of keys for that algorithm.
255	351	"""
256	352
257	353	def _count_e2e_one_time_keys(txn):

266	362	result[algorithm] = key_count
267	363	return result
268	364
269		return self.db_pool.runInteraction(
	365	return await self.db_pool.runInteraction(
270	366	"count_e2e_one_time_keys", _count_e2e_one_time_keys
271	367	)
272	368

304	400	list_name="user_ids",
305	401	num_args=1,
306	402	)
307		def _get_bare_e2e_cross_signing_keys_bulk(
	403	async def _get_bare_e2e_cross_signing_keys_bulk(
308	404	self, user_ids: List[str]
309	405	) -> Dict[str, Dict[str, dict]]:
310	406	"""Returns the cross-signing keys for a set of users. The output of this

312	408	the signatures for the calling user need to be fetched.
313	409
314	410	Args:
315		user_ids (list[str]): the users whose keys are being requested
316
317		Returns:
318		dict[str, dict[str, dict]]: mapping from user ID to key type to key
319		data. If a user's cross-signing keys were not found, either
320		their user ID will not be in the dict, or their user ID will map
321		to None.
322
323		"""
324		return self.db_pool.runInteraction(
	411	user_ids: the users whose keys are being requested
	412
	413	Returns:
	414	A mapping from user ID to key type to key data. If a user's cross-signing
	415	keys were not found, either their user ID will not be in the dict, or
	416	their user ID will map to None.
	417
	418	"""
	419	return await self.db_pool.runInteraction(
325	420	"get_bare_e2e_cross_signing_keys_bulk",
326	421	self._get_bare_e2e_cross_signing_keys_bulk_txn,
327	422	user_ids,

537	632	_get_all_user_signature_changes_for_remotes_txn,
538	633	)
539	634
	635	@abc.abstractmethod
	636	def get_device_stream_token(self) -> int:
	637	"""Get the current stream id from the _device_list_id_gen"""
	638	...
	639
540	640
541	641	class EndToEndKeyStore(EndToEndKeyWorkerStore, SQLBaseStore):
542		def set_e2e_device_keys(self, user_id, device_id, time_now, device_keys):
	642	async def set_e2e_device_keys(
	643	self, user_id: str, device_id: str, time_now: int, device_keys: JsonDict
	644	) -> bool:
543	645	"""Stores device keys for a device. Returns whether there was a change
544	646	or the keys were already in the database.
545	647	"""

575	677	log_kv({"message": "Device keys stored."})
576	678	return True
577	679
578		return self.db_pool.runInteraction(
	680	return await self.db_pool.runInteraction(
579	681	"set_e2e_device_keys", _set_e2e_device_keys_txn
580	682	)
581	683
582		def claim_e2e_one_time_keys(self, query_list):
583		"""Take a list of one time keys out of the database"""
	684	async def claim_e2e_one_time_keys(
	685	self, query_list: Iterable[Tuple[str, str, str]]
	686	) -> Dict[str, Dict[str, Dict[str, bytes]]]:
	687	"""Take a list of one time keys out of the database.
	688
	689	Args:
	690	query_list: An iterable of tuples of (user ID, device ID, algorithm).
	691
	692	Returns:
	693	A map of user ID -> a map device ID -> a map of key ID -> JSON bytes.
	694	"""
584	695
585	696	@trace
586	697	def _claim_e2e_one_time_keys(txn):

616	727	)
617	728	return result
618	729
619		return self.db_pool.runInteraction(
	730	return await self.db_pool.runInteraction(
620	731	"claim_e2e_one_time_keys", _claim_e2e_one_time_keys
621	732	)
622	733
623		def delete_e2e_keys_by_device(self, user_id, device_id):
	734	async def delete_e2e_keys_by_device(self, user_id: str, device_id: str) -> None:
624	735	def delete_e2e_keys_by_device_txn(txn):
625	736	log_kv(
626	737	{

643	754	txn, self.count_e2e_one_time_keys, (user_id, device_id)
644	755	)
645	756
646		return self.db_pool.runInteraction(
	757	await self.db_pool.runInteraction(
647	758	"delete_e2e_keys_by_device", delete_e2e_keys_by_device_txn
648	759	)
649	760
650		def _set_e2e_cross_signing_key_txn(self, txn, user_id, key_type, key):
	761	def _set_e2e_cross_signing_key_txn(self, txn, user_id, key_type, key, stream_id):
651	762	"""Set a user's cross-signing key.
652	763
653	764	Args:

657	768	for a master key, 'self_signing' for a self-signing key, or
658	769	'user_signing' for a user-signing key
659	770	key (dict): the key data
	771	stream_id (int)
660	772	"""
661	773	# the 'key' dict will look something like:
662	774	# {

694	806	)
695	807
696	808	# and finally, store the key itself
697		with self._cross_signing_id_gen.get_next() as stream_id:
698		self.db_pool.simple_insert_txn(
699		txn,
700		"e2e_cross_signing_keys",
701		values={
702		"user_id": user_id,
703		"keytype": key_type,
704		"keydata": json_encoder.encode(key),
705		"stream_id": stream_id,
706		},
707		)
	809	self.db_pool.simple_insert_txn(
	810	txn,
	811	"e2e_cross_signing_keys",
	812	values={
	813	"user_id": user_id,
	814	"keytype": key_type,
	815	"keydata": json_encoder.encode(key),
	816	"stream_id": stream_id,
	817	},
	818	)
708	819
709	820	self._invalidate_cache_and_stream(
710	821	txn, self._get_bare_e2e_cross_signing_keys, (user_id,)
711	822	)
712	823
713		def set_e2e_cross_signing_key(self, user_id, key_type, key):
	824	async def set_e2e_cross_signing_key(self, user_id, key_type, key):
714	825	"""Set a user's cross-signing key.
715	826
716	827	Args:

718	829	key_type (str): the type of cross-signing key to set
719	830	key (dict): the key data
720	831	"""
721		return self.db_pool.runInteraction(
722		"add_e2e_cross_signing_key",
723		self._set_e2e_cross_signing_key_txn,
724		user_id,
725		key_type,
726		key,
727		)
728
729		def store_e2e_cross_signing_signatures(self, user_id, signatures):
	832
	833	with await self._cross_signing_id_gen.get_next() as stream_id:
	834	return await self.db_pool.runInteraction(
	835	"add_e2e_cross_signing_key",
	836	self._set_e2e_cross_signing_key_txn,
	837	user_id,
	838	key_type,
	839	key,
	840	stream_id,
	841	)
	842
	843	async def store_e2e_cross_signing_signatures(
	844	self, user_id: str, signatures: "Iterable[SignatureListItem]"
	845	) -> None:
730	846	"""Stores cross-signing signatures.
731	847
732	848	Args:
733		user_id (str): the user who made the signatures
734		signatures (iterable[SignatureListItem]): signatures to add
735		"""
736		return self.db_pool.simple_insert_many(
	849	user_id: the user who made the signatures
	850	signatures: signatures to add
	851	"""
	852	await self.db_pool.simple_insert_many(
737	853	"e2e_cross_signing_signatures",
738	854	[
739	855	{

+80

-98

synapse/storage/databases/main/event_federation.py less more

14	14	import itertools
15	15	import logging
16	16	from queue import Empty, PriorityQueue
17		from typing import Dict, Iterable, List, Optional, Set, Tuple
	17	from typing import Dict, Iterable, List, Set, Tuple
18	18
19	19	from synapse.api.errors import StoreError
	20	from synapse.events import EventBase
20	21	from synapse.metrics.background_process_metrics import run_as_background_process
21	22	from synapse.storage._base import SQLBaseStore, make_in_list_sql_clause
22		from synapse.storage.database import DatabasePool
	23	from synapse.storage.database import DatabasePool, LoggingTransaction
23	24	from synapse.storage.databases.main.events_worker import EventsWorkerStore
24	25	from synapse.storage.databases.main.signatures import SignatureWorkerStore
	26	from synapse.types import Collection
25	27	from synapse.util.caches.descriptors import cached
26	28	from synapse.util.iterutils import batch_iter
27	29

29	31
30	32
31	33	class EventFederationWorkerStore(EventsWorkerStore, SignatureWorkerStore, SQLBaseStore):
32		def get_auth_chain(self, event_ids, include_given=False):
	34	async def get_auth_chain(
	35	self, event_ids: Collection[str], include_given: bool = False
	36	) -> List[EventBase]:
33	37	"""Get auth events for given event_ids. The events must be state events.
34	38
35	39	Args:
36		event_ids (list): state events
37		include_given (bool): include the given events in result
	40	event_ids: state events
	41	include_given: include the given events in result
38	42
39	43	Returns:
40	44	list of events
41	45	"""
42		return self.get_auth_chain_ids(
	46	event_ids = await self.get_auth_chain_ids(
43	47	event_ids, include_given=include_given
44		).addCallback(self.get_events_as_list)
45
46		def get_auth_chain_ids(
47		self,
48		event_ids: List[str],
49		include_given: bool = False,
50		ignore_events: Optional[Set[str]] = None,
51		):
	48	)
	49	return await self.get_events_as_list(event_ids)
	50
	51	async def get_auth_chain_ids(
	52	self, event_ids: Collection[str], include_given: bool = False,
	53	) -> List[str]:
52	54	"""Get auth events for given event_ids. The events must be state events.
53	55
54	56	Args:
55	57	event_ids: state events
56	58	include_given: include the given events in result
57		ignore_events: Set of events to exclude from the returned auth
58		chain. This is useful if the caller will just discard the
59		given events anyway, and saves us from figuring out their auth
60		chains if not required.
61	59
62	60	Returns:
63		list of event_ids
64		"""
65		return self.db_pool.runInteraction(
	61	An awaitable which resolve to a list of event_ids
	62	"""
	63	return await self.db_pool.runInteraction(
66	64	"get_auth_chain_ids",
67	65	self._get_auth_chain_ids_txn,
68	66	event_ids,
69	67	include_given,
70		ignore_events,
71		)
72
73		def _get_auth_chain_ids_txn(self, txn, event_ids, include_given, ignore_events):
74		if ignore_events is None:
75		ignore_events = set()
76
	68	)
	69
	70	def _get_auth_chain_ids_txn(
	71	self, txn: LoggingTransaction, event_ids: Collection[str], include_given: bool
	72	) -> List[str]:
77	73	if include_given:
78	74	results = set(event_ids)
79	75	else:
80	76	results = set()
81	77
82		base_sql = "SELECT auth_id FROM event_auth WHERE "
	78	base_sql = "SELECT DISTINCT auth_id FROM event_auth WHERE "
83	79
84	80	front = set(event_ids)
85	81	while front:

91	87	txn.execute(base_sql + clause, args)
92	88	new_front.update(r[0] for r in txn)
93	89
94		new_front -= ignore_events
95	90	new_front -= results
96	91
97	92	front = new_front

99	94
100	95	return list(results)
101	96
102		def get_auth_chain_difference(self, state_sets: List[Set[str]]):
	97	async def get_auth_chain_difference(self, state_sets: List[Set[str]]) -> Set[str]:
103	98	"""Given sets of state events figure out the auth chain difference (as
104	99	per state res v2 algorithm).
105	100

108	103	chain.
109	104
110	105	Returns:
111		Deferred[Set[str]]
112		"""
113
114		return self.db_pool.runInteraction(
	106	The set of the difference in auth chains.
	107	"""
	108
	109	return await self.db_pool.runInteraction(
115	110	"get_auth_chain_difference",
116	111	self._get_auth_chain_difference_txn,
117	112	state_sets,

256	251	# Return all events where not all sets can reach them.
257	252	return {eid for eid, n in event_to_missing_sets.items() if n}
258	253
259		def get_oldest_events_in_room(self, room_id):
260		return self.db_pool.runInteraction(
261		"get_oldest_events_in_room", self._get_oldest_events_in_room_txn, room_id
262		)
263
264		def get_oldest_events_with_depth_in_room(self, room_id):
265		return self.db_pool.runInteraction(
	254	async def get_oldest_events_with_depth_in_room(self, room_id):
	255	return await self.db_pool.runInteraction(
266	256	"get_oldest_events_with_depth_in_room",
267	257	self.get_oldest_events_with_depth_in_room_txn,
268	258	room_id,

302	292	else:
303	293	return max(row["depth"] for row in rows)
304	294
305		def _get_oldest_events_in_room_txn(self, txn, room_id):
306		return self.db_pool.simple_select_onecol_txn(
307		txn,
308		table="event_backward_extremities",
309		keyvalues={"room_id": room_id},
310		retcol="event_id",
311		)
312
313		def get_prev_events_for_room(self, room_id: str):
	295	async def get_prev_events_for_room(self, room_id: str) -> List[str]:
314	296	"""
315	297	Gets a subset of the current forward extremities in the given room.
316	298

318	300	events which refer to hundreds of prev_events.
319	301
320	302	Args:
321		room_id (str): room_id
	303	room_id: room_id
322	304
323	305	Returns:
324		Deferred[List[str]]: the event ids of the forward extremites
325
326		"""
327
328		return self.db_pool.runInteraction(
	306	The event ids of the forward extremities.
	307
	308	"""
	309
	310	return await self.db_pool.runInteraction(
329	311	"get_prev_events_for_room", self._get_prev_events_for_room_txn, room_id
330	312	)
331	313

345	327
346	328	return [row[0] for row in txn]
347	329
348		def get_rooms_with_many_extremities(self, min_count, limit, room_id_filter):
	330	async def get_rooms_with_many_extremities(
	331	self, min_count: int, limit: int, room_id_filter: Iterable[str]
	332	) -> List[str]:
349	333	"""Get the top rooms with at least N extremities.
350	334
351	335	Args:
352		min_count (int): The minimum number of extremities
353		limit (int): The maximum number of rooms to return.
354		room_id_filter (iterable[str]): room_ids to exclude from the results
	336	min_count: The minimum number of extremities
	337	limit: The maximum number of rooms to return.
	338	room_id_filter: room_ids to exclude from the results
355	339
356	340	Returns:
357		Deferred[list]: At most `limit` room IDs that have at least
358		`min_count` extremities, sorted by extremity count.
	341	At most `limit` room IDs that have at least `min_count` extremities,
	342	sorted by extremity count.
359	343	"""
360	344
361	345	def _get_rooms_with_many_extremities_txn(txn):

380	364	txn.execute(sql, query_args)
381	365	return [room_id for room_id, in txn]
382	366
383		return self.db_pool.runInteraction(
	367	return await self.db_pool.runInteraction(
384	368	"get_rooms_with_many_extremities", _get_rooms_with_many_extremities_txn
385	369	)
386	370
387	371	@cached(max_entries=5000, iterable=True)
388		def get_latest_event_ids_in_room(self, room_id):
389		return self.db_pool.simple_select_onecol(
	372	async def get_latest_event_ids_in_room(self, room_id: str) -> List[str]:
	373	return await self.db_pool.simple_select_onecol(
390	374	table="event_forward_extremities",
391	375	keyvalues={"room_id": room_id},
392	376	retcol="event_id",
393	377	desc="get_latest_event_ids_in_room",
394	378	)
395	379
396		def get_min_depth(self, room_id):
397		""" For hte given room, get the minimum depth we have seen for it.
398		"""
399		return self.db_pool.runInteraction(
	380	async def get_min_depth(self, room_id: str) -> int:
	381	"""For the given room, get the minimum depth we have seen for it.
	382	"""
	383	return await self.db_pool.runInteraction(
400	384	"get_min_depth", self._get_min_depth_interaction, room_id
401	385	)
402	386

411	395
412	396	return int(min_depth) if min_depth is not None else None
413	397
414		def get_forward_extremeties_for_room(self, room_id, stream_ordering):
	398	async def get_forward_extremeties_for_room(
	399	self, room_id: str, stream_ordering: int
	400	) -> List[str]:
415	401	"""For a given room_id and stream_ordering, return the forward
416	402	extremeties of the room at that point in "time".
417	403

419	405	stream_orderings from that point.
420	406
421	407	Args:
422		room_id (str):
423		stream_ordering (int):
	408	room_id:
	409	stream_ordering:
424	410
425	411	Returns:
426		deferred, which resolves to a list of event_ids
	412	A list of event_ids
427	413	"""
428	414	# We want to make the cache more effective, so we clamp to the last
429	415	# change before the given ordering.

439	425	if last_change > self.stream_ordering_month_ago:
440	426	stream_ordering = min(last_change, stream_ordering)
441	427
442		return self._get_forward_extremeties_for_room(room_id, stream_ordering)
	428	return await self._get_forward_extremeties_for_room(room_id, stream_ordering)
443	429
444	430	@cached(max_entries=5000, num_args=2)
445		def _get_forward_extremeties_for_room(self, room_id, stream_ordering):
	431	async def _get_forward_extremeties_for_room(self, room_id, stream_ordering):
446	432	"""For a given room_id and stream_ordering, return the forward
447	433	extremeties of the room at that point in "time".
448	434

467	453	txn.execute(sql, (stream_ordering, room_id))
468	454	return [event_id for event_id, in txn]
469	455
470		return self.db_pool.runInteraction(
	456	return await self.db_pool.runInteraction(
471	457	"get_forward_extremeties_for_room", get_forward_extremeties_for_room_txn
472	458	)
473	459
474		def get_backfill_events(self, room_id, event_list, limit):
	460	async def get_backfill_events(self, room_id: str, event_list: list, limit: int):
475	461	"""Get a list of Events for a given topic that occurred before (and
476	462	including) the events in event_list. Return a list of max size `limit`
477	463
478	464	Args:
479		txn
480		room_id (str)
481		event_list (list)
482		limit (int)
483		"""
484		return (
485		self.db_pool.runInteraction(
486		"get_backfill_events",
487		self._get_backfill_events,
488		room_id,
489		event_list,
490		limit,
491		)
492		.addCallback(self.get_events_as_list)
493		.addCallback(lambda l: sorted(l, key=lambda e: -e.depth))
494		)
	465	room_id
	466	event_list
	467	limit
	468	"""
	469	event_ids = await self.db_pool.runInteraction(
	470	"get_backfill_events",
	471	self._get_backfill_events,
	472	room_id,
	473	event_list,
	474	limit,
	475	)
	476	events = await self.get_events_as_list(event_ids)
	477	return sorted(events, key=lambda e: -e.depth)
495	478
496	479	def _get_backfill_events(self, txn, room_id, event_list, limit):
497	480	logger.debug("_get_backfill_events: %s, %r, %s", room_id, event_list, limit)

552	535	latest_events,
553	536	limit,
554	537	)
555		events = await self.get_events_as_list(ids)
556		return events
	538	return await self.get_events_as_list(ids)
557	539
558	540	def _get_missing_events(self, txn, room_id, earliest_events, latest_events, limit):
559	541

651	633	_delete_old_forward_extrem_cache_txn,
652	634	)
653	635
654		def clean_room_for_join(self, room_id):
655		return self.db_pool.runInteraction(
	636	async def clean_room_for_join(self, room_id):
	637	return await self.db_pool.runInteraction(
656	638	"clean_room_for_join", self._clean_room_for_join_txn, room_id
657	639	)
658	640

+192

-95

synapse/storage/databases/main/event_push_actions.py less more

14	14	# limitations under the License.
15	15
16	16	import logging
17		from typing import List
	17	from typing import Dict, List, Optional, Tuple, Union
	18
	19	import attr
18	20
19	21	from synapse.metrics.background_process_metrics import run_as_background_process
20	22	from synapse.storage._base import LoggingTransaction, SQLBaseStore, db_to_json
21	23	from synapse.storage.database import DatabasePool
22	24	from synapse.util import json_encoder
23		from synapse.util.caches.descriptors import cachedInlineCallbacks
	25	from synapse.util.caches.descriptors import cached
24	26
25	27	logger = logging.getLogger(__name__)
26	28

85	87	self._rotate_delay = 3
86	88	self._rotate_count = 10000
87	89
88		@cachedInlineCallbacks(num_args=3, tree=True, max_entries=5000)
89		def get_unread_event_push_actions_by_room_for_user(
90		self, room_id, user_id, last_read_event_id
91		):
92		ret = yield self.db_pool.runInteraction(
	90	@cached(num_args=3, tree=True, max_entries=5000)
	91	async def get_unread_event_push_actions_by_room_for_user(
	92	self, room_id: str, user_id: str, last_read_event_id: Optional[str],
	93	) -> Dict[str, int]:
	94	"""Get the notification count, the highlight count and the unread message count
	95	for a given user in a given room after the given read receipt.
	96
	97	Note that this function assumes the user to be a current member of the room,
	98	since it's either called by the sync handler to handle joined room entries, or by
	99	the HTTP pusher to calculate the badge of unread joined rooms.
	100
	101	Args:
	102	room_id: The room to retrieve the counts in.
	103	user_id: The user to retrieve the counts for.
	104	last_read_event_id: The event associated with the latest read receipt for
	105	this user in this room. None if no receipt for this user in this room.
	106
	107	Returns
	108	A dict containing the counts mentioned earlier in this docstring,
	109	respectively under the keys "notify_count", "highlight_count" and
	110	"unread_count".
	111	"""
	112	return await self.db_pool.runInteraction(
93	113	"get_unread_event_push_actions_by_room",
94	114	self._get_unread_counts_by_receipt_txn,
95	115	room_id,
96	116	user_id,
97	117	last_read_event_id,
98	118	)
99		return ret
100	119
101	120	def _get_unread_counts_by_receipt_txn(
102		self, txn, room_id, user_id, last_read_event_id
	121	self, txn, room_id, user_id, last_read_event_id,
103	122	):
104		sql = (
105		"SELECT stream_ordering"
106		" FROM events"
107		" WHERE room_id = ? AND event_id = ?"
108		)
109		txn.execute(sql, (room_id, last_read_event_id))
110		results = txn.fetchall()
111		if len(results) == 0:
112		return {"notify_count": 0, "highlight_count": 0}
113
114		stream_ordering = results[0][0]
	123	stream_ordering = None
	124
	125	if last_read_event_id is not None:
	126	stream_ordering = self.get_stream_id_for_event_txn(
	127	txn, last_read_event_id, allow_none=True,
	128	)
	129
	130	if stream_ordering is None:
	131	# Either last_read_event_id is None, or it's an event we don't have (e.g.
	132	# because it's been purged), in which case retrieve the stream ordering for
	133	# the latest membership event from this user in this room (which we assume is
	134	# a join).
	135	event_id = self.db_pool.simple_select_one_onecol_txn(
	136	txn=txn,
	137	table="local_current_membership",
	138	keyvalues={"room_id": room_id, "user_id": user_id},
	139	retcol="event_id",
	140	)
	141
	142	stream_ordering = self.get_stream_id_for_event_txn(txn, event_id)
115	143
116	144	return self._get_unread_counts_by_pos_txn(
117	145	txn, room_id, user_id, stream_ordering
118	146	)
119	147
120	148	def _get_unread_counts_by_pos_txn(self, txn, room_id, user_id, stream_ordering):
121
122		# First get number of notifications.
123		# We don't need to put a notif=1 clause as all rows always have
124		# notif=1
125	149	sql = (
126		"SELECT count(*)"
	150	"SELECT"
	151	" COUNT(CASE WHEN notif = 1 THEN 1 END),"
	152	" COUNT(CASE WHEN highlight = 1 THEN 1 END),"
	153	" COUNT(CASE WHEN unread = 1 THEN 1 END)"
127	154	" FROM event_push_actions ea"
128		" WHERE"
129		" user_id = ?"
130		" AND room_id = ?"
131		" AND stream_ordering > ?"
	155	" WHERE user_id = ?"
	156	" AND room_id = ?"
	157	" AND stream_ordering > ?"
132	158	)
133	159
134	160	txn.execute(sql, (user_id, room_id, stream_ordering))
135	161	row = txn.fetchone()
136		notify_count = row[0] if row else 0
	162
	163	(notif_count, highlight_count, unread_count) = (0, 0, 0)
	164
	165	if row:
	166	(notif_count, highlight_count, unread_count) = row
137	167
138	168	txn.execute(
139	169	"""
140		SELECT notif_count FROM event_push_summary
141		WHERE room_id = ? AND user_id = ? AND stream_ordering > ?
142		""",
	170	SELECT notif_count, unread_count FROM event_push_summary
	171	WHERE room_id = ? AND user_id = ? AND stream_ordering > ?
	172	""",
143	173	(room_id, user_id, stream_ordering),
144	174	)
145		rows = txn.fetchall()
146		if rows:
147		notify_count += rows[0][0]
148
149		# Now get the number of highlights
150		sql = (
151		"SELECT count(*)"
152		" FROM event_push_actions ea"
153		" WHERE"
154		" highlight = 1"
155		" AND user_id = ?"
156		" AND room_id = ?"
157		" AND stream_ordering > ?"
158		)
159
160		txn.execute(sql, (user_id, room_id, stream_ordering))
161	175	row = txn.fetchone()
162		highlight_count = row[0] if row else 0
163
164		return {"notify_count": notify_count, "highlight_count": highlight_count}
	176
	177	if row:
	178	notif_count += row[0]
	179
	180	if row[1] is not None:
	181	# The unread_count column of event_push_summary is NULLable, so we need
	182	# to make sure we don't try increasing the unread counts if it's NULL
	183	# for this row.
	184	unread_count += row[1]
	185
	186	return {
	187	"notify_count": notif_count,
	188	"unread_count": unread_count,
	189	"highlight_count": highlight_count,
	190	}
165	191
166	192	async def get_push_action_users_in_range(
167	193	self, min_stream_ordering, max_stream_ordering

169	195	def f(txn):
170	196	sql = (
171	197	"SELECT DISTINCT(user_id) FROM event_push_actions WHERE"
172		" stream_ordering >= ? AND stream_ordering <= ?"
	198	" stream_ordering >= ? AND stream_ordering <= ? AND notif = 1"
173	199	)
174	200	txn.execute(sql, (min_stream_ordering, max_stream_ordering))
175	201	return [r[0] for r in txn]

222	248	" AND ep.user_id = ?"
223	249	" AND ep.stream_ordering > ?"
224	250	" AND ep.stream_ordering <= ?"
	251	" AND ep.notif = 1"
225	252	" ORDER BY ep.stream_ordering ASC LIMIT ?"
226	253	)
227	254	args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]

250	277	" AND ep.user_id = ?"
251	278	" AND ep.stream_ordering > ?"
252	279	" AND ep.stream_ordering <= ?"
	280	" AND ep.notif = 1"
253	281	" ORDER BY ep.stream_ordering ASC LIMIT ?"
254	282	)
255	283	args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]

324	352	" AND ep.user_id = ?"
325	353	" AND ep.stream_ordering > ?"
326	354	" AND ep.stream_ordering <= ?"
	355	" AND ep.notif = 1"
327	356	" ORDER BY ep.stream_ordering DESC LIMIT ?"
328	357	)
329	358	args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]

352	381	" AND ep.user_id = ?"
353	382	" AND ep.stream_ordering > ?"
354	383	" AND ep.stream_ordering <= ?"
	384	" AND ep.notif = 1"
355	385	" ORDER BY ep.stream_ordering DESC LIMIT ?"
356	386	)
357	387	args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]

383	413	# Now return the first `limit`
384	414	return notifs[:limit]
385	415
386		def get_if_maybe_push_in_range_for_user(self, user_id, min_stream_ordering):
	416	async def get_if_maybe_push_in_range_for_user(
	417	self, user_id: str, min_stream_ordering: int
	418	) -> bool:
387	419	"""A fast check to see if there might be something to push for the
388	420	user since the given stream ordering. May return false positives.
389	421
390	422	Useful to know whether to bother starting a pusher on start up or not.
391	423
392	424	Args:
393		user_id (str)
394		min_stream_ordering (int)
	425	user_id
	426	min_stream_ordering
395	427
396	428	Returns:
397		Deferred[bool]: True if there may be push to process, False if
398		there definitely isn't.
	429	True if there may be push to process, False if there definitely isn't.
399	430	"""
400	431
401	432	def _get_if_maybe_push_in_range_for_user_txn(txn):
402	433	sql = """
403	434	SELECT 1 FROM event_push_actions
404		WHERE user_id = ? AND stream_ordering > ?
	435	WHERE user_id = ? AND stream_ordering > ? AND notif = 1
405	436	LIMIT 1
406	437	"""
407	438
408	439	txn.execute(sql, (user_id, min_stream_ordering))
409	440	return bool(txn.fetchone())
410	441
411		return self.db_pool.runInteraction(
	442	return await self.db_pool.runInteraction(
412	443	"get_if_maybe_push_in_range_for_user",
413	444	_get_if_maybe_push_in_range_for_user_txn,
414	445	)
415	446
416		async def add_push_actions_to_staging(self, event_id, user_id_actions):
	447	async def add_push_actions_to_staging(
	448	self,
	449	event_id: str,
	450	user_id_actions: Dict[str, List[Union[dict, str]]],
	451	count_as_unread: bool,
	452	) -> None:
417	453	"""Add the push actions for the event to the push action staging area.
418	454
419	455	Args:
420		event_id (str)
421		user_id_actions (dict[str, list[dict\|str])]): A dictionary mapping
422		user_id to list of push actions, where an action can either be
423		a string or dict.
424
425		Returns:
426		Deferred
427		"""
428
	456	event_id
	457	user_id_actions: A mapping of user_id to list of push actions, where
	458	an action can either be a string or dict.
	459	count_as_unread: Whether this event should increment unread counts.
	460	"""
429	461	if not user_id_actions:
430	462	return
431	463
432	464	# This is a helper function for generating the necessary tuple that
433		# can be used to inert into the `event_push_actions_staging` table.
	465	# can be used to insert into the `event_push_actions_staging` table.
434	466	def _gen_entry(user_id, actions):
435	467	is_highlight = 1 if _action_has_highlight(actions) else 0
	468	notif = 1 if "notify" in actions else 0
436	469	return (
437	470	event_id, # event_id column
438	471	user_id, # user_id column
439	472	_serialize_action(actions, is_highlight), # actions column
440		1, # notif column
	473	notif, # notif column
441	474	is_highlight, # highlight column
	475	int(count_as_unread), # unread column
442	476	)
443	477
444	478	def _add_push_actions_to_staging_txn(txn):

447	481
448	482	sql = """
449	483	INSERT INTO event_push_actions_staging
450		(event_id, user_id, actions, notif, highlight)
451		VALUES (?, ?, ?, ?, ?)
	484	(event_id, user_id, actions, notif, highlight, unread)
	485	VALUES (?, ?, ?, ?, ?, ?)
452	486	"""
453	487
454	488	txn.executemany(

507	541	"Found stream ordering 1 day ago: it's %d", self.stream_ordering_day_ago
508	542	)
509	543
510		def find_first_stream_ordering_after_ts(self, ts):
	544	async def find_first_stream_ordering_after_ts(self, ts: int) -> int:
511	545	"""Gets the stream ordering corresponding to a given timestamp.
512	546
513	547	Specifically, finds the stream_ordering of the first event that was

516	550	relatively slow.
517	551
518	552	Args:
519		ts (int): timestamp in millis
	553	ts: timestamp in millis
520	554
521	555	Returns:
522		Deferred[int]: stream ordering of the first event received on/after
523		the timestamp
524		"""
525		return self.db_pool.runInteraction(
	556	stream ordering of the first event received on/after the timestamp
	557	"""
	558	return await self.db_pool.runInteraction(
526	559	"_find_first_stream_ordering_after_ts_txn",
527	560	self._find_first_stream_ordering_after_ts_txn,
528	561	ts,

610	643	"SELECT e.received_ts"
611	644	" FROM event_push_actions AS ep"
612	645	" JOIN events e ON ep.room_id = e.room_id AND ep.event_id = e.event_id"
613		" WHERE ep.stream_ordering > ?"
	646	" WHERE ep.stream_ordering > ? AND notif = 1"
614	647	" ORDER BY ep.stream_ordering ASC"
615	648	" LIMIT 1"
616	649	)

674	707	" FROM event_push_actions epa, events e"
675	708	" WHERE epa.event_id = e.event_id"
676	709	" AND epa.user_id = ? %s"
	710	" AND epa.notif = 1"
677	711	" ORDER BY epa.stream_ordering DESC"
678	712	" LIMIT ?" % (before_clause,)
679	713	)

813	847	# Calculate the new counts that should be upserted into event_push_summary
814	848	sql = """
815	849	SELECT user_id, room_id,
816		coalesce(old.notif_count, 0) + upd.notif_count,
	850	coalesce(old.%s, 0) + upd.cnt,
817	851	upd.stream_ordering,
818	852	old.user_id
819	853	FROM (
820		SELECT user_id, room_id, count(*) as notif_count,
	854	SELECT user_id, room_id, count(*) as cnt,
821	855	max(stream_ordering) as stream_ordering
822	856	FROM event_push_actions
823	857	WHERE ? <= stream_ordering AND stream_ordering < ?
824	858	AND highlight = 0
	859	AND %s = 1
825	860	GROUP BY user_id, room_id
826	861	) AS upd
827	862	LEFT JOIN event_push_summary AS old USING (user_id, room_id)
828	863	"""
829	864
830		txn.execute(sql, (old_rotate_stream_ordering, rotate_to_stream_ordering))
831		rows = txn.fetchall()
832
833		logger.info("Rotating notifications, handling %d rows", len(rows))
	865	# First get the count of unread messages.
	866	txn.execute(
	867	sql % ("unread_count", "unread"),
	868	(old_rotate_stream_ordering, rotate_to_stream_ordering),
	869	)
	870
	871	# We need to merge results from the two requests (the one that retrieves the
	872	# unread count and the one that retrieves the notifications count) into a single
	873	# object because we might not have the same amount of rows in each of them. To do
	874	# this, we use a dict indexed on the user ID and room ID to make it easier to
	875	# populate.
	876	summaries = {} # type: Dict[Tuple[str, str], _EventPushSummary]
	877	for row in txn:
	878	summaries[(row[0], row[1])] = _EventPushSummary(
	879	unread_count=row[2],
	880	stream_ordering=row[3],
	881	old_user_id=row[4],
	882	notif_count=0,
	883	)
	884
	885	# Then get the count of notifications.
	886	txn.execute(
	887	sql % ("notif_count", "notif"),
	888	(old_rotate_stream_ordering, rotate_to_stream_ordering),
	889	)
	890
	891	for row in txn:
	892	if (row[0], row[1]) in summaries:
	893	summaries[(row[0], row[1])].notif_count = row[2]
	894	else:
	895	# Because the rules on notifying are different than the rules on marking
	896	# a message unread, we might end up with messages that notify but aren't
	897	# marked unread, so we might not have a summary for this (user, room)
	898	# tuple to complete.
	899	summaries[(row[0], row[1])] = _EventPushSummary(
	900	unread_count=0,
	901	stream_ordering=row[3],
	902	old_user_id=row[4],
	903	notif_count=row[2],
	904	)
	905
	906	logger.info("Rotating notifications, handling %d rows", len(summaries))
834	907
835	908	# If the `old.user_id` above is NULL then we know there isn't already an
836	909	# entry in the table, so we simply insert it. Otherwise we update the

840	913	table="event_push_summary",
841	914	values=[
842	915	{
843		"user_id": row[0],
844		"room_id": row[1],
845		"notif_count": row[2],
846		"stream_ordering": row[3],
	916	"user_id": user_id,
	917	"room_id": room_id,
	918	"notif_count": summary.notif_count,
	919	"unread_count": summary.unread_count,
	920	"stream_ordering": summary.stream_ordering,
847	921	}
848		for row in rows
849		if row[4] is None
	922	for ((user_id, room_id), summary) in summaries.items()
	923	if summary.old_user_id is None
850	924	],
851	925	)
852	926
853	927	txn.executemany(
854	928	"""
855		UPDATE event_push_summary SET notif_count = ?, stream_ordering = ?
	929	UPDATE event_push_summary
	930	SET notif_count = ?, unread_count = ?, stream_ordering = ?
856	931	WHERE user_id = ? AND room_id = ?
857	932	""",
858		((row[2], row[3], row[0], row[1]) for row in rows if row[4] is not None),
	933	(
	934	(
	935	summary.notif_count,
	936	summary.unread_count,
	937	summary.stream_ordering,
	938	user_id,
	939	room_id,
	940	)
	941	for ((user_id, room_id), summary) in summaries.items()
	942	if summary.old_user_id is not None
	943	),
859	944	)
860	945
861	946	txn.execute(

881	966	pass
882	967
883	968	return False
	969
	970
	971	@attr.s
	972	class _EventPushSummary:
	973	"""Summary of pending event push actions for a given user in a given room.
	974	Used in _rotate_notifs_before_txn to manipulate results from event_push_actions.
	975	"""
	976
	977	unread_count = attr.ib(type=int)
	978	stream_ordering = attr.ib(type=int)
	979	old_user_id = attr.ib(type=str)
	980	notif_count = attr.ib(type=int)

+19

-24

synapse/storage/databases/main/events.py less more

16	16	import itertools
17	17	import logging
18	18	from collections import OrderedDict, namedtuple
19		from typing import TYPE_CHECKING, Dict, Iterable, List, Tuple
	19	from typing import TYPE_CHECKING, Dict, Iterable, List, Set, Tuple
20	20
21	21	import attr
22	22	from prometheus_client import Counter
23
24		from twisted.internet import defer
25	23
26	24	import synapse.metrics
27	25	from synapse.api.constants import EventContentFields, EventTypes, RelationTypes

112	110	hs.config.worker.writers.events == hs.get_instance_name()
113	111	), "Can only instantiate EventsStore on master"
114	112
115		@defer.inlineCallbacks
116		def _persist_events_and_state_updates(
	113	async def _persist_events_and_state_updates(
117	114	self,
118	115	events_and_contexts: List[Tuple[EventBase, EventContext]],
119	116	current_state_for_room: Dict[str, StateMap[str]],
120	117	state_delta_for_room: Dict[str, DeltaState],
121	118	new_forward_extremeties: Dict[str, List[str]],
122	119	backfilled: bool = False,
123		):
	120	) -> None:
124	121	"""Persist a set of events alongside updates to the current state and
125	122	forward extremities tables.
126	123

135	132	backfilled
136	133
137	134	Returns:
138		Deferred: resolves when the events have been persisted
	135	Resolves when the events have been persisted
139	136	"""
140	137
141	138	# We want to calculate the stream orderings as late as possible, as

155	152	# Note: Multiple instances of this function cannot be in flight at
156	153	# the same time for the same room.
157	154	if backfilled:
158		stream_ordering_manager = self._backfill_id_gen.get_next_mult(
	155	stream_ordering_manager = await self._backfill_id_gen.get_next_mult(
159	156	len(events_and_contexts)
160	157	)
161	158	else:
162		stream_ordering_manager = self._stream_id_gen.get_next_mult(
	159	stream_ordering_manager = await self._stream_id_gen.get_next_mult(
163	160	len(events_and_contexts)
164	161	)
165	162

167	164	for (event, context), stream in zip(events_and_contexts, stream_orderings):
168	165	event.internal_metadata.stream_ordering = stream
169	166
170		yield self.db_pool.runInteraction(
	167	await self.db_pool.runInteraction(
171	168	"persist_events",
172	169	self._persist_events_txn,
173	170	events_and_contexts=events_and_contexts,

205	202	(room_id,), list(latest_event_ids)
206	203	)
207	204
208		@defer.inlineCallbacks
209		def _get_events_which_are_prevs(self, event_ids):
	205	async def _get_events_which_are_prevs(self, event_ids: Iterable[str]) -> List[str]:
210	206	"""Filter the supplied list of event_ids to get those which are prev_events of
211	207	existing (non-outlier/rejected) events.
212	208
213	209	Args:
214		event_ids (Iterable[str]): event ids to filter
	210	event_ids: event ids to filter
215	211
216	212	Returns:
217		Deferred[List[str]]: filtered event ids
	213	Filtered event ids
218	214	"""
219	215	results = []
220	216

239	235	results.extend(r[0] for r in txn if not db_to_json(r[1]).get("soft_failed"))
240	236
241	237	for chunk in batch_iter(event_ids, 100):
242		yield self.db_pool.runInteraction(
	238	await self.db_pool.runInteraction(
243	239	"_get_events_which_are_prevs", _get_events_which_are_prevs_txn, chunk
244	240	)
245	241
246	242	return results
247	243
248		@defer.inlineCallbacks
249		def _get_prevs_before_rejected(self, event_ids):
	244	async def _get_prevs_before_rejected(self, event_ids: Iterable[str]) -> Set[str]:
250	245	"""Get soft-failed ancestors to remove from the extremities.
251	246
252	247	Given a set of events, find all those that have been soft-failed or

258	253	are separated by soft failed events.
259	254
260	255	Args:
261		event_ids (Iterable[str]): Events to find prev events for. Note
262		that these must have already been persisted.
	256	event_ids: Events to find prev events for. Note that these must have
	257	already been persisted.
263	258
264	259	Returns:
265		Deferred[set[str]]
	260	The previous events.
266	261	"""
267	262
268	263	# The set of event_ids to return. This includes all soft-failed events

303	298	existing_prevs.add(prev_event_id)
304	299
305	300	for chunk in batch_iter(event_ids, 100):
306		yield self.db_pool.runInteraction(
	301	await self.db_pool.runInteraction(
307	302	"_get_prevs_before_rejected", _get_prevs_before_rejected_txn, chunk
308	303	)
309	304

1300	1295	sql = """
1301	1296	INSERT INTO event_push_actions (
1302	1297	room_id, event_id, user_id, actions, stream_ordering,
1303		topological_ordering, notif, highlight
1304		)
1305		SELECT ?, event_id, user_id, actions, ?, ?, notif, highlight
	1298	topological_ordering, notif, highlight, unread
	1299	)
	1300	SELECT ?, event_id, user_id, actions, ?, ?, notif, highlight, unread
1306	1301	FROM event_push_actions_staging
1307	1302	WHERE event_id = ?
1308	1303	"""

+19

-27

synapse/storage/databases/main/events_bg_updates.py less more

14	14
15	15	import logging
16	16
17		from twisted.internet import defer
18
19	17	from synapse.api.constants import EventContentFields
20	18	from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
21	19	from synapse.storage.database import DatabasePool

93	91	where_clause="NOT have_censored",
94	92	)
95	93
96		@defer.inlineCallbacks
97		def _background_reindex_fields_sender(self, progress, batch_size):
	94	async def _background_reindex_fields_sender(self, progress, batch_size):
98	95	target_min_stream_id = progress["target_min_stream_id_inclusive"]
99	96	max_stream_id = progress["max_stream_id_exclusive"]
100	97	rows_inserted = progress.get("rows_inserted", 0)

154	151
155	152	return len(rows)
156	153
157		result = yield self.db_pool.runInteraction(
	154	result = await self.db_pool.runInteraction(
158	155	self.EVENT_FIELDS_SENDER_URL_UPDATE_NAME, reindex_txn
159	156	)
160	157
161	158	if not result:
162		yield self.db_pool.updates._end_background_update(
	159	await self.db_pool.updates._end_background_update(
163	160	self.EVENT_FIELDS_SENDER_URL_UPDATE_NAME
164	161	)
165	162
166	163	return result
167	164
168		@defer.inlineCallbacks
169		def _background_reindex_origin_server_ts(self, progress, batch_size):
	165	async def _background_reindex_origin_server_ts(self, progress, batch_size):
170	166	target_min_stream_id = progress["target_min_stream_id_inclusive"]
171	167	max_stream_id = progress["max_stream_id_exclusive"]
172	168	rows_inserted = progress.get("rows_inserted", 0)

233	229
234	230	return len(rows_to_update)
235	231
236		result = yield self.db_pool.runInteraction(
	232	result = await self.db_pool.runInteraction(
237	233	self.EVENT_ORIGIN_SERVER_TS_NAME, reindex_search_txn
238	234	)
239	235
240	236	if not result:
241		yield self.db_pool.updates._end_background_update(
	237	await self.db_pool.updates._end_background_update(
242	238	self.EVENT_ORIGIN_SERVER_TS_NAME
243	239	)
244	240
245	241	return result
246	242
247		@defer.inlineCallbacks
248		def _cleanup_extremities_bg_update(self, progress, batch_size):
	243	async def _cleanup_extremities_bg_update(self, progress, batch_size):
249	244	"""Background update to clean out extremities that should have been
250	245	deleted previously.
251	246

413	408
414	409	return len(original_set)
415	410
416		num_handled = yield self.db_pool.runInteraction(
	411	num_handled = await self.db_pool.runInteraction(
417	412	"_cleanup_extremities_bg_update", _cleanup_extremities_bg_update_txn
418	413	)
419	414
420	415	if not num_handled:
421		yield self.db_pool.updates._end_background_update(
	416	await self.db_pool.updates._end_background_update(
422	417	self.DELETE_SOFT_FAILED_EXTREMITIES
423	418	)
424	419
425	420	def _drop_table_txn(txn):
426	421	txn.execute("DROP TABLE _extremities_to_check")
427	422
428		yield self.db_pool.runInteraction(
	423	await self.db_pool.runInteraction(
429	424	"_cleanup_extremities_bg_update_drop_table", _drop_table_txn
430	425	)
431	426
432	427	return num_handled
433	428
434		@defer.inlineCallbacks
435		def _redactions_received_ts(self, progress, batch_size):
	429	async def _redactions_received_ts(self, progress, batch_size):
436	430	"""Handles filling out the `received_ts` column in redactions.
437	431	"""
438	432	last_event_id = progress.get("last_event_id", "")

479	473
480	474	return len(rows)
481	475
482		count = yield self.db_pool.runInteraction(
	476	count = await self.db_pool.runInteraction(
483	477	"_redactions_received_ts", _redactions_received_ts_txn
484	478	)
485	479
486	480	if not count:
487		yield self.db_pool.updates._end_background_update("redactions_received_ts")
	481	await self.db_pool.updates._end_background_update("redactions_received_ts")
488	482
489	483	return count
490	484
491		@defer.inlineCallbacks
492		def _event_fix_redactions_bytes(self, progress, batch_size):
	485	async def _event_fix_redactions_bytes(self, progress, batch_size):
493	486	"""Undoes hex encoded censored redacted event JSON.
494	487	"""
495	488

510	503
511	504	txn.execute("DROP INDEX redactions_censored_redacts")
512	505
513		yield self.db_pool.runInteraction(
	506	await self.db_pool.runInteraction(
514	507	"_event_fix_redactions_bytes", _event_fix_redactions_bytes_txn
515	508	)
516	509
517		yield self.db_pool.updates._end_background_update("event_fix_redactions_bytes")
	510	await self.db_pool.updates._end_background_update("event_fix_redactions_bytes")
518	511
519	512	return 1
520	513
521		@defer.inlineCallbacks
522		def _event_store_labels(self, progress, batch_size):
	514	async def _event_store_labels(self, progress, batch_size):
523	515	"""Background update handler which will store labels for existing events."""
524	516	last_event_id = progress.get("last_event_id", "")
525	517

574	566
575	567	return nbrows
576	568
577		num_rows = yield self.db_pool.runInteraction(
	569	num_rows = await self.db_pool.runInteraction(
578	570	desc="event_store_labels", func=_event_store_labels_txn
579	571	)
580	572
581	573	if not num_rows:
582		yield self.db_pool.updates._end_background_update("event_store_labels")
	574	await self.db_pool.updates._end_background_update("event_store_labels")
583	575
584	576	return num_rows

+153

-264

synapse/storage/databases/main/events_worker.py less more

18	18	import logging
19	19	import threading
20	20	from collections import namedtuple
21		from typing import List, Optional, Tuple
	21	from typing import Dict, Iterable, List, Optional, Tuple, overload
22	22
23	23	from constantly import NamedConstant, Names
	24	from typing_extensions import Literal
24	25
25	26	from twisted.internet import defer
26	27

31	32	EventFormatVersions,
32	33	RoomVersions,
33	34	)
34		from synapse.events import make_event_from_dict
	35	from synapse.events import EventBase, make_event_from_dict
35	36	from synapse.events.utils import prune_event
36	37	from synapse.logging.context import PreserveLoggingContext, current_context
37	38	from synapse.metrics.background_process_metrics import run_as_background_process

41	42	from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
42	43	from synapse.storage.database import DatabasePool
43	44	from synapse.storage.util.id_generators import StreamIdGenerator
44		from synapse.types import get_domain_from_id
45		from synapse.util.caches.descriptors import Cache, cached, cachedInlineCallbacks
	45	from synapse.types import Collection, get_domain_from_id
	46	from synapse.util.caches.descriptors import Cache, cached
46	47	from synapse.util.iterutils import batch_iter
47	48	from synapse.util.metrics import Measure
48	49

111	112
112	113	def process_replication_rows(self, stream_name, instance_name, token, rows):
113	114	if stream_name == EventsStream.NAME:
114		self._stream_id_gen.advance(token)
	115	self._stream_id_gen.advance(instance_name, token)
115	116	elif stream_name == BackfillStream.NAME:
116		self._backfill_id_gen.advance(-token)
	117	self._backfill_id_gen.advance(instance_name, -token)
117	118
118	119	super().process_replication_rows(stream_name, instance_name, token, rows)
119	120
120		def get_received_ts(self, event_id):
	121	async def get_received_ts(self, event_id: str) -> Optional[int]:
121	122	"""Get received_ts (when it was persisted) for the event.
122	123
123	124	Raises an exception for unknown events.
124	125
125	126	Args:
126		event_id (str)
127
128		Returns:
129		Deferred[int\|None]: Timestamp in milliseconds, or None for events
130		that were persisted before received_ts was implemented.
131		"""
132		return self.db_pool.simple_select_one_onecol(
	127	event_id: The event ID to query.
	128
	129	Returns:
	130	Timestamp in milliseconds, or None for events that were persisted
	131	before received_ts was implemented.
	132	"""
	133	return await self.db_pool.simple_select_one_onecol(
133	134	table="events",
134	135	keyvalues={"event_id": event_id},
135	136	retcol="received_ts",
136	137	desc="get_received_ts",
137	138	)
138	139
139		def get_received_ts_by_stream_pos(self, stream_ordering):
140		"""Given a stream ordering get an approximate timestamp of when it
141		happened.
142
143		This is done by simply taking the received ts of the first event that
144		has a stream ordering greater than or equal to the given stream pos.
145		If none exists returns the current time, on the assumption that it must
146		have happened recently.
147
148		Args:
149		stream_ordering (int)
150
151		Returns:
152		Deferred[int]
153		"""
154
155		def _get_approximate_received_ts_txn(txn):
156		sql = """
157		SELECT received_ts FROM events
158		WHERE stream_ordering >= ?
159		LIMIT 1
160		"""
161
162		txn.execute(sql, (stream_ordering,))
163		row = txn.fetchone()
164		if row and row[0]:
165		ts = row[0]
166		else:
167		ts = self.clock.time_msec()
168
169		return ts
170
171		return self.db_pool.runInteraction(
172		"get_approximate_received_ts", _get_approximate_received_ts_txn
173		)
174
175		@defer.inlineCallbacks
176		def get_event(
	140	# Inform mypy that if allow_none is False (the default) then get_event
	141	# always returns an EventBase.
	142	@overload
	143	async def get_event(
	144	self,
	145	event_id: str,
	146	redact_behaviour: EventRedactBehaviour = EventRedactBehaviour.REDACT,
	147	get_prev_content: bool = False,
	148	allow_rejected: bool = False,
	149	allow_none: Literal[False] = False,
	150	check_room_id: Optional[str] = None,
	151	) -> EventBase:
	152	...
	153
	154	@overload
	155	async def get_event(
	156	self,
	157	event_id: str,
	158	redact_behaviour: EventRedactBehaviour = EventRedactBehaviour.REDACT,
	159	get_prev_content: bool = False,
	160	allow_rejected: bool = False,
	161	allow_none: Literal[True] = False,
	162	check_room_id: Optional[str] = None,
	163	) -> Optional[EventBase]:
	164	...
	165
	166	async def get_event(
177	167	self,
178	168	event_id: str,
179	169	redact_behaviour: EventRedactBehaviour = EventRedactBehaviour.REDACT,

181	171	allow_rejected: bool = False,
182	172	allow_none: bool = False,
183	173	check_room_id: Optional[str] = None,
184		):
	174	) -> Optional[EventBase]:
185	175	"""Get an event from the database by event_id.
186	176
187	177	Args:

206	196	If there is a mismatch, behave as per allow_none.
207	197
208	198	Returns:
209		Deferred[EventBase\|None]
	199	The event, or None if the event was not found.
210	200	"""
211	201	if not isinstance(event_id, str):
212	202	raise TypeError("Invalid event event_id %r" % (event_id,))
213	203
214		events = yield self.get_events_as_list(
	204	events = await self.get_events_as_list(
215	205	[event_id],
216	206	redact_behaviour=redact_behaviour,
217	207	get_prev_content=get_prev_content,

229	219
230	220	return event
231	221
232		@defer.inlineCallbacks
233		def get_events(
	222	async def get_events(
234	223	self,
235		event_ids: List[str],
	224	event_ids: Iterable[str],
236	225	redact_behaviour: EventRedactBehaviour = EventRedactBehaviour.REDACT,
237	226	get_prev_content: bool = False,
238	227	allow_rejected: bool = False,
239		):
	228	) -> Dict[str, EventBase]:
240	229	"""Get events from the database
241	230
242	231	Args:

255	244	omits rejeted events from the response.
256	245
257	246	Returns:
258		Deferred : Dict from event_id to event.
259		"""
260		events = yield self.get_events_as_list(
	247	A mapping from event_id to event.
	248	"""
	249	events = await self.get_events_as_list(
261	250	event_ids,
262	251	redact_behaviour=redact_behaviour,
263	252	get_prev_content=get_prev_content,

266	255
267	256	return {e.event_id: e for e in events}
268	257
269		@defer.inlineCallbacks
270		def get_events_as_list(
	258	async def get_events_as_list(
271	259	self,
272		event_ids: List[str],
	260	event_ids: Collection[str],
273	261	redact_behaviour: EventRedactBehaviour = EventRedactBehaviour.REDACT,
274	262	get_prev_content: bool = False,
275	263	allow_rejected: bool = False,
276		):
	264	) -> List[EventBase]:
277	265	"""Get events from the database and return in a list in the same order
278	266	as given by `event_ids` arg.
279	267

294	282	omits rejected events from the response.
295	283
296	284	Returns:
297		Deferred[list[EventBase]]: List of events fetched from the database. The
298		events are in the same order as `event_ids` arg.
	285	List of events fetched from the database. The events are in the same
	286	order as `event_ids` arg.
299	287
300	288	Note that the returned list may be smaller than the list of event
301	289	IDs if not all events could be fetched.

305	293	return []
306	294
307	295	# there may be duplicates so we cast the list to a set
308		event_entry_map = yield self._get_events_from_cache_or_db(
	296	event_entry_map = await self._get_events_from_cache_or_db(
309	297	set(event_ids), allow_rejected=allow_rejected
310	298	)
311	299

340	328	continue
341	329
342	330	redacted_event_id = entry.event.redacts
343		event_map = yield self._get_events_from_cache_or_db([redacted_event_id])
	331	event_map = await self._get_events_from_cache_or_db([redacted_event_id])
344	332	original_event_entry = event_map.get(redacted_event_id)
345	333	if not original_event_entry:
346	334	# we don't have the redacted event (or it was rejected).

406	394
407	395	if get_prev_content:
408	396	if "replaces_state" in event.unsigned:
409		prev = yield self.get_event(
	397	prev = await self.get_event(
410	398	event.unsigned["replaces_state"],
411	399	get_prev_content=False,
412	400	allow_none=True,

418	406
419	407	return events
420	408
421		@defer.inlineCallbacks
422		def _get_events_from_cache_or_db(self, event_ids, allow_rejected=False):
	409	async def _get_events_from_cache_or_db(self, event_ids, allow_rejected=False):
423	410	"""Fetch a bunch of events from the cache or the database.
424	411
425	412	If events are pulled from the database, they will be cached for future lookups.

434	421	rejected events are omitted from the response.
435	422
436	423	Returns:
437		Deferred[Dict[str, _EventCacheEntry]]:
	424	Dict[str, _EventCacheEntry]:
438	425	map from event id to result
439	426	"""
440	427	event_entry_map = self._get_events_from_cache(

452	439	# the events have been redacted, and if so pulling the redaction event out
453	440	# of the database to check it.
454	441	#
455		missing_events = yield self._get_events_from_db(
	442	missing_events = await self._get_events_from_db(
456	443	missing_events_ids, allow_rejected=allow_rejected
457	444	)
458	445

560	547	with PreserveLoggingContext():
561	548	self.hs.get_reactor().callFromThread(fire, event_list, e)
562	549
563		@defer.inlineCallbacks
564		def _get_events_from_db(self, event_ids, allow_rejected=False):
	550	async def _get_events_from_db(self, event_ids, allow_rejected=False):
565	551	"""Fetch a bunch of events from the database.
566	552
567	553	Returned events will be added to the cache for future lookups.

575	561	rejected events are omitted from the response.
576	562
577	563	Returns:
578		Deferred[Dict[str, _EventCacheEntry]]:
	564	Dict[str, _EventCacheEntry]:
579	565	map from event id to result. May return extra events which
580	566	weren't asked for.
581	567	"""

583	569	events_to_fetch = event_ids
584	570
585	571	while events_to_fetch:
586		row_map = yield self._enqueue_events(events_to_fetch)
	572	row_map = await self._enqueue_events(events_to_fetch)
587	573
588	574	# we need to recursively fetch any redactions of those events
589	575	redaction_ids = set()

609	595	if not allow_rejected and rejected_reason:
610	596	continue
611	597
612		d = db_to_json(row["json"])
613		internal_metadata = db_to_json(row["internal_metadata"])
	598	# If the event or metadata cannot be parsed, log the error and act
	599	# as if the event is unknown.
	600	try:
	601	d = db_to_json(row["json"])
	602	except ValueError:
	603	logger.error("Unable to parse json from event: %s", event_id)
	604	continue
	605	try:
	606	internal_metadata = db_to_json(row["internal_metadata"])
	607	except ValueError:
	608	logger.error(
	609	"Unable to parse internal_metadata from event: %s", event_id
	610	)
	611	continue
614	612
615	613	format_version = row["format_version"]
616	614	if format_version is None:

621	619	room_version_id = row["room_version_id"]
622	620
623	621	if not room_version_id:
624		# this should only happen for out-of-band membership events
625		if not internal_metadata.get("out_of_band_membership"):
626		logger.warning(
627		"Room %s for event %s is unknown", d["room_id"], event_id
	622	# this should only happen for out-of-band membership events which
	623	# arrived before #6983 landed. For all other events, we should have
	624	# an entry in the 'rooms' table.
	625	#
	626	# However, the 'out_of_band_membership' flag is unreliable for older
	627	# invites, so just accept it for all membership events.
	628	#
	629	if d["type"] != EventTypes.Member:
	630	raise Exception(
	631	"Room %s for event %s is unknown" % (d["room_id"], event_id)
628	632	)
629		continue
630
631		# take a wild stab at the room version based on the event format
	633
	634	# so, assuming this is an out-of-band-invite that arrived before #6983
	635	# landed, we know that the room version must be v5 or earlier (because
	636	# v6 hadn't been invented at that point, so invites from such rooms
	637	# would have been rejected.)
	638	#
	639	# The main reason we need to know the room version here (other than
	640	# choosing the right python Event class) is in case the event later has
	641	# to be redacted - and all the room versions up to v5 used the same
	642	# redaction algorithm.
	643	#
	644	# So, the following approximations should be adequate.
	645
632	646	if format_version == EventFormatVersions.V1:
	647	# if it's event format v1 then it must be room v1 or v2
633	648	room_version = RoomVersions.V1
634	649	elif format_version == EventFormatVersions.V2:
	650	# if it's event format v2 then it must be room v3
635	651	room_version = RoomVersions.V3
636	652	else:
	653	# if it's event format v3 then it must be room v4 or v5
637	654	room_version = RoomVersions.V5
638	655	else:
639	656	room_version = KNOWN_ROOM_VERSIONS.get(room_version_id)

685	702
686	703	return result_map
687	704
688		@defer.inlineCallbacks
689		def _enqueue_events(self, events):
	705	async def _enqueue_events(self, events):
690	706	"""Fetches events from the database using the _event_fetch_list. This
691	707	allows batch and bulk fetching of events - it allows us to fetch events
692	708	without having to create a new transaction for each request for events.

695	711	events (Iterable[str]): events to be fetched.
696	712
697	713	Returns:
698		Deferred[Dict[str, Dict]]: map from event id to row data from the database.
	714	Dict[str, Dict]: map from event id to row data from the database.
699	715	May contain events that weren't requested.
700	716	"""
701	717

718	734
719	735	logger.debug("Loading %d events: %s", len(events), events)
720	736	with PreserveLoggingContext():
721		row_map = yield events_d
	737	row_map = await events_d
722	738	logger.debug("Loaded %d events (%d rows)", len(events), len(row_map))
723	739
724	740	return row_map

806	822
807	823	return event_dict
808	824
809		def _maybe_redact_event_row(self, original_ev, redactions, event_map):
	825	def _maybe_redact_event_row(
	826	self,
	827	original_ev: EventBase,
	828	redactions: Iterable[str],
	829	event_map: Dict[str, EventBase],
	830	) -> Optional[EventBase]:
810	831	"""Given an event object and a list of possible redacting event ids,
811	832	determine whether to honour any of those redactions and if so return a redacted
812	833	event.
813	834
814	835	Args:
815		original_ev (EventBase):
816		redactions (iterable[str]): list of event ids of potential redaction events
817		event_map (dict[str, EventBase]): other events which have been fetched, in
818		which we can look up the redaaction events. Map from event id to event.
819
820		Returns:
821		Deferred[EventBase\|None]: if the event should be redacted, a pruned
822		event object. Otherwise, None.
	836	original_ev: The original event.
	837	redactions: list of event ids of potential redaction events
	838	event_map: other events which have been fetched, in which we can
	839	look up the redaaction events. Map from event id to event.
	840
	841	Returns:
	842	If the event should be redacted, a pruned event object. Otherwise, None.
823	843	"""
824	844	if original_ev.type == "m.room.create":
825	845	# we choose to ignore redactions of m.room.create events.

877	897	# no valid redaction found for this event
878	898	return None
879	899
880		@defer.inlineCallbacks
881		def have_events_in_timeline(self, event_ids):
	900	async def have_events_in_timeline(self, event_ids):
882	901	"""Given a list of event ids, check if we have already processed and
883	902	stored them as non outliers.
884	903	"""
885		rows = yield self.db_pool.simple_select_many_batch(
	904	rows = await self.db_pool.simple_select_many_batch(
886	905	table="events",
887	906	retcols=("event_id",),
888	907	column="event_id",

893	912
894	913	return {r["event_id"] for r in rows}
895	914
896		@defer.inlineCallbacks
897		def have_seen_events(self, event_ids):
	915	async def have_seen_events(self, event_ids):
898	916	"""Given a list of event ids, check if we have already processed them.
899	917
900	918	Args:
901	919	event_ids (iterable[str]):
902	920
903	921	Returns:
904		Deferred[set[str]]: The events we have already seen.
	922	set[str]: The events we have already seen.
905	923	"""
906	924	results = set()
907	925

917	935	# break the input up into chunks of 100
918	936	input_iterator = iter(event_ids)
919	937	for chunk in iter(lambda: list(itertools.islice(input_iterator, 100)), []):
920		yield self.db_pool.runInteraction(
	938	await self.db_pool.runInteraction(
921	939	"have_seen_events", have_seen_events_txn, chunk
922	940	)
923	941	return results
924
925		def _get_total_state_event_counts_txn(self, txn, room_id):
926		"""
927		See get_total_state_event_counts.
928		"""
929		# We join against the events table as that has an index on room_id
930		sql = """
931		SELECT COUNT(*) FROM state_events
932		INNER JOIN events USING (room_id, event_id)
933		WHERE room_id=?
934		"""
935		txn.execute(sql, (room_id,))
936		row = txn.fetchone()
937		return row[0] if row else 0
938
939		def get_total_state_event_counts(self, room_id):
940		"""
941		Gets the total number of state events in a room.
942
943		Args:
944		room_id (str)
945
946		Returns:
947		Deferred[int]
948		"""
949		return self.db_pool.runInteraction(
950		"get_total_state_event_counts",
951		self._get_total_state_event_counts_txn,
952		room_id,
953		)
954	942
955	943	def _get_current_state_event_counts_txn(self, txn, room_id):
956	944	"""

961	949	row = txn.fetchone()
962	950	return row[0] if row else 0
963	951
964		def get_current_state_event_counts(self, room_id):
	952	async def get_current_state_event_counts(self, room_id: str) -> int:
965	953	"""
966	954	Gets the current number of state events in a room.
967	955
968	956	Args:
969		room_id (str)
970
971		Returns:
972		Deferred[int]
973		"""
974		return self.db_pool.runInteraction(
	957	room_id: The room ID to query.
	958
	959	Returns:
	960	The current number of state events.
	961	"""
	962	return await self.db_pool.runInteraction(
975	963	"get_current_state_event_counts",
976	964	self._get_current_state_event_counts_txn,
977	965	room_id,
978	966	)
979	967
980		@defer.inlineCallbacks
981		def get_room_complexity(self, room_id):
	968	async def get_room_complexity(self, room_id):
982	969	"""
983	970	Get a rough approximation of the complexity of the room. This is used by
984	971	remote servers to decide whether they wish to join the room or not.

989	976	room_id (str)
990	977
991	978	Returns:
992		Deferred[dict[str:int]] of complexity version to complexity.
993		"""
994		state_events = yield self.get_current_state_event_counts(room_id)
	979	dict[str:int] of complexity version to complexity.
	980	"""
	981	state_events = await self.get_current_state_event_counts(room_id)
995	982
996	983	# Call this one "v1", so we can introduce new ones as we want to develop
997	984	# it.

1007	994	"""The current maximum token that events have reached"""
1008	995	return self._stream_id_gen.get_current_token()
1009	996
1010		def get_all_new_forward_event_rows(self, last_id, current_id, limit):
	997	async def get_all_new_forward_event_rows(
	998	self, last_id: int, current_id: int, limit: int
	999	) -> List[Tuple]:
1011	1000	"""Returns new events, for the Events replication stream
1012	1001
1013	1002	Args:

1015	1004	current_id: the maximum stream_id to return up to
1016	1005	limit: the maximum number of rows to return
1017	1006
1018		Returns: Deferred[List[Tuple]]
	1007	Returns:
1019	1008	a list of events stream rows. Each tuple consists of a stream id as
1020	1009	the first element, followed by fields suitable for casting into an
1021	1010	EventsStreamRow.

1036	1025	txn.execute(sql, (last_id, current_id, limit))
1037	1026	return txn.fetchall()
1038	1027
1039		return self.db_pool.runInteraction(
	1028	return await self.db_pool.runInteraction(
1040	1029	"get_all_new_forward_event_rows", get_all_new_forward_event_rows
1041	1030	)
1042	1031
1043		def get_ex_outlier_stream_rows(self, last_id, current_id):
	1032	async def get_ex_outlier_stream_rows(
	1033	self, last_id: int, current_id: int
	1034	) -> List[Tuple]:
1044	1035	"""Returns de-outliered events, for the Events replication stream
1045	1036
1046	1037	Args:
1047	1038	last_id: the last stream_id from the previous batch.
1048	1039	current_id: the maximum stream_id to return up to
1049	1040
1050		Returns: Deferred[List[Tuple]]
	1041	Returns:
1051	1042	a list of events stream rows. Each tuple consists of a stream id as
1052	1043	the first element, followed by fields suitable for casting into an
1053	1044	EventsStreamRow.

1070	1061	txn.execute(sql, (last_id, current_id))
1071	1062	return txn.fetchall()
1072	1063
1073		return self.db_pool.runInteraction(
	1064	return await self.db_pool.runInteraction(
1074	1065	"get_ex_outlier_stream_rows", get_ex_outlier_stream_rows_txn
1075	1066	)
1076	1067

1221	1212
1222	1213	return rows, to_token, True
1223	1214
1224		@cached(num_args=5, max_entries=10)
1225		def get_all_new_events(
1226		self,
1227		last_backfill_id,
1228		last_forward_id,
1229		current_backfill_id,
1230		current_forward_id,
1231		limit,
1232		):
1233		"""Get all the new events that have arrived at the server either as
1234		new events or as backfilled events"""
1235		have_backfill_events = last_backfill_id != current_backfill_id
1236		have_forward_events = last_forward_id != current_forward_id
1237
1238		if not have_backfill_events and not have_forward_events:
1239		return defer.succeed(AllNewEventsResult([], [], [], [], []))
1240
1241		def get_all_new_events_txn(txn):
1242		sql = (
1243		"SELECT e.stream_ordering, e.event_id, e.room_id, e.type,"
1244		" state_key, redacts"
1245		" FROM events AS e"
1246		" LEFT JOIN redactions USING (event_id)"
1247		" LEFT JOIN state_events USING (event_id)"
1248		" WHERE ? < stream_ordering AND stream_ordering <= ?"
1249		" ORDER BY stream_ordering ASC"
1250		" LIMIT ?"
1251		)
1252		if have_forward_events:
1253		txn.execute(sql, (last_forward_id, current_forward_id, limit))
1254		new_forward_events = txn.fetchall()
1255
1256		if len(new_forward_events) == limit:
1257		upper_bound = new_forward_events[-1][0]
1258		else:
1259		upper_bound = current_forward_id
1260
1261		sql = (
1262		"SELECT event_stream_ordering, event_id, state_group"
1263		" FROM ex_outlier_stream"
1264		" WHERE ? > event_stream_ordering"
1265		" AND event_stream_ordering >= ?"
1266		" ORDER BY event_stream_ordering DESC"
1267		)
1268		txn.execute(sql, (last_forward_id, upper_bound))
1269		forward_ex_outliers = txn.fetchall()
1270		else:
1271		new_forward_events = []
1272		forward_ex_outliers = []
1273
1274		sql = (
1275		"SELECT -e.stream_ordering, e.event_id, e.room_id, e.type,"
1276		" state_key, redacts"
1277		" FROM events AS e"
1278		" LEFT JOIN redactions USING (event_id)"
1279		" LEFT JOIN state_events USING (event_id)"
1280		" WHERE ? > stream_ordering AND stream_ordering >= ?"
1281		" ORDER BY stream_ordering DESC"
1282		" LIMIT ?"
1283		)
1284		if have_backfill_events:
1285		txn.execute(sql, (-last_backfill_id, -current_backfill_id, limit))
1286		new_backfill_events = txn.fetchall()
1287
1288		if len(new_backfill_events) == limit:
1289		upper_bound = new_backfill_events[-1][0]
1290		else:
1291		upper_bound = current_backfill_id
1292
1293		sql = (
1294		"SELECT -event_stream_ordering, event_id, state_group"
1295		" FROM ex_outlier_stream"
1296		" WHERE ? > event_stream_ordering"
1297		" AND event_stream_ordering >= ?"
1298		" ORDER BY event_stream_ordering DESC"
1299		)
1300		txn.execute(sql, (-last_backfill_id, -upper_bound))
1301		backward_ex_outliers = txn.fetchall()
1302		else:
1303		new_backfill_events = []
1304		backward_ex_outliers = []
1305
1306		return AllNewEventsResult(
1307		new_forward_events,
1308		new_backfill_events,
1309		forward_ex_outliers,
1310		backward_ex_outliers,
1311		)
1312
1313		return self.db_pool.runInteraction("get_all_new_events", get_all_new_events_txn)
1314
1315	1215	async def is_event_after(self, event_id1, event_id2):
1316	1216	"""Returns True if event_id1 is after event_id2 in the stream
1317	1217	"""

1319	1219	to_2, so_2 = await self.get_event_ordering(event_id2)
1320	1220	return (to_1, so_1) > (to_2, so_2)
1321	1221
1322		@cachedInlineCallbacks(max_entries=5000)
1323		def get_event_ordering(self, event_id):
1324		res = yield self.db_pool.simple_select_one(
	1222	@cached(max_entries=5000)
	1223	async def get_event_ordering(self, event_id):
	1224	res = await self.db_pool.simple_select_one(
1325	1225	table="events",
1326	1226	retcols=["topological_ordering", "stream_ordering"],
1327	1227	keyvalues={"event_id": event_id},

1333	1233
1334	1234	return (int(res["topological_ordering"]), int(res["stream_ordering"]))
1335	1235
1336		def get_next_event_to_expire(self):
	1236	async def get_next_event_to_expire(self) -> Optional[Tuple[str, int]]:
1337	1237	"""Retrieve the entry with the lowest expiry timestamp in the event_expiry
1338	1238	table, or None if there's no more event to expire.
1339	1239
1340		Returns: Deferred[Optional[Tuple[str, int]]]
	1240	Returns:
1341	1241	A tuple containing the event ID as its first element and an expiry timestamp
1342	1242	as its second one, if there's at least one row in the event_expiry table.
1343	1243	None otherwise.

1353	1253
1354	1254	return txn.fetchone()
1355	1255
1356		return self.db_pool.runInteraction(
	1256	return await self.db_pool.runInteraction(
1357	1257	desc="get_next_event_to_expire", func=get_next_event_to_expire_txn
1358	1258	)
1359
1360
1361		AllNewEventsResult = namedtuple(
1362		"AllNewEventsResult",
1363		[
1364		"new_forward_events",
1365		"new_backfill_events",
1366		"forward_ex_outliers",
1367		"backward_ex_outliers",
1368		],
1369		)

+3

-2

synapse/storage/databases/main/filtering.py less more

16	16
17	17	from synapse.api.errors import Codes, SynapseError
18	18	from synapse.storage._base import SQLBaseStore, db_to_json
	19	from synapse.types import JsonDict
19	20	from synapse.util.caches.descriptors import cached
20	21
21	22

39	40
40	41	return db_to_json(def_json)
41	42
42		def add_user_filter(self, user_localpart, user_filter):
	43	async def add_user_filter(self, user_localpart: str, user_filter: JsonDict) -> str:
43	44	def_json = encode_canonical_json(user_filter)
44	45
45	46	# Need an atomic transaction to SELECT the maximal ID so far then

70	71
71	72	return filter_id
72	73
73		return self.db_pool.runInteraction("add_user_filter", _do_txn)
	74	return await self.db_pool.runInteraction("add_user_filter", _do_txn)

+209

-124

synapse/storage/databases/main/group_server.py less more

13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
15	15
16		from typing import List, Optional, Tuple
	16	from typing import Any, Dict, List, Optional, Tuple, Union
17	17
18	18	from synapse.api.errors import SynapseError
19	19	from synapse.storage._base import SQLBaseStore, db_to_json

27	27
28	28
29	29	class GroupServerWorkerStore(SQLBaseStore):
30		def get_group(self, group_id):
31		return self.db_pool.simple_select_one(
	30	async def get_group(self, group_id: str) -> Optional[Dict[str, Any]]:
	31	return await self.db_pool.simple_select_one(
32	32	table="groups",
33	33	keyvalues={"group_id": group_id},
34	34	retcols=(

43	43	desc="get_group",
44	44	)
45	45
46		def get_users_in_group(self, group_id, include_private=False):
	46	async def get_users_in_group(
	47	self, group_id: str, include_private: bool = False
	48	) -> List[Dict[str, Any]]:
47	49	# TODO: Pagination
48	50
49	51	keyvalues = {"group_id": group_id}
50	52	if not include_private:
51	53	keyvalues["is_public"] = True
52	54
53		return self.db_pool.simple_select_list(
	55	return await self.db_pool.simple_select_list(
54	56	table="group_users",
55	57	keyvalues=keyvalues,
56	58	retcols=("user_id", "is_public", "is_admin"),
57	59	desc="get_users_in_group",
58	60	)
59	61
60		def get_invited_users_in_group(self, group_id):
	62	async def get_invited_users_in_group(self, group_id: str) -> List[str]:
61	63	# TODO: Pagination
62	64
63		return self.db_pool.simple_select_onecol(
	65	return await self.db_pool.simple_select_onecol(
64	66	table="group_invites",
65	67	keyvalues={"group_id": group_id},
66	68	retcol="user_id",
67	69	desc="get_invited_users_in_group",
68	70	)
69	71
70		def get_rooms_in_group(self, group_id: str, include_private: bool = False):
	72	async def get_rooms_in_group(
	73	self, group_id: str, include_private: bool = False
	74	) -> List[Dict[str, Union[str, bool]]]:
71	75	"""Retrieve the rooms that belong to a given group. Does not return rooms that
72	76	lack members.
73	77

76	80	include_private: Whether to return private rooms in results
77	81
78	82	Returns:
79		Deferred[List[Dict[str, str\|bool]]]: A list of dictionaries, each in the
80		form of:
	83	A list of dictionaries, each in the form of:
81	84
82	85	{
83	86	"room_id": "!a_room_id:example.com", # The ID of the room

114	117	for room_id, is_public in txn
115	118	]
116	119
117		return self.db_pool.runInteraction(
	120	return await self.db_pool.runInteraction(
118	121	"get_rooms_in_group", _get_rooms_in_group_txn
119	122	)
120	123
121		def get_rooms_for_summary_by_category(
	124	async def get_rooms_for_summary_by_category(
122	125	self, group_id: str, include_private: bool = False,
123		):
	126	) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
124	127	"""Get the rooms and categories that should be included in a summary request
125	128
126	129	Args:

128	131	include_private: Whether to return private rooms in results
129	132
130	133	Returns:
131		Deferred[Tuple[List, Dict]]: A tuple containing:
	134	A tuple containing:
132	135
133	136	* A list of dictionaries with the keys:
134	137	* "room_id": str, the room ID

204	207
205	208	return rooms, categories
206	209
207		return self.db_pool.runInteraction(
	210	return await self.db_pool.runInteraction(
208	211	"get_rooms_for_summary", _get_rooms_for_summary_txn
209	212	)
210	213

264	267
265	268	return role
266	269
267		def get_local_groups_for_room(self, room_id):
	270	async def get_local_groups_for_room(self, room_id: str) -> List[str]:
268	271	"""Get all of the local group that contain a given room
269	272	Args:
270		room_id (str): The ID of a room
	273	room_id: The ID of a room
271	274	Returns:
272		Deferred[list[str]]: A twisted.Deferred containing a list of group ids
273		containing this room
274		"""
275		return self.db_pool.simple_select_onecol(
	275	A list of group ids containing this room
	276	"""
	277	return await self.db_pool.simple_select_onecol(
276	278	table="group_rooms",
277	279	keyvalues={"room_id": room_id},
278	280	retcol="group_id",
279	281	desc="get_local_groups_for_room",
280	282	)
281	283
282		def get_users_for_summary_by_role(self, group_id, include_private=False):
	284	async def get_users_for_summary_by_role(self, group_id, include_private=False):
283	285	"""Get the users and roles that should be included in a summary request
284	286
285		Returns ([users], [roles])
	287	Returns:
	288	([users], [roles])
286	289	"""
287	290
288	291	def _get_users_for_summary_txn(txn):

336	339
337	340	return users, roles
338	341
339		return self.db_pool.runInteraction(
	342	return await self.db_pool.runInteraction(
340	343	"get_users_for_summary_by_role", _get_users_for_summary_txn
341	344	)
342	345
343		def is_user_in_group(self, user_id, group_id):
344		return self.db_pool.simple_select_one_onecol(
	346	async def is_user_in_group(self, user_id: str, group_id: str) -> bool:
	347	result = await self.db_pool.simple_select_one_onecol(
345	348	table="group_users",
346	349	keyvalues={"group_id": group_id, "user_id": user_id},
347	350	retcol="user_id",
348	351	allow_none=True,
349	352	desc="is_user_in_group",
350		).addCallback(lambda r: bool(r))
351
352		def is_user_admin_in_group(self, group_id, user_id):
353		return self.db_pool.simple_select_one_onecol(
	353	)
	354	return bool(result)
	355
	356	async def is_user_admin_in_group(
	357	self, group_id: str, user_id: str
	358	) -> Optional[bool]:
	359	return await self.db_pool.simple_select_one_onecol(
354	360	table="group_users",
355	361	keyvalues={"group_id": group_id, "user_id": user_id},
356	362	retcol="is_admin",

358	364	desc="is_user_admin_in_group",
359	365	)
360	366
361		def is_user_invited_to_local_group(self, group_id, user_id):
	367	async def is_user_invited_to_local_group(
	368	self, group_id: str, user_id: str
	369	) -> Optional[bool]:
362	370	"""Has the group server invited a user?
363	371	"""
364		return self.db_pool.simple_select_one_onecol(
	372	return await self.db_pool.simple_select_one_onecol(
365	373	table="group_invites",
366	374	keyvalues={"group_id": group_id, "user_id": user_id},
367	375	retcol="user_id",

369	377	allow_none=True,
370	378	)
371	379
372		def get_users_membership_info_in_group(self, group_id, user_id):
	380	async def get_users_membership_info_in_group(self, group_id, user_id):
373	381	"""Get a dict describing the membership of a user in a group.
374	382
375	383	Example if joined:

380	388	"is_privileged": False,
381	389	}
382	390
383		Returns an empty dict if the user is not join/invite/etc
	391	Returns:
	392	An empty dict if the user is not join/invite/etc
384	393	"""
385	394
386	395	def _get_users_membership_in_group_txn(txn):

412	421
413	422	return {}
414	423
415		return self.db_pool.runInteraction(
	424	return await self.db_pool.runInteraction(
416	425	"get_users_membership_info_in_group", _get_users_membership_in_group_txn
417	426	)
418	427
419		def get_publicised_groups_for_user(self, user_id):
	428	async def get_publicised_groups_for_user(self, user_id: str) -> List[str]:
420	429	"""Get all groups a user is publicising
421	430	"""
422		return self.db_pool.simple_select_onecol(
	431	return await self.db_pool.simple_select_onecol(
423	432	table="local_group_membership",
424	433	keyvalues={"user_id": user_id, "membership": "join", "is_publicised": True},
425	434	retcol="group_id",
426	435	desc="get_publicised_groups_for_user",
427	436	)
428	437
429		def get_attestations_need_renewals(self, valid_until_ms):
	438	async def get_attestations_need_renewals(self, valid_until_ms):
430	439	"""Get all attestations that need to be renewed until givent time
431	440	"""
432	441

438	447	txn.execute(sql, (valid_until_ms,))
439	448	return self.db_pool.cursor_to_dict(txn)
440	449
441		return self.db_pool.runInteraction(
	450	return await self.db_pool.runInteraction(
442	451	"get_attestations_need_renewals", _get_attestations_need_renewals_txn
443	452	)
444	453

460	469
461	470	return None
462	471
463		def get_joined_groups(self, user_id):
464		return self.db_pool.simple_select_onecol(
	472	async def get_joined_groups(self, user_id: str) -> List[str]:
	473	return await self.db_pool.simple_select_onecol(
465	474	table="local_group_membership",
466	475	keyvalues={"user_id": user_id, "membership": "join"},
467	476	retcol="group_id",
468	477	desc="get_joined_groups",
469	478	)
470	479
471		def get_all_groups_for_user(self, user_id, now_token):
	480	async def get_all_groups_for_user(self, user_id, now_token):
472	481	def _get_all_groups_for_user_txn(txn):
473	482	sql = """
474	483	SELECT group_id, type, membership, u.content

488	497	for row in txn
489	498	]
490	499
491		return self.db_pool.runInteraction(
	500	return await self.db_pool.runInteraction(
492	501	"get_all_groups_for_user", _get_all_groups_for_user_txn
493	502	)
494	503

579	588
580	589
581	590	class GroupServerStore(GroupServerWorkerStore):
582		def set_group_join_policy(self, group_id, join_policy):
	591	async def set_group_join_policy(self, group_id: str, join_policy: str) -> None:
583	592	"""Set the join policy of a group.
584	593
585	594	join_policy can be one of:
586	595	* "invite"
587	596	* "open"
588	597	"""
589		return self.db_pool.simple_update_one(
	598	await self.db_pool.simple_update_one(
590	599	table="groups",
591	600	keyvalues={"group_id": group_id},
592	601	updatevalues={"join_policy": join_policy},
593	602	desc="set_group_join_policy",
594	603	)
595	604
596		def add_room_to_summary(self, group_id, room_id, category_id, order, is_public):
597		return self.db_pool.runInteraction(
	605	async def add_room_to_summary(
	606	self,
	607	group_id: str,
	608	room_id: str,
	609	category_id: str,
	610	order: int,
	611	is_public: Optional[bool],
	612	) -> None:
	613	"""Add (or update) room's entry in summary.
	614
	615	Args:
	616	group_id
	617	room_id
	618	category_id: If not None then adds the category to the end of
	619	the summary if its not already there.
	620	order: If not None inserts the room at that position, e.g. an order
	621	of 1 will put the room first. Otherwise, the room gets added to
	622	the end.
	623	is_public
	624	"""
	625	await self.db_pool.runInteraction(
598	626	"add_room_to_summary",
599	627	self._add_room_to_summary_txn,
600	628	group_id,

605	633	)
606	634
607	635	def _add_room_to_summary_txn(
608		self, txn, group_id, room_id, category_id, order, is_public
609		):
	636	self,
	637	txn,
	638	group_id: str,
	639	room_id: str,
	640	category_id: str,
	641	order: int,
	642	is_public: Optional[bool],
	643	) -> None:
610	644	"""Add (or update) room's entry in summary.
611	645
612	646	Args:
613		group_id (str)
614		room_id (str)
615		category_id (str): If not None then adds the category to the end of
616		the summary if its not already there. [Optional]
617		order (int): If not None inserts the room at that position, e.g.
618		an order of 1 will put the room first. Otherwise, the room gets
619		added to the end.
	647	txn
	648	group_id
	649	room_id
	650	category_id: If not None then adds the category to the end of
	651	the summary if its not already there.
	652	order: If not None inserts the room at that position, e.g. an order
	653	of 1 will put the room first. Otherwise, the room gets added to
	654	the end.
	655	is_public
620	656	"""
621	657	room_in_group = self.db_pool.simple_select_one_onecol_txn(
622	658	txn,

721	757	},
722	758	)
723	759
724		def remove_room_from_summary(self, group_id, room_id, category_id):
	760	async def remove_room_from_summary(
	761	self, group_id: str, room_id: str, category_id: str
	762	) -> int:
725	763	if category_id is None:
726	764	category_id = _DEFAULT_CATEGORY_ID
727	765
728		return self.db_pool.simple_delete(
	766	return await self.db_pool.simple_delete(
729	767	table="group_summary_rooms",
730	768	keyvalues={
731	769	"group_id": group_id,

735	773	desc="remove_room_from_summary",
736	774	)
737	775
738		def upsert_group_category(self, group_id, category_id, profile, is_public):
	776	async def upsert_group_category(
	777	self,
	778	group_id: str,
	779	category_id: str,
	780	profile: Optional[JsonDict],
	781	is_public: Optional[bool],
	782	) -> None:
739	783	"""Add/update room category for group
740	784	"""
741	785	insertion_values = {}

751	795	else:
752	796	update_values["is_public"] = is_public
753	797
754		return self.db_pool.simple_upsert(
	798	await self.db_pool.simple_upsert(
755	799	table="group_room_categories",
756	800	keyvalues={"group_id": group_id, "category_id": category_id},
757	801	values=update_values,

759	803	desc="upsert_group_category",
760	804	)
761	805
762		def remove_group_category(self, group_id, category_id):
763		return self.db_pool.simple_delete(
	806	async def remove_group_category(self, group_id: str, category_id: str) -> int:
	807	return await self.db_pool.simple_delete(
764	808	table="group_room_categories",
765	809	keyvalues={"group_id": group_id, "category_id": category_id},
766	810	desc="remove_group_category",
767	811	)
768	812
769		def upsert_group_role(self, group_id, role_id, profile, is_public):
	813	async def upsert_group_role(
	814	self,
	815	group_id: str,
	816	role_id: str,
	817	profile: Optional[JsonDict],
	818	is_public: Optional[bool],
	819	) -> None:
770	820	"""Add/remove user role
771	821	"""
772	822	insertion_values = {}

782	832	else:
783	833	update_values["is_public"] = is_public
784	834
785		return self.db_pool.simple_upsert(
	835	await self.db_pool.simple_upsert(
786	836	table="group_roles",
787	837	keyvalues={"group_id": group_id, "role_id": role_id},
788	838	values=update_values,

790	840	desc="upsert_group_role",
791	841	)
792	842
793		def remove_group_role(self, group_id, role_id):
794		return self.db_pool.simple_delete(
	843	async def remove_group_role(self, group_id: str, role_id: str) -> int:
	844	return await self.db_pool.simple_delete(
795	845	table="group_roles",
796	846	keyvalues={"group_id": group_id, "role_id": role_id},
797	847	desc="remove_group_role",
798	848	)
799	849
800		def add_user_to_summary(self, group_id, user_id, role_id, order, is_public):
801		return self.db_pool.runInteraction(
	850	async def add_user_to_summary(
	851	self,
	852	group_id: str,
	853	user_id: str,
	854	role_id: str,
	855	order: int,
	856	is_public: Optional[bool],
	857	) -> None:
	858	"""Add (or update) user's entry in summary.
	859
	860	Args:
	861	group_id
	862	user_id
	863	role_id: If not None then adds the role to the end of the summary if
	864	its not already there.
	865	order: If not None inserts the user at that position, e.g. an order
	866	of 1 will put the user first. Otherwise, the user gets added to
	867	the end.
	868	is_public
	869	"""
	870	await self.db_pool.runInteraction(
802	871	"add_user_to_summary",
803	872	self._add_user_to_summary_txn,
804	873	group_id,

809	878	)
810	879
811	880	def _add_user_to_summary_txn(
812		self, txn, group_id, user_id, role_id, order, is_public
	881	self,
	882	txn,
	883	group_id: str,
	884	user_id: str,
	885	role_id: str,
	886	order: int,
	887	is_public: Optional[bool],
813	888	):
814	889	"""Add (or update) user's entry in summary.
815	890
816	891	Args:
817		group_id (str)
818		user_id (str)
819		role_id (str): If not None then adds the role to the end of
820		the summary if its not already there. [Optional]
821		order (int): If not None inserts the user at that position, e.g.
822		an order of 1 will put the user first. Otherwise, the user gets
823		added to the end.
	892	txn
	893	group_id
	894	user_id
	895	role_id: If not None then adds the role to the end of the summary if
	896	its not already there.
	897	order: If not None inserts the user at that position, e.g. an order
	898	of 1 will put the user first. Otherwise, the user gets added to
	899	the end.
	900	is_public
824	901	"""
825	902	user_in_group = self.db_pool.simple_select_one_onecol_txn(
826	903	txn,

921	998	},
922	999	)
923	1000
924		def remove_user_from_summary(self, group_id, user_id, role_id):
	1001	async def remove_user_from_summary(
	1002	self, group_id: str, user_id: str, role_id: str
	1003	) -> int:
925	1004	if role_id is None:
926	1005	role_id = _DEFAULT_ROLE_ID
927	1006
928		return self.db_pool.simple_delete(
	1007	return await self.db_pool.simple_delete(
929	1008	table="group_summary_users",
930	1009	keyvalues={"group_id": group_id, "role_id": role_id, "user_id": user_id},
931	1010	desc="remove_user_from_summary",
932	1011	)
933	1012
934		def add_group_invite(self, group_id, user_id):
	1013	async def add_group_invite(self, group_id: str, user_id: str) -> None:
935	1014	"""Record that the group server has invited a user
936	1015	"""
937		return self.db_pool.simple_insert(
	1016	await self.db_pool.simple_insert(
938	1017	table="group_invites",
939	1018	values={"group_id": group_id, "user_id": user_id},
940	1019	desc="add_group_invite",
941	1020	)
942	1021
943		def add_user_to_group(
	1022	async def add_user_to_group(
944	1023	self,
945		group_id,
946		user_id,
947		is_admin=False,
948		is_public=True,
949		local_attestation=None,
950		remote_attestation=None,
951		):
	1024	group_id: str,
	1025	user_id: str,
	1026	is_admin: bool = False,
	1027	is_public: bool = True,
	1028	local_attestation: dict = None,
	1029	remote_attestation: dict = None,
	1030	) -> None:
952	1031	"""Add a user to the group server.
953	1032
954	1033	Args:
955		group_id (str)
956		user_id (str)
957		is_admin (bool)
958		is_public (bool)
959		local_attestation (dict): The attestation the GS created to give
960		to the remote server. Optional if the user and group are on the
961		same server
962		remote_attestation (dict): The attestation given to GS by remote
	1034	group_id
	1035	user_id
	1036	is_admin
	1037	is_public
	1038	local_attestation: The attestation the GS created to give to the remote
963	1039	server. Optional if the user and group are on the same server
	1040	remote_attestation: The attestation given to GS by remote server.
	1041	Optional if the user and group are on the same server
964	1042	"""
965	1043
966	1044	def _add_user_to_group_txn(txn):

1003	1081	},
1004	1082	)
1005	1083
1006		return self.db_pool.runInteraction("add_user_to_group", _add_user_to_group_txn)
1007
1008		def remove_user_from_group(self, group_id, user_id):
	1084	await self.db_pool.runInteraction("add_user_to_group", _add_user_to_group_txn)
	1085
	1086	async def remove_user_from_group(self, group_id: str, user_id: str) -> None:
1009	1087	def _remove_user_from_group_txn(txn):
1010	1088	self.db_pool.simple_delete_txn(
1011	1089	txn,

1033	1111	keyvalues={"group_id": group_id, "user_id": user_id},
1034	1112	)
1035	1113
1036		return self.db_pool.runInteraction(
	1114	await self.db_pool.runInteraction(
1037	1115	"remove_user_from_group", _remove_user_from_group_txn
1038	1116	)
1039	1117
1040		def add_room_to_group(self, group_id, room_id, is_public):
1041		return self.db_pool.simple_insert(
	1118	async def add_room_to_group(
	1119	self, group_id: str, room_id: str, is_public: bool
	1120	) -> None:
	1121	await self.db_pool.simple_insert(
1042	1122	table="group_rooms",
1043	1123	values={"group_id": group_id, "room_id": room_id, "is_public": is_public},
1044	1124	desc="add_room_to_group",
1045	1125	)
1046	1126
1047		def update_room_in_group_visibility(self, group_id, room_id, is_public):
1048		return self.db_pool.simple_update(
	1127	async def update_room_in_group_visibility(
	1128	self, group_id: str, room_id: str, is_public: bool
	1129	) -> int:
	1130	return await self.db_pool.simple_update(
1049	1131	table="group_rooms",
1050	1132	keyvalues={"group_id": group_id, "room_id": room_id},
1051	1133	updatevalues={"is_public": is_public},
1052	1134	desc="update_room_in_group_visibility",
1053	1135	)
1054	1136
1055		def remove_room_from_group(self, group_id, room_id):
	1137	async def remove_room_from_group(self, group_id: str, room_id: str) -> None:
1056	1138	def _remove_room_from_group_txn(txn):
1057	1139	self.db_pool.simple_delete_txn(
1058	1140	txn,

1066	1148	keyvalues={"group_id": group_id, "room_id": room_id},
1067	1149	)
1068	1150
1069		return self.db_pool.runInteraction(
	1151	await self.db_pool.runInteraction(
1070	1152	"remove_room_from_group", _remove_room_from_group_txn
1071	1153	)
1072	1154
1073		def update_group_publicity(self, group_id, user_id, publicise):
	1155	async def update_group_publicity(
	1156	self, group_id: str, user_id: str, publicise: bool
	1157	) -> None:
1074	1158	"""Update whether the user is publicising their membership of the group
1075	1159	"""
1076		return self.db_pool.simple_update_one(
	1160	await self.db_pool.simple_update_one(
1077	1161	table="local_group_membership",
1078	1162	keyvalues={"group_id": group_id, "user_id": user_id},
1079	1163	updatevalues={"is_publicised": publicise},

1180	1264
1181	1265	return next_id
1182	1266
1183		with self._group_updates_id_gen.get_next() as next_id:
	1267	with await self._group_updates_id_gen.get_next() as next_id:
1184	1268	res = await self.db_pool.runInteraction(
1185	1269	"register_user_group_membership",
1186	1270	_register_user_group_membership_txn,

1212	1296	desc="update_group_profile",
1213	1297	)
1214	1298
1215		def update_attestation_renewal(self, group_id, user_id, attestation):
	1299	async def update_attestation_renewal(
	1300	self, group_id: str, user_id: str, attestation: dict
	1301	) -> None:
1216	1302	"""Update an attestation that we have renewed
1217	1303	"""
1218		return self.db_pool.simple_update_one(
	1304	await self.db_pool.simple_update_one(
1219	1305	table="group_attestations_renewals",
1220	1306	keyvalues={"group_id": group_id, "user_id": user_id},
1221	1307	updatevalues={"valid_until_ms": attestation["valid_until_ms"]},
1222	1308	desc="update_attestation_renewal",
1223	1309	)
1224	1310
1225		def update_remote_attestion(self, group_id, user_id, attestation):
	1311	async def update_remote_attestion(
	1312	self, group_id: str, user_id: str, attestation: dict
	1313	) -> None:
1226	1314	"""Update an attestation that a remote has renewed
1227	1315	"""
1228		return self.db_pool.simple_update_one(
	1316	await self.db_pool.simple_update_one(
1229	1317	table="group_attestations_remote",
1230	1318	keyvalues={"group_id": group_id, "user_id": user_id},
1231	1319	updatevalues={

1235	1323	desc="update_remote_attestion",
1236	1324	)
1237	1325
1238		def remove_attestation_renewal(self, group_id, user_id):
	1326	async def remove_attestation_renewal(self, group_id: str, user_id: str) -> int:
1239	1327	"""Remove an attestation that we thought we should renew, but actually
1240	1328	shouldn't. Ideally this would never get called as we would never
1241	1329	incorrectly try and do attestations for local users on local groups.
1242	1330
1243	1331	Args:
1244		group_id (str)
1245		user_id (str)
1246		"""
1247		return self.db_pool.simple_delete(
	1332	group_id
	1333	user_id
	1334	"""
	1335	return await self.db_pool.simple_delete(
1248	1336	table="group_attestations_renewals",
1249	1337	keyvalues={"group_id": group_id, "user_id": user_id},
1250	1338	desc="remove_attestation_renewal",

1253	1341	def get_group_stream_token(self):
1254	1342	return self._group_updates_id_gen.get_current_token()
1255	1343
1256		def delete_group(self, group_id):
	1344	async def delete_group(self, group_id: str) -> None:
1257	1345	"""Deletes a group fully from the database.
1258	1346
1259	1347	Args:
1260		group_id (str)
1261
1262		Returns:
1263		Deferred
	1348	group_id: The group ID to delete.
1264	1349	"""
1265	1350
1266	1351	def _delete_group_txn(txn):

1284	1369	txn, table=table, keyvalues={"group_id": group_id}
1285	1370	)
1286	1371
1287		return self.db_pool.runInteraction("delete_group", _delete_group_txn)
	1372	await self.db_pool.runInteraction("delete_group", _delete_group_txn)

+49

-37

synapse/storage/databases/main/keys.py less more

15	15
16	16	import itertools
17	17	import logging
	18	from typing import Dict, Iterable, List, Optional, Tuple
18	19
19	20	from signedjson.key import decode_verify_key_bytes
20	21

40	41	@cachedList(
41	42	cached_method_name="_get_server_verify_key", list_name="server_name_and_key_ids"
42	43	)
43		def get_server_verify_keys(self, server_name_and_key_ids):
44		"""
45		Args:
46		server_name_and_key_ids (iterable[Tuple[str, str]]):
	44	async def get_server_verify_keys(
	45	self, server_name_and_key_ids: Iterable[Tuple[str, str]]
	46	) -> Dict[Tuple[str, str], Optional[FetchKeyResult]]:
	47	"""
	48	Args:
	49	server_name_and_key_ids:
47	50	iterable of (server_name, key-id) tuples to fetch keys for
48	51
49	52	Returns:
50		Deferred: resolves to dict[Tuple[str, str], FetchKeyResult\|None]:
51		map from (server_name, key_id) -> FetchKeyResult, or None if the key is
52		unknown
	53	A map from (server_name, key_id) -> FetchKeyResult, or None if the
	54	key is unknown
53	55	"""
54	56	keys = {}
55	57

85	87	_get_keys(txn, batch)
86	88	return keys
87	89
88		return self.db_pool.runInteraction("get_server_verify_keys", _txn)
89
90		def store_server_verify_keys(self, from_server, ts_added_ms, verify_keys):
	90	return await self.db_pool.runInteraction("get_server_verify_keys", _txn)
	91
	92	async def store_server_verify_keys(
	93	self,
	94	from_server: str,
	95	ts_added_ms: int,
	96	verify_keys: Iterable[Tuple[str, str, FetchKeyResult]],
	97	) -> None:
91	98	"""Stores NACL verification keys for remote servers.
92	99	Args:
93		from_server (str): Where the verification keys were looked up
94		ts_added_ms (int): The time to record that the key was added
95		verify_keys (iterable[tuple[str, str, FetchKeyResult]]):
	100	from_server: Where the verification keys were looked up
	101	ts_added_ms: The time to record that the key was added
	102	verify_keys:
96	103	keys to be stored. Each entry is a triplet of
97	104	(server_name, key_id, key).
98	105	"""

114	121	# param, which is itself the 2-tuple (server_name, key_id).
115	122	invalidations.append((server_name, key_id))
116	123
117		def _invalidate(res):
118		f = self._get_server_verify_key.invalidate
119		for i in invalidations:
120		f((i,))
121		return res
122
123		return self.db_pool.runInteraction(
	124	await self.db_pool.runInteraction(
124	125	"store_server_verify_keys",
125	126	self.db_pool.simple_upsert_many_txn,
126	127	table="server_signature_keys",

133	134	"verify_key",
134	135	),
135	136	value_values=value_values,
136		).addCallback(_invalidate)
137
138		def store_server_keys_json(
139		self, server_name, key_id, from_server, ts_now_ms, ts_expires_ms, key_json_bytes
140		):
	137	)
	138
	139	invalidate = self._get_server_verify_key.invalidate
	140	for i in invalidations:
	141	invalidate((i,))
	142
	143	async def store_server_keys_json(
	144	self,
	145	server_name: str,
	146	key_id: str,
	147	from_server: str,
	148	ts_now_ms: int,
	149	ts_expires_ms: int,
	150	key_json_bytes: bytes,
	151	) -> None:
141	152	"""Stores the JSON bytes for a set of keys from a server
142	153	The JSON should be signed by the originating server, the intermediate
143	154	server, and by this server. Updates the value for the
144	155	(server_name, key_id, from_server) triplet if one already existed.
145	156	Args:
146		server_name (str): The name of the server.
147		key_id (str): The identifer of the key this JSON is for.
148		from_server (str): The server this JSON was fetched from.
149		ts_now_ms (int): The time now in milliseconds.
150		ts_valid_until_ms (int): The time when this json stops being valid.
151		key_json (bytes): The encoded JSON.
152		"""
153		return self.db_pool.simple_upsert(
	157	server_name: The name of the server.
	158	key_id: The identifer of the key this JSON is for.
	159	from_server: The server this JSON was fetched from.
	160	ts_now_ms: The time now in milliseconds.
	161	ts_valid_until_ms: The time when this json stops being valid.
	162	key_json_bytes: The encoded JSON.
	163	"""
	164	await self.db_pool.simple_upsert(
154	165	table="server_keys_json",
155	166	keyvalues={
156	167	"server_name": server_name,

168	179	desc="store_server_keys_json",
169	180	)
170	181
171		def get_server_keys_json(self, server_keys):
	182	async def get_server_keys_json(
	183	self, server_keys: Iterable[Tuple[str, Optional[str], Optional[str]]]
	184	) -> Dict[Tuple[str, Optional[str], Optional[str]], List[dict]]:
172	185	"""Retrive the key json for a list of server_keys and key ids.
173	186	If no keys are found for a given server, key_id and source then
174	187	that server, key_id, and source triplet entry will be an empty list.

177	190	Args:
178	191	server_keys (list): List of (server_name, key_id, source) triplets.
179	192	Returns:
180		Deferred[dict[Tuple[str, str, str\|None], list[dict]]]:
181		Dict mapping (server_name, key_id, source) triplets to lists of dicts
	193	A mapping from (server_name, key_id, source) triplets to a list of dicts
182	194	"""
183	195
184	196	def _get_server_keys_json_txn(txn):

204	216	results[(server_name, key_id, from_server)] = rows
205	217	return results
206	218
207		return self.db_pool.runInteraction(
	219	return await self.db_pool.runInteraction(
208	220	"get_server_keys_json", _get_server_keys_json_txn
209	221	)

+48

-36

synapse/storage/databases/main/media_repository.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14	from typing import Any, Dict, Iterable, List, Optional, Tuple
	15
14	16	from synapse.storage._base import SQLBaseStore
15	17	from synapse.storage.database import DatabasePool
16	18

36	38	def __init__(self, database: DatabasePool, db_conn, hs):
37	39	super(MediaRepositoryStore, self).__init__(database, db_conn, hs)
38	40
39		def get_local_media(self, media_id):
	41	async def get_local_media(self, media_id: str) -> Optional[Dict[str, Any]]:
40	42	"""Get the metadata for a local piece of media
	43
41	44	Returns:
42	45	None if the media_id doesn't exist.
43	46	"""
44		return self.db_pool.simple_select_one(
	47	return await self.db_pool.simple_select_one(
45	48	"local_media_repository",
46	49	{"media_id": media_id},
47	50	(

56	59	desc="get_local_media",
57	60	)
58	61
59		def store_local_media(
	62	async def store_local_media(
60	63	self,
61	64	media_id,
62	65	media_type,

65	68	media_length,
66	69	user_id,
67	70	url_cache=None,
68		):
69		return self.db_pool.simple_insert(
	71	) -> None:
	72	await self.db_pool.simple_insert(
70	73	"local_media_repository",
71	74	{
72	75	"media_id": media_id,

80	83	desc="store_local_media",
81	84	)
82	85
83		def mark_local_media_as_safe(self, media_id: str):
	86	async def mark_local_media_as_safe(self, media_id: str) -> None:
84	87	"""Mark a local media as safe from quarantining."""
85		return self.db_pool.simple_update_one(
	88	await self.db_pool.simple_update_one(
86	89	table="local_media_repository",
87	90	keyvalues={"media_id": media_id},
88	91	updatevalues={"safe_from_quarantine": True},
89	92	desc="mark_local_media_as_safe",
90	93	)
91	94
92		def get_url_cache(self, url, ts):
	95	async def get_url_cache(self, url: str, ts: int) -> Optional[Dict[str, Any]]:
93	96	"""Get the media_id and ts for a cached URL as of the given timestamp
94	97	Returns:
95	98	None if the URL isn't cached.

135	138	)
136	139	)
137	140
138		return self.db_pool.runInteraction("get_url_cache", get_url_cache_txn)
139
140		def store_url_cache(
	141	return await self.db_pool.runInteraction("get_url_cache", get_url_cache_txn)
	142
	143	async def store_url_cache(
141	144	self, url, response_code, etag, expires_ts, og, media_id, download_ts
142	145	):
143		return self.db_pool.simple_insert(
	146	await self.db_pool.simple_insert(
144	147	"local_media_repository_url_cache",
145	148	{
146	149	"url": url,

154	157	desc="store_url_cache",
155	158	)
156	159
157		def get_local_media_thumbnails(self, media_id):
158		return self.db_pool.simple_select_list(
	160	async def get_local_media_thumbnails(self, media_id: str) -> List[Dict[str, Any]]:
	161	return await self.db_pool.simple_select_list(
159	162	"local_media_repository_thumbnails",
160	163	{"media_id": media_id},
161	164	(

168	171	desc="get_local_media_thumbnails",
169	172	)
170	173
171		def store_local_thumbnail(
	174	async def store_local_thumbnail(
172	175	self,
173	176	media_id,
174	177	thumbnail_width,

177	180	thumbnail_method,
178	181	thumbnail_length,
179	182	):
180		return self.db_pool.simple_insert(
	183	await self.db_pool.simple_insert(
181	184	"local_media_repository_thumbnails",
182	185	{
183	186	"media_id": media_id,

190	193	desc="store_local_thumbnail",
191	194	)
192	195
193		def get_cached_remote_media(self, origin, media_id):
194		return self.db_pool.simple_select_one(
	196	async def get_cached_remote_media(
	197	self, origin, media_id: str
	198	) -> Optional[Dict[str, Any]]:
	199	return await self.db_pool.simple_select_one(
195	200	"remote_media_cache",
196	201	{"media_origin": origin, "media_id": media_id},
197	202	(

206	211	desc="get_cached_remote_media",
207	212	)
208	213
209		def store_cached_remote_media(
	214	async def store_cached_remote_media(
210	215	self,
211	216	origin,
212	217	media_id,

216	221	upload_name,
217	222	filesystem_id,
218	223	):
219		return self.db_pool.simple_insert(
	224	await self.db_pool.simple_insert(
220	225	"remote_media_cache",
221	226	{
222	227	"media_origin": origin,

231	236	desc="store_cached_remote_media",
232	237	)
233	238
234		def update_cached_last_access_time(self, local_media, remote_media, time_ms):
	239	async def update_cached_last_access_time(
	240	self,
	241	local_media: Iterable[str],
	242	remote_media: Iterable[Tuple[str, str]],
	243	time_ms: int,
	244	):
235	245	"""Updates the last access time of the given media
236	246
237	247	Args:
238		local_media (iterable[str]): Set of media_ids
239		remote_media (iterable[(str, str)]): Set of (server_name, media_id)
	248	local_media: Set of media_ids
	249	remote_media: Set of (server_name, media_id)
240	250	time_ms: Current time in milliseconds
241	251	"""
242	252

261	271
262	272	txn.executemany(sql, ((time_ms, media_id) for media_id in local_media))
263	273
264		return self.db_pool.runInteraction(
	274	return await self.db_pool.runInteraction(
265	275	"update_cached_last_access_time", update_cache_txn
266	276	)
267	277
268		def get_remote_media_thumbnails(self, origin, media_id):
269		return self.db_pool.simple_select_list(
	278	async def get_remote_media_thumbnails(
	279	self, origin: str, media_id: str
	280	) -> List[Dict[str, Any]]:
	281	return await self.db_pool.simple_select_list(
270	282	"remote_media_cache_thumbnails",
271	283	{"media_origin": origin, "media_id": media_id},
272	284	(

280	292	desc="get_remote_media_thumbnails",
281	293	)
282	294
283		def store_remote_media_thumbnail(
	295	async def store_remote_media_thumbnail(
284	296	self,
285	297	origin,
286	298	media_id,

291	303	thumbnail_method,
292	304	thumbnail_length,
293	305	):
294		return self.db_pool.simple_insert(
	306	await self.db_pool.simple_insert(
295	307	"remote_media_cache_thumbnails",
296	308	{
297	309	"media_origin": origin,

306	318	desc="store_remote_media_thumbnail",
307	319	)
308	320
309		def get_remote_media_before(self, before_ts):
	321	async def get_remote_media_before(self, before_ts):
310	322	sql = (
311	323	"SELECT media_origin, media_id, filesystem_id"
312	324	" FROM remote_media_cache"
313	325	" WHERE last_access_ts < ?"
314	326	)
315	327
316		return self.db_pool.execute(
	328	return await self.db_pool.execute(
317	329	"get_remote_media_before", self.db_pool.cursor_to_dict, sql, before_ts
318	330	)
319	331
320		def delete_remote_media(self, media_origin, media_id):
	332	async def delete_remote_media(self, media_origin: str, media_id: str) -> None:
321	333	def delete_remote_media_txn(txn):
322	334	self.db_pool.simple_delete_txn(
323	335	txn,

330	342	keyvalues={"media_origin": media_origin, "media_id": media_id},
331	343	)
332	344
333		return self.db_pool.runInteraction(
	345	await self.db_pool.runInteraction(
334	346	"delete_remote_media", delete_remote_media_txn
335	347	)
336	348
337		def get_expired_url_cache(self, now_ts):
	349	async def get_expired_url_cache(self, now_ts: int) -> List[str]:
338	350	sql = (
339	351	"SELECT media_id FROM local_media_repository_url_cache"
340	352	" WHERE expires_ts < ?"

346	358	txn.execute(sql, (now_ts,))
347	359	return [row[0] for row in txn]
348	360
349		return self.db_pool.runInteraction(
	361	return await self.db_pool.runInteraction(
350	362	"get_expired_url_cache", _get_expired_url_cache_txn
351	363	)
352	364

363	375	"delete_url_cache", _delete_url_cache_txn
364	376	)
365	377
366		def get_url_cache_media_before(self, before_ts):
	378	async def get_url_cache_media_before(self, before_ts: int) -> List[str]:
367	379	sql = (
368	380	"SELECT media_id FROM local_media_repository"
369	381	" WHERE created_ts < ? AND url_cache IS NOT NULL"

375	387	txn.execute(sql, (before_ts,))
376	388	return [row[0] for row in txn]
377	389
378		return self.db_pool.runInteraction(
	390	return await self.db_pool.runInteraction(
379	391	"get_url_cache_media_before", _get_url_cache_media_before_txn
380	392	)
381	393

+19

-19

synapse/storage/databases/main/monthly_active_users.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
15		from typing import List
	15	from typing import Dict, List
16	16
17	17	from synapse.storage._base import SQLBaseStore
18	18	from synapse.storage.database import DatabasePool, make_in_list_sql_clause

32	32	self.hs = hs
33	33
34	34	@cached(num_args=0)
35		def get_monthly_active_count(self):
	35	async def get_monthly_active_count(self) -> int:
36	36	"""Generates current count of monthly active users
37	37
38	38	Returns:
39		Defered[int]: Number of current monthly active users
	39	Number of current monthly active users
40	40	"""
41	41
42	42	def _count_users(txn):

45	45	(count,) = txn.fetchone()
46	46	return count
47	47
48		return self.db_pool.runInteraction("count_users", _count_users)
	48	return await self.db_pool.runInteraction("count_users", _count_users)
49	49
50	50	@cached(num_args=0)
51		def get_monthly_active_count_by_service(self):
	51	async def get_monthly_active_count_by_service(self) -> Dict[str, int]:
52	52	"""Generates current count of monthly active users broken down by service.
53	53	A service is typically an appservice but also includes native matrix users.
54	54	Since the `monthly_active_users` table is populated from the `user_ips` table

56	56	method to return anything other than native matrix users.
57	57
58	58	Returns:
59		Deferred[dict]: dict that includes a mapping between app_service_id
60		and the number of occurrences.
	59	A mapping between app_service_id and the number of occurrences.
61	60
62	61	"""
63	62

73	72	result = txn.fetchall()
74	73	return dict(result)
75	74
76		return self.db_pool.runInteraction(
	75	return await self.db_pool.runInteraction(
77	76	"count_users_by_service", _count_users_by_service
78	77	)
79	78

98	97	return users
99	98
100	99	@cached(num_args=1)
101		def user_last_seen_monthly_active(self, user_id):
102		"""
103		Checks if a given user is part of the monthly active user group
104		Arguments:
105		user_id (str): user to add/update
106		Return:
107		Deferred[int] : timestamp since last seen, None if never seen
108
109		"""
110
111		return self.db_pool.simple_select_one_onecol(
	100	async def user_last_seen_monthly_active(self, user_id: str) -> int:
	101	"""
	102	Checks if a given user is part of the monthly active user group
	103
	104	Arguments:
	105	user_id: user to add/update
	106
	107	Return:
	108	Timestamp since last seen, None if never seen
	109	"""
	110
	111	return await self.db_pool.simple_select_one_onecol(
112	112	table="monthly_active_users",
113	113	keyvalues={"user_id": user_id},
114	114	retcol="timestamp",

+10

-4

synapse/storage/databases/main/openid.py less more

	0	from typing import Optional
	1
0	2	from synapse.storage._base import SQLBaseStore
1	3
2	4
3	5	class OpenIdStore(SQLBaseStore):
4		def insert_open_id_token(self, token, ts_valid_until_ms, user_id):
5		return self.db_pool.simple_insert(
	6	async def insert_open_id_token(
	7	self, token: str, ts_valid_until_ms: int, user_id: str
	8	) -> None:
	9	await self.db_pool.simple_insert(
6	10	table="open_id_tokens",
7	11	values={
8	12	"token": token,

12	16	desc="insert_open_id_token",
13	17	)
14	18
15		def get_user_id_for_open_id_token(self, token, ts_now_ms):
	19	async def get_user_id_for_open_id_token(
	20	self, token: str, ts_now_ms: int
	21	) -> Optional[str]:
16	22	def get_user_id_for_token_txn(txn):
17	23	sql = (
18	24	"SELECT user_id FROM open_id_tokens"

27	33	else:
28	34	return rows[0][0]
29	35
30		return self.db_pool.runInteraction(
	36	return await self.db_pool.runInteraction(
31	37	"get_user_id_for_token", get_user_id_for_token_txn
32	38	)

+5

-29

synapse/storage/databases/main/presence.py less more

14	14
15	15	from typing import List, Tuple
16	16
	17	from synapse.api.presence import UserPresenceState
17	18	from synapse.storage._base import SQLBaseStore, make_in_list_sql_clause
18		from synapse.storage.presence import UserPresenceState
19	19	from synapse.util.caches.descriptors import cached, cachedList
20	20	from synapse.util.iterutils import batch_iter
21	21
22	22
23	23	class PresenceStore(SQLBaseStore):
24	24	async def update_presence(self, presence_states):
25		stream_ordering_manager = self._presence_id_gen.get_next_mult(
	25	stream_ordering_manager = await self._presence_id_gen.get_next_mult(
26	26	len(presence_states)
27	27	)
28	28

129	129	raise NotImplementedError()
130	130
131	131	@cachedList(
132		cached_method_name="_get_presence_for_user",
133		list_name="user_ids",
134		num_args=1,
135		inlineCallbacks=True,
	132	cached_method_name="_get_presence_for_user", list_name="user_ids", num_args=1,
136	133	)
137		def get_presence_for_users(self, user_ids):
138		rows = yield self.db_pool.simple_select_many_batch(
	134	async def get_presence_for_users(self, user_ids):
	135	rows = await self.db_pool.simple_select_many_batch(
139	136	table="presence_stream",
140	137	column="user_id",
141	138	iterable=user_ids,

159	156
160	157	def get_current_presence_token(self):
161	158	return self._presence_id_gen.get_current_token()
162
163		def allow_presence_visible(self, observed_localpart, observer_userid):
164		return self.db_pool.simple_insert(
165		table="presence_allow_inbound",
166		values={
167		"observed_user_id": observed_localpart,
168		"observer_user_id": observer_userid,
169		},
170		desc="allow_presence_visible",
171		or_ignore=True,
172		)
173
174		def disallow_presence_visible(self, observed_localpart, observer_userid):
175		return self.db_pool.simple_delete_one(
176		table="presence_allow_inbound",
177		keyvalues={
178		"observed_user_id": observed_localpart,
179		"observer_user_id": observer_userid,
180		},
181		desc="disallow_presence_visible",
182		)

+32

-19

synapse/storage/databases/main/profile.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14	from typing import Any, Dict, Optional
14	15
15	16	from synapse.api.errors import StoreError
16	17	from synapse.storage._base import SQLBaseStore

18	19
19	20
20	21	class ProfileWorkerStore(SQLBaseStore):
21		async def get_profileinfo(self, user_localpart):
	22	async def get_profileinfo(self, user_localpart: str) -> ProfileInfo:
22	23	try:
23	24	profile = await self.db_pool.simple_select_one(
24	25	table="profiles",

37	38	avatar_url=profile["avatar_url"], display_name=profile["displayname"]
38	39	)
39	40
40		def get_profile_displayname(self, user_localpart):
41		return self.db_pool.simple_select_one_onecol(
	41	async def get_profile_displayname(self, user_localpart: str) -> str:
	42	return await self.db_pool.simple_select_one_onecol(
42	43	table="profiles",
43	44	keyvalues={"user_id": user_localpart},
44	45	retcol="displayname",
45	46	desc="get_profile_displayname",
46	47	)
47	48
48		def get_profile_avatar_url(self, user_localpart):
49		return self.db_pool.simple_select_one_onecol(
	49	async def get_profile_avatar_url(self, user_localpart: str) -> str:
	50	return await self.db_pool.simple_select_one_onecol(
50	51	table="profiles",
51	52	keyvalues={"user_id": user_localpart},
52	53	retcol="avatar_url",
53	54	desc="get_profile_avatar_url",
54	55	)
55	56
56		def get_from_remote_profile_cache(self, user_id):
57		return self.db_pool.simple_select_one(
	57	async def get_from_remote_profile_cache(
	58	self, user_id: str
	59	) -> Optional[Dict[str, Any]]:
	60	return await self.db_pool.simple_select_one(
58	61	table="remote_profile_cache",
59	62	keyvalues={"user_id": user_id},
60	63	retcols=("displayname", "avatar_url"),

62	65	desc="get_from_remote_profile_cache",
63	66	)
64	67
65		def create_profile(self, user_localpart):
66		return self.db_pool.simple_insert(
	68	async def create_profile(self, user_localpart: str) -> None:
	69	await self.db_pool.simple_insert(
67	70	table="profiles", values={"user_id": user_localpart}, desc="create_profile"
68	71	)
69	72
70		def set_profile_displayname(self, user_localpart, new_displayname):
71		return self.db_pool.simple_update_one(
	73	async def set_profile_displayname(
	74	self, user_localpart: str, new_displayname: str
	75	) -> None:
	76	await self.db_pool.simple_update_one(
72	77	table="profiles",
73	78	keyvalues={"user_id": user_localpart},
74	79	updatevalues={"displayname": new_displayname},
75	80	desc="set_profile_displayname",
76	81	)
77	82
78		def set_profile_avatar_url(self, user_localpart, new_avatar_url):
79		return self.db_pool.simple_update_one(
	83	async def set_profile_avatar_url(
	84	self, user_localpart: str, new_avatar_url: str
	85	) -> None:
	86	await self.db_pool.simple_update_one(
80	87	table="profiles",
81	88	keyvalues={"user_id": user_localpart},
82	89	updatevalues={"avatar_url": new_avatar_url},

85	92
86	93
87	94	class ProfileStore(ProfileWorkerStore):
88		def add_remote_profile_cache(self, user_id, displayname, avatar_url):
	95	async def add_remote_profile_cache(
	96	self, user_id: str, displayname: str, avatar_url: str
	97	) -> None:
89	98	"""Ensure we are caching the remote user's profiles.
90	99
91	100	This should only be called when `is_subscribed_remote_profile_for_user`
92	101	would return true for the user.
93	102	"""
94		return self.db_pool.simple_upsert(
	103	await self.db_pool.simple_upsert(
95	104	table="remote_profile_cache",
96	105	keyvalues={"user_id": user_id},
97	106	values={

102	111	desc="add_remote_profile_cache",
103	112	)
104	113
105		def update_remote_profile_cache(self, user_id, displayname, avatar_url):
106		return self.db_pool.simple_update(
	114	async def update_remote_profile_cache(
	115	self, user_id: str, displayname: str, avatar_url: str
	116	) -> int:
	117	return await self.db_pool.simple_update(
107	118	table="remote_profile_cache",
108	119	keyvalues={"user_id": user_id},
109	120	updatevalues={

126	137	desc="delete_remote_profile_cache",
127	138	)
128	139
129		def get_remote_profile_cache_entries_that_expire(self, last_checked):
	140	async def get_remote_profile_cache_entries_that_expire(
	141	self, last_checked: int
	142	) -> Dict[str, str]:
130	143	"""Get all users who haven't been checked since `last_checked`
131	144	"""
132	145

141	154
142	155	return self.db_pool.cursor_to_dict(txn)
143	156
144		return self.db_pool.runInteraction(
	157	return await self.db_pool.runInteraction(
145	158	"get_remote_profile_cache_entries_that_expire",
146	159	_get_remote_profile_cache_entries_that_expire_txn,
147	160	)

+15

-15

synapse/storage/databases/main/purge_events.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
16		from typing import Any, Tuple
	16	from typing import Any, List, Set, Tuple
17	17
18	18	from synapse.api.errors import SynapseError
19	19	from synapse.storage._base import SQLBaseStore

24	24
25	25
26	26	class PurgeEventsStore(StateGroupWorkerStore, SQLBaseStore):
27		def purge_history(self, room_id, token, delete_local_events):
	27	async def purge_history(
	28	self, room_id: str, token: str, delete_local_events: bool
	29	) -> Set[int]:
28	30	"""Deletes room history before a certain point
29	31
30	32	Args:
31		room_id (str):
32
33		token (str): A topological token to delete events before
34
35		delete_local_events (bool):
	33	room_id:
	34	token: A topological token to delete events before
	35	delete_local_events:
36	36	if True, we will delete local events as well as remote ones
37	37	(instead of just marking them as outliers and deleting their
38	38	state groups).
39	39
40	40	Returns:
41		Deferred[set[int]]: The set of state groups that are referenced by
42		deleted events.
	41	The set of state groups that are referenced by deleted events.
43	42	"""
44	43
45		return self.db_pool.runInteraction(
	44	return await self.db_pool.runInteraction(
46	45	"purge_history",
47	46	self._purge_history_txn,
48	47	room_id,

282	281
283	282	return referenced_state_groups
284	283
285		def purge_room(self, room_id):
	284	async def purge_room(self, room_id: str) -> List[int]:
286	285	"""Deletes all record of a room
287	286
288	287	Args:
289		room_id (str)
	288	room_id
290	289
291	290	Returns:
292		Deferred[List[int]]: The list of state groups to delete.
	291	The list of state groups to delete.
293	292	"""
294
295		return self.db_pool.runInteraction("purge_room", self._purge_room_txn, room_id)
	293	return await self.db_pool.runInteraction(
	294	"purge_room", self._purge_room_txn, room_id
	295	)
296	296
297	297	def _purge_room_txn(self, txn, room_id):
298	298	# First we fetch all the state groups that should be deleted, before

+67

-77

synapse/storage/databases/main/push_rule.py less more

17	17	import logging
18	18	from typing import List, Tuple, Union
19	19
20		from twisted.internet import defer
21
22	20	from synapse.push.baserules import list_with_base_rules
23	21	from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
24	22	from synapse.storage._base import SQLBaseStore, db_to_json

29	27	from synapse.storage.databases.main.receipts import ReceiptsWorkerStore
30	28	from synapse.storage.databases.main.roommember import RoomMemberWorkerStore
31	29	from synapse.storage.push_rule import InconsistentRuleException, RuleNotFoundException
32		from synapse.storage.util.id_generators import ChainedIdGenerator
	30	from synapse.storage.util.id_generators import StreamIdGenerator
33	31	from synapse.util import json_encoder
34		from synapse.util.caches.descriptors import cachedInlineCallbacks, cachedList
	32	from synapse.util.caches.descriptors import cached, cachedList
35	33	from synapse.util.caches.stream_change_cache import StreamChangeCache
36	34
37	35	logger = logging.getLogger(__name__)

81	79	super(PushRulesWorkerStore, self).__init__(database, db_conn, hs)
82	80
83	81	if hs.config.worker.worker_app is None:
84		self._push_rules_stream_id_gen = ChainedIdGenerator(
85		self._stream_id_gen, db_conn, "push_rules_stream", "stream_id"
86		) # type: Union[ChainedIdGenerator, SlavedIdTracker]
	82	self._push_rules_stream_id_gen = StreamIdGenerator(
	83	db_conn, "push_rules_stream", "stream_id"
	84	) # type: Union[StreamIdGenerator, SlavedIdTracker]
87	85	else:
88	86	self._push_rules_stream_id_gen = SlavedIdTracker(
89	87	db_conn, "push_rules_stream", "stream_id"

114	112	"""
115	113	raise NotImplementedError()
116	114
117		@cachedInlineCallbacks(max_entries=5000)
118		def get_push_rules_for_user(self, user_id):
119		rows = yield self.db_pool.simple_select_list(
	115	@cached(max_entries=5000)
	116	async def get_push_rules_for_user(self, user_id):
	117	rows = await self.db_pool.simple_select_list(
120	118	table="push_rules",
121	119	keyvalues={"user_name": user_id},
122	120	retcols=(

132	130
133	131	rows.sort(key=lambda row: (-int(row["priority_class"]), -int(row["priority"])))
134	132
135		enabled_map = yield self.get_push_rules_enabled_for_user(user_id)
	133	enabled_map = await self.get_push_rules_enabled_for_user(user_id)
136	134
137	135	use_new_defaults = user_id in self._users_new_default_push_rules
138	136
139		rules = _load_rules(rows, enabled_map, use_new_defaults)
140
141		return rules
142
143		@cachedInlineCallbacks(max_entries=5000)
144		def get_push_rules_enabled_for_user(self, user_id):
145		results = yield self.db_pool.simple_select_list(
	137	return _load_rules(rows, enabled_map, use_new_defaults)
	138
	139	@cached(max_entries=5000)
	140	async def get_push_rules_enabled_for_user(self, user_id):
	141	results = await self.db_pool.simple_select_list(
146	142	table="push_rules_enable",
147	143	keyvalues={"user_name": user_id},
148	144	retcols=("user_name", "rule_id", "enabled"),

150	146	)
151	147	return {r["rule_id"]: False if r["enabled"] == 0 else True for r in results}
152	148
153		def have_push_rules_changed_for_user(self, user_id, last_id):
	149	async def have_push_rules_changed_for_user(
	150	self, user_id: str, last_id: int
	151	) -> bool:
154	152	if not self.push_rules_stream_cache.has_entity_changed(user_id, last_id):
155		return defer.succeed(False)
	153	return False
156	154	else:
157	155
158	156	def have_push_rules_changed_txn(txn):

164	162	(count,) = txn.fetchone()
165	163	return bool(count)
166	164
167		return self.db_pool.runInteraction(
	165	return await self.db_pool.runInteraction(
168	166	"have_push_rules_changed", have_push_rules_changed_txn
169	167	)
170	168
171	169	@cachedList(
172		cached_method_name="get_push_rules_for_user",
173		list_name="user_ids",
174		num_args=1,
175		inlineCallbacks=True,
	170	cached_method_name="get_push_rules_for_user", list_name="user_ids", num_args=1,
176	171	)
177		def bulk_get_push_rules(self, user_ids):
	172	async def bulk_get_push_rules(self, user_ids):
178	173	if not user_ids:
179	174	return {}
180	175
181	176	results = {user_id: [] for user_id in user_ids}
182	177
183		rows = yield self.db_pool.simple_select_many_batch(
	178	rows = await self.db_pool.simple_select_many_batch(
184	179	table="push_rules",
185	180	column="user_name",
186	181	iterable=user_ids,

193	188	for row in rows:
194	189	results.setdefault(row["user_name"], []).append(row)
195	190
196		enabled_map_by_user = yield self.bulk_get_push_rules_enabled(user_ids)
	191	enabled_map_by_user = await self.bulk_get_push_rules_enabled(user_ids)
197	192
198	193	for user_id, rules in results.items():
199	194	use_new_defaults = user_id in self._users_new_default_push_rules

204	199
205	200	return results
206	201
207		@defer.inlineCallbacks
208		def copy_push_rule_from_room_to_room(self, new_room_id, user_id, rule):
	202	async def copy_push_rule_from_room_to_room(
	203	self, new_room_id: str, user_id: str, rule: dict
	204	) -> None:
209	205	"""Copy a single push rule from one room to another for a specific user.
210	206
211	207	Args:
212		new_room_id (str): ID of the new room.
213		user_id (str): ID of user the push rule belongs to.
214		rule (Dict): A push rule.
	208	new_room_id: ID of the new room.
	209	user_id : ID of user the push rule belongs to.
	210	rule: A push rule.
215	211	"""
216	212	# Create new rule id
217	213	rule_id_scope = "/".join(rule["rule_id"].split("/")[:-1])

223	219	condition["pattern"] = new_room_id
224	220
225	221	# Add the rule for the new room
226		yield self.add_push_rule(
	222	await self.add_push_rule(
227	223	user_id=user_id,
228	224	rule_id=new_rule_id,
229	225	priority_class=rule["priority_class"],

231	227	actions=rule["actions"],
232	228	)
233	229
234		@defer.inlineCallbacks
235		def copy_push_rules_from_room_to_room_for_user(
236		self, old_room_id, new_room_id, user_id
237		):
	230	async def copy_push_rules_from_room_to_room_for_user(
	231	self, old_room_id: str, new_room_id: str, user_id: str
	232	) -> None:
238	233	"""Copy all of the push rules from one room to another for a specific
239	234	user.
240	235
241	236	Args:
242		old_room_id (str): ID of the old room.
243		new_room_id (str): ID of the new room.
244		user_id (str): ID of user to copy push rules for.
	237	old_room_id: ID of the old room.
	238	new_room_id: ID of the new room.
	239	user_id: ID of user to copy push rules for.
245	240	"""
246	241	# Retrieve push rules for this user
247		user_push_rules = yield self.get_push_rules_for_user(user_id)
	242	user_push_rules = await self.get_push_rules_for_user(user_id)
248	243
249	244	# Get rules relating to the old room and copy them to the new room
250	245	for rule in user_push_rules:

253	248	(c.get("key") == "room_id" and c.get("pattern") == old_room_id)
254	249	for c in conditions
255	250	):
256		yield self.copy_push_rule_from_room_to_room(new_room_id, user_id, rule)
	251	await self.copy_push_rule_from_room_to_room(new_room_id, user_id, rule)
257	252
258	253	@cachedList(
259	254	cached_method_name="get_push_rules_enabled_for_user",
260	255	list_name="user_ids",
261	256	num_args=1,
262		inlineCallbacks=True,
263	257	)
264		def bulk_get_push_rules_enabled(self, user_ids):
	258	async def bulk_get_push_rules_enabled(self, user_ids):
265	259	if not user_ids:
266	260	return {}
267	261
268	262	results = {user_id: {} for user_id in user_ids}
269	263
270		rows = yield self.db_pool.simple_select_many_batch(
	264	rows = await self.db_pool.simple_select_many_batch(
271	265	table="push_rules_enable",
272	266	column="user_name",
273	267	iterable=user_ids,

331	325
332	326
333	327	class PushRuleStore(PushRulesWorkerStore):
334		@defer.inlineCallbacks
335		def add_push_rule(
	328	async def add_push_rule(
336	329	self,
337	330	user_id,
338	331	rule_id,

341	334	actions,
342	335	before=None,
343	336	after=None,
344		):
	337	) -> None:
345	338	conditions_json = json_encoder.encode(conditions)
346	339	actions_json = json_encoder.encode(actions)
347		with self._push_rules_stream_id_gen.get_next() as ids:
348		stream_id, event_stream_ordering = ids
	340	with await self._push_rules_stream_id_gen.get_next() as stream_id:
	341	event_stream_ordering = self._stream_id_gen.get_current_token()
	342
349	343	if before or after:
350		yield self.db_pool.runInteraction(
	344	await self.db_pool.runInteraction(
351	345	"_add_push_rule_relative_txn",
352	346	self._add_push_rule_relative_txn,
353	347	stream_id,

361	355	after,
362	356	)
363	357	else:
364		yield self.db_pool.runInteraction(
	358	await self.db_pool.runInteraction(
365	359	"_add_push_rule_highest_priority_txn",
366	360	self._add_push_rule_highest_priority_txn,
367	361	stream_id,

545	539	},
546	540	)
547	541
548		@defer.inlineCallbacks
549		def delete_push_rule(self, user_id, rule_id):
	542	async def delete_push_rule(self, user_id: str, rule_id: str) -> None:
550	543	"""
551	544	Delete a push rule. Args specify the row to be deleted and can be
552	545	any of the columns in the push_rule table, but below are the
553	546	standard ones
554	547
555	548	Args:
556		user_id (str): The matrix ID of the push rule owner
557		rule_id (str): The rule_id of the rule to be deleted
	549	user_id: The matrix ID of the push rule owner
	550	rule_id: The rule_id of the rule to be deleted
558	551	"""
559	552
560	553	def delete_push_rule_txn(txn, stream_id, event_stream_ordering):

566	559	txn, stream_id, event_stream_ordering, user_id, rule_id, op="DELETE"
567	560	)
568	561
569		with self._push_rules_stream_id_gen.get_next() as ids:
570		stream_id, event_stream_ordering = ids
571		yield self.db_pool.runInteraction(
	562	with await self._push_rules_stream_id_gen.get_next() as stream_id:
	563	event_stream_ordering = self._stream_id_gen.get_current_token()
	564
	565	await self.db_pool.runInteraction(
572	566	"delete_push_rule",
573	567	delete_push_rule_txn,
574	568	stream_id,
575	569	event_stream_ordering,
576	570	)
577	571
578		@defer.inlineCallbacks
579		def set_push_rule_enabled(self, user_id, rule_id, enabled):
580		with self._push_rules_stream_id_gen.get_next() as ids:
581		stream_id, event_stream_ordering = ids
582		yield self.db_pool.runInteraction(
	572	async def set_push_rule_enabled(self, user_id, rule_id, enabled) -> None:
	573	with await self._push_rules_stream_id_gen.get_next() as stream_id:
	574	event_stream_ordering = self._stream_id_gen.get_current_token()
	575
	576	await self.db_pool.runInteraction(
583	577	"_set_push_rule_enabled_txn",
584	578	self._set_push_rule_enabled_txn,
585	579	stream_id,

610	604	op="ENABLE" if enabled else "DISABLE",
611	605	)
612	606
613		@defer.inlineCallbacks
614		def set_push_rule_actions(self, user_id, rule_id, actions, is_default_rule):
	607	async def set_push_rule_actions(
	608	self, user_id, rule_id, actions, is_default_rule
	609	) -> None:
615	610	actions_json = json_encoder.encode(actions)
616	611
617	612	def set_push_rule_actions_txn(txn, stream_id, event_stream_ordering):

650	645	data={"actions": actions_json},
651	646	)
652	647
653		with self._push_rules_stream_id_gen.get_next() as ids:
654		stream_id, event_stream_ordering = ids
655		yield self.db_pool.runInteraction(
	648	with await self._push_rules_stream_id_gen.get_next() as stream_id:
	649	event_stream_ordering = self._stream_id_gen.get_current_token()
	650
	651	await self.db_pool.runInteraction(
656	652	"set_push_rule_actions",
657	653	set_push_rule_actions_txn,
658	654	stream_id,

680	676	self.push_rules_stream_cache.entity_has_changed, user_id, stream_id
681	677	)
682	678
683		def get_push_rules_stream_token(self):
684		"""Get the position of the push rules stream.
685		Returns a pair of a stream id for the push_rules stream and the
686		room stream ordering it corresponds to."""
	679	def get_max_push_rules_stream_id(self):
687	680	return self._push_rules_stream_id_gen.get_current_token()
688
689		def get_max_push_rules_stream_id(self):
690		return self.get_push_rules_stream_token()[0]

+51

-57

synapse/storage/databases/main/pusher.py less more

18	18
19	19	from canonicaljson import encode_canonical_json
20	20
21		from twisted.internet import defer
22
23	21	from synapse.storage._base import SQLBaseStore, db_to_json
24		from synapse.util.caches.descriptors import cachedInlineCallbacks, cachedList
	22	from synapse.util.caches.descriptors import cached, cachedList
25	23
26	24	logger = logging.getLogger(__name__)
27	25

33	31	Drops any rows whose data cannot be decoded
34	32	"""
35	33	for r in rows:
36		dataJson = r["data"]
	34	data_json = r["data"]
37	35	try:
38		r["data"] = db_to_json(dataJson)
	36	r["data"] = db_to_json(data_json)
39	37	except Exception as e:
40	38	logger.warning(
41	39	"Invalid JSON in data for pusher %d: %s, %s",
42	40	r["id"],
43		dataJson,
	41	data_json,
44	42	e.args[0],
45	43	)
46	44	continue
47	45
48	46	yield r
49	47
50		@defer.inlineCallbacks
51		def user_has_pusher(self, user_id):
52		ret = yield self.db_pool.simple_select_one_onecol(
	48	async def user_has_pusher(self, user_id):
	49	ret = await self.db_pool.simple_select_one_onecol(
53	50	"pushers", {"user_name": user_id}, "id", allow_none=True
54	51	)
55	52	return ret is not None

60	57	def get_pushers_by_user_id(self, user_id):
61	58	return self.get_pushers_by({"user_name": user_id})
62	59
63		@defer.inlineCallbacks
64		def get_pushers_by(self, keyvalues):
65		ret = yield self.db_pool.simple_select_list(
	60	async def get_pushers_by(self, keyvalues):
	61	ret = await self.db_pool.simple_select_list(
66	62	"pushers",
67	63	keyvalues,
68	64	[

86	82	)
87	83	return self._decode_pushers_rows(ret)
88	84
89		@defer.inlineCallbacks
90		def get_all_pushers(self):
	85	async def get_all_pushers(self):
91	86	def get_pushers(txn):
92	87	txn.execute("SELECT * FROM pushers")
93	88	rows = self.db_pool.cursor_to_dict(txn)
94	89
95	90	return self._decode_pushers_rows(rows)
96	91
97		rows = yield self.db_pool.runInteraction("get_all_pushers", get_pushers)
98		return rows
	92	return await self.db_pool.runInteraction("get_all_pushers", get_pushers)
99	93
100	94	async def get_all_updated_pushers_rows(
101	95	self, instance_name: str, last_id: int, current_id: int, limit: int

163	157	"get_all_updated_pushers_rows", get_all_updated_pushers_rows_txn
164	158	)
165	159
166		@cachedInlineCallbacks(num_args=1, max_entries=15000)
167		def get_if_user_has_pusher(self, user_id):
	160	@cached(num_args=1, max_entries=15000)
	161	async def get_if_user_has_pusher(self, user_id):
168	162	# This only exists for the cachedList decorator
169	163	raise NotImplementedError()
170	164
171	165	@cachedList(
172		cached_method_name="get_if_user_has_pusher",
173		list_name="user_ids",
174		num_args=1,
175		inlineCallbacks=True,
	166	cached_method_name="get_if_user_has_pusher", list_name="user_ids", num_args=1,
176	167	)
177		def get_if_users_have_pushers(self, user_ids):
178		rows = yield self.db_pool.simple_select_many_batch(
	168	async def get_if_users_have_pushers(self, user_ids):
	169	rows = await self.db_pool.simple_select_many_batch(
179	170	table="pushers",
180	171	column="user_name",
181	172	iterable=user_ids,

188	179
189	180	return result
190	181
191		@defer.inlineCallbacks
192		def update_pusher_last_stream_ordering(
	182	async def update_pusher_last_stream_ordering(
193	183	self, app_id, pushkey, user_id, last_stream_ordering
194		):
195		yield self.db_pool.simple_update_one(
	184	) -> None:
	185	await self.db_pool.simple_update_one(
196	186	"pushers",
197	187	{"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
198	188	{"last_stream_ordering": last_stream_ordering},
199	189	desc="update_pusher_last_stream_ordering",
200	190	)
201	191
202		@defer.inlineCallbacks
203		def update_pusher_last_stream_ordering_and_success(
204		self, app_id, pushkey, user_id, last_stream_ordering, last_success
205		):
	192	async def update_pusher_last_stream_ordering_and_success(
	193	self,
	194	app_id: str,
	195	pushkey: str,
	196	user_id: str,
	197	last_stream_ordering: int,
	198	last_success: int,
	199	) -> bool:
206	200	"""Update the last stream ordering position we've processed up to for
207	201	the given pusher.
208	202
209	203	Args:
210		app_id (str)
211		pushkey (str)
212		last_stream_ordering (int)
213		last_success (int)
	204	app_id
	205	pushkey
	206	user_id
	207	last_stream_ordering
	208	last_success
214	209
215	210	Returns:
216		Deferred[bool]: True if the pusher still exists; False if it has been deleted.
	211	True if the pusher still exists; False if it has been deleted.
217	212	"""
218		updated = yield self.db_pool.simple_update(
	213	updated = await self.db_pool.simple_update(
219	214	table="pushers",
220	215	keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
221	216	updatevalues={

227	222
228	223	return bool(updated)
229	224
230		@defer.inlineCallbacks
231		def update_pusher_failing_since(self, app_id, pushkey, user_id, failing_since):
232		yield self.db_pool.simple_update(
	225	async def update_pusher_failing_since(
	226	self, app_id, pushkey, user_id, failing_since
	227	) -> None:
	228	await self.db_pool.simple_update(
233	229	table="pushers",
234	230	keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
235	231	updatevalues={"failing_since": failing_since},
236	232	desc="update_pusher_failing_since",
237	233	)
238	234
239		@defer.inlineCallbacks
240		def get_throttle_params_by_room(self, pusher_id):
241		res = yield self.db_pool.simple_select_list(
	235	async def get_throttle_params_by_room(self, pusher_id):
	236	res = await self.db_pool.simple_select_list(
242	237	"pusher_throttle",
243	238	{"pusher": pusher_id},
244	239	["room_id", "last_sent_ts", "throttle_ms"],

254	249
255	250	return params_by_room
256	251
257		@defer.inlineCallbacks
258		def set_throttle_params(self, pusher_id, room_id, params):
	252	async def set_throttle_params(self, pusher_id, room_id, params) -> None:
259	253	# no need to lock because `pusher_throttle` has a primary key on
260	254	# (pusher, room_id) so simple_upsert will retry
261		yield self.db_pool.simple_upsert(
	255	await self.db_pool.simple_upsert(
262	256	"pusher_throttle",
263	257	{"pusher": pusher_id, "room_id": room_id},
264	258	params,

271	265	def get_pushers_stream_token(self):
272	266	return self._pushers_id_gen.get_current_token()
273	267
274		@defer.inlineCallbacks
275		def add_pusher(
	268	async def add_pusher(
276	269	self,
277	270	user_id,
278	271	access_token,

286	279	data,
287	280	last_stream_ordering,
288	281	profile_tag="",
289		):
290		with self._pushers_id_gen.get_next() as stream_id:
	282	) -> None:
	283	with await self._pushers_id_gen.get_next() as stream_id:
291	284	# no need to lock because `pushers` has a unique key on
292	285	# (app_id, pushkey, user_name) so simple_upsert will retry
293		yield self.db_pool.simple_upsert(
	286	await self.db_pool.simple_upsert(
294	287	table="pushers",
295	288	keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
296	289	values={

315	308
316	309	if user_has_pusher is not True:
317	310	# invalidate, since we the user might not have had a pusher before
318		yield self.db_pool.runInteraction(
	311	await self.db_pool.runInteraction(
319	312	"add_pusher",
320	313	self._invalidate_cache_and_stream,
321	314	self.get_if_user_has_pusher,
322	315	(user_id,),
323	316	)
324	317
325		@defer.inlineCallbacks
326		def delete_pusher_by_app_id_pushkey_user_id(self, app_id, pushkey, user_id):
	318	async def delete_pusher_by_app_id_pushkey_user_id(
	319	self, app_id, pushkey, user_id
	320	) -> None:
327	321	def delete_pusher_txn(txn, stream_id):
328	322	self._invalidate_cache_and_stream(
329	323	txn, self.get_if_user_has_pusher, (user_id,)

349	343	},
350	344	)
351	345
352		with self._pushers_id_gen.get_next() as stream_id:
353		yield self.db_pool.runInteraction(
	346	with await self._pushers_id_gen.get_next() as stream_id:
	347	await self.db_pool.runInteraction(
354	348	"delete_pusher", delete_pusher_txn, stream_id
355	349	)

+68

-52

synapse/storage/databases/main/receipts.py less more

15	15
16	16	import abc
17	17	import logging
18		from typing import List, Tuple
	18	from typing import Any, Dict, List, Optional, Tuple
19	19
20	20	from twisted.internet import defer
21	21

24	24	from synapse.storage.util.id_generators import StreamIdGenerator
25	25	from synapse.util import json_encoder
26	26	from synapse.util.async_helpers import ObservableDeferred
27		from synapse.util.caches.descriptors import cached, cachedInlineCallbacks, cachedList
	27	from synapse.util.caches.descriptors import cached, cachedList
28	28	from synapse.util.caches.stream_change_cache import StreamChangeCache
29	29
30	30	logger = logging.getLogger(__name__)

55	55	"""
56	56	raise NotImplementedError()
57	57
58		@cachedInlineCallbacks()
59		def get_users_with_read_receipts_in_room(self, room_id):
60		receipts = yield self.get_receipts_for_room(room_id, "m.read")
	58	@cached()
	59	async def get_users_with_read_receipts_in_room(self, room_id):
	60	receipts = await self.get_receipts_for_room(room_id, "m.read")
61	61	return {r["user_id"] for r in receipts}
62	62
63	63	@cached(num_args=2)
64		def get_receipts_for_room(self, room_id, receipt_type):
65		return self.db_pool.simple_select_list(
	64	async def get_receipts_for_room(
	65	self, room_id: str, receipt_type: str
	66	) -> List[Dict[str, Any]]:
	67	return await self.db_pool.simple_select_list(
66	68	table="receipts_linearized",
67	69	keyvalues={"room_id": room_id, "receipt_type": receipt_type},
68	70	retcols=("user_id", "event_id"),

70	72	)
71	73
72	74	@cached(num_args=3)
73		def get_last_receipt_event_id_for_user(self, user_id, room_id, receipt_type):
74		return self.db_pool.simple_select_one_onecol(
	75	async def get_last_receipt_event_id_for_user(
	76	self, user_id: str, room_id: str, receipt_type: str
	77	) -> Optional[str]:
	78	return await self.db_pool.simple_select_one_onecol(
75	79	table="receipts_linearized",
76	80	keyvalues={
77	81	"room_id": room_id,

83	87	allow_none=True,
84	88	)
85	89
86		@cachedInlineCallbacks(num_args=2)
87		def get_receipts_for_user(self, user_id, receipt_type):
88		rows = yield self.db_pool.simple_select_list(
	90	@cached(num_args=2)
	91	async def get_receipts_for_user(self, user_id, receipt_type):
	92	rows = await self.db_pool.simple_select_list(
89	93	table="receipts_linearized",
90	94	keyvalues={"user_id": user_id, "receipt_type": receipt_type},
91	95	retcols=("room_id", "event_id"),

94	98
95	99	return {row["room_id"]: row["event_id"] for row in rows}
96	100
97		@defer.inlineCallbacks
98		def get_receipts_for_user_with_orderings(self, user_id, receipt_type):
	101	async def get_receipts_for_user_with_orderings(self, user_id, receipt_type):
99	102	def f(txn):
100	103	sql = (
101	104	"SELECT rl.room_id, rl.event_id,"

109	112	txn.execute(sql, (user_id,))
110	113	return txn.fetchall()
111	114
112		rows = yield self.db_pool.runInteraction(
	115	rows = await self.db_pool.runInteraction(
113	116	"get_receipts_for_user_with_orderings", f
114	117	)
115	118	return {

121	124	for row in rows
122	125	}
123	126
124		@defer.inlineCallbacks
125		def get_linearized_receipts_for_rooms(self, room_ids, to_key, from_key=None):
	127	async def get_linearized_receipts_for_rooms(
	128	self, room_ids: List[str], to_key: int, from_key: Optional[int] = None
	129	) -> List[dict]:
126	130	"""Get receipts for multiple rooms for sending to clients.
127	131
128	132	Args:
129		room_ids (list): List of room_ids.
130		to_key (int): Max stream id to fetch receipts upto.
131		from_key (int): Min stream id to fetch receipts from. None fetches
	133	room_id: List of room_ids.
	134	to_key: Max stream id to fetch receipts upto.
	135	from_key: Min stream id to fetch receipts from. None fetches
132	136	from the start.
133	137
134	138	Returns:
135		list: A list of receipts.
	139	A list of receipts.
136	140	"""
137	141	room_ids = set(room_ids)
138	142
139	143	if from_key is not None:
140	144	# Only ask the database about rooms where there have been new
141	145	# receipts added since `from_key`
142		room_ids = yield self._receipts_stream_cache.get_entities_changed(
	146	room_ids = self._receipts_stream_cache.get_entities_changed(
143	147	room_ids, from_key
144	148	)
145	149
146		results = yield self._get_linearized_receipts_for_rooms(
	150	results = await self._get_linearized_receipts_for_rooms(
147	151	room_ids, to_key, from_key=from_key
148	152	)
149	153
150	154	return [ev for res in results.values() for ev in res]
151	155
152		def get_linearized_receipts_for_room(self, room_id, to_key, from_key=None):
	156	async def get_linearized_receipts_for_room(
	157	self, room_id: str, to_key: int, from_key: Optional[int] = None
	158	) -> List[dict]:
153	159	"""Get receipts for a single room for sending to clients.
154	160
155	161	Args:
156		room_ids (str): The room id.
157		to_key (int): Max stream id to fetch receipts upto.
158		from_key (int): Min stream id to fetch receipts from. None fetches
	162	room_ids: The room id.
	163	to_key: Max stream id to fetch receipts upto.
	164	from_key: Min stream id to fetch receipts from. None fetches
159	165	from the start.
160	166
161	167	Returns:
162		Deferred[list]: A list of receipts.
	168	A list of receipts.
163	169	"""
164	170	if from_key is not None:
165	171	# Check the cache first to see if any new receipts have been added
166	172	# since`from_key`. If not we can no-op.
167	173	if not self._receipts_stream_cache.has_entity_changed(room_id, from_key):
168		defer.succeed([])
169
170		return self._get_linearized_receipts_for_room(room_id, to_key, from_key)
171
172		@cachedInlineCallbacks(num_args=3, tree=True)
173		def _get_linearized_receipts_for_room(self, room_id, to_key, from_key=None):
	174	return []
	175
	176	return await self._get_linearized_receipts_for_room(room_id, to_key, from_key)
	177
	178	@cached(num_args=3, tree=True)
	179	async def _get_linearized_receipts_for_room(
	180	self, room_id: str, to_key: int, from_key: Optional[int] = None
	181	) -> List[dict]:
174	182	"""See get_linearized_receipts_for_room
175	183	"""
176	184

194	202
195	203	return rows
196	204
197		rows = yield self.db_pool.runInteraction("get_linearized_receipts_for_room", f)
	205	rows = await self.db_pool.runInteraction("get_linearized_receipts_for_room", f)
198	206
199	207	if not rows:
200	208	return []

211	219	cached_method_name="_get_linearized_receipts_for_room",
212	220	list_name="room_ids",
213	221	num_args=3,
214		inlineCallbacks=True,
215	222	)
216		def _get_linearized_receipts_for_rooms(self, room_ids, to_key, from_key=None):
	223	async def _get_linearized_receipts_for_rooms(self, room_ids, to_key, from_key=None):
217	224	if not room_ids:
218	225	return {}
219	226

242	249
243	250	return self.db_pool.cursor_to_dict(txn)
244	251
245		txn_results = yield self.db_pool.runInteraction(
	252	txn_results = await self.db_pool.runInteraction(
246	253	"_get_linearized_receipts_for_rooms", f
247	254	)
248	255

268	275	}
269	276	return results
270	277
271		def get_users_sent_receipts_between(self, last_id: int, current_id: int):
	278	async def get_users_sent_receipts_between(
	279	self, last_id: int, current_id: int
	280	) -> List[str]:
272	281	"""Get all users who sent receipts between `last_id` exclusive and
273	282	`current_id` inclusive.
274	283
275	284	Returns:
276		Deferred[List[str]]
	285	The list of users.
277	286	"""
278	287
279	288	if last_id == current_id:

288	297
289	298	return [r[0] for r in txn]
290	299
291		return self.db_pool.runInteraction(
	300	return await self.db_pool.runInteraction(
292	301	"get_users_sent_receipts_between", _get_users_sent_receipts_between_txn
293	302	)
294	303

345	354	)
346	355
347	356	def _invalidate_get_users_with_receipts_in_room(
348		self, room_id, receipt_type, user_id
	357	self, room_id: str, receipt_type: str, user_id: str
349	358	):
350	359	if receipt_type != "m.read":
351	360	return

471	480
472	481	return rx_ts
473	482
474		@defer.inlineCallbacks
475		def insert_receipt(self, room_id, receipt_type, user_id, event_ids, data):
	483	async def insert_receipt(
	484	self,
	485	room_id: str,
	486	receipt_type: str,
	487	user_id: str,
	488	event_ids: List[str],
	489	data: dict,
	490	) -> Optional[Tuple[int, int]]:
476	491	"""Insert a receipt, either from local client or remote server.
477	492
478	493	Automatically does conversion between linearized and graph
479	494	representations.
480	495	"""
481	496	if not event_ids:
482		return
	497	return None
483	498
484	499	if len(event_ids) == 1:
485	500	linearized_event_id = event_ids[0]

506	521	else:
507	522	raise RuntimeError("Unrecognized event_ids: %r" % (event_ids,))
508	523
509		linearized_event_id = yield self.db_pool.runInteraction(
	524	linearized_event_id = await self.db_pool.runInteraction(
510	525	"insert_receipt_conv", graph_to_linear
511	526	)
512	527
513		stream_id_manager = self._receipts_id_gen.get_next()
514		with stream_id_manager as stream_id:
515		event_ts = yield self.db_pool.runInteraction(
	528	with await self._receipts_id_gen.get_next() as stream_id:
	529	event_ts = await self.db_pool.runInteraction(
516	530	"insert_linearized_receipt",
517	531	self.insert_linearized_receipt_txn,
518	532	room_id,

534	548	now - event_ts,
535	549	)
536	550
537		yield self.insert_graph_receipt(room_id, receipt_type, user_id, event_ids, data)
	551	await self.insert_graph_receipt(room_id, receipt_type, user_id, event_ids, data)
538	552
539	553	max_persisted_id = self._receipts_id_gen.get_current_token()
540	554
541	555	return stream_id, max_persisted_id
542	556
543		def insert_graph_receipt(self, room_id, receipt_type, user_id, event_ids, data):
544		return self.db_pool.runInteraction(
	557	async def insert_graph_receipt(
	558	self, room_id, receipt_type, user_id, event_ids, data
	559	):
	560	return await self.db_pool.runInteraction(
545	561	"insert_graph_receipt",
546	562	self.insert_graph_receipt_txn,
547	563	room_id,

+204

-215

synapse/storage/databases/main/registration.py less more

16	16
17	17	import logging
18	18	import re
19		from typing import Dict, List, Optional
20
21		from twisted.internet.defer import Deferred
	19	from typing import Any, Dict, List, Optional, Tuple
22	20
23	21	from synapse.api.constants import UserTypes
24	22	from synapse.api.errors import Codes, StoreError, SynapseError, ThreepidValidationError

47	45	)
48	46
49	47	@cached()
50		def get_user_by_id(self, user_id):
51		return self.db_pool.simple_select_one(
	48	async def get_user_by_id(self, user_id: str) -> Optional[Dict[str, Any]]:
	49	return await self.db_pool.simple_select_one(
52	50	table="users",
53	51	keyvalues={"name": user_id},
54	52	retcols=[

85	83	return is_trial
86	84
87	85	@cached()
88		def get_user_by_access_token(self, token):
	86	async def get_user_by_access_token(self, token: str) -> Optional[dict]:
89	87	"""Get a user from the given access token.
90	88
91	89	Args:
92		token (str): The access token of a user.
93		Returns:
94		defer.Deferred: None, if the token did not match, otherwise dict
95		including the keys `name`, `is_guest`, `device_id`, `token_id`,
96		`valid_until_ms`.
97		"""
98		return self.db_pool.runInteraction(
	90	token: The access token of a user.
	91	Returns:
	92	None, if the token did not match, otherwise dict
	93	including the keys `name`, `is_guest`, `device_id`, `token_id`,
	94	`valid_until_ms`.
	95	"""
	96	return await self.db_pool.runInteraction(
99	97	"get_user_by_access_token", self._query_for_auth, token
100	98	)
101	99
102	100	@cached()
103		async def get_expiration_ts_for_user(self, user_id: str) -> Optional[None]:
	101	async def get_expiration_ts_for_user(self, user_id: str) -> Optional[int]:
104	102	"""Get the expiration timestamp for the account bearing a given user ID.
105	103
106	104	Args:

282	280
283	281	return bool(res) if res else False
284	282
285		def set_server_admin(self, user, admin):
	283	async def set_server_admin(self, user: UserID, admin: bool) -> None:
286	284	"""Sets whether a user is an admin of this homeserver.
287	285
288	286	Args:
289		user (UserID): user ID of the user to test
290		admin (bool): true iff the user is to be a server admin,
291		false otherwise.
	287	user: user ID of the user to test
	288	admin: true iff the user is to be a server admin, false otherwise.
292	289	"""
293	290
294	291	def set_server_admin_txn(txn):

299	296	txn, self.get_user_by_id, (user.to_string(),)
300	297	)
301	298
302		return self.db_pool.runInteraction("set_server_admin", set_server_admin_txn)
	299	await self.db_pool.runInteraction("set_server_admin", set_server_admin_txn)
303	300
304	301	def _query_for_auth(self, txn, token):
305	302	sql = (
306		"SELECT users.name, users.is_guest, access_tokens.id as token_id,"
	303	"SELECT users.name, users.is_guest, users.shadow_banned, access_tokens.id as token_id,"
307	304	" access_tokens.device_id, access_tokens.valid_until_ms"
308	305	" FROM users"
309	306	" INNER JOIN access_tokens on users.name = access_tokens.user_id"

365	362	)
366	363	return True if res == UserTypes.SUPPORT else False
367	364
368		def get_users_by_id_case_insensitive(self, user_id):
	365	async def get_users_by_id_case_insensitive(self, user_id: str) -> Dict[str, str]:
369	366	"""Gets users that match user_id case insensitively.
370		Returns a mapping of user_id -> password_hash.
	367
	368	Returns:
	369	A mapping of user_id -> password_hash.
371	370	"""
372	371
373	372	def f(txn):

375	374	txn.execute(sql, (user_id,))
376	375	return dict(txn)
377	376
378		return self.db_pool.runInteraction("get_users_by_id_case_insensitive", f)
	377	return await self.db_pool.runInteraction("get_users_by_id_case_insensitive", f)
379	378
380	379	async def get_user_by_external_id(
381	380	self, auth_provider: str, external_id: str

409	408
410	409	return await self.db_pool.runInteraction("count_users", _count_users)
411	410
412		def count_daily_user_type(self):
	411	async def count_daily_user_type(self) -> Dict[str, int]:
413	412	"""
414	413	Counts 1) native non guest users
415	414	2) native guests users

438	437	results[row[0]] = row[1]
439	438	return results
440	439
441		return self.db_pool.runInteraction(
	440	return await self.db_pool.runInteraction(
442	441	"count_daily_user_type", _count_daily_user_type
443	442	)
444	443

530	529	"user_get_threepids",
531	530	)
532	531
533		def user_delete_threepid(self, user_id, medium, address):
534		return self.db_pool.simple_delete(
	532	async def user_delete_threepid(self, user_id, medium, address) -> None:
	533	await self.db_pool.simple_delete(
535	534	"user_threepids",
536	535	keyvalues={"user_id": user_id, "medium": medium, "address": address},
537	536	desc="user_delete_threepid",
538	537	)
539	538
540		def user_delete_threepids(self, user_id: str):
	539	async def user_delete_threepids(self, user_id: str) -> None:
541	540	"""Delete all threepid this user has bound
542	541
543	542	Args:
544	543	user_id: The user id to delete all threepids of
545	544
546	545	"""
547		return self.db_pool.simple_delete(
	546	await self.db_pool.simple_delete(
548	547	"user_threepids",
549	548	keyvalues={"user_id": user_id},
550	549	desc="user_delete_threepids",
551	550	)
552	551
553		def add_user_bound_threepid(self, user_id, medium, address, id_server):
	552	async def add_user_bound_threepid(
	553	self, user_id: str, medium: str, address: str, id_server: str
	554	):
554	555	"""The server proxied a bind request to the given identity server on
555	556	behalf of the given user. We need to remember this in case the user
556	557	asks us to unbind the threepid.
557	558
558	559	Args:
559		user_id (str)
560		medium (str)
561		address (str)
562		id_server (str)
563
564		Returns:
565		Deferred
	560	user_id
	561	medium
	562	address
	563	id_server
566	564	"""
567	565	# We need to use an upsert, in case they user had already bound the
568	566	# threepid
569		return self.db_pool.simple_upsert(
	567	await self.db_pool.simple_upsert(
570	568	table="user_threepid_id_server",
571	569	keyvalues={
572	570	"user_id": user_id,

579	577	desc="add_user_bound_threepid",
580	578	)
581	579
582		def user_get_bound_threepids(self, user_id):
	580	async def user_get_bound_threepids(self, user_id: str) -> List[Dict[str, Any]]:
583	581	"""Get the threepids that a user has bound to an identity server through the homeserver
584	582	The homeserver remembers where binds to an identity server occurred. Using this
585	583	method can retrieve those threepids.
586	584
587	585	Args:
588		user_id (str): The ID of the user to retrieve threepids for
589
590		Returns:
591		Deferred[list[dict]]: List of dictionaries containing the following:
	586	user_id: The ID of the user to retrieve threepids for
	587
	588	Returns:
	589	List of dictionaries containing the following keys:
592	590	medium (str): The medium of the threepid (e.g "email")
593	591	address (str): The address of the threepid (e.g "bob@example.com")
594	592	"""
595		return self.db_pool.simple_select_list(
	593	return await self.db_pool.simple_select_list(
596	594	table="user_threepid_id_server",
597	595	keyvalues={"user_id": user_id},
598	596	retcols=["medium", "address"],
599	597	desc="user_get_bound_threepids",
600	598	)
601	599
602		def remove_user_bound_threepid(self, user_id, medium, address, id_server):
	600	async def remove_user_bound_threepid(
	601	self, user_id: str, medium: str, address: str, id_server: str
	602	) -> None:
603	603	"""The server proxied an unbind request to the given identity server on
604	604	behalf of the given user, so we remove the mapping of threepid to
605	605	identity server.
606	606
607	607	Args:
608		user_id (str)
609		medium (str)
610		address (str)
611		id_server (str)
612
613		Returns:
614		Deferred
615		"""
616		return self.db_pool.simple_delete(
	608	user_id
	609	medium
	610	address
	611	id_server
	612	"""
	613	await self.db_pool.simple_delete(
617	614	table="user_threepid_id_server",
618	615	keyvalues={
619	616	"user_id": user_id,

624	621	desc="remove_user_bound_threepid",
625	622	)
626	623
627		def get_id_servers_user_bound(self, user_id, medium, address):
	624	async def get_id_servers_user_bound(
	625	self, user_id: str, medium: str, address: str
	626	) -> List[str]:
628	627	"""Get the list of identity servers that the server proxied bind
629	628	requests to for given user and threepid
630	629
631	630	Args:
632		user_id (str)
633		medium (str)
634		address (str)
635
636		Returns:
637		Deferred[list[str]]: Resolves to a list of identity servers
638		"""
639		return self.db_pool.simple_select_onecol(
	631	user_id: The user to query for identity servers.
	632	medium: The medium to query for identity servers.
	633	address: The address to query for identity servers.
	634
	635	Returns:
	636	A list of identity servers
	637	"""
	638	return await self.db_pool.simple_select_onecol(
640	639	table="user_threepid_id_server",
641	640	keyvalues={"user_id": user_id, "medium": medium, "address": address},
642	641	retcol="id_server",

664	663	# Convert the integer into a boolean.
665	664	return res == 1
666	665
667		def get_threepid_validation_session(
668		self, medium, client_secret, address=None, sid=None, validated=True
669		):
	666	async def get_threepid_validation_session(
	667	self,
	668	medium: Optional[str],
	669	client_secret: str,
	670	address: Optional[str] = None,
	671	sid: Optional[str] = None,
	672	validated: Optional[bool] = True,
	673	) -> Optional[Dict[str, Any]]:
670	674	"""Gets a session_id and last_send_attempt (if available) for a
671	675	combination of validation metadata
672	676
673	677	Args:
674		medium (str\|None): The medium of the 3PID
675		address (str\|None): The address of the 3PID
676		sid (str\|None): The ID of the validation session
677		client_secret (str): A unique string provided by the client to help identify this
	678	medium: The medium of the 3PID
	679	client_secret: A unique string provided by the client to help identify this
678	680	validation attempt
679		validated (bool\|None): Whether sessions should be filtered by
	681	address: The address of the 3PID
	682	sid: The ID of the validation session
	683	validated: Whether sessions should be filtered by
680	684	whether they have been validated already or not. None to
681	685	perform no filtering
682	686
683	687	Returns:
684		Deferred[dict\|None]: A dict containing the following:
	688	A dict containing the following:
685	689	* address - address of the 3pid
686	690	* medium - medium of the 3pid
687	691	* client_secret - a secret provided by the client for this validation session

727	731
728	732	return rows[0]
729	733
730		return self.db_pool.runInteraction(
	734	return await self.db_pool.runInteraction(
731	735	"get_threepid_validation_session", get_threepid_validation_session_txn
732	736	)
733	737
734		def delete_threepid_session(self, session_id):
	738	async def delete_threepid_session(self, session_id: str) -> None:
735	739	"""Removes a threepid validation session from the database. This can
736	740	be done after validation has been performed and whatever action was
737	741	waiting on it has been carried out
738	742
739	743	Args:
740		session_id (str): The ID of the session to delete
	744	session_id: The ID of the session to delete
741	745	"""
742	746
743	747	def delete_threepid_session_txn(txn):

752	756	keyvalues={"session_id": session_id},
753	757	)
754	758
755		return self.db_pool.runInteraction(
	759	await self.db_pool.runInteraction(
756	760	"delete_threepid_session", delete_threepid_session_txn
757	761	)
758	762

890	894	super(RegistrationStore, self).__init__(database, db_conn, hs)
891	895
892	896	self._account_validity = hs.config.account_validity
	897	self._ignore_unknown_session_error = hs.config.request_token_inhibit_3pid_errors
893	898
894	899	if self._account_validity.enabled:
895	900	self._clock.call_later(

941	946	desc="add_access_token_to_user",
942	947	)
943	948
944		def register_user(
	949	async def register_user(
945	950	self,
946		user_id,
947		password_hash=None,
948		was_guest=False,
949		make_guest=False,
950		appservice_id=None,
951		create_profile_with_displayname=None,
952		admin=False,
953		user_type=None,
954		):
	951	user_id: str,
	952	password_hash: Optional[str] = None,
	953	was_guest: bool = False,
	954	make_guest: bool = False,
	955	appservice_id: Optional[str] = None,
	956	create_profile_with_displayname: Optional[str] = None,
	957	admin: bool = False,
	958	user_type: Optional[str] = None,
	959	shadow_banned: bool = False,
	960	) -> None:
955	961	"""Attempts to register an account.
956	962
957	963	Args:
958		user_id (str): The desired user ID to register.
959		password_hash (str\|None): Optional. The password hash for this user.
960		was_guest (bool): Optional. Whether this is a guest account being
961		upgraded to a non-guest account.
962		make_guest (boolean): True if the the new user should be guest,
963		false to add a regular user account.
964		appservice_id (str): The ID of the appservice registering the user.
965		create_profile_with_displayname (unicode): Optionally create a profile for
	964	user_id: The desired user ID to register.
	965	password_hash: Optional. The password hash for this user.
	966	was_guest: Whether this is a guest account being upgraded to a
	967	non-guest account.
	968	make_guest: True if the the new user should be guest, false to add a
	969	regular user account.
	970	appservice_id: The ID of the appservice registering the user.
	971	create_profile_with_displayname: Optionally create a profile for
966	972	the user, setting their displayname to the given value
967		admin (boolean): is an admin user?
968		user_type (str\|None): type of user. One of the values from
969		api.constants.UserTypes, or None for a normal user.
	973	admin: is an admin user?
	974	user_type: type of user. One of the values from api.constants.UserTypes,
	975	or None for a normal user.
	976	shadow_banned: Whether the user is shadow-banned, i.e. they may be
	977	told their requests succeeded but we ignore them.
970	978
971	979	Raises:
972	980	StoreError if the user_id could not be registered.
973
974		Returns:
975		Deferred
976		"""
977		return self.db_pool.runInteraction(
	981	"""
	982	await self.db_pool.runInteraction(
978	983	"register_user",
979	984	self._register_user,
980	985	user_id,

985	990	create_profile_with_displayname,
986	991	admin,
987	992	user_type,
	993	shadow_banned,
988	994	)
989	995
990	996	def _register_user(

998	1004	create_profile_with_displayname,
999	1005	admin,
1000	1006	user_type,
	1007	shadow_banned,
1001	1008	):
1002	1009	user_id_obj = UserID.from_string(user_id)
1003	1010

1027	1034	"appservice_id": appservice_id,
1028	1035	"admin": 1 if admin else 0,
1029	1036	"user_type": user_type,
	1037	"shadow_banned": shadow_banned,
1030	1038	},
1031	1039	)
1032	1040	else:

1041	1049	"appservice_id": appservice_id,
1042	1050	"admin": 1 if admin else 0,
1043	1051	"user_type": user_type,
	1052	"shadow_banned": shadow_banned,
1044	1053	},
1045	1054	)
1046	1055

1074	1083
1075	1084	self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
1076	1085
1077		def record_user_external_id(
	1086	async def record_user_external_id(
1078	1087	self, auth_provider: str, external_id: str, user_id: str
1079		) -> Deferred:
	1088	) -> None:
1080	1089	"""Record a mapping from an external user id to a mxid
1081	1090
1082	1091	Args:

1084	1093	external_id: id on that system
1085	1094	user_id: complete mxid that it is mapped to
1086	1095	"""
1087		return self.db_pool.simple_insert(
	1096	await self.db_pool.simple_insert(
1088	1097	table="user_external_ids",
1089	1098	values={
1090	1099	"auth_provider": auth_provider,

1094	1103	desc="record_user_external_id",
1095	1104	)
1096	1105
1097		def user_set_password_hash(self, user_id, password_hash):
	1106	async def user_set_password_hash(self, user_id: str, password_hash: str) -> None:
1098	1107	"""
1099	1108	NB. This does not evict any cache because the one use for this
1100	1109	removes most of the entries subsequently anyway so it would be

1107	1116	)
1108	1117	self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
1109	1118
1110		return self.db_pool.runInteraction(
	1119	await self.db_pool.runInteraction(
1111	1120	"user_set_password_hash", user_set_password_hash_txn
1112	1121	)
1113	1122
1114		def user_set_consent_version(self, user_id, consent_version):
	1123	async def user_set_consent_version(
	1124	self, user_id: str, consent_version: str
	1125	) -> None:
1115	1126	"""Updates the user table to record privacy policy consent
1116	1127
1117	1128	Args:
1118		user_id (str): full mxid of the user to update
1119		consent_version (str): version of the policy the user has consented
1120		to
	1129	user_id: full mxid of the user to update
	1130	consent_version: version of the policy the user has consented to
1121	1131
1122	1132	Raises:
1123	1133	StoreError(404) if user not found

1132	1142	)
1133	1143	self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
1134	1144
1135		return self.db_pool.runInteraction("user_set_consent_version", f)
1136
1137		def user_set_consent_server_notice_sent(self, user_id, consent_version):
	1145	await self.db_pool.runInteraction("user_set_consent_version", f)
	1146
	1147	async def user_set_consent_server_notice_sent(
	1148	self, user_id: str, consent_version: str
	1149	) -> None:
1138	1150	"""Updates the user table to record that we have sent the user a server
1139	1151	notice about privacy policy consent
1140	1152
1141	1153	Args:
1142		user_id (str): full mxid of the user to update
1143		consent_version (str): version of the policy we have notified the
1144		user about
	1154	user_id: full mxid of the user to update
	1155	consent_version: version of the policy we have notified the user about
1145	1156
1146	1157	Raises:
1147	1158	StoreError(404) if user not found

1156	1167	)
1157	1168	self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
1158	1169
1159		return self.db_pool.runInteraction("user_set_consent_server_notice_sent", f)
1160
1161		def user_delete_access_tokens(self, user_id, except_token_id=None, device_id=None):
	1170	await self.db_pool.runInteraction("user_set_consent_server_notice_sent", f)
	1171
	1172	async def user_delete_access_tokens(
	1173	self,
	1174	user_id: str,
	1175	except_token_id: Optional[str] = None,
	1176	device_id: Optional[str] = None,
	1177	) -> List[Tuple[str, int, Optional[str]]]:
1162	1178	"""
1163	1179	Invalidate access tokens belonging to a user
1164	1180
1165	1181	Args:
1166		user_id (str): ID of user the tokens belong to
1167		except_token_id (str): list of access_tokens IDs which should
1168		not be deleted
1169		device_id (str\|None): ID of device the tokens are associated with.
	1182	user_id: ID of user the tokens belong to
	1183	except_token_id: access_tokens ID which should not be deleted
	1184	device_id: ID of device the tokens are associated with.
1170	1185	If None, tokens associated with any device (or no device) will
1171	1186	be deleted
1172	1187	Returns:
1173		defer.Deferred[list[str, int, str\|None, int]]: a list of
1174		(token, token id, device id) for each of the deleted tokens
	1188	A tuple of (token, token id, device id) for each of the deleted tokens
1175	1189	"""
1176	1190
1177	1191	def f(txn):

1202	1216
1203	1217	return tokens_and_devices
1204	1218
1205		return self.db_pool.runInteraction("user_delete_access_tokens", f)
1206
1207		def delete_access_token(self, access_token):
	1219	return await self.db_pool.runInteraction("user_delete_access_tokens", f)
	1220
	1221	async def delete_access_token(self, access_token: str) -> None:
1208	1222	def f(txn):
1209	1223	self.db_pool.simple_delete_one_txn(
1210	1224	txn, table="access_tokens", keyvalues={"token": access_token}

1214	1228	txn, self.get_user_by_access_token, (access_token,)
1215	1229	)
1216	1230
1217		return self.db_pool.runInteraction("delete_access_token", f)
	1231	await self.db_pool.runInteraction("delete_access_token", f)
1218	1232
1219	1233	@cached()
1220	1234	async def is_guest(self, user_id: str) -> bool:

1228	1242
1229	1243	return res if res else False
1230	1244
1231		def add_user_pending_deactivation(self, user_id):
	1245	async def add_user_pending_deactivation(self, user_id: str) -> None:
1232	1246	"""
1233	1247	Adds a user to the table of users who need to be parted from all the rooms they're
1234	1248	in
1235	1249	"""
1236		return self.db_pool.simple_insert(
	1250	await self.db_pool.simple_insert(
1237	1251	"users_pending_deactivation",
1238	1252	values={"user_id": user_id},
1239	1253	desc="add_user_pending_deactivation",
1240	1254	)
1241	1255
1242		def del_user_pending_deactivation(self, user_id):
	1256	async def del_user_pending_deactivation(self, user_id: str) -> None:
1243	1257	"""
1244	1258	Removes the given user to the table of users who need to be parted from all the
1245	1259	rooms they're in, effectively marking that user as fully deactivated.
1246	1260	"""
1247	1261	# XXX: This should be simple_delete_one but we failed to put a unique index on
1248	1262	# the table, so somehow duplicate entries have ended up in it.
1249		return self.db_pool.simple_delete(
	1263	await self.db_pool.simple_delete(
1250	1264	"users_pending_deactivation",
1251	1265	keyvalues={"user_id": user_id},
1252	1266	desc="del_user_pending_deactivation",
1253	1267	)
1254	1268
1255		def get_user_pending_deactivation(self):
	1269	async def get_user_pending_deactivation(self) -> Optional[str]:
1256	1270	"""
1257	1271	Gets one user from the table of users waiting to be parted from all the rooms
1258	1272	they're in.
1259	1273	"""
1260		return self.db_pool.simple_select_one_onecol(
	1274	return await self.db_pool.simple_select_one_onecol(
1261	1275	"users_pending_deactivation",
1262	1276	keyvalues={},
1263	1277	retcol="user_id",

1265	1279	desc="get_users_pending_deactivation",
1266	1280	)
1267	1281
1268		def validate_threepid_session(self, session_id, client_secret, token, current_ts):
	1282	async def validate_threepid_session(
	1283	self, session_id: str, client_secret: str, token: str, current_ts: int
	1284	) -> Optional[str]:
1269	1285	"""Attempt to validate a threepid session using a token
1270	1286
1271	1287	Args:
1272		session_id (str): The id of a validation session
1273		client_secret (str): A unique string provided by the client to
1274		help identify this validation attempt
1275		token (str): A validation token
1276		current_ts (int): The current unix time in milliseconds. Used for
1277		checking token expiry status
	1288	session_id: The id of a validation session
	1289	client_secret: A unique string provided by the client to help identify
	1290	this validation attempt
	1291	token: A validation token
	1292	current_ts: The current unix time in milliseconds. Used for checking
	1293	token expiry status
1278	1294
1279	1295	Raises:
1280	1296	ThreepidValidationError: if a matching validation token was not found or has
1281	1297	expired
1282	1298
1283	1299	Returns:
1284		deferred str\|None: A str representing a link to redirect the user
1285		to if there is one.
	1300	A str representing a link to redirect the user to if there is one.
1286	1301	"""
1287	1302
1288	1303	# Insert everything into a transaction in order to run atomically

1296	1311	)
1297	1312
1298	1313	if not row:
1299		raise ThreepidValidationError(400, "Unknown session_id")
	1314	if self._ignore_unknown_session_error:
	1315	# If we need to inhibit the error caused by an incorrect session ID,
	1316	# use None as placeholder values for the client secret and the
	1317	# validation timestamp.
	1318	# It shouldn't be an issue because they're both only checked after
	1319	# the token check, which should fail. And if it doesn't for some
	1320	# reason, the next check is on the client secret, which is NOT NULL,
	1321	# so we don't have to worry about the client secret matching by
	1322	# accident.
	1323	row = {"client_secret": None, "validated_at": None}
	1324	else:
	1325	raise ThreepidValidationError(400, "Unknown session_id")
	1326
1300	1327	retrieved_client_secret = row["client_secret"]
1301	1328	validated_at = row["validated_at"]
1302
1303		if retrieved_client_secret != client_secret:
1304		raise ThreepidValidationError(
1305		400, "This client_secret does not match the provided session_id"
1306		)
1307	1329
1308	1330	row = self.db_pool.simple_select_one_txn(
1309	1331	txn,

1320	1342	expires = row["expires"]
1321	1343	next_link = row["next_link"]
1322	1344
	1345	if retrieved_client_secret != client_secret:
	1346	raise ThreepidValidationError(
	1347	400, "This client_secret does not match the provided session_id"
	1348	)
	1349
1323	1350	# If the session is already validated, no need to revalidate
1324	1351	if validated_at:
1325	1352	return next_link

1340	1367	return next_link
1341	1368
1342	1369	# Return next_link if it exists
1343		return self.db_pool.runInteraction(
	1370	return await self.db_pool.runInteraction(
1344	1371	"validate_threepid_session_txn", validate_threepid_session_txn
1345	1372	)
1346	1373
1347		def upsert_threepid_validation_session(
	1374	async def start_or_continue_validation_session(
1348	1375	self,
1349		medium,
1350		address,
1351		client_secret,
1352		send_attempt,
1353		session_id,
1354		validated_at=None,
1355		):
1356		"""Upsert a threepid validation session
1357		Args:
1358		medium (str): The medium of the 3PID
1359		address (str): The address of the 3PID
1360		client_secret (str): A unique string provided by the client to
1361		help identify this validation attempt
1362		send_attempt (int): The latest send_attempt on this session
1363		session_id (str): The id of this validation session
1364		validated_at (int\|None): The unix timestamp in milliseconds of
1365		when the session was marked as valid
1366		"""
1367		insertion_values = {
1368		"medium": medium,
1369		"address": address,
1370		"client_secret": client_secret,
1371		}
1372
1373		if validated_at:
1374		insertion_values["validated_at"] = validated_at
1375
1376		return self.db_pool.simple_upsert(
1377		table="threepid_validation_session",
1378		keyvalues={"session_id": session_id},
1379		values={"last_send_attempt": send_attempt},
1380		insertion_values=insertion_values,
1381		desc="upsert_threepid_validation_session",
1382		)
1383
1384		def start_or_continue_validation_session(
1385		self,
1386		medium,
1387		address,
1388		session_id,
1389		client_secret,
1390		send_attempt,
1391		next_link,
1392		token,
1393		token_expires,
1394		):
	1376	medium: str,
	1377	address: str,
	1378	session_id: str,
	1379	client_secret: str,
	1380	send_attempt: int,
	1381	next_link: Optional[str],
	1382	token: str,
	1383	token_expires: int,
	1384	) -> None:
1395	1385	"""Creates a new threepid validation session if it does not already
1396	1386	exist and associates a new validation token with it
1397	1387
1398	1388	Args:
1399		medium (str): The medium of the 3PID
1400		address (str): The address of the 3PID
1401		session_id (str): The id of this validation session
1402		client_secret (str): A unique string provided by the client to
1403		help identify this validation attempt
1404		send_attempt (int): The latest send_attempt on this session
1405		next_link (str\|None): The link to redirect the user to upon
1406		successful validation
1407		token (str): The validation token
1408		token_expires (int): The timestamp for which after the token
1409		will no longer be valid
	1389	medium: The medium of the 3PID
	1390	address: The address of the 3PID
	1391	session_id: The id of this validation session
	1392	client_secret: A unique string provided by the client to help
	1393	identify this validation attempt
	1394	send_attempt: The latest send_attempt on this session
	1395	next_link: The link to redirect the user to upon successful validation
	1396	token: The validation token
	1397	token_expires: The timestamp for which after the token will no
	1398	longer be valid
1410	1399	"""
1411	1400
1412	1401	def start_or_continue_validation_session_txn(txn):

1435	1424	},
1436	1425	)
1437	1426
1438		return self.db_pool.runInteraction(
	1427	await self.db_pool.runInteraction(
1439	1428	"start_or_continue_validation_session",
1440	1429	start_or_continue_validation_session_txn,
1441	1430	)
1442	1431
1443		def cull_expired_threepid_validation_tokens(self):
	1432	async def cull_expired_threepid_validation_tokens(self) -> None:
1444	1433	"""Remove threepid validation tokens with expiry dates that have passed"""
1445	1434
1446	1435	def cull_expired_threepid_validation_tokens_txn(txn, ts):

1448	1437	DELETE FROM threepid_validation_token WHERE
1449	1438	expires < ?
1450	1439	"""
1451		return txn.execute(sql, (ts,))
1452
1453		return self.db_pool.runInteraction(
	1440	txn.execute(sql, (ts,))
	1441
	1442	await self.db_pool.runInteraction(
1454	1443	"cull_expired_threepid_validation_tokens",
1455	1444	cull_expired_threepid_validation_tokens_txn,
1456	1445	self.clock.time_msec(),

+3

-2

synapse/storage/databases/main/rejections.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
	16	from typing import Optional
16	17
17	18	from synapse.storage._base import SQLBaseStore
18	19

20	21
21	22
22	23	class RejectionsStore(SQLBaseStore):
23		def get_rejection_reason(self, event_id):
24		return self.db_pool.simple_select_one_onecol(
	24	async def get_rejection_reason(self, event_id: str) -> Optional[str]:
	25	return await self.db_pool.simple_select_one_onecol(
25	26	table="rejections",
26	27	retcol="reason",
27	28	keyvalues={"event_id": event_id},

+48

-55

synapse/storage/databases/main/relations.py less more

33	33
34	34	class RelationsWorkerStore(SQLBaseStore):
35	35	@cached(tree=True)
36		def get_relations_for_event(
	36	async def get_relations_for_event(
37	37	self,
38		event_id,
39		relation_type=None,
40		event_type=None,
41		aggregation_key=None,
42		limit=5,
43		direction="b",
44		from_token=None,
45		to_token=None,
46		):
	38	event_id: str,
	39	relation_type: Optional[str] = None,
	40	event_type: Optional[str] = None,
	41	aggregation_key: Optional[str] = None,
	42	limit: int = 5,
	43	direction: str = "b",
	44	from_token: Optional[RelationPaginationToken] = None,
	45	to_token: Optional[RelationPaginationToken] = None,
	46	) -> PaginationChunk:
47	47	"""Get a list of relations for an event, ordered by topological ordering.
48	48
49	49	Args:
50		event_id (str): Fetch events that relate to this event ID.
51		relation_type (str\|None): Only fetch events with this relation
52		type, if given.
53		event_type (str\|None): Only fetch events with this event type, if
54		given.
55		aggregation_key (str\|None): Only fetch events with this aggregation
56		key, if given.
57		limit (int): Only fetch the most recent `limit` events.
58		direction (str): Whether to fetch the most recent first (`"b"`) or
59		the oldest first (`"f"`).
60		from_token (RelationPaginationToken\|None): Fetch rows from the given
61		token, or from the start if None.
62		to_token (RelationPaginationToken\|None): Fetch rows up to the given
63		token, or up to the end if None.
	50	event_id: Fetch events that relate to this event ID.
	51	relation_type: Only fetch events with this relation type, if given.
	52	event_type: Only fetch events with this event type, if given.
	53	aggregation_key: Only fetch events with this aggregation key, if given.
	54	limit: Only fetch the most recent `limit` events.
	55	direction: Whether to fetch the most recent first (`"b"`) or the
	56	oldest first (`"f"`).
	57	from_token: Fetch rows from the given token, or from the start if None.
	58	to_token: Fetch rows up to the given token, or up to the end if None.
64	59
65	60	Returns:
66		Deferred[PaginationChunk]: List of event IDs that match relations
67		requested. The rows are of the form `{"event_id": "..."}`.
	61	List of event IDs that match relations requested. The rows are of
	62	the form `{"event_id": "..."}`.
68	63	"""
69	64
70	65	where_clause = ["relates_to_id = ?"]

130	125	chunk=list(events[:limit]), next_batch=next_batch, prev_batch=from_token
131	126	)
132	127
133		return self.db_pool.runInteraction(
	128	return await self.db_pool.runInteraction(
134	129	"get_recent_references_for_event", _get_recent_references_for_event_txn
135	130	)
136	131
137	132	@cached(tree=True)
138		def get_aggregation_groups_for_event(
	133	async def get_aggregation_groups_for_event(
139	134	self,
140		event_id,
141		event_type=None,
142		limit=5,
143		direction="b",
144		from_token=None,
145		to_token=None,
146		):
	135	event_id: str,
	136	event_type: Optional[str] = None,
	137	limit: int = 5,
	138	direction: str = "b",
	139	from_token: Optional[AggregationPaginationToken] = None,
	140	to_token: Optional[AggregationPaginationToken] = None,
	141	) -> PaginationChunk:
147	142	"""Get a list of annotations on the event, grouped by event type and
148	143	aggregation key, sorted by count.
149	144

151	146	on an event.
152	147
153	148	Args:
154		event_id (str): Fetch events that relate to this event ID.
155		event_type (str\|None): Only fetch events with this event type, if
156		given.
157		limit (int): Only fetch the `limit` groups.
158		direction (str): Whether to fetch the highest count first (`"b"`) or
	149	event_id: Fetch events that relate to this event ID.
	150	event_type: Only fetch events with this event type, if given.
	151	limit: Only fetch the `limit` groups.
	152	direction: Whether to fetch the highest count first (`"b"`) or
159	153	the lowest count first (`"f"`).
160		from_token (AggregationPaginationToken\|None): Fetch rows from the
161		given token, or from the start if None.
162		to_token (AggregationPaginationToken\|None): Fetch rows up to the
163		given token, or up to the end if None.
164
	154	from_token: Fetch rows from the given token, or from the start if None.
	155	to_token: Fetch rows up to the given token, or up to the end if None.
165	156
166	157	Returns:
167		Deferred[PaginationChunk]: List of groups of annotations that
168		match. Each row is a dict with `type`, `key` and `count` fields.
	158	List of groups of annotations that match. Each row is a dict with
	159	`type`, `key` and `count` fields.
169	160	"""
170	161
171	162	where_clause = ["relates_to_id = ?", "relation_type = ?"]

224	215	chunk=list(events[:limit]), next_batch=next_batch, prev_batch=from_token
225	216	)
226	217
227		return self.db_pool.runInteraction(
	218	return await self.db_pool.runInteraction(
228	219	"get_aggregation_groups_for_event", _get_aggregation_groups_for_event_txn
229	220	)
230	221

278	269
279	270	return await self.get_event(edit_id, allow_none=True)
280	271
281		def has_user_annotated_event(self, parent_id, event_type, aggregation_key, sender):
	272	async def has_user_annotated_event(
	273	self, parent_id: str, event_type: str, aggregation_key: str, sender: str
	274	) -> bool:
282	275	"""Check if a user has already annotated an event with the same key
283	276	(e.g. already liked an event).
284	277
285	278	Args:
286		parent_id (str): The event being annotated
287		event_type (str): The event type of the annotation
288		aggregation_key (str): The aggregation key of the annotation
289		sender (str): The sender of the annotation
	279	parent_id: The event being annotated
	280	event_type: The event type of the annotation
	281	aggregation_key: The aggregation key of the annotation
	282	sender: The sender of the annotation
290	283
291	284	Returns:
292		Deferred[bool]
	285	True if the event is already annotated.
293	286	"""
294	287
295	288	sql = """

318	311
319	312	return bool(txn.fetchone())
320	313
321		return self.db_pool.runInteraction(
	314	return await self.db_pool.runInteraction(
322	315	"get_if_user_has_annotated_event", _get_if_user_has_annotated_event
323	316	)
324	317

+52

-44

synapse/storage/databases/main/room.py less more

20	20	from enum import Enum
21	21	from typing import Any, Dict, List, Optional, Tuple
22	22
23		from canonicaljson import json
24
25	23	from synapse.api.constants import EventTypes
26	24	from synapse.api.errors import StoreError
27	25	from synapse.api.room_versions import RoomVersion, RoomVersions
28	26	from synapse.storage._base import SQLBaseStore, db_to_json
29	27	from synapse.storage.database import DatabasePool, LoggingTransaction
30	28	from synapse.storage.databases.main.search import SearchStore
31		from synapse.types import ThirdPartyInstanceID
	29	from synapse.types import JsonDict, ThirdPartyInstanceID
	30	from synapse.util import json_encoder
32	31	from synapse.util.caches.descriptors import cached
33	32
34	33	logger = logging.getLogger(__name__)
35	34
36
37		OpsLevel = collections.namedtuple(
38		"OpsLevel", ("ban_level", "kick_level", "redact_level")
39		)
40	35
41	36	RatelimitOverride = collections.namedtuple(
42	37	"RatelimitOverride", ("messages_per_second", "burst_count")

77	72
78	73	self.config = hs.config
79	74
80		def get_room(self, room_id):
	75	async def get_room(self, room_id: str) -> dict:
81	76	"""Retrieve a room.
82	77
83	78	Args:
84		room_id (str): The ID of the room to retrieve.
	79	room_id: The ID of the room to retrieve.
85	80	Returns:
86	81	A dict containing the room information, or None if the room is unknown.
87	82	"""
88		return self.db_pool.simple_select_one(
	83	return await self.db_pool.simple_select_one(
89	84	table="rooms",
90	85	keyvalues={"room_id": room_id},
91	86	retcols=("room_id", "is_public", "creator"),

93	88	allow_none=True,
94	89	)
95	90
96		def get_room_with_stats(self, room_id: str):
	91	async def get_room_with_stats(self, room_id: str) -> Optional[Dict[str, Any]]:
97	92	"""Retrieve room with statistics.
98	93
99	94	Args:

125	120	res["public"] = bool(res["public"])
126	121	return res
127	122
128		return self.db_pool.runInteraction(
	123	return await self.db_pool.runInteraction(
129	124	"get_room_with_stats", get_room_with_stats_txn, room_id
130	125	)
131	126
132		def get_public_room_ids(self):
133		return self.db_pool.simple_select_onecol(
	127	async def get_public_room_ids(self) -> List[str]:
	128	return await self.db_pool.simple_select_onecol(
134	129	table="rooms",
135	130	keyvalues={"is_public": True},
136	131	retcol="room_id",
137	132	desc="get_public_room_ids",
138	133	)
139	134
140		def count_public_rooms(self, network_tuple, ignore_non_federatable):
	135	async def count_public_rooms(
	136	self,
	137	network_tuple: Optional[ThirdPartyInstanceID],
	138	ignore_non_federatable: bool,
	139	) -> int:
141	140	"""Counts the number of public rooms as tracked in the room_stats_current
142	141	and room_stats_state table.
143	142
144	143	Args:
145		network_tuple (ThirdPartyInstanceID\|None)
146		ignore_non_federatable (bool): If true filters out non-federatable rooms
	144	network_tuple
	145	ignore_non_federatable: If true filters out non-federatable rooms
147	146	"""
148	147
149	148	def _count_public_rooms_txn(txn):

187	186	txn.execute(sql, query_args)
188	187	return txn.fetchone()[0]
189	188
190		return self.db_pool.runInteraction(
	189	return await self.db_pool.runInteraction(
191	190	"count_public_rooms", _count_public_rooms_txn
192	191	)
193	192

334	333	return ret_val
335	334
336	335	@cached(max_entries=10000)
337		def is_room_blocked(self, room_id):
338		return self.db_pool.simple_select_one_onecol(
	336	async def is_room_blocked(self, room_id: str) -> Optional[bool]:
	337	return await self.db_pool.simple_select_one_onecol(
339	338	table="blocked_rooms",
340	339	keyvalues={"room_id": room_id},
341	340	retcol="1",

590	589
591	590	return row
592	591
593		def get_media_mxcs_in_room(self, room_id):
	592	async def get_media_mxcs_in_room(self, room_id: str) -> Tuple[List[str], List[str]]:
594	593	"""Retrieves all the local and remote media MXC URIs in a given room
595	594
596	595	Args:
597		room_id (str)
	596	room_id
598	597
599	598	Returns:
600		The local and remote media as a lists of tuples where the key is
601		the hostname and the value is the media ID.
	599	The local and remote media as a lists of the media IDs.
602	600	"""
603	601
604	602	def _get_media_mxcs_in_room_txn(txn):

614	612
615	613	return local_media_mxcs, remote_media_mxcs
616	614
617		return self.db_pool.runInteraction(
	615	return await self.db_pool.runInteraction(
618	616	"get_media_ids_in_room", _get_media_mxcs_in_room_txn
619	617	)
620	618
621		def quarantine_media_ids_in_room(self, room_id, quarantined_by):
	619	async def quarantine_media_ids_in_room(
	620	self, room_id: str, quarantined_by: str
	621	) -> int:
622	622	"""For a room loops through all events with media and quarantines
623	623	the associated media
624	624	"""

631	631	txn, local_mxcs, remote_mxcs, quarantined_by
632	632	)
633	633
634		return self.db_pool.runInteraction(
	634	return await self.db_pool.runInteraction(
635	635	"quarantine_media_in_room", _quarantine_media_in_room_txn
636	636	)
637	637

694	694
695	695	return local_media_mxcs, remote_media_mxcs
696	696
697		def quarantine_media_by_id(
	697	async def quarantine_media_by_id(
698	698	self, server_name: str, media_id: str, quarantined_by: str,
699		):
	699	) -> int:
700	700	"""quarantines a single local or remote media id
701	701
702	702	Args:

715	715	txn, local_mxcs, remote_mxcs, quarantined_by
716	716	)
717	717
718		return self.db_pool.runInteraction(
	718	return await self.db_pool.runInteraction(
719	719	"quarantine_media_by_user", _quarantine_media_by_id_txn
720	720	)
721	721
722		def quarantine_media_ids_by_user(self, user_id: str, quarantined_by: str):
	722	async def quarantine_media_ids_by_user(
	723	self, user_id: str, quarantined_by: str
	724	) -> int:
723	725	"""quarantines all local media associated with a single user
724	726
725	727	Args:

731	733	local_media_ids = self._get_media_ids_by_user_txn(txn, user_id)
732	734	return self._quarantine_media_txn(txn, local_media_ids, [], quarantined_by)
733	735
734		return self.db_pool.runInteraction(
	736	return await self.db_pool.runInteraction(
735	737	"quarantine_media_by_user", _quarantine_media_by_user_txn
736	738	)
737	739

1133	1135	},
1134	1136	)
1135	1137
1136		with self._public_room_id_gen.get_next() as next_id:
	1138	with await self._public_room_id_gen.get_next() as next_id:
1137	1139	await self.db_pool.runInteraction(
1138	1140	"store_room_txn", store_room_txn, next_id
1139	1141	)

1200	1202	},
1201	1203	)
1202	1204
1203		with self._public_room_id_gen.get_next() as next_id:
	1205	with await self._public_room_id_gen.get_next() as next_id:
1204	1206	await self.db_pool.runInteraction(
1205	1207	"set_room_is_public", set_room_is_public_txn, next_id
1206	1208	)

1280	1282	},
1281	1283	)
1282	1284
1283		with self._public_room_id_gen.get_next() as next_id:
	1285	with await self._public_room_id_gen.get_next() as next_id:
1284	1286	await self.db_pool.runInteraction(
1285	1287	"set_room_is_public_appservice",
1286	1288	set_room_is_public_appservice_txn,

1288	1290	)
1289	1291	self.hs.get_notifier().on_new_replication_data()
1290	1292
1291		def get_room_count(self):
1292		"""Retrieve a list of all rooms
	1293	async def get_room_count(self) -> int:
	1294	"""Retrieve the total number of rooms.
1293	1295	"""
1294	1296
1295	1297	def f(txn):

1298	1300	row = txn.fetchone()
1299	1301	return row[0] or 0
1300	1302
1301		return self.db_pool.runInteraction("get_rooms", f)
1302
1303		def add_event_report(
1304		self, room_id, event_id, user_id, reason, content, received_ts
1305		):
	1303	return await self.db_pool.runInteraction("get_rooms", f)
	1304
	1305	async def add_event_report(
	1306	self,
	1307	room_id: str,
	1308	event_id: str,
	1309	user_id: str,
	1310	reason: str,
	1311	content: JsonDict,
	1312	received_ts: int,
	1313	) -> None:
1306	1314	next_id = self._event_reports_id_gen.get_next()
1307		return self.db_pool.simple_insert(
	1315	await self.db_pool.simple_insert(
1308	1316	table="event_reports",
1309	1317	values={
1310	1318	"id": next_id,

1313	1321	"event_id": event_id,
1314	1322	"user_id": user_id,
1315	1323	"reason": reason,
1316		"content": json.dumps(content),
	1324	"content": json_encoder.encode(content),
1317	1325	},
1318	1326	desc="add_event_report",
1319	1327	)

+44

-47

synapse/storage/databases/main/roommember.py less more

14	14	# limitations under the License.
15	15
16	16	import logging
17		from typing import TYPE_CHECKING, Awaitable, Iterable, List, Optional, Set
18
19		from twisted.internet import defer
	17	from typing import TYPE_CHECKING, Dict, FrozenSet, Iterable, List, Optional, Set
20	18
21	19	from synapse.api.constants import EventTypes, Membership
22	20	from synapse.events import EventBase

91	89	lambda: self._known_servers_count,
92	90	)
93	91
94		@defer.inlineCallbacks
95		def _count_known_servers(self):
	92	async def _count_known_servers(self):
96	93	"""
97	94	Count the servers that this server knows about.
98	95

120	117	txn.execute(query)
121	118	return list(txn)[0][0]
122	119
123		count = yield self.db_pool.runInteraction("get_known_servers", _transact)
	120	count = await self.db_pool.runInteraction("get_known_servers", _transact)
124	121
125	122	# We always know about ourselves, even if we have nothing in
126	123	# room_memberships (for example, the server is new).

154	151	)
155	152
156	153	@cached(max_entries=100000, iterable=True)
157		def get_users_in_room(self, room_id: str):
158		return self.db_pool.runInteraction(
	154	async def get_users_in_room(self, room_id: str) -> List[str]:
	155	return await self.db_pool.runInteraction(
159	156	"get_users_in_room", self.get_users_in_room_txn, room_id
160	157	)
161	158

182	179	return [r[0] for r in txn]
183	180
184	181	@cached(max_entries=100000)
185		def get_room_summary(self, room_id: str):
	182	async def get_room_summary(self, room_id: str) -> Dict[str, MemberSummary]:
186	183	""" Get the details of a room roughly suitable for use by the room
187	184	summary extension to /sync. Useful when lazy loading room members.
188	185	Args:
189	186	room_id: The room ID to query
190	187	Returns:
191		Deferred[dict[str, MemberSummary]:
192		dict of membership states, pointing to a MemberSummary named tuple.
	188	dict of membership states, pointing to a MemberSummary named tuple.
193	189	"""
194	190
195	191	def _get_room_summary_txn(txn):

263	259
264	260	return res
265	261
266		return self.db_pool.runInteraction("get_room_summary", _get_room_summary_txn)
	262	return await self.db_pool.runInteraction(
	263	"get_room_summary", _get_room_summary_txn
	264	)
267	265
268	266	@cached()
269		def get_invited_rooms_for_local_user(self, user_id: str) -> Awaitable[RoomsForUser]:
	267	async def get_invited_rooms_for_local_user(self, user_id: str) -> RoomsForUser:
270	268	"""Get all the rooms the local user is invited to.
271	269
272	270	Args:
273	271	user_id: The user ID.
274	272
275	273	Returns:
276		A awaitable list of RoomsForUser.
277		"""
278
279		return self.get_rooms_for_local_user_where_membership_is(
	274	A list of RoomsForUser.
	275	"""
	276
	277	return await self.get_rooms_for_local_user_where_membership_is(
280	278	user_id, [Membership.INVITE]
281	279	)
282	280

299	297	return None
300	298
301	299	async def get_rooms_for_local_user_where_membership_is(
302		self, user_id: str, membership_list: List[str]
303		) -> Optional[List[RoomsForUser]]:
	300	self, user_id: str, membership_list: Collection[str]
	301	) -> List[RoomsForUser]:
304	302	"""Get all the rooms for this local user where the membership for this user
305	303	matches one in the membership list.
306	304

315	313	The RoomsForUser that the user matches the membership types.
316	314	"""
317	315	if not membership_list:
318		return None
	316	return []
319	317
320	318	rooms = await self.db_pool.runInteraction(
321	319	"get_rooms_for_local_user_where_membership_is",

359	357	return results
360	358
361	359	@cached(max_entries=500000, iterable=True)
362		def get_rooms_for_user_with_stream_ordering(self, user_id: str):
	360	async def get_rooms_for_user_with_stream_ordering(
	361	self, user_id: str
	362	) -> FrozenSet[GetRoomsForUserWithStreamOrdering]:
363	363	"""Returns a set of room_ids the user is currently joined to.
364	364
365	365	If a remote user only returns rooms this server is currently

369	369	user_id
370	370
371	371	Returns:
372		Deferred[frozenset[GetRoomsForUserWithStreamOrdering]]: Returns
373		the rooms the user is in currently, along with the stream ordering
374		of the most recent join for that user and room.
375		"""
376		return self.db_pool.runInteraction(
	372	Returns the rooms the user is in currently, along with the stream
	373	ordering of the most recent join for that user and room.
	374	"""
	375	return await self.db_pool.runInteraction(
377	376	"get_rooms_for_user_with_stream_ordering",
378	377	self._get_rooms_for_user_with_stream_ordering_txn,
379	378	user_id,
380	379	)
381	380
382		def _get_rooms_for_user_with_stream_ordering_txn(self, txn, user_id: str):
	381	def _get_rooms_for_user_with_stream_ordering_txn(
	382	self, txn, user_id: str
	383	) -> FrozenSet[GetRoomsForUserWithStreamOrdering]:
383	384	# We use `current_state_events` here and not `local_current_membership`
384	385	# as a) this gets called with remote users and b) this only gets called
385	386	# for rooms the server is participating in.

406	407	"""
407	408
408	409	txn.execute(sql, (user_id, Membership.JOIN))
409		results = frozenset(GetRoomsForUserWithStreamOrdering(*row) for row in txn)
410
411		return results
	410	return frozenset(GetRoomsForUserWithStreamOrdering(*row) for row in txn)
412	411
413	412	async def get_users_server_still_shares_room_with(
414	413	self, user_ids: Collection[str]

588	587	raise NotImplementedError()
589	588
590	589	@cachedList(
591		cached_method_name="_get_joined_profile_from_event_id",
592		list_name="event_ids",
593		inlineCallbacks=True,
	590	cached_method_name="_get_joined_profile_from_event_id", list_name="event_ids",
594	591	)
595		def _get_joined_profiles_from_event_ids(self, event_ids: Iterable[str]):
	592	async def _get_joined_profiles_from_event_ids(self, event_ids: Iterable[str]):
596	593	"""For given set of member event_ids check if they point to a join
597	594	event and if so return the associated user and profile info.
598	595

600	597	event_ids: The member event IDs to lookup
601	598
602	599	Returns:
603		Deferred[dict[str, Tuple[str, ProfileInfo]\|None]]: Map from event ID
	600	dict[str, Tuple[str, ProfileInfo]\|None]: Map from event ID
604	601	to `user_id` and ProfileInfo (or None if not join event).
605	602	"""
606	603
607		rows = yield self.db_pool.simple_select_many_batch(
	604	rows = await self.db_pool.simple_select_many_batch(
608	605	table="room_memberships",
609	606	column="event_id",
610	607	iterable=event_ids,

715	712	return count == 0
716	713
717	714	@cached()
718		def get_forgotten_rooms_for_user(self, user_id: str):
	715	async def get_forgotten_rooms_for_user(self, user_id: str) -> Set[str]:
719	716	"""Gets all rooms the user has forgotten.
720	717
721	718	Args:
722		user_id
	719	user_id: The user ID to query the rooms of.
723	720
724	721	Returns:
725		Deferred[set[str]]
	722	The forgotten rooms.
726	723	"""
727	724
728	725	def _get_forgotten_rooms_for_user_txn(txn):

748	745	txn.execute(sql, (user_id,))
749	746	return {row[0] for row in txn if row[1] == 0}
750	747
751		return self.db_pool.runInteraction(
	748	return await self.db_pool.runInteraction(
752	749	"get_forgotten_rooms_for_user", _get_forgotten_rooms_for_user_txn
753	750	)
754	751

771	768
772	769	return set(room_ids)
773	770
774		def get_membership_from_event_ids(
	771	async def get_membership_from_event_ids(
775	772	self, member_event_ids: Iterable[str]
776	773	) -> List[dict]:
777	774	"""Get user_id and membership of a set of event IDs.
778	775	"""
779	776
780		return self.db_pool.simple_select_many_batch(
	777	return await self.db_pool.simple_select_many_batch(
781	778	table="room_memberships",
782	779	column="event_id",
783	780	iterable=member_event_ids,

977	974	def __init__(self, database: DatabasePool, db_conn, hs):
978	975	super(RoomMemberStore, self).__init__(database, db_conn, hs)
979	976
980		def forget(self, user_id: str, room_id: str):
	977	async def forget(self, user_id: str, room_id: str) -> None:
981	978	"""Indicate that user_id wishes to discard history for room_id."""
982	979
983	980	def f(txn):

998	995	txn, self.get_forgotten_rooms_for_user, (user_id,)
999	996	)
1000	997
1001		return self.db_pool.runInteraction("forget_membership", f)
1002
1003
1004		class _JoinedHostsCache(object):
	998	await self.db_pool.runInteraction("forget_membership", f)
	999
	1000
	1001	class _JoinedHostsCache:
1005	1002	"""Cache for joined hosts in a room that is optimised to handle updates
1006	1003	via state deltas.
1007	1004	"""

+25

-0

synapse/storage/databases/main/schema/delta/58/07persist_ui_auth_ips.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- A table of the IP address and user-agent used to complete each step of a
	16	-- user-interactive authentication session.
	17	CREATE TABLE IF NOT EXISTS ui_auth_sessions_ips(
	18	session_id TEXT NOT NULL,
	19	ip TEXT NOT NULL,
	20	user_agent TEXT NOT NULL,
	21	UNIQUE (session_id, ip, user_agent),
	22	FOREIGN KEY (session_id)
	23	REFERENCES ui_auth_sessions (session_id)
	24	);

+18

-0

synapse/storage/databases/main/schema/delta/58/09shadow_ban.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- A shadow-banned user may be told that their requests succeeded when they were
	16	-- actually ignored.
	17	ALTER TABLE users ADD COLUMN shadow_banned BOOLEAN;

+17

-0

synapse/storage/databases/main/schema/delta/58/13remove_presence_allow_inbound.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- This table is no longer used.
	16	DROP TABLE IF EXISTS presence_allow_inbound;

+26

-0

synapse/storage/databases/main/schema/delta/58/15unread_count.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- We're hijacking the push actions to store unread messages and unread counts (specified
	16	-- in MSC2654) because doing otherwise would result in either performance issues or
	17	-- reimplementing a consequent bit of the push actions.
	18
	19	-- Add columns to event_push_actions and event_push_actions_staging to track unread
	20	-- messages and calculate unread counts.
	21	ALTER TABLE event_push_actions_staging ADD COLUMN unread SMALLINT;
	22	ALTER TABLE event_push_actions ADD COLUMN unread SMALLINT;
	23
	24	-- Add column to event_push_summary
	25	ALTER TABLE event_push_summary ADD COLUMN unread_count BIGINT;⏎

+9

-6

synapse/storage/databases/main/search.py less more

15	15	import logging
16	16	import re
17	17	from collections import namedtuple
18		from typing import List, Optional
	18	from typing import List, Optional, Set
19	19
20	20	from synapse.api.errors import SynapseError
	21	from synapse.events import EventBase
21	22	from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
22	23	from synapse.storage.database import DatabasePool
23	24	from synapse.storage.databases.main.events_worker import EventRedactBehaviour

619	620	"count": count,
620	621	}
621	622
622		def _find_highlights_in_postgres(self, search_query, events):
	623	async def _find_highlights_in_postgres(
	624	self, search_query: str, events: List[EventBase]
	625	) -> Set[str]:
623	626	"""Given a list of events and a search term, return a list of words
624	627	that match from the content of the event.
625	628

627	630	highlight the matching parts.
628	631
629	632	Args:
630		search_query (str)
631		events (list): A list of events
	633	search_query
	634	events: A list of events
632	635
633	636	Returns:
634		deferred : A set of strings.
	637	A set of strings.
635	638	"""
636	639
637	640	def f(txn):

684	687
685	688	return highlight_words
686	689
687		return self.db_pool.runInteraction("_find_highlights", f)
	690	return await self.db_pool.runInteraction("_find_highlights", f)
688	691
689	692
690	693	def _to_postgres_options(options_dict):

+33

-7

synapse/storage/databases/main/signatures.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
	15	from typing import Dict, Iterable, List, Tuple
	16
15	17	from unpaddedbase64 import encode_base64
16	18
17	19	from synapse.storage._base import SQLBaseStore
	20	from synapse.storage.types import Cursor
18	21	from synapse.util.caches.descriptors import cached, cachedList
19	22
20	23

28	31	@cachedList(
29	32	cached_method_name="get_event_reference_hash", list_name="event_ids", num_args=1
30	33	)
31		def get_event_reference_hashes(self, event_ids):
	34	async def get_event_reference_hashes(
	35	self, event_ids: Iterable[str]
	36	) -> Dict[str, Dict[str, bytes]]:
	37	"""Get all hashes for given events.
	38
	39	Args:
	40	event_ids: The event IDs to get hashes for.
	41
	42	Returns:
	43	A mapping of event ID to a mapping of algorithm to hash.
	44	"""
	45
32	46	def f(txn):
33	47	return {
34	48	event_id: self._get_event_reference_hashes_txn(txn, event_id)
35	49	for event_id in event_ids
36	50	}
37	51
38		return self.db_pool.runInteraction("get_event_reference_hashes", f)
	52	return await self.db_pool.runInteraction("get_event_reference_hashes", f)
39	53
40		async def add_event_hashes(self, event_ids):
	54	async def add_event_hashes(
	55	self, event_ids: Iterable[str]
	56	) -> List[Tuple[str, Dict[str, str]]]:
	57	"""
	58
	59	Args:
	60	event_ids: The event IDs
	61
	62	Returns:
	63	A list of tuples of event ID and a mapping of algorithm to base-64 encoded hash.
	64	"""
41	65	hashes = await self.get_event_reference_hashes(event_ids)
42	66	hashes = {
43	67	e_id: {k: encode_base64(v) for k, v in h.items() if k == "sha256"}

46	70
47	71	return list(hashes.items())
48	72
49		def _get_event_reference_hashes_txn(self, txn, event_id):
	73	def _get_event_reference_hashes_txn(
	74	self, txn: Cursor, event_id: str
	75	) -> Dict[str, bytes]:
50	76	"""Get all the hashes for a given PDU.
51	77	Args:
52		txn (cursor):
53		event_id (str): Id for the Event.
	78	txn:
	79	event_id: Id for the Event.
54	80	Returns:
55		A dict[unicode, bytes] of algorithm -> hash.
	81	A mapping of algorithm -> hash.
56	82	"""
57	83	query = (
58	84	"SELECT algorithm, hash"

+14

-14

synapse/storage/databases/main/state.py less more

26	26	from synapse.storage.databases.main.events_worker import EventsWorkerStore
27	27	from synapse.storage.databases.main.roommember import RoomMemberWorkerStore
28	28	from synapse.storage.state import StateFilter
	29	from synapse.types import StateMap
29	30	from synapse.util.caches import intern_string
30	31	from synapse.util.caches.descriptors import cached, cachedList
31	32

162	163	return create_event
163	164
164	165	@cached(max_entries=100000, iterable=True)
165		def get_current_state_ids(self, room_id):
	166	async def get_current_state_ids(self, room_id: str) -> StateMap[str]:
166	167	"""Get the current state event ids for a room based on the
167	168	current_state_events table.
168	169
169	170	Args:
170		room_id (str)
	171	room_id: The room to get the state IDs of.
171	172
172	173	Returns:
173		deferred: dict of (type, state_key) -> event_id
	174	The current state of the room.
174	175	"""
175	176
176	177	def _get_current_state_ids_txn(txn):

183	184
184	185	return {(intern_string(r[0]), intern_string(r[1])): r[2] for r in txn}
185	186
186		return self.db_pool.runInteraction(
	187	return await self.db_pool.runInteraction(
187	188	"get_current_state_ids", _get_current_state_ids_txn
188	189	)
189	190
190	191	# FIXME: how should this be cached?
191		def get_filtered_current_state_ids(
	192	async def get_filtered_current_state_ids(
192	193	self, room_id: str, state_filter: StateFilter = StateFilter.all()
193		):
	194	) -> StateMap[str]:
194	195	"""Get the current state event of a given type for a room based on the
195	196	current_state_events table. This may not be as up-to-date as the result
196	197	of doing a fresh state resolution as per state_handler.get_current_state

201	202	from the database.
202	203
203	204	Returns:
204		defer.Deferred[StateMap[str]]: Map from type/state_key to event ID.
	205	Map from type/state_key to event ID.
205	206	"""
206	207
207	208	where_clause, where_args = state_filter.make_sql_filter_clause()
208	209
209	210	if not where_clause:
210	211	# We delegate to the cached version
211		return self.get_current_state_ids(room_id)
	212	return await self.get_current_state_ids(room_id)
212	213
213	214	def _get_filtered_current_state_ids_txn(txn):
214	215	results = {}

230	231
231	232	return results
232	233
233		return self.db_pool.runInteraction(
	234	return await self.db_pool.runInteraction(
234	235	"get_filtered_current_state_ids", _get_filtered_current_state_ids_txn
235	236	)
236	237

259	260	return event.content.get("canonical_alias")
260	261
261	262	@cached(max_entries=50000)
262		def _get_state_group_for_event(self, event_id):
263		return self.db_pool.simple_select_one_onecol(
	263	async def _get_state_group_for_event(self, event_id: str) -> Optional[int]:
	264	return await self.db_pool.simple_select_one_onecol(
264	265	table="event_to_state_groups",
265	266	keyvalues={"event_id": event_id},
266	267	retcol="state_group",

272	273	cached_method_name="_get_state_group_for_event",
273	274	list_name="event_ids",
274	275	num_args=1,
275		inlineCallbacks=True,
276	276	)
277		def _get_state_group_for_events(self, event_ids):
	277	async def _get_state_group_for_events(self, event_ids):
278	278	"""Returns mapping event_id -> state_group
279	279	"""
280		rows = yield self.db_pool.simple_select_many_batch(
	280	rows = await self.db_pool.simple_select_many_batch(
281	281	table="event_to_state_groups",
282	282	column="event_id",
283	283	iterable=event_ids,

+11

-10

synapse/storage/databases/main/state_deltas.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
16
17		from twisted.internet import defer
	16	from typing import Any, Dict, List, Tuple
18	17
19	18	from synapse.storage._base import SQLBaseStore
20	19

22	21
23	22
24	23	class StateDeltasStore(SQLBaseStore):
25		def get_current_state_deltas(self, prev_stream_id: int, max_stream_id: int):
	24	async def get_current_state_deltas(
	25	self, prev_stream_id: int, max_stream_id: int
	26	) -> Tuple[int, List[Dict[str, Any]]]:
26	27	"""Fetch a list of room state changes since the given stream id
27	28
28	29	Each entry in the result contains the following fields:

36	37	if it's new state.
37	38
38	39	Args:
39		prev_stream_id (int): point to get changes since (exclusive)
40		max_stream_id (int): the point that we know has been correctly persisted
	40	prev_stream_id: point to get changes since (exclusive)
	41	max_stream_id: the point that we know has been correctly persisted
41	42	- ie, an upper limit to return changes from.
42	43
43	44	Returns:
44		Deferred[tuple[int, list[dict]]: A tuple consisting of:
	45	A tuple consisting of:
45	46	- the stream id which these results go up to
46	47	- list of current_state_delta_stream rows. If it is empty, we are
47	48	up to date.

57	58	# if the CSDs haven't changed between prev_stream_id and now, we
58	59	# know for certain that they haven't changed between prev_stream_id and
59	60	# max_stream_id.
60		return defer.succeed((max_stream_id, []))
	61	return (max_stream_id, [])
61	62
62	63	def get_current_state_deltas_txn(txn):
63	64	# First we calculate the max stream id that will give us less than

101	102	txn.execute(sql, (prev_stream_id, clipped_stream_id))
102	103	return clipped_stream_id, self.db_pool.cursor_to_dict(txn)
103	104
104		return self.db_pool.runInteraction(
	105	return await self.db_pool.runInteraction(
105	106	"get_current_state_deltas", get_current_state_deltas_txn
106	107	)
107	108

113	114	retcol="COALESCE(MAX(stream_id), -1)",
114	115	)
115	116
116		def get_max_stream_id_in_current_state_deltas(self):
117		return self.db_pool.runInteraction(
	117	async def get_max_stream_id_in_current_state_deltas(self):
	118	return await self.db_pool.runInteraction(
118	119	"get_max_stream_id_in_current_state_deltas",
119	120	self._get_max_stream_id_in_current_state_deltas_txn,
120	121	)

+80

-60

synapse/storage/databases/main/stats.py less more

14	14	# limitations under the License.
15	15
16	16	import logging
	17	from collections import Counter
17	18	from itertools import chain
18		from typing import Tuple
	19	from typing import Any, Dict, List, Optional, Tuple
19	20
20	21	from twisted.internet.defer import DeferredLock
21	22

210	211
211	212	return len(rooms_to_work_on)
212	213
213		def get_stats_positions(self):
	214	async def get_stats_positions(self) -> int:
214	215	"""
215	216	Returns the stats processor positions.
216	217	"""
217		return self.db_pool.simple_select_one_onecol(
	218	return await self.db_pool.simple_select_one_onecol(
218	219	table="stats_incremental_position",
219	220	keyvalues={},
220	221	retcol="stream_id",
221	222	desc="stats_incremental_position",
222	223	)
223	224
224		def update_room_state(self, room_id, fields):
225		"""
	225	async def update_room_state(self, room_id: str, fields: Dict[str, Any]) -> None:
	226	"""Update the state of a room.
	227
	228	fields can contain the following keys with string values:
	229	* join_rules
	230	* history_visibility
	231	* encryption
	232	* name
	233	* topic
	234	* avatar
	235	* canonical_alias
	236
	237	A is_federatable key can also be included with a boolean value.
	238
226	239	Args:
227		room_id (str)
228		fields (dict[str:Any])
229		"""
230
231		# For whatever reason some of the fields may contain null bytes, which
232		# postgres isn't a fan of, so we replace those fields with null.
	240	room_id: The room ID to update the state of.
	241	fields: The fields to update. This can include a partial list of the
	242	above fields to only update some room information.
	243	"""
	244	# Ensure that the values to update are valid, they should be strings and
	245	# not contain any null bytes.
	246	#
	247	# Invalid data gets overwritten with null.
	248	#
	249	# Note that a missing value should not be overwritten (it keeps the
	250	# previous value).
	251	sentinel = object()
233	252	for col in (
234	253	"join_rules",
235	254	"history_visibility",

239	258	"avatar",
240	259	"canonical_alias",
241	260	):
242		field = fields.get(col)
243		if field and "\0" in field:
	261	field = fields.get(col, sentinel)
	262	if field is not sentinel and (not isinstance(field, str) or "\0" in field):
244	263	fields[col] = None
245	264
246		return self.db_pool.simple_upsert(
	265	await self.db_pool.simple_upsert(
247	266	table="room_stats_state",
248	267	keyvalues={"room_id": room_id},
249	268	values=fields,
250	269	desc="update_room_state",
251	270	)
252	271
253		def get_statistics_for_subject(self, stats_type, stats_id, start, size=100):
	272	async def get_statistics_for_subject(
	273	self, stats_type: str, stats_id: str, start: str, size: int = 100
	274	) -> List[dict]:
254	275	"""
255	276	Get statistics for a given subject.
256	277
257	278	Args:
258		stats_type (str): The type of subject
259		stats_id (str): The ID of the subject (e.g. room_id or user_id)
260		start (int): Pagination start. Number of entries, not timestamp.
261		size (int): How many entries to return.
	279	stats_type: The type of subject
	280	stats_id: The ID of the subject (e.g. room_id or user_id)
	281	start: Pagination start. Number of entries, not timestamp.
	282	size: How many entries to return.
262	283
263	284	Returns:
264		Deferred[list[dict]], where the dict has the keys of
	285	A list of dicts, where the dict has the keys of
265	286	ABSOLUTE_STATS_FIELDS[stats_type], and "bucket_size" and "end_ts".
266	287	"""
267		return self.db_pool.runInteraction(
	288	return await self.db_pool.runInteraction(
268	289	"get_statistics_for_subject",
269	290	self._get_statistics_for_subject_txn,
270	291	stats_type,

299	320	return slice_list
300	321
301	322	@cached()
302		def get_earliest_token_for_stats(self, stats_type, id):
	323	async def get_earliest_token_for_stats(self, stats_type: str, id: str) -> int:
303	324	"""
304	325	Fetch the "earliest token". This is used by the room stats delta
305	326	processor to ignore deltas that have been processed between the

307	328	being calculated.
308	329
309	330	Returns:
310		Deferred[int]
	331	The earliest token.
311	332	"""
312	333	table, id_col = TYPE_TO_TABLE[stats_type]
313	334
314		return self.db_pool.simple_select_one_onecol(
	335	return await self.db_pool.simple_select_one_onecol(
315	336	"%s_current" % (table,),
316	337	keyvalues={id_col: id},
317	338	retcol="completed_delta_stream_id",
318	339	allow_none=True,
319	340	)
320	341
321		def bulk_update_stats_delta(self, ts, updates, stream_id):
	342	async def bulk_update_stats_delta(
	343	self, ts: int, updates: Dict[str, Dict[str, Dict[str, Counter]]], stream_id: int
	344	) -> None:
322	345	"""Bulk update stats tables for a given stream_id and updates the stats
323	346	incremental position.
324	347
325	348	Args:
326		ts (int): Current timestamp in ms
327		updates(dict[str, dict[str, dict[str, Counter]]]): The updates to
328		commit as a mapping stats_type -> stats_id -> field -> delta.
329		stream_id (int): Current position.
330
331		Returns:
332		Deferred
	349	ts: Current timestamp in ms
	350	updates: The updates to commit as a mapping of
	351	stats_type -> stats_id -> field -> delta.
	352	stream_id: Current position.
333	353	"""
334	354
335	355	def _bulk_update_stats_delta_txn(txn):

354	374	updatevalues={"stream_id": stream_id},
355	375	)
356	376
357		return self.db_pool.runInteraction(
	377	await self.db_pool.runInteraction(
358	378	"bulk_update_stats_delta", _bulk_update_stats_delta_txn
359	379	)
360	380
361		def update_stats_delta(
	381	async def update_stats_delta(
362	382	self,
363		ts,
364		stats_type,
365		stats_id,
366		fields,
367		complete_with_stream_id,
368		absolute_field_overrides=None,
369		):
	383	ts: int,
	384	stats_type: str,
	385	stats_id: str,
	386	fields: Dict[str, int],
	387	complete_with_stream_id: Optional[int],
	388	absolute_field_overrides: Optional[Dict[str, int]] = None,
	389	) -> None:
370	390	"""
371	391	Updates the statistics for a subject, with a delta (difference/relative
372	392	change).
373	393
374	394	Args:
375		ts (int): timestamp of the change
376		stats_type (str): "room" or "user" – the kind of subject
377		stats_id (str): the subject's ID (room ID or user ID)
378		fields (dict[str, int]): Deltas of stats values.
379		complete_with_stream_id (int, optional):
	395	ts: timestamp of the change
	396	stats_type: "room" or "user" – the kind of subject
	397	stats_id: the subject's ID (room ID or user ID)
	398	fields: Deltas of stats values.
	399	complete_with_stream_id:
380	400	If supplied, converts an incomplete row into a complete row,
381	401	with the supplied stream_id marked as the stream_id where the
382	402	row was completed.
383		absolute_field_overrides (dict[str, int]): Current stats values
384		(i.e. not deltas) of absolute fields.
385		Does not work with per-slice fields.
386		"""
387
388		return self.db_pool.runInteraction(
	403	absolute_field_overrides: Current stats values (i.e. not deltas) of
	404	absolute fields. Does not work with per-slice fields.
	405	"""
	406
	407	await self.db_pool.runInteraction(
389	408	"update_stats_delta",
390	409	self._update_stats_delta_txn,
391	410	ts,

645	664	txn, into_table, all_dest_keyvalues, src_row
646	665	)
647	666
648		def get_changes_room_total_events_and_bytes(self, min_pos, max_pos):
	667	async def get_changes_room_total_events_and_bytes(
	668	self, min_pos: int, max_pos: int
	669	) -> Dict[str, Dict[str, int]]:
649	670	"""Fetches the counts of events in the given range of stream IDs.
650	671
651	672	Args:
652		min_pos (int)
653		max_pos (int)
	673	min_pos
	674	max_pos
654	675
655	676	Returns:
656		Deferred[dict[str, dict[str, int]]]: Mapping of room ID to field
657		changes.
658		"""
659
660		return self.db_pool.runInteraction(
	677	Mapping of room ID to field changes.
	678	"""
	679
	680	return await self.db_pool.runInteraction(
661	681	"stats_incremental_total_events_and_bytes",
662	682	self.get_changes_room_total_events_and_bytes_txn,
663	683	min_pos,

+275

-227

synapse/storage/databases/main/stream.py less more

38	38	import abc
39	39	import logging
40	40	from collections import namedtuple
41		from typing import Optional
	41	from typing import TYPE_CHECKING, Dict, List, Optional, Set, Tuple
42	42
43	43	from twisted.internet import defer
44	44
	45	from synapse.api.filtering import Filter
	46	from synapse.events import EventBase
45	47	from synapse.logging.context import make_deferred_yieldable, run_in_background
46	48	from synapse.storage._base import SQLBaseStore
47		from synapse.storage.database import DatabasePool, make_in_list_sql_clause
	49	from synapse.storage.database import (
	50	DatabasePool,
	51	LoggingTransaction,
	52	make_in_list_sql_clause,
	53	)
48	54	from synapse.storage.databases.main.events_worker import EventsWorkerStore
49		from synapse.storage.engines import PostgresEngine
50		from synapse.types import RoomStreamToken
	55	from synapse.storage.engines import BaseDatabaseEngine, PostgresEngine
	56	from synapse.types import Collection, RoomStreamToken
51	57	from synapse.util.caches.stream_change_cache import StreamChangeCache
	58
	59	if TYPE_CHECKING:
	60	from synapse.server import HomeServer
52	61
53	62	logger = logging.getLogger(__name__)
54	63

67	76
68	77
69	78	def generate_pagination_where_clause(
70		direction, column_names, from_token, to_token, engine
71		):
	79	direction: str,
	80	column_names: Tuple[str, str],
	81	from_token: Optional[Tuple[int, int]],
	82	to_token: Optional[Tuple[int, int]],
	83	engine: BaseDatabaseEngine,
	84	) -> str:
72	85	"""Creates an SQL expression to bound the columns by the pagination
73	86	tokens.
74	87

89	102	token, but include those that match the to token.
90	103
91	104	Args:
92		direction (str): Whether we're paginating backwards("b") or
93		forwards ("f").
94		column_names (tuple[str, str]): The column names to bound. Must not
95		be user defined as these get inserted directly into the SQL
96		statement without escapes.
97		from_token (tuple[int, int]\|None): The start point for the pagination.
98		This is an exclusive minimum bound if direction is "f", and an
99		inclusive maximum bound if direction is "b".
100		to_token (tuple[int, int]\|None): The endpoint point for the pagination.
101		This is an inclusive maximum bound if direction is "f", and an
102		exclusive minimum bound if direction is "b".
	105	direction: Whether we're paginating backwards("b") or forwards ("f").
	106	column_names: The column names to bound. Must not be user defined as
	107	these get inserted directly into the SQL statement without escapes.
	108	from_token: The start point for the pagination. This is an exclusive
	109	minimum bound if direction is "f", and an inclusive maximum bound if
	110	direction is "b".
	111	to_token: The endpoint point for the pagination. This is an inclusive
	112	maximum bound if direction is "f", and an exclusive minimum bound if
	113	direction is "b".
103	114	engine: The database engine to generate the clauses for
104	115
105	116	Returns:
106		str: The sql expression
	117	The sql expression
107	118	"""
108	119	assert direction in ("b", "f")
109	120

131	142	return " AND ".join(where_clause)
132	143
133	144
134		def _make_generic_sql_bound(bound, column_names, values, engine):
	145	def _make_generic_sql_bound(
	146	bound: str,
	147	column_names: Tuple[str, str],
	148	values: Tuple[Optional[int], int],
	149	engine: BaseDatabaseEngine,
	150	) -> str:
135	151	"""Create an SQL expression that bounds the given column names by the
136	152	values, e.g. create the equivalent of `(1, 2) < (col1, col2)`.
137	153

141	157	out manually.
142	158
143	159	Args:
144		bound (str): The comparison operator to use. One of ">", "<", ">=",
	160	bound: The comparison operator to use. One of ">", "<", ">=",
145	161	"<=", where the values are on the left and columns on the right.
146		names (tuple[str, str]): The column names. Must not be user defined
	162	names: The column names. Must not be user defined
147	163	as these get inserted directly into the SQL statement without
148	164	escapes.
149		values (tuple[int\|None, int]): The values to bound the columns by. If
	165	values: The values to bound the columns by. If
150	166	the first value is None then only creates a bound on the second
151	167	column.
152	168	engine: The database engine to generate the SQL for
153	169
154	170	Returns:
155		str
	171	The SQL statement
156	172	"""
157	173
158	174	assert bound in (">", "<", ">=", "<=")

192	208	)
193	209
194	210
195		def filter_to_clause(event_filter):
	211	def filter_to_clause(event_filter: Optional[Filter]) -> Tuple[str, List[str]]:
196	212	# NB: This may create SQL clauses that don't optimise well (and we don't
197	213	# have indices on all possible clauses). E.g. it may create
198	214	# "room_id == X AND room_id != X", which postgres doesn't optimise.

250	266
251	267	__metaclass__ = abc.ABCMeta
252	268
253		def __init__(self, database: DatabasePool, db_conn, hs):
	269	def __init__(self, database: DatabasePool, db_conn, hs: "HomeServer"):
254	270	super(StreamWorkerStore, self).__init__(database, db_conn, hs)
255	271
256	272	self._instance_name = hs.get_instance_name()

283	299	self._stream_order_on_start = self.get_room_max_stream_ordering()
284	300
285	301	@abc.abstractmethod
286		def get_room_max_stream_ordering(self):
	302	def get_room_max_stream_ordering(self) -> int:
287	303	raise NotImplementedError()
288	304
289	305	@abc.abstractmethod
290		def get_room_min_stream_ordering(self):
	306	def get_room_min_stream_ordering(self) -> int:
291	307	raise NotImplementedError()
292	308
293		@defer.inlineCallbacks
294		def get_room_events_stream_for_rooms(
295		self, room_ids, from_key, to_key, limit=0, order="DESC"
296		):
	309	async def get_room_events_stream_for_rooms(
	310	self,
	311	room_ids: Collection[str],
	312	from_key: str,
	313	to_key: str,
	314	limit: int = 0,
	315	order: str = "DESC",
	316	) -> Dict[str, Tuple[List[EventBase], str]]:
297	317	"""Get new room events in stream ordering since `from_key`.
298	318
299	319	Args:
300		room_id (str)
301		from_key (str): Token from which no events are returned before
302		to_key (str): Token from which no events are returned after. (This
	320	room_ids
	321	from_key: Token from which no events are returned before
	322	to_key: Token from which no events are returned after. (This
303	323	is typically the current stream token)
304		limit (int): Maximum number of events to return
305		order (str): Either "DESC" or "ASC". Determines which events are
	324	limit: Maximum number of events to return
	325	order: Either "DESC" or "ASC". Determines which events are
306	326	returned when the result is limited. If "DESC" then the most
307	327	recent `limit` events are returned, otherwise returns the
308	328	oldest `limit` events.
309	329
310	330	Returns:
311		Deferred[dict[str,tuple[list[FrozenEvent], str]]]
312		A map from room id to a tuple containing:
313		- list of recent events in the room
314		- stream ordering key for the start of the chunk of events returned.
	331	A map from room id to a tuple containing:
	332	- list of recent events in the room
	333	- stream ordering key for the start of the chunk of events returned.
315	334	"""
316	335	from_id = RoomStreamToken.parse_stream_token(from_key).stream
317	336
318		room_ids = yield self._events_stream_cache.get_entities_changed(
319		room_ids, from_id
320		)
	337	room_ids = self._events_stream_cache.get_entities_changed(room_ids, from_id)
321	338
322	339	if not room_ids:
323	340	return {}

325	342	results = {}
326	343	room_ids = list(room_ids)
327	344	for rm_ids in (room_ids[i : i + 20] for i in range(0, len(room_ids), 20)):
328		res = yield make_deferred_yieldable(
	345	res = await make_deferred_yieldable(
329	346	defer.gatherResults(
330	347	[
331	348	run_in_background(

345	362
346	363	return results
347	364
348		def get_rooms_that_changed(self, room_ids, from_key):
	365	def get_rooms_that_changed(
	366	self, room_ids: Collection[str], from_key: str
	367	) -> Set[str]:
349	368	"""Given a list of rooms and a token, return rooms where there may have
350	369	been changes.
351	370
352	371	Args:
353		room_ids (list)
354		from_key (str): The room_key portion of a StreamToken
355		"""
356		from_key = RoomStreamToken.parse_stream_token(from_key).stream
	372	room_ids
	373	from_key: The room_key portion of a StreamToken
	374	"""
	375	from_id = RoomStreamToken.parse_stream_token(from_key).stream
357	376	return {
358	377	room_id
359	378	for room_id in room_ids
360		if self._events_stream_cache.has_entity_changed(room_id, from_key)
	379	if self._events_stream_cache.has_entity_changed(room_id, from_id)
361	380	}
362	381
363		@defer.inlineCallbacks
364		def get_room_events_stream_for_room(
365		self, room_id, from_key, to_key, limit=0, order="DESC"
366		):
367
	382	async def get_room_events_stream_for_room(
	383	self,
	384	room_id: str,
	385	from_key: str,
	386	to_key: str,
	387	limit: int = 0,
	388	order: str = "DESC",
	389	) -> Tuple[List[EventBase], str]:
368	390	"""Get new room events in stream ordering since `from_key`.
369	391
370	392	Args:
371		room_id (str)
372		from_key (str): Token from which no events are returned before
373		to_key (str): Token from which no events are returned after. (This
	393	room_id
	394	from_key: Token from which no events are returned before
	395	to_key: Token from which no events are returned after. (This
374	396	is typically the current stream token)
375		limit (int): Maximum number of events to return
376		order (str): Either "DESC" or "ASC". Determines which events are
	397	limit: Maximum number of events to return
	398	order: Either "DESC" or "ASC". Determines which events are
377	399	returned when the result is limited. If "DESC" then the most
378	400	recent `limit` events are returned, otherwise returns the
379	401	oldest `limit` events.
380	402
381	403	Returns:
382		Deferred[tuple[list[FrozenEvent], str]]: Returns the list of
383		events (in ascending order) and the token from the start of
384		the chunk of events returned.
	404	The list of events (in ascending order) and the token from the start
	405	of the chunk of events returned.
385	406	"""
386	407	if from_key == to_key:
387	408	return [], from_key

389	410	from_id = RoomStreamToken.parse_stream_token(from_key).stream
390	411	to_id = RoomStreamToken.parse_stream_token(to_key).stream
391	412
392		has_changed = yield self._events_stream_cache.has_entity_changed(
393		room_id, from_id
394		)
	413	has_changed = self._events_stream_cache.has_entity_changed(room_id, from_id)
395	414
396	415	if not has_changed:
397	416	return [], from_key

409	428	rows = [_EventDictReturn(row[0], None, row[1]) for row in txn]
410	429	return rows
411	430
412		rows = yield self.db_pool.runInteraction("get_room_events_stream_for_room", f)
413
414		ret = yield self.get_events_as_list(
	431	rows = await self.db_pool.runInteraction("get_room_events_stream_for_room", f)
	432
	433	ret = await self.get_events_as_list(
415	434	[r.event_id for r in rows], get_prev_content=True
416	435	)
417	436

429	448
430	449	return ret, key
431	450
432		@defer.inlineCallbacks
433		def get_membership_changes_for_user(self, user_id, from_key, to_key):
	451	async def get_membership_changes_for_user(
	452	self, user_id: str, from_key: str, to_key: str
	453	) -> List[EventBase]:
434	454	from_id = RoomStreamToken.parse_stream_token(from_key).stream
435	455	to_id = RoomStreamToken.parse_stream_token(to_key).stream
436	456

459	479
460	480	return rows
461	481
462		rows = yield self.db_pool.runInteraction("get_membership_changes_for_user", f)
463
464		ret = yield self.get_events_as_list(
	482	rows = await self.db_pool.runInteraction("get_membership_changes_for_user", f)
	483
	484	ret = await self.get_events_as_list(
465	485	[r.event_id for r in rows], get_prev_content=True
466	486	)
467	487

469	489
470	490	return ret
471	491
472		@defer.inlineCallbacks
473		def get_recent_events_for_room(self, room_id, limit, end_token):
	492	async def get_recent_events_for_room(
	493	self, room_id: str, limit: int, end_token: str
	494	) -> Tuple[List[EventBase], str]:
474	495	"""Get the most recent events in the room in topological ordering.
475	496
476	497	Args:
477		room_id (str)
478		limit (int)
479		end_token (str): The stream token representing now.
	498	room_id
	499	limit
	500	end_token: The stream token representing now.
480	501
481	502	Returns:
482		Deferred[tuple[list[FrozenEvent], str]]: Returns a list of
483		events and a token pointing to the start of the returned
484		events.
485		The events returned are in ascending order.
486		"""
487
488		rows, token = yield self.get_recent_event_ids_for_room(
	503	A list of events and a token pointing to the start of the returned
	504	events. The events returned are in ascending order.
	505	"""
	506
	507	rows, token = await self.get_recent_event_ids_for_room(
489	508	room_id, limit, end_token
490	509	)
491	510
492		events = yield self.get_events_as_list(
	511	events = await self.get_events_as_list(
493	512	[r.event_id for r in rows], get_prev_content=True
494	513	)
495	514

497	516
498	517	return (events, token)
499	518
500		@defer.inlineCallbacks
501		def get_recent_event_ids_for_room(self, room_id, limit, end_token):
	519	async def get_recent_event_ids_for_room(
	520	self, room_id: str, limit: int, end_token: str
	521	) -> Tuple[List[_EventDictReturn], str]:
502	522	"""Get the most recent events in the room in topological ordering.
503	523
504	524	Args:
505		room_id (str)
506		limit (int)
507		end_token (str): The stream token representing now.
	525	room_id
	526	limit
	527	end_token: The stream token representing now.
508	528
509	529	Returns:
510		Deferred[tuple[list[_EventDictReturn], str]]: Returns a list of
511		_EventDictReturn and a token pointing to the start of the returned
512		events.
513		The events returned are in ascending order.
	530	A list of _EventDictReturn and a token pointing to the start of the
	531	returned events. The events returned are in ascending order.
514	532	"""
515	533	# Allow a zero limit here, and no-op.
516	534	if limit == 0:

518	536
519	537	end_token = RoomStreamToken.parse(end_token)
520	538
521		rows, token = yield self.db_pool.runInteraction(
	539	rows, token = await self.db_pool.runInteraction(
522	540	"get_recent_event_ids_for_room",
523	541	self._paginate_room_events_txn,
524	542	room_id,

531	549
532	550	return rows, token
533	551
534		def get_room_event_before_stream_ordering(self, room_id, stream_ordering):
	552	async def get_room_event_before_stream_ordering(
	553	self, room_id: str, stream_ordering: int
	554	) -> Tuple[int, int, str]:
535	555	"""Gets details of the first event in a room at or before a stream ordering
536	556
537	557	Args:
538		room_id (str):
539		stream_ordering (int):
	558	room_id:
	559	stream_ordering:
540	560
541	561	Returns:
542		Deferred[(int, int, str)]:
543		(stream ordering, topological ordering, event_id)
	562	A tuple of (stream ordering, topological ordering, event_id)
544	563	"""
545	564
546	565	def _f(txn):

555	574	txn.execute(sql, (room_id, stream_ordering))
556	575	return txn.fetchone()
557	576
558		return self.db_pool.runInteraction("get_room_event_before_stream_ordering", _f)
	577	return await self.db_pool.runInteraction(
	578	"get_room_event_before_stream_ordering", _f
	579	)
559	580
560	581	async def get_room_events_max_id(self, room_id: Optional[str] = None) -> str:
561	582	"""Returns the current token for rooms stream.

573	594	)
574	595	return "t%d-%d" % (topo, token)
575	596
576		def get_stream_token_for_event(self, event_id):
577		"""The stream token for an event
578		Args:
579		event_id(str): The id of the event to look up a stream token for.
	597	async def get_stream_id_for_event(self, event_id: str) -> int:
	598	"""The stream ID for an event
	599	Args:
	600	event_id: The id of the event to look up a stream token for.
580	601	Raises:
581	602	StoreError if the event wasn't in the database.
582	603	Returns:
583		A deferred "s%d" stream token.
584		"""
585		return self.db_pool.simple_select_one_onecol(
586		table="events", keyvalues={"event_id": event_id}, retcol="stream_ordering"
587		).addCallback(lambda row: "s%d" % (row,))
588
589		def get_topological_token_for_event(self, event_id):
	604	A stream ID.
	605	"""
	606	return await self.db_pool.runInteraction(
	607	"get_stream_id_for_event", self.get_stream_id_for_event_txn, event_id,
	608	)
	609
	610	def get_stream_id_for_event_txn(
	611	self, txn: LoggingTransaction, event_id: str, allow_none=False,
	612	) -> int:
	613	return self.db_pool.simple_select_one_onecol_txn(
	614	txn=txn,
	615	table="events",
	616	keyvalues={"event_id": event_id},
	617	retcol="stream_ordering",
	618	allow_none=allow_none,
	619	)
	620
	621	async def get_stream_token_for_event(self, event_id: str) -> str:
590	622	"""The stream token for an event
591	623	Args:
592		event_id(str): The id of the event to look up a stream token for.
	624	event_id: The id of the event to look up a stream token for.
593	625	Raises:
594	626	StoreError if the event wasn't in the database.
595	627	Returns:
596		A deferred "t%d-%d" topological token.
597		"""
598		return self.db_pool.simple_select_one(
	628	A "s%d" stream token.
	629	"""
	630	stream_id = await self.get_stream_id_for_event(event_id)
	631	return "s%d" % (stream_id,)
	632
	633	async def get_topological_token_for_event(self, event_id: str) -> str:
	634	"""The stream token for an event
	635	Args:
	636	event_id: The id of the event to look up a stream token for.
	637	Raises:
	638	StoreError if the event wasn't in the database.
	639	Returns:
	640	A "t%d-%d" topological token.
	641	"""
	642	row = await self.db_pool.simple_select_one(
599	643	table="events",
600	644	keyvalues={"event_id": event_id},
601	645	retcols=("stream_ordering", "topological_ordering"),
602	646	desc="get_topological_token_for_event",
603		).addCallback(
604		lambda row: "t%d-%d" % (row["topological_ordering"], row["stream_ordering"])
605		)
606
607		def get_max_topological_token(self, room_id, stream_key):
608		"""Get the max topological token in a room before the given stream
	647	)
	648	return "t%d-%d" % (row["topological_ordering"], row["stream_ordering"])
	649
	650	async def get_current_topological_token(self, room_id: str, stream_key: int) -> int:
	651	"""Gets the topological token in a room after or at the given stream
609	652	ordering.
610	653
611	654	Args:
612		room_id (str)
613		stream_key (int)
614
615		Returns:
616		Deferred[int]
	655	room_id
	656	stream_key
617	657	"""
618	658	sql = (
619		"SELECT coalesce(max(topological_ordering), 0) FROM events"
620		" WHERE room_id = ? AND stream_ordering < ?"
621		)
622		return self.db_pool.execute(
623		"get_max_topological_token", None, sql, room_id, stream_key
624		).addCallback(lambda r: r[0][0] if r else 0)
625
626		def _get_max_topological_txn(self, txn, room_id):
	659	"SELECT coalesce(MIN(topological_ordering), 0) FROM events"
	660	" WHERE room_id = ? AND stream_ordering >= ?"
	661	)
	662	row = await self.db_pool.execute(
	663	"get_current_topological_token", None, sql, room_id, stream_key
	664	)
	665	return row[0][0] if row else 0
	666
	667	def _get_max_topological_txn(self, txn: LoggingTransaction, room_id: str) -> int:
627	668	txn.execute(
628	669	"SELECT MAX(topological_ordering) FROM events WHERE room_id = ?",
629	670	(room_id,),

633	674	return rows[0][0] if rows else 0
634	675
635	676	@staticmethod
636		def _set_before_and_after(events, rows, topo_order=True):
	677	def _set_before_and_after(
	678	events: List[EventBase], rows: List[_EventDictReturn], topo_order: bool = True
	679	):
637	680	"""Inserts ordering information to events' internal metadata from
638	681	the DB rows.
639	682
640	683	Args:
641		events (list[FrozenEvent])
642		rows (list[_EventDictReturn])
643		topo_order (bool): Whether the events were ordered topologically
644		or by stream ordering. If true then all rows should have a non
645		null topological_ordering.
	684	events
	685	rows
	686	topo_order: Whether the events were ordered topologically or by stream
	687	ordering. If true then all rows should have a non null
	688	topological_ordering.
646	689	"""
647	690	for event, row in zip(events, rows):
648	691	stream = row.stream_ordering

655	698	internal.after = str(RoomStreamToken(topo, stream))
656	699	internal.order = (int(topo) if topo else 0, int(stream))
657	700
658		@defer.inlineCallbacks
659		def get_events_around(
660		self, room_id, event_id, before_limit, after_limit, event_filter=None
661		):
	701	async def get_events_around(
	702	self,
	703	room_id: str,
	704	event_id: str,
	705	before_limit: int,
	706	after_limit: int,
	707	event_filter: Optional[Filter] = None,
	708	) -> dict:
662	709	"""Retrieve events and pagination tokens around a given event in a
663	710	room.
664
665		Args:
666		room_id (str)
667		event_id (str)
668		before_limit (int)
669		after_limit (int)
670		event_filter (Filter\|None)
671
672		Returns:
673		dict
674		"""
675
676		results = yield self.db_pool.runInteraction(
	711	"""
	712
	713	results = await self.db_pool.runInteraction(
677	714	"get_events_around",
678	715	self._get_events_around_txn,
679	716	room_id,

683	720	event_filter,
684	721	)
685	722
686		events_before = yield self.get_events_as_list(
	723	events_before = await self.get_events_as_list(
687	724	list(results["before"]["event_ids"]), get_prev_content=True
688	725	)
689	726
690		events_after = yield self.get_events_as_list(
	727	events_after = await self.get_events_as_list(
691	728	list(results["after"]["event_ids"]), get_prev_content=True
692	729	)
693	730

699	736	}
700	737
701	738	def _get_events_around_txn(
702		self, txn, room_id, event_id, before_limit, after_limit, event_filter
703		):
	739	self,
	740	txn: LoggingTransaction,
	741	room_id: str,
	742	event_id: str,
	743	before_limit: int,
	744	after_limit: int,
	745	event_filter: Optional[Filter],
	746	) -> dict:
704	747	"""Retrieves event_ids and pagination tokens around a given event in a
705	748	room.
706	749
707	750	Args:
708		room_id (str)
709		event_id (str)
710		before_limit (int)
711		after_limit (int)
712		event_filter (Filter\|None)
	751	room_id
	752	event_id
	753	before_limit
	754	after_limit
	755	event_filter
713	756
714	757	Returns:
715	758	dict

721	764	keyvalues={"event_id": event_id, "room_id": room_id},
722	765	retcols=["stream_ordering", "topological_ordering"],
723	766	)
	767
	768	# This cannot happen as `allow_none=False`.
	769	assert results is not None
724	770
725	771	# Paginating backwards includes the event at the token, but paginating
726	772	# forward doesn't.

757	803	"after": {"event_ids": events_after, "token": end_token},
758	804	}
759	805
760		@defer.inlineCallbacks
761		def get_all_new_events_stream(self, from_id, current_id, limit):
	806	async def get_all_new_events_stream(
	807	self, from_id: int, current_id: int, limit: int
	808	) -> Tuple[int, List[EventBase]]:
762	809	"""Get all new events
763	810
764	811	Returns all events with from_id < stream_ordering <= current_id.
765	812
766	813	Args:
767		from_id (int): the stream_ordering of the last event we processed
768		current_id (int): the stream_ordering of the most recently processed event
769		limit (int): the maximum number of events to return
	814	from_id: the stream_ordering of the last event we processed
	815	current_id: the stream_ordering of the most recently processed event
	816	limit: the maximum number of events to return
770	817
771	818	Returns:
772		Deferred[Tuple[int, list[FrozenEvent]]]: A tuple of (next_id, events), where
773		`next_id` is the next value to pass as `from_id` (it will either be the
774		stream_ordering of the last returned event, or, if fewer than `limit` events
775		were found, `current_id`.
	819	A tuple of (next_id, events), where `next_id` is the next value to
	820	pass as `from_id` (it will either be the stream_ordering of the
	821	last returned event, or, if fewer than `limit` events were found,
	822	the `current_id`).
776	823	"""
777	824
778	825	def get_all_new_events_stream_txn(txn):

794	841
795	842	return upper_bound, [row[1] for row in rows]
796	843
797		upper_bound, event_ids = yield self.db_pool.runInteraction(
	844	upper_bound, event_ids = await self.db_pool.runInteraction(
798	845	"get_all_new_events_stream", get_all_new_events_stream_txn
799	846	)
800	847
801		events = yield self.get_events_as_list(event_ids)
	848	events = await self.get_events_as_list(event_ids)
802	849
803	850	return upper_bound, events
804	851

816	863	desc="get_federation_out_pos",
817	864	)
818	865
819		async def update_federation_out_pos(self, typ, stream_id):
	866	async def update_federation_out_pos(self, typ: str, stream_id: int) -> None:
820	867	if self._need_to_reset_federation_stream_positions:
821	868	await self.db_pool.runInteraction(
822	869	"_reset_federation_positions_txn", self._reset_federation_positions_txn
823	870	)
824	871	self._need_to_reset_federation_stream_positions = False
825	872
826		return await self.db_pool.simple_update_one(
	873	await self.db_pool.simple_update_one(
827	874	table="federation_stream_position",
828	875	keyvalues={"type": typ, "instance_name": self._instance_name},
829	876	updatevalues={"stream_id": stream_id},
830	877	desc="update_federation_out_pos",
831	878	)
832	879
833		def _reset_federation_positions_txn(self, txn):
	880	def _reset_federation_positions_txn(self, txn: LoggingTransaction) -> None:
834	881	"""Fiddles with the `federation_stream_position` table to make it match
835	882	the configured federation sender instances during start up.
836	883	"""

869	916	GROUP BY type
870	917	"""
871	918	txn.execute(sql)
872		min_positions = dict(txn) # Map from type -> min position
	919	min_positions = {typ: pos for typ, pos in txn} # Map from type -> min position
873	920
874	921	# Ensure we do actually have some values here
875	922	assert set(min_positions) == {"federation", "events"}

891	938	values={"stream_id": stream_id},
892	939	)
893	940
894		def has_room_changed_since(self, room_id, stream_id):
	941	def has_room_changed_since(self, room_id: str, stream_id: int) -> bool:
895	942	return self._events_stream_cache.has_entity_changed(room_id, stream_id)
896	943
897	944	def _paginate_room_events_txn(
898	945	self,
899		txn,
900		room_id,
901		from_token,
902		to_token=None,
903		direction="b",
904		limit=-1,
905		event_filter=None,
906		):
	946	txn: LoggingTransaction,
	947	room_id: str,
	948	from_token: RoomStreamToken,
	949	to_token: Optional[RoomStreamToken] = None,
	950	direction: str = "b",
	951	limit: int = -1,
	952	event_filter: Optional[Filter] = None,
	953	) -> Tuple[List[_EventDictReturn], str]:
907	954	"""Returns list of events before or after a given token.
908	955
909	956	Args:
910	957	txn
911		room_id (str)
912		from_token (RoomStreamToken): The token used to stream from
913		to_token (RoomStreamToken\|None): A token which if given limits the
914		results to only those before
915		direction(char): Either 'b' or 'f' to indicate whether we are
916		paginating forwards or backwards from `from_key`.
917		limit (int): The maximum number of events to return.
918		event_filter (Filter\|None): If provided filters the events to
	958	room_id
	959	from_token: The token used to stream from
	960	to_token: A token which if given limits the results to only those before
	961	direction: Either 'b' or 'f' to indicate whether we are paginating
	962	forwards or backwards from `from_key`.
	963	limit: The maximum number of events to return.
	964	event_filter: If provided filters the events to
919	965	those that match the filter.
920	966
921	967	Returns:
922		Deferred[tuple[list[_EventDictReturn], str]]: Returns the results
923		as a list of _EventDictReturn and a token that points to the end
924		of the result set. If no events are returned then the end of the
925		stream has been reached (i.e. there are no events between
926		`from_token` and `to_token`), or `limit` is zero.
	968	A list of _EventDictReturn and a token that points to the end of the
	969	result set. If no events are returned then the end of the stream has
	970	been reached (i.e. there are no events between `from_token` and
	971	`to_token`), or `limit` is zero.
927	972	"""
928	973
929	974	assert int(limit) >= 0

1007	1052
1008	1053	return rows, str(next_token)
1009	1054
1010		@defer.inlineCallbacks
1011		def paginate_room_events(
1012		self, room_id, from_key, to_key=None, direction="b", limit=-1, event_filter=None
1013		):
	1055	async def paginate_room_events(
	1056	self,
	1057	room_id: str,
	1058	from_key: str,
	1059	to_key: Optional[str] = None,
	1060	direction: str = "b",
	1061	limit: int = -1,
	1062	event_filter: Optional[Filter] = None,
	1063	) -> Tuple[List[EventBase], str]:
1014	1064	"""Returns list of events before or after a given token.
1015	1065
1016	1066	Args:
1017		room_id (str)
1018		from_key (str): The token used to stream from
1019		to_key (str\|None): A token which if given limits the results to
1020		only those before
1021		direction(char): Either 'b' or 'f' to indicate whether we are
1022		paginating forwards or backwards from `from_key`.
1023		limit (int): The maximum number of events to return.
1024		event_filter (Filter\|None): If provided filters the events to
1025		those that match the filter.
	1067	room_id
	1068	from_key: The token used to stream from
	1069	to_key: A token which if given limits the results to only those before
	1070	direction: Either 'b' or 'f' to indicate whether we are paginating
	1071	forwards or backwards from `from_key`.
	1072	limit: The maximum number of events to return.
	1073	event_filter: If provided filters the events to those that match the filter.
1026	1074
1027	1075	Returns:
1028		tuple[list[FrozenEvent], str]: Returns the results as a list of
1029		events and a token that points to the end of the result set. If no
1030		events are returned then the end of the stream has been reached
1031		(i.e. there are no events between `from_key` and `to_key`).
	1076	The results as a list of events and a token that points to the end
	1077	of the result set. If no events are returned then the end of the
	1078	stream has been reached (i.e. there are no events between `from_key`
	1079	and `to_key`).
1032	1080	"""
1033	1081
1034	1082	from_key = RoomStreamToken.parse(from_key)
1035	1083	if to_key:
1036	1084	to_key = RoomStreamToken.parse(to_key)
1037	1085
1038		rows, token = yield self.db_pool.runInteraction(
	1086	rows, token = await self.db_pool.runInteraction(
1039	1087	"paginate_room_events",
1040	1088	self._paginate_room_events_txn,
1041	1089	room_id,

1046	1094	event_filter,
1047	1095	)
1048	1096
1049		events = yield self.get_events_as_list(
	1097	events = await self.get_events_as_list(
1050	1098	[r.event_id for r in rows], get_prev_content=True
1051	1099	)
1052	1100

1056	1104
1057	1105
1058	1106	class StreamStore(StreamWorkerStore):
1059		def get_room_max_stream_ordering(self):
	1107	def get_room_max_stream_ordering(self) -> int:
1060	1108	return self._stream_id_gen.get_current_token()
1061	1109
1062		def get_room_min_stream_ordering(self):
	1110	def get_room_min_stream_ordering(self) -> int:
1063	1111	return self._backfill_id_gen.get_current_token()

+7

-8

synapse/storage/databases/main/tags.py less more

16	16	import logging
17	17	from typing import Dict, List, Tuple
18	18
19		from canonicaljson import json
20
21	19	from synapse.storage._base import db_to_json
22	20	from synapse.storage.databases.main.account_data import AccountDataWorkerStore
23	21	from synapse.types import JsonDict
	22	from synapse.util import json_encoder
24	23	from synapse.util.caches.descriptors import cached
25	24
26	25	logger = logging.getLogger(__name__)

43	42	"room_tags", {"user_id": user_id}, ["room_id", "tag", "content"]
44	43	)
45	44
46		tags_by_room = {}
	45	tags_by_room = {} # type: Dict[str, Dict[str, JsonDict]]
47	46	for row in rows:
48	47	room_tags = tags_by_room.setdefault(row["room_id"], {})
49	48	room_tags[row["tag"]] = db_to_json(row["content"])

97	96	txn.execute(sql, (user_id, room_id))
98	97	tags = []
99	98	for tag, content in txn:
100		tags.append(json.dumps(tag) + ":" + content)
	99	tags.append(json_encoder.encode(tag) + ":" + content)
101	100	tag_json = "{" + ",".join(tags) + "}"
102	101	results.append((stream_id, (user_id, room_id, tag_json)))
103	102

123	122
124	123	async def get_updated_tags(
125	124	self, user_id: str, stream_id: int
126		) -> Dict[str, List[str]]:
	125	) -> Dict[str, Dict[str, JsonDict]]:
127	126	"""Get all the tags for the rooms where the tags have changed since the
128	127	given version
129	128

199	198	Returns:
200	199	The next account data ID.
201	200	"""
202		content_json = json.dumps(content)
	201	content_json = json_encoder.encode(content)
203	202
204	203	def add_tag_txn(txn, next_id):
205	204	self.db_pool.simple_upsert_txn(

210	209	)
211	210	self._update_revision_txn(txn, user_id, room_id, next_id)
212	211
213		with self._account_data_id_gen.get_next() as next_id:
	212	with await self._account_data_id_gen.get_next() as next_id:
214	213	await self.db_pool.runInteraction("add_tag", add_tag_txn, next_id)
215	214
216	215	self.get_tags_for_user.invalidate((user_id,))

232	231	txn.execute(sql, (user_id, room_id, tag))
233	232	self._update_revision_txn(txn, user_id, room_id, next_id)
234	233
235		with self._account_data_id_gen.get_next() as next_id:
	234	with await self._account_data_id_gen.get_next() as next_id:
236	235	await self.db_pool.runInteraction("remove_tag", remove_tag_txn, next_id)
237	236
238	237	self.get_tags_for_user.invalidate((user_id,))

+37

-28

synapse/storage/databases/main/transactions.py less more

14	14
15	15	import logging
16	16	from collections import namedtuple
	17	from typing import Optional, Tuple
17	18
18	19	from canonicaljson import encode_canonical_json
19	20
20	21	from synapse.metrics.background_process_metrics import run_as_background_process
21	22	from synapse.storage._base import SQLBaseStore, db_to_json
22	23	from synapse.storage.database import DatabasePool
	24	from synapse.types import JsonDict
23	25	from synapse.util.caches.expiringcache import ExpiringCache
24	26
25	27	db_binary_type = memoryview

54	56	expiry_ms=5 * 60 * 1000,
55	57	)
56	58
57		def get_received_txn_response(self, transaction_id, origin):
	59	async def get_received_txn_response(
	60	self, transaction_id: str, origin: str
	61	) -> Optional[Tuple[int, JsonDict]]:
58	62	"""For an incoming transaction from a given origin, check if we have
59	63	already responded to it. If so, return the response code and response
60	64	body (as a dict).
61	65
62	66	Args:
63		transaction_id (str)
64		origin(str)
	67	transaction_id
	68	origin
65	69
66	70	Returns:
67		tuple: None if we have not previously responded to
68		this transaction or a 2-tuple of (int, dict)
69		"""
70
71		return self.db_pool.runInteraction(
	71	None if we have not previously responded to this transaction or a
	72	2-tuple of (int, dict)
	73	"""
	74
	75	return await self.db_pool.runInteraction(
72	76	"get_received_txn_response",
73	77	self._get_received_txn_response,
74	78	transaction_id,

97	101	else:
98	102	return None
99	103
100		def set_received_txn_response(self, transaction_id, origin, code, response_dict):
101		"""Persist the response we returened for an incoming transaction, and
	104	async def set_received_txn_response(
	105	self, transaction_id: str, origin: str, code: int, response_dict: JsonDict
	106	) -> None:
	107	"""Persist the response we returned for an incoming transaction, and
102	108	should return for subsequent transactions with the same transaction_id
103	109	and origin.
104	110
105	111	Args:
106		txn
107		transaction_id (str)
108		origin (str)
109		code (int)
110		response_json (str)
111		"""
112
113		return self.db_pool.simple_insert(
	112	transaction_id: The incoming transaction ID.
	113	origin: The origin server.
	114	code: The response code.
	115	response_dict: The response, to be encoded into JSON.
	116	"""
	117
	118	await self.db_pool.simple_insert(
114	119	table="received_transactions",
115	120	values={
116	121	"transaction_id": transaction_id,

163	168	else:
164	169	return None
165	170
166		def set_destination_retry_timings(
167		self, destination, failure_ts, retry_last_ts, retry_interval
168		):
	171	async def set_destination_retry_timings(
	172	self,
	173	destination: str,
	174	failure_ts: Optional[int],
	175	retry_last_ts: int,
	176	retry_interval: int,
	177	) -> None:
169	178	"""Sets the current retry timings for a given destination.
170	179	Both timings should be zero if retrying is no longer occuring.
171	180
172	181	Args:
173		destination (str)
174		failure_ts (int\|None) - when the server started failing (ms since epoch)
175		retry_last_ts (int) - time of last retry attempt in unix epoch ms
176		retry_interval (int) - how long until next retry in ms
	182	destination
	183	failure_ts: when the server started failing (ms since epoch)
	184	retry_last_ts: time of last retry attempt in unix epoch ms
	185	retry_interval: how long until next retry in ms
177	186	"""
178	187
179	188	self._destination_retry_cache.pop(destination, None)
180		return self.db_pool.runInteraction(
	189	return await self.db_pool.runInteraction(
181	190	"set_destination_retry_timings",
182	191	self._set_destination_retry_timings,
183	192	destination,

253	262	"cleanup_transactions", self._cleanup_transactions
254	263	)
255	264
256		def _cleanup_transactions(self):
	265	async def _cleanup_transactions(self) -> None:
257	266	now = self._clock.time_msec()
258	267	month_ago = now - 30 * 24 * 60 * 60 * 1000
259	268
260	269	def _cleanup_transactions_txn(txn):
261	270	txn.execute("DELETE FROM received_transactions WHERE ts < ?", (month_ago,))
262	271
263		return self.db_pool.runInteraction(
	272	await self.db_pool.runInteraction(
264	273	"_cleanup_transactions", _cleanup_transactions_txn
265	274	)

+53

-12

synapse/storage/databases/main/ui_auth.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14		from typing import Any, Dict, Optional, Union
	14	from typing import Any, Dict, List, Optional, Tuple, Union
15	15
16	16	import attr
17		from canonicaljson import json
18	17
19	18	from synapse.api.errors import StoreError
20	19	from synapse.storage._base import SQLBaseStore, db_to_json
	20	from synapse.storage.database import LoggingTransaction
21	21	from synapse.types import JsonDict
22		from synapse.util import stringutils as stringutils
	22	from synapse.util import json_encoder, stringutils
23	23
24	24
25	25	@attr.s

71	71	StoreError if a unique session ID cannot be generated.
72	72	"""
73	73	# The clientdict gets stored as JSON.
74		clientdict_json = json.dumps(clientdict)
	74	clientdict_json = json_encoder.encode(clientdict)
75	75
76	76	# autogen a session ID and try to create it. We may clash, so just
77	77	# try a few times till one goes through, giving up eventually.

142	142	await self.db_pool.simple_upsert(
143	143	table="ui_auth_sessions_credentials",
144	144	keyvalues={"session_id": session_id, "stage_type": stage_type},
145		values={"result": json.dumps(result)},
	145	values={"result": json_encoder.encode(result)},
146	146	desc="mark_ui_auth_stage_complete",
147	147	)
148	148	except self.db_pool.engine.module.IntegrityError:

183	183	The dictionary from the client root level, not the 'auth' key.
184	184	"""
185	185	# The clientdict gets stored as JSON.
186		clientdict_json = json.dumps(clientdict)
	186	clientdict_json = json_encoder.encode(clientdict)
187	187
188	188	await self.db_pool.simple_update_one(
189	189	table="ui_auth_sessions",

213	213	value,
214	214	)
215	215
216		def _set_ui_auth_session_data_txn(self, txn, session_id: str, key: str, value: Any):
	216	def _set_ui_auth_session_data_txn(
	217	self, txn: LoggingTransaction, session_id: str, key: str, value: Any
	218	):
217	219	# Get the current value.
218	220	result = self.db_pool.simple_select_one_txn(
219	221	txn,
220	222	table="ui_auth_sessions",
221	223	keyvalues={"session_id": session_id},
222	224	retcols=("serverdict",),
223		)
	225	) # type: Dict[str, Any] # type: ignore
224	226
225	227	# Update it and add it back to the database.
226	228	serverdict = db_to_json(result["serverdict"])

230	232	txn,
231	233	table="ui_auth_sessions",
232	234	keyvalues={"session_id": session_id},
233		updatevalues={"serverdict": json.dumps(serverdict)},
	235	updatevalues={"serverdict": json_encoder.encode(serverdict)},
234	236	)
235	237
236	238	async def get_ui_auth_session_data(

257	259
258	260	return serverdict.get(key, default)
259	261
	262	async def add_user_agent_ip_to_ui_auth_session(
	263	self, session_id: str, user_agent: str, ip: str,
	264	):
	265	"""Add the given user agent / IP to the tracking table
	266	"""
	267	await self.db_pool.simple_upsert(
	268	table="ui_auth_sessions_ips",
	269	keyvalues={"session_id": session_id, "user_agent": user_agent, "ip": ip},
	270	values={},
	271	desc="add_user_agent_ip_to_ui_auth_session",
	272	)
	273
	274	async def get_user_agents_ips_to_ui_auth_session(
	275	self, session_id: str,
	276	) -> List[Tuple[str, str]]:
	277	"""Get the given user agents / IPs used during the ui auth process
	278
	279	Returns:
	280	List of user_agent/ip pairs
	281	"""
	282	rows = await self.db_pool.simple_select_list(
	283	table="ui_auth_sessions_ips",
	284	keyvalues={"session_id": session_id},
	285	retcols=("user_agent", "ip"),
	286	desc="get_user_agents_ips_to_ui_auth_session",
	287	)
	288	return [(row["user_agent"], row["ip"]) for row in rows]
	289
260	290
261	291	class UIAuthStore(UIAuthWorkerStore):
262		def delete_old_ui_auth_sessions(self, expiration_time: int):
	292	async def delete_old_ui_auth_sessions(self, expiration_time: int) -> None:
263	293	"""
264	294	Remove sessions which were last used earlier than the expiration time.
265	295

268	298	This is an epoch time in milliseconds.
269	299
270	300	"""
271		return self.db_pool.runInteraction(
	301	await self.db_pool.runInteraction(
272	302	"delete_old_ui_auth_sessions",
273	303	self._delete_old_ui_auth_sessions_txn,
274	304	expiration_time,
275	305	)
276	306
277		def _delete_old_ui_auth_sessions_txn(self, txn, expiration_time: int):
	307	def _delete_old_ui_auth_sessions_txn(
	308	self, txn: LoggingTransaction, expiration_time: int
	309	):
278	310	# Get the expired sessions.
279	311	sql = "SELECT session_id FROM ui_auth_sessions WHERE creation_time <= ?"
280	312	txn.execute(sql, [expiration_time])
281	313	session_ids = [r[0] for r in txn.fetchall()]
282	314
	315	# Delete the corresponding IP/user agents.
	316	self.db_pool.simple_delete_many_txn(
	317	txn,
	318	table="ui_auth_sessions_ips",
	319	column="session_id",
	320	iterable=session_ids,
	321	keyvalues={},
	322	)
	323
283	324	# Delete the corresponding completed credentials.
284	325	self.db_pool.simple_delete_many_txn(
285	326	txn,

+78

-24

synapse/storage/databases/main/user_directory.py less more

14	14
15	15	import logging
16	16	import re
	17	from typing import Any, Dict, Iterable, Optional, Set, Tuple
17	18
18	19	from synapse.api.constants import EventTypes, JoinRules
19	20	from synapse.storage.database import DatabasePool

363	364
364	365	return False
365	366
366		def update_profile_in_user_dir(self, user_id, display_name, avatar_url):
	367	async def update_profile_in_user_dir(
	368	self, user_id: str, display_name: str, avatar_url: str
	369	) -> None:
367	370	"""
368	371	Update or add a user's profile in the user directory.
369	372	"""
	373	# If the display name or avatar URL are unexpected types, overwrite them.
	374	if not isinstance(display_name, str):
	375	display_name = None
	376	if not isinstance(avatar_url, str):
	377	avatar_url = None
370	378
371	379	def _update_profile_in_user_dir_txn(txn):
372	380	new_entry = self.db_pool.simple_upsert_txn(

456	464
457	465	txn.call_after(self.get_user_in_directory.invalidate, (user_id,))
458	466
459		return self.db_pool.runInteraction(
	467	await self.db_pool.runInteraction(
460	468	"update_profile_in_user_dir", _update_profile_in_user_dir_txn
461	469	)
462	470
463		def add_users_who_share_private_room(self, room_id, user_id_tuples):
	471	async def add_users_who_share_private_room(
	472	self, room_id: str, user_id_tuples: Iterable[Tuple[str, str]]
	473	) -> None:
464	474	"""Insert entries into the users_who_share_private_rooms table. The first
465	475	user should be a local user.
466	476
467	477	Args:
468		room_id (str)
469		user_id_tuples([(str, str)]): iterable of 2-tuple of user IDs.
	478	room_id
	479	user_id_tuples: iterable of 2-tuple of user IDs.
470	480	"""
471	481
472	482	def _add_users_who_share_room_txn(txn):

482	492	value_values=None,
483	493	)
484	494
485		return self.db_pool.runInteraction(
	495	await self.db_pool.runInteraction(
486	496	"add_users_who_share_room", _add_users_who_share_room_txn
487	497	)
488	498
489		def add_users_in_public_rooms(self, room_id, user_ids):
	499	async def add_users_in_public_rooms(
	500	self, room_id: str, user_ids: Iterable[str]
	501	) -> None:
490	502	"""Insert entries into the users_who_share_private_rooms table. The first
491	503	user should be a local user.
492	504
493	505	Args:
494		room_id (str)
495		user_ids (list[str])
	506	room_id
	507	user_ids
496	508	"""
497	509
498	510	def _add_users_in_public_rooms_txn(txn):

506	518	value_values=None,
507	519	)
508	520
509		return self.db_pool.runInteraction(
	521	await self.db_pool.runInteraction(
510	522	"add_users_in_public_rooms", _add_users_in_public_rooms_txn
511	523	)
512	524
513		def delete_all_from_user_dir(self):
	525	async def delete_all_from_user_dir(self) -> None:
514	526	"""Delete the entire user directory
515	527	"""
516	528

521	533	txn.execute("DELETE FROM users_who_share_private_rooms")
522	534	txn.call_after(self.get_user_in_directory.invalidate_all)
523	535
524		return self.db_pool.runInteraction(
	536	await self.db_pool.runInteraction(
525	537	"delete_all_from_user_dir", _delete_all_from_user_dir_txn
526	538	)
527	539
528	540	@cached()
529		def get_user_in_directory(self, user_id):
530		return self.db_pool.simple_select_one(
	541	async def get_user_in_directory(self, user_id: str) -> Optional[Dict[str, Any]]:
	542	return await self.db_pool.simple_select_one(
531	543	table="user_directory",
532	544	keyvalues={"user_id": user_id},
533	545	retcols=("display_name", "avatar_url"),

535	547	desc="get_user_in_directory",
536	548	)
537	549
538		def update_user_directory_stream_pos(self, stream_id):
539		return self.db_pool.simple_update_one(
	550	async def update_user_directory_stream_pos(self, stream_id: str) -> None:
	551	await self.db_pool.simple_update_one(
540	552	table="user_directory_stream_pos",
541	553	keyvalues={},
542	554	updatevalues={"stream_id": stream_id},

553	565	def __init__(self, database: DatabasePool, db_conn, hs):
554	566	super(UserDirectoryStore, self).__init__(database, db_conn, hs)
555	567
556		def remove_from_user_dir(self, user_id):
	568	async def remove_from_user_dir(self, user_id: str) -> None:
557	569	def _remove_from_user_dir_txn(txn):
558	570	self.db_pool.simple_delete_txn(
559	571	txn, table="user_directory", keyvalues={"user_id": user_id}

576	588	)
577	589	txn.call_after(self.get_user_in_directory.invalidate, (user_id,))
578	590
579		return self.db_pool.runInteraction(
	591	await self.db_pool.runInteraction(
580	592	"remove_from_user_dir", _remove_from_user_dir_txn
581	593	)
582	594

603	615
604	616	return user_ids
605	617
606		def remove_user_who_share_room(self, user_id, room_id):
	618	async def remove_user_who_share_room(self, user_id: str, room_id: str) -> None:
607	619	"""
608	620	Deletes entries in the users_who_share_*_rooms table. The first
609	621	user should be a local user.
610	622
611	623	Args:
612		user_id (str)
613		room_id (str)
	624	user_id
	625	room_id
614	626	"""
615	627
616	628	def _remove_user_who_share_room_txn(txn):

630	642	keyvalues={"user_id": user_id, "room_id": room_id},
631	643	)
632	644
633		return self.db_pool.runInteraction(
	645	await self.db_pool.runInteraction(
634	646	"remove_user_who_share_room", _remove_user_who_share_room_txn
635	647	)
636	648

662	674	users.update(rows)
663	675	return list(users)
664	676
665		def get_user_directory_stream_pos(self):
666		return self.db_pool.simple_select_one_onecol(
	677	@cached()
	678	async def get_shared_rooms_for_users(
	679	self, user_id: str, other_user_id: str
	680	) -> Set[str]:
	681	"""
	682	Returns the rooms that a local user shares with another local or remote user.
	683
	684	Args:
	685	user_id: The MXID of a local user
	686	other_user_id: The MXID of the other user
	687
	688	Returns:
	689	A set of room ID's that the users share.
	690	"""
	691
	692	def _get_shared_rooms_for_users_txn(txn):
	693	txn.execute(
	694	"""
	695	SELECT p1.room_id
	696	FROM users_in_public_rooms as p1
	697	INNER JOIN users_in_public_rooms as p2
	698	ON p1.room_id = p2.room_id
	699	AND p1.user_id = ?
	700	AND p2.user_id = ?
	701	UNION
	702	SELECT room_id
	703	FROM users_who_share_private_rooms
	704	WHERE
	705	user_id = ?
	706	AND other_user_id = ?
	707	""",
	708	(user_id, other_user_id, user_id, other_user_id),
	709	)
	710	rows = self.db_pool.cursor_to_dict(txn)
	711	return rows
	712
	713	rows = await self.db_pool.runInteraction(
	714	"get_shared_rooms_for_users", _get_shared_rooms_for_users_txn
	715	)
	716
	717	return {row["room_id"] for row in rows}
	718
	719	async def get_user_directory_stream_pos(self) -> int:
	720	return await self.db_pool.simple_select_one_onecol(
667	721	table="user_directory_stream_pos",
668	722	keyvalues={},
669	723	retcol="stream_id",

+15

-19

synapse/storage/databases/main/user_erasure_store.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		import operator
16
17	15	from synapse.storage._base import SQLBaseStore
18	16	from synapse.util.caches.descriptors import cached, cachedList
19	17
20	18
21	19	class UserErasureWorkerStore(SQLBaseStore):
22	20	@cached()
23		def is_user_erased(self, user_id):
	21	async def is_user_erased(self, user_id: str) -> bool:
24	22	"""
25	23	Check if the given user id has requested erasure
26	24
27	25	Args:
28		user_id (str): full user id to check
	26	user_id: full user id to check
29	27
30	28	Returns:
31		Deferred[bool]: True if the user has requested erasure
	29	True if the user has requested erasure
32	30	"""
33		return self.db_pool.simple_select_onecol(
	31	result = await self.db_pool.simple_select_onecol(
34	32	table="erased_users",
35	33	keyvalues={"user_id": user_id},
36	34	retcol="1",
37	35	desc="is_user_erased",
38		).addCallback(operator.truth)
	36	)
	37	return bool(result)
39	38
40		@cachedList(
41		cached_method_name="is_user_erased", list_name="user_ids", inlineCallbacks=True
42		)
43		def are_users_erased(self, user_ids):
	39	@cachedList(cached_method_name="is_user_erased", list_name="user_ids")
	40	async def are_users_erased(self, user_ids):
44	41	"""
45	42	Checks which users in a list have requested erasure
46	43

48	45	user_ids (iterable[str]): full user id to check
49	46
50	47	Returns:
51		Deferred[dict[str, bool]]:
	48	dict[str, bool]:
52	49	for each user, whether the user has requested erasure.
53	50	"""
54	51	# this serves the dual purpose of (a) making sure we can do len and
55	52	# iterate it multiple times, and (b) avoiding duplicates.
56	53	user_ids = tuple(set(user_ids))
57	54
58		rows = yield self.db_pool.simple_select_many_batch(
	55	rows = await self.db_pool.simple_select_many_batch(
59	56	table="erased_users",
60	57	column="user_id",
61	58	iterable=user_ids,

64	61	)
65	62	erased_users = {row["user_id"] for row in rows}
66	63
67		res = {u: u in erased_users for u in user_ids}
68		return res
	64	return {u: u in erased_users for u in user_ids}
69	65
70	66
71	67	class UserErasureStore(UserErasureWorkerStore):
72		def mark_user_erased(self, user_id: str) -> None:
	68	async def mark_user_erased(self, user_id: str) -> None:
73	69	"""Indicate that user_id wishes their message history to be erased.
74	70
75	71	Args:

87	83
88	84	self._invalidate_cache_and_stream(txn, self.is_user_erased, (user_id,))
89	85
90		return self.db_pool.runInteraction("mark_user_erased", f)
	86	await self.db_pool.runInteraction("mark_user_erased", f)
91	87
92		def mark_user_not_erased(self, user_id: str) -> None:
	88	async def mark_user_not_erased(self, user_id: str) -> None:
93	89	"""Indicate that user_id is no longer erased.
94	90
95	91	Args:

109	105
110	106	self._invalidate_cache_and_stream(txn, self.is_user_erased, (user_id,))
111	107
112		return self.db_pool.runInteraction("mark_user_not_erased", f)
	108	await self.db_pool.runInteraction("mark_user_not_erased", f)

+14

-14

synapse/storage/databases/state/store.py less more

15	15	import logging
16	16	from collections import namedtuple
17	17	from typing import Dict, Iterable, List, Set, Tuple
18
19		from twisted.internet import defer
20	18
21	19	from synapse.api.constants import EventTypes
22	20	from synapse.storage._base import SQLBaseStore

102	100	)
103	101
104	102	@cached(max_entries=10000, iterable=True)
105		def get_state_group_delta(self, state_group):
	103	async def get_state_group_delta(self, state_group):
106	104	"""Given a state group try to return a previous group and a delta between
107	105	the old and the new.
108	106

134	132	{(row["type"], row["state_key"]): row["event_id"] for row in delta_ids},
135	133	)
136	134
137		return self.db_pool.runInteraction(
	135	return await self.db_pool.runInteraction(
138	136	"get_state_group_delta", _get_state_group_delta_txn
139	137	)
140	138

366	364	fetched_keys=non_member_types,
367	365	)
368	366
369		def store_state_group(
	367	async def store_state_group(
370	368	self, event_id, room_id, prev_group, delta_ids, current_state_ids
371		):
	369	) -> int:
372	370	"""Store a new set of state, returning a newly assigned state group.
373	371
374	372	Args:

382	380	to event_id.
383	381
384	382	Returns:
385		Deferred[int]: The state group ID
	383	The state group ID
386	384	"""
387	385
388	386	def _store_state_group_txn(txn):

483	481
484	482	return state_group
485	483
486		return self.db_pool.runInteraction("store_state_group", _store_state_group_txn)
487
488		def purge_unreferenced_state_groups(
	484	return await self.db_pool.runInteraction(
	485	"store_state_group", _store_state_group_txn
	486	)
	487
	488	async def purge_unreferenced_state_groups(
489	489	self, room_id: str, state_groups_to_delete
490		) -> defer.Deferred:
	490	) -> None:
491	491	"""Deletes no longer referenced state groups and de-deltas any state
492	492	groups that reference them.
493	493

498	498	to delete.
499	499	"""
500	500
501		return self.db_pool.runInteraction(
	501	await self.db_pool.runInteraction(
502	502	"purge_unreferenced_state_groups",
503	503	self._purge_unreferenced_state_groups,
504	504	room_id,

593	593
594	594	return {row["state_group"]: row["prev_state_group"] for row in rows}
595	595
596		def purge_room_state(self, room_id, state_groups_to_delete):
	596	async def purge_room_state(self, room_id, state_groups_to_delete):
597	597	"""Deletes all record of a room from state tables
598	598
599	599	Args:

601	601	state_groups_to_delete (list[int]): State groups to delete
602	602	"""
603	603
604		return self.db_pool.runInteraction(
	604	await self.db_pool.runInteraction(
605	605	"purge_room_state",
606	606	self._purge_room_state_txn,
607	607	room_id,

+1

-1

synapse/storage/keys.py less more

21	21
22	22
23	23	@attr.s(slots=True, frozen=True)
24		class FetchKeyResult(object):
	24	class FetchKeyResult:
25	25	verify_key = attr.ib() # VerifyKey: the key itself
26	26	valid_until_ts = attr.ib() # int: how long we can use this key for

+2

-2

synapse/storage/persist_events.py less more

68	68	)
69	69
70	70
71		class _EventPeristenceQueue(object):
	71	class _EventPeristenceQueue:
72	72	"""Queues up events so that they can be persisted in bulk with only one
73	73	concurrent transaction per room.
74	74	"""

171	171	pass
172	172
173	173
174		class EventsPersistenceStorage(object):
	174	class EventsPersistenceStorage:
175	175	"""High level interface for handling persisting newly received events.
176	176
177	177	Takes care of batching up events by room, and calculating the necessary

+73

-19

synapse/storage/prepare_database.py less more

46	46	pass
47	47
48	48
	49	OUTDATED_SCHEMA_ON_WORKER_ERROR = (
	50	"Expected database schema version %i but got %i: run the main synapse process to "
	51	"upgrade the database schema before starting worker processes."
	52	)
	53
	54	EMPTY_DATABASE_ON_WORKER_ERROR = (
	55	"Uninitialised database: run the main synapse process to prepare the database "
	56	"schema before starting worker processes."
	57	)
	58
	59	UNAPPLIED_DELTA_ON_WORKER_ERROR = (
	60	"Database schema delta %s has not been applied: run the main synapse process to "
	61	"upgrade the database schema before starting worker processes."
	62	)
	63
	64
49	65	def prepare_database(db_conn, database_engine, config, databases=["main", "state"]):
50	66	"""Prepares a physical database for usage. Will either create all necessary tables
51	67	or upgrade from an older schema version.

65	81
66	82	try:
67	83	cur = db_conn.cursor()
	84
	85	logger.info("%r: Checking existing schema version", databases)
68	86	version_info = _get_or_create_schema_state(cur, database_engine)
69	87
70	88	if version_info:
71	89	user_version, delta_files, upgraded = version_info
72
	90	logger.info(
	91	"%r: Existing schema is %i (+%i deltas)",
	92	databases,
	93	user_version,
	94	len(delta_files),
	95	)
	96
	97	# config should only be None when we are preparing an in-memory SQLite db,
	98	# which should be empty.
73	99	if config is None:
74		if user_version != SCHEMA_VERSION:
75		# If we don't pass in a config file then we are expecting to
76		# have already upgraded the DB.
77		raise UpgradeDatabaseException(
78		"Expected database schema version %i but got %i"
79		% (SCHEMA_VERSION, user_version)
80		)
81		else:
82		_upgrade_existing_database(
83		cur,
84		user_version,
85		delta_files,
86		upgraded,
87		database_engine,
88		config,
89		databases=databases,
	100	raise ValueError(
	101	"config==None in prepare_database, but databse is not empty"
90	102	)
	103
	104	# if it's a worker app, refuse to upgrade the database, to avoid multiple
	105	# workers doing it at once.
	106	if config.worker_app is not None and user_version != SCHEMA_VERSION:
	107	raise UpgradeDatabaseException(
	108	OUTDATED_SCHEMA_ON_WORKER_ERROR % (SCHEMA_VERSION, user_version)
	109	)
	110
	111	_upgrade_existing_database(
	112	cur,
	113	user_version,
	114	delta_files,
	115	upgraded,
	116	database_engine,
	117	config,
	118	databases=databases,
	119	)
91	120	else:
	121	logger.info("%r: Initialising new database", databases)
	122
	123	# if it's a worker app, refuse to upgrade the database, to avoid multiple
	124	# workers doing it at once.
	125	if config and config.worker_app is not None:
	126	raise UpgradeDatabaseException(EMPTY_DATABASE_ON_WORKER_ERROR)
	127
92	128	_setup_new_database(cur, database_engine, databases=databases)
93	129
94	130	# check if any of our configured dynamic modules want a database

294	330	else:
295	331	assert config
296	332
	333	is_worker = config and config.worker_app is not None
	334
297	335	if current_version > SCHEMA_VERSION:
298	336	raise ValueError(
299	337	"Cannot use this database as it is too "

321	359	specific_engine_extensions = (".sqlite", ".postgres")
322	360
323	361	for v in range(start_ver, SCHEMA_VERSION + 1):
324		logger.info("Upgrading schema to v%d", v)
	362	logger.info("Applying schema deltas for v%d", v)
325	363
326	364	# We need to search both the global and per data store schema
327	365	# directories for schema updates.

381	419	continue
382	420
383	421	root_name, ext = os.path.splitext(file_name)
	422
384	423	if ext == ".py":
385	424	# This is a python upgrade module. We need to import into some
386	425	# package and then execute its `run_upgrade` function.
	426	if is_worker:
	427	raise PrepareDatabaseException(
	428	UNAPPLIED_DELTA_ON_WORKER_ERROR % relative_path
	429	)
	430
387	431	module_name = "synapse.storage.v%d_%s" % (v, root_name)
388	432	with open(absolute_path) as python_file:
389	433	module = imp.load_source(module_name, absolute_path, python_file)

398	442	continue
399	443	elif ext == ".sql":
400	444	# A plain old .sql file, just read and execute it
	445	if is_worker:
	446	raise PrepareDatabaseException(
	447	UNAPPLIED_DELTA_ON_WORKER_ERROR % relative_path
	448	)
401	449	logger.info("Applying schema %s", relative_path)
402	450	executescript(cur, absolute_path)
403	451	elif ext == specific_engine_extension and root_name.endswith(".sql"):
404	452	# A .sql file specific to our engine; just read and execute it
	453	if is_worker:
	454	raise PrepareDatabaseException(
	455	UNAPPLIED_DELTA_ON_WORKER_ERROR % relative_path
	456	)
405	457	logger.info("Applying engine-specific schema %s", relative_path)
406	458	executescript(cur, absolute_path)
407	459	elif ext in specific_engine_extensions and root_name.endswith(".sql"):

430	482	),
431	483	(v, True),
432	484	)
	485
	486	logger.info("Schema now up to date")
433	487
434	488
435	489	def _apply_module_schemas(txn, database_engine, config):

568	622
569	623
570	624	@attr.s()
571		class _DirectoryListing(object):
	625	class _DirectoryListing:
572	626	"""Helper class to store schema file name and the
573	627	absolute path to it.
574	628

+0

-69

~~synapse/storage/presence.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		from collections import namedtuple
16
17		from synapse.api.constants import PresenceState
18
19
20		class UserPresenceState(
21		namedtuple(
22		"UserPresenceState",
23		(
24		"user_id",
25		"state",
26		"last_active_ts",
27		"last_federation_update_ts",
28		"last_user_sync_ts",
29		"status_msg",
30		"currently_active",
31		),
32		)
33		):
34		"""Represents the current presence state of the user.
35
36		user_id (str)
37		last_active (int): Time in msec that the user last interacted with server.
38		last_federation_update (int): Time in msec since either a) we sent a presence
39		update to other servers or b) we received a presence update, depending
40		on if is a local user or not.
41		last_user_sync (int): Time in msec that the user last completed a sync
42		(or event stream).
43		status_msg (str): User set status message.
44		"""
45
46		def as_dict(self):
47		return dict(self._asdict())
48
49		@staticmethod
50		def from_dict(d):
51		return UserPresenceState(**d)
52
53		def copy_and_replace(self, **kwargs):
54		return self._replace(**kwargs)
55
56		@classmethod
57		def default(cls, user_id):
58		"""Returns a default presence state.
59		"""
60		return cls(
61		user_id=user_id,
62		state=PresenceState.OFFLINE,
63		last_active_ts=0,
64		last_federation_update_ts=0,
65		last_user_sync_ts=0,
66		status_msg=None,
67		currently_active=False,
68		)

+1

-1

synapse/storage/purge_events.py less more

19	19	logger = logging.getLogger(__name__)
20	20
21	21
22		class PurgeEventsStorage(object):
	22	class PurgeEventsStorage:
23	23	"""High level interface for purging rooms and event history.
24	24	"""
25	25

+3

-3

synapse/storage/relations.py less more

22	22
23	23
24	24	@attr.s
25		class PaginationChunk(object):
	25	class PaginationChunk:
26	26	"""Returned by relation pagination APIs.
27	27
28	28	Attributes:

50	50
51	51
52	52	@attr.s(frozen=True, slots=True)
53		class RelationPaginationToken(object):
	53	class RelationPaginationToken:
54	54	"""Pagination token for relation pagination API.
55	55
56	56	As the results are in topological order, we can use the

81	81
82	82
83	83	@attr.s(frozen=True, slots=True)
84		class AggregationPaginationToken(object):
	84	class AggregationPaginationToken:
85	85	"""Pagination token for relation aggregation pagination API.
86	86
87	87	As the results are order by count and then MAX(stream_ordering) of the

+11

-11

synapse/storage/state.py less more

28	28
29	29
30	30	@attr.s(slots=True)
31		class StateFilter(object):
	31	class StateFilter:
32	32	"""A filter used when querying for state.
33	33
34	34	Attributes:

325	325	return member_filter, non_member_filter
326	326
327	327
328		class StateGroupStorage(object):
	328	class StateGroupStorage:
329	329	"""High level interface to fetching state for event.
330	330	"""
331	331
332	332	def __init__(self, hs, stores):
333	333	self.stores = stores
334	334
335		def get_state_group_delta(self, state_group: int):
	335	async def get_state_group_delta(self, state_group: int):
336	336	"""Given a state group try to return a previous group and a delta between
337	337	the old and the new.
338	338

340	340	state_group: The state group used to retrieve state deltas.
341	341
342	342	Returns:
343		Deferred[Tuple[Optional[int], Optional[StateMap[str]]]]:
	343	Tuple[Optional[int], Optional[StateMap[str]]]:
344	344	(prev_group, delta_ids)
345	345	"""
346	346
347		return self.stores.state.get_state_group_delta(state_group)
	347	return await self.stores.state.get_state_group_delta(state_group)
348	348
349	349	async def get_state_groups_ids(
350	350	self, _room_id: str, event_ids: Iterable[str]

524	524	state_filter: The state filter used to fetch state from the database.
525	525
526	526	Returns:
527		A deferred dict from (type, state_key) -> state_event
	527	A dict from (type, state_key) -> state_event
528	528	"""
529	529	state_map = await self.get_state_ids_for_events([event_id], state_filter)
530	530	return state_map[event_id]

545	545	"""
546	546	return self.stores.state._get_state_for_groups(groups, state_filter)
547	547
548		def store_state_group(
	548	async def store_state_group(
549	549	self,
550	550	event_id: str,
551	551	room_id: str,
552	552	prev_group: Optional[int],
553	553	delta_ids: Optional[dict],
554	554	current_state_ids: dict,
555		):
	555	) -> int:
556	556	"""Store a new set of state, returning a newly assigned state group.
557	557
558	558	Args:

566	566	to event_id.
567	567
568	568	Returns:
569		Deferred[int]: The state group ID
570		"""
571		return self.stores.state.store_state_group(
	569	The state group ID
	570	"""
	571	return await self.stores.state.store_state_group(
572	572	event_id, room_id, prev_group, delta_ids, current_state_ids
573	573	)

+162

-87

synapse/storage/util/id_generators.py less more

13	13	# limitations under the License.
14	14
15	15	import contextlib
	16	import heapq
	17	import logging
16	18	import threading
17	19	from collections import deque
18		from typing import Dict, Set, Tuple
	20	from typing import Dict, List, Set
19	21
20	22	from typing_extensions import Deque
21	23
22	24	from synapse.storage.database import DatabasePool, LoggingTransaction
23	25	from synapse.storage.util.sequence import PostgresSequenceGenerator
24	26
25
26		class IdGenerator(object):
	27	logger = logging.getLogger(__name__)
	28
	29
	30	class IdGenerator:
27	31	def __init__(self, db_conn, table, column):
28	32	self._lock = threading.Lock()
29	33	self._next_id = _load_current_id(db_conn, table, column)

46	50	Returns:
47	51	int
48	52	"""
	53	# debug logging for https://github.com/matrix-org/synapse/issues/7968
	54	logger.info("initialising stream generator for %s(%s)", table, column)
49	55	cur = db_conn.cursor()
50	56	if step == 1:
51	57	cur.execute("SELECT MAX(%s) FROM %s" % (column, table))

57	63	return (max if step > 0 else min)(current_id, step)
58	64
59	65
60		class StreamIdGenerator(object):
	66	class StreamIdGenerator:
61	67	"""Used to generate new stream ids when persisting events while keeping
62	68	track of which transactions have been completed.
63	69

79	85	upwards, -1 to grow downwards.
80	86
81	87	Usage:
82		with stream_id_gen.get_next() as stream_id:
	88	with await stream_id_gen.get_next() as stream_id:
83	89	# ... persist event ...
84	90	"""
85	91

94	100	)
95	101	self._unfinished_ids = deque() # type: Deque[int]
96	102
97		def get_next(self):
	103	async def get_next(self):
98	104	"""
99	105	Usage:
100		with stream_id_gen.get_next() as stream_id:
	106	with await stream_id_gen.get_next() as stream_id:
101	107	# ... persist event ...
102	108	"""
103	109	with self._lock:

116	122
117	123	return manager()
118	124
119		def get_next_mult(self, n):
	125	async def get_next_mult(self, n):
120	126	"""
121	127	Usage:
122		with stream_id_gen.get_next(n) as stream_ids:
	128	with await stream_id_gen.get_next(n) as stream_ids:
123	129	# ... persist events ...
124	130	"""
125	131	with self._lock:

157	163
158	164	return self._current
159	165
160
161		class ChainedIdGenerator(object):
162		"""Used to generate new stream ids where the stream must be kept in sync
163		with another stream. It generates pairs of IDs, the first element is an
164		integer ID for this stream, the second element is the ID for the stream
165		that this stream needs to be kept in sync with."""
166
167		def __init__(self, chained_generator, db_conn, table, column):
168		self.chained_generator = chained_generator
169		self._table = table
170		self._lock = threading.Lock()
171		self._current_max = _load_current_id(db_conn, table, column)
172		self._unfinished_ids = deque() # type: Deque[Tuple[int, int]]
173
174		def get_next(self):
175		"""
176		Usage:
177		with stream_id_gen.get_next() as (stream_id, chained_id):
178		# ... persist event ...
179		"""
180		with self._lock:
181		self._current_max += 1
182		next_id = self._current_max
183		chained_id = self.chained_generator.get_current_token()
184
185		self._unfinished_ids.append((next_id, chained_id))
186
187		@contextlib.contextmanager
188		def manager():
189		try:
190		yield (next_id, chained_id)
191		finally:
192		with self._lock:
193		self._unfinished_ids.remove((next_id, chained_id))
194
195		return manager()
196
197		def get_current_token(self):
198		"""Returns the maximum stream id such that all stream ids less than or
199		equal to it have been successfully persisted.
200		"""
201		with self._lock:
202		if self._unfinished_ids:
203		stream_id, chained_id = self._unfinished_ids[0]
204		return stream_id - 1, chained_id
205
206		return self._current_max, self.chained_generator.get_current_token()
207
208		def advance(self, token: int):
209		"""Stub implementation for advancing the token when receiving updates
210		over replication; raises an exception as this instance should be the
211		only source of updates.
212		"""
213
214		raise Exception(
215		"Attempted to advance token on source for table %r", self._table
216		)
	166	def get_current_token_for_writer(self, instance_name: str) -> int:
	167	"""Returns the position of the given writer.
	168
	169	For streams with single writers this is equivalent to
	170	`get_current_token`.
	171	"""
	172	return self.get_current_token()
217	173
218	174
219	175	class MultiWriterIdGenerator:

233	189	id_column: Column that stores the stream ID.
234	190	sequence_name: The name of the postgres sequence used to generate new
235	191	IDs.
	192	positive: Whether the IDs are positive (true) or negative (false).
	193	When using negative IDs we go backwards from -1 to -2, -3, etc.
236	194	"""
237	195
238	196	def __init__(

244	202	instance_column: str,
245	203	id_column: str,
246	204	sequence_name: str,
	205	positive: bool = True,
247	206	):
248	207	self._db = db
249	208	self._instance_name = instance_name
	209	self._positive = positive
	210	self._return_factor = 1 if positive else -1
250	211
251	212	# We lock as some functions may be called from DB threads.
252	213	self._lock = threading.Lock()
253	214
	215	# Note: If we are a negative stream then we still store all the IDs as
	216	# positive to make life easier for us, and simply negate the IDs when we
	217	# return them.
254	218	self._current_positions = self._load_current_ids(
255	219	db_conn, table, instance_column, id_column
256	220	)

259	223	# should be less than the minimum of this set (if not empty).
260	224	self._unfinished_ids = set() # type: Set[int]
261	225
	226	# We track the max position where we know everything before has been
	227	# persisted. This is done by a) looking at the min across all instances
	228	# and b) noting that if we have seen a run of persisted positions
	229	# without gaps (e.g. 5, 6, 7) then we can skip forward (e.g. to 7).
	230	#
	231	# Note: There is no guarentee that the IDs generated by the sequence
	232	# will be gapless; gaps can form when e.g. a transaction was rolled
	233	# back. This means that sometimes we won't be able to skip forward the
	234	# position even though everything has been persisted. However, since
	235	# gaps should be relatively rare it's still worth doing the book keeping
	236	# that allows us to skip forwards when there are gapless runs of
	237	# positions.
	238	self._persisted_upto_position = (
	239	min(self._current_positions.values()) if self._current_positions else 0
	240	)
	241	self._known_persisted_positions = [] # type: List[int]
	242
262	243	self._sequence_gen = PostgresSequenceGenerator(sequence_name)
263	244
264	245	def _load_current_ids(
265	246	self, db_conn, table: str, instance_column: str, id_column: str
266	247	) -> Dict[str, int]:
	248	# If positive stream aggregate via MAX. For negative stream use MIN
	249	# and negate the result to get a positive number.
267	250	sql = """
268		SELECT %(instance)s, MAX(%(id)s) FROM %(table)s
	251	SELECT %(instance)s, %(agg)s(%(id)s) FROM %(table)s
269	252	GROUP BY %(instance)s
270	253	""" % {
271	254	"instance": instance_column,
272	255	"id": id_column,
273	256	"table": table,
	257	"agg": "MAX" if self._positive else "-MIN",
274	258	}
275	259
276	260	cur = db_conn.cursor()

283	267
284	268	return current_positions
285	269
286		def _load_next_id_txn(self, txn):
	270	def _load_next_id_txn(self, txn) -> int:
287	271	return self._sequence_gen.get_next_id_txn(txn)
	272
	273	def _load_next_mult_id_txn(self, txn, n: int) -> List[int]:
	274	return self._sequence_gen.get_next_mult_txn(txn, n)
288	275
289	276	async def get_next(self):
290	277	"""

297	284	# Assert the fetched ID is actually greater than what we currently
298	285	# believe the ID to be. If not, then the sequence and table have got
299	286	# out of sync somehow.
300		assert self.get_current_token() < next_id
301
302		with self._lock:
	287	with self._lock:
	288	assert self._current_positions.get(self._instance_name, 0) < next_id
	289
303	290	self._unfinished_ids.add(next_id)
304	291
305	292	@contextlib.contextmanager
306	293	def manager():
307	294	try:
308		yield next_id
	295	# Multiply by the return factor so that the ID has correct sign.
	296	yield self._return_factor * next_id
309	297	finally:
310	298	self._mark_id_as_finished(next_id)
311	299
312	300	return manager()
313	301
	302	async def get_next_mult(self, n: int):
	303	"""
	304	Usage:
	305	with await stream_id_gen.get_next_mult(5) as stream_ids:
	306	# ... persist events ...
	307	"""
	308	next_ids = await self._db.runInteraction(
	309	"_load_next_mult_id", self._load_next_mult_id_txn, n
	310	)
	311
	312	# Assert the fetched ID is actually greater than any ID we've already
	313	# seen. If not, then the sequence and table have got out of sync
	314	# somehow.
	315	with self._lock:
	316	assert max(self._current_positions.values(), default=0) < min(next_ids)
	317
	318	self._unfinished_ids.update(next_ids)
	319
	320	@contextlib.contextmanager
	321	def manager():
	322	try:
	323	yield [self._return_factor * i for i in next_ids]
	324	finally:
	325	for i in next_ids:
	326	self._mark_id_as_finished(i)
	327
	328	return manager()
	329
314	330	def get_next_txn(self, txn: LoggingTransaction):
315	331	"""
316	332	Usage:

327	343	txn.call_after(self._mark_id_as_finished, next_id)
328	344	txn.call_on_exception(self._mark_id_as_finished, next_id)
329	345
330		return next_id
	346	return self._return_factor * next_id
331	347
332	348	def _mark_id_as_finished(self, next_id: int):
333	349	"""The ID has finished being processed so we should advance the

343	359	curr = self._current_positions.get(self._instance_name, 0)
344	360	self._current_positions[self._instance_name] = max(curr, next_id)
345	361
346		def get_current_token(self, instance_name: str = None) -> int:
347		"""Gets the current position of a named writer (defaults to current
348		instance).
349
350		Returns 0 if we don't have a position for the named writer (likely due
351		to it being a new writer).
352		"""
353
354		if instance_name is None:
355		instance_name = self._instance_name
356
357		with self._lock:
358		return self._current_positions.get(instance_name, 0)
	362	self._add_persisted_position(next_id)
	363
	364	def get_current_token(self) -> int:
	365	"""Returns the maximum stream id such that all stream ids less than or
	366	equal to it have been successfully persisted.
	367	"""
	368
	369	# Currently we don't support this operation, as it's not obvious how to
	370	# condense the stream positions of multiple writers into a single int.
	371	raise NotImplementedError()
	372
	373	def get_current_token_for_writer(self, instance_name: str) -> int:
	374	"""Returns the position of the given writer.
	375	"""
	376
	377	with self._lock:
	378	return self._return_factor * self._current_positions.get(instance_name, 0)
359	379
360	380	def get_positions(self) -> Dict[str, int]:
361	381	"""Get a copy of the current positon map.
362	382	"""
363	383
364	384	with self._lock:
365		return dict(self._current_positions)
	385	return {
	386	name: self._return_factor * i
	387	for name, i in self._current_positions.items()
	388	}
366	389
367	390	def advance(self, instance_name: str, new_id: int):
368	391	"""Advance the postion of the named writer to the given ID, if greater
369	392	than existing entry.
370	393	"""
371	394
	395	new_id *= self._return_factor
	396
372	397	with self._lock:
373	398	self._current_positions[instance_name] = max(
374	399	new_id, self._current_positions.get(instance_name, 0)
375	400	)
	401
	402	self._add_persisted_position(new_id)
	403
	404	def get_persisted_upto_position(self) -> int:
	405	"""Get the max position where all previous positions have been
	406	persisted.
	407
	408	Note: In the worst case scenario this will be equal to the minimum
	409	position across writers. This means that the returned position here can
	410	lag if one writer doesn't write very often.
	411	"""
	412
	413	with self._lock:
	414	return self._return_factor * self._persisted_upto_position
	415
	416	def _add_persisted_position(self, new_id: int):
	417	"""Record that we have persisted a position.
	418
	419	This is used to keep the `_current_positions` up to date.
	420	"""
	421
	422	# We require that the lock is locked by caller
	423	assert self._lock.locked()
	424
	425	heapq.heappush(self._known_persisted_positions, new_id)
	426
	427	# We move the current min position up if the minimum current positions
	428	# of all instances is higher (since by definition all positions less
	429	# that that have been persisted).
	430	min_curr = min(self._current_positions.values())
	431	self._persisted_upto_position = max(min_curr, self._persisted_upto_position)
	432
	433	# We now iterate through the seen positions, discarding those that are
	434	# less than the current min positions, and incrementing the min position
	435	# if its exactly one greater.
	436	#
	437	# This is also where we discard items from `_known_persisted_positions`
	438	# (to ensure the list doesn't infinitely grow).
	439	while self._known_persisted_positions:
	440	if self._known_persisted_positions[0] <= self._persisted_upto_position:
	441	heapq.heappop(self._known_persisted_positions)
	442	elif (
	443	self._known_persisted_positions[0] == self._persisted_upto_position + 1
	444	):
	445	heapq.heappop(self._known_persisted_positions)
	446	self._persisted_upto_position += 1
	447	else:
	448	# There was a gap in seen positions, so there is nothing more to
	449	# do.
	450	break

+7

-1

synapse/storage/util/sequence.py less more

13	13	# limitations under the License.
14	14	import abc
15	15	import threading
16		from typing import Callable, Optional
	16	from typing import Callable, List, Optional
17	17
18	18	from synapse.storage.engines import BaseDatabaseEngine, PostgresEngine
19	19	from synapse.storage.types import Cursor

37	37	def get_next_id_txn(self, txn: Cursor) -> int:
38	38	txn.execute("SELECT nextval(?)", (self._sequence_name,))
39	39	return txn.fetchone()[0]
	40
	41	def get_next_mult_txn(self, txn: Cursor, n: int) -> List[int]:
	42	txn.execute(
	43	"SELECT nextval(?) FROM generate_series(1, ?)", (self._sequence_name, n)
	44	)
	45	return [i for (i,) in txn]
40	46
41	47
42	48	GetFirstCallbackType = Callable[[Cursor], int]

+2

-2

synapse/streams/config.py less more

24	24	MAX_LIMIT = 1000
25	25
26	26
27		class SourcePaginationConfig(object):
	27	class SourcePaginationConfig:
28	28
29	29	"""A configuration object which stores pagination parameters for a
30	30	specific event source."""

44	44	)
45	45
46	46
47		class PaginationConfig(object):
	47	class PaginationConfig:
48	48
49	49	"""A configuration object which stores pagination parameters."""
50	50

+2

-2

synapse/streams/events.py less more

22	22	from synapse.types import StreamToken
23	23
24	24
25		class EventSources(object):
	25	class EventSources:
26	26	SOURCE_TYPES = {
27	27	"room": RoomEventSource,
28	28	"presence": PresenceEventSource,

38	38	self.store = hs.get_datastore()
39	39
40	40	def get_current_token(self) -> StreamToken:
41		push_rules_key, _ = self.store.get_push_rules_stream_token()
	41	push_rules_key = self.store.get_max_push_rules_stream_id()
42	42	to_device_key = self.store.get_to_device_stream_token()
43	43	device_list_key = self.store.get_device_stream_token()
44	44	groups_key = self.store.get_group_stream_token()

+27

-7

synapse/types.py less more

17	17	import string
18	18	import sys
19	19	from collections import namedtuple
20		from typing import Any, Dict, Tuple, Type, TypeVar
	20	from typing import Any, Dict, Mapping, MutableMapping, Tuple, Type, TypeVar
21	21
22	22	import attr
23	23	from signedjson.key import decode_verify_key_bytes

40	40	# Define a state map type from type/state_key to T (usually an event ID or
41	41	# event)
42	42	T = TypeVar("T")
43		StateMap = Dict[Tuple[str, str], T]
44
	43	StateKey = Tuple[str, str]
	44	StateMap = Mapping[StateKey, T]
	45	MutableStateMap = MutableMapping[StateKey, T]
45	46
46	47	# the type of a JSON-serialisable dict. This could be made stronger, but it will
47	48	# do for now.

50	51
51	52	class Requester(
52	53	namedtuple(
53		"Requester", ["user", "access_token_id", "is_guest", "device_id", "app_service"]
	54	"Requester",
	55	[
	56	"user",
	57	"access_token_id",
	58	"is_guest",
	59	"shadow_banned",
	60	"device_id",
	61	"app_service",
	62	],
54	63	)
55	64	):
56	65	"""

61	70	access_token_id (int\|None): ID of the access token used for this
62	71	request, or None if it came via the appservice API or similar
63	72	is_guest (bool): True if the user making this request is a guest user
	73	shadow_banned (bool): True if the user making this request has been shadow-banned.
64	74	device_id (str\|None): device_id which was set at authentication time
65	75	app_service (ApplicationService\|None): the AS requesting on behalf of the user
66	76	"""

76	86	"user_id": self.user.to_string(),
77	87	"access_token_id": self.access_token_id,
78	88	"is_guest": self.is_guest,
	89	"shadow_banned": self.shadow_banned,
79	90	"device_id": self.device_id,
80	91	"app_server_id": self.app_service.id if self.app_service else None,
81	92	}

100	111	user=UserID.from_string(input["user_id"]),
101	112	access_token_id=input["access_token_id"],
102	113	is_guest=input["is_guest"],
	114	shadow_banned=input["shadow_banned"],
103	115	device_id=input["device_id"],
104	116	app_service=appservice,
105	117	)
106	118
107	119
108	120	def create_requester(
109		user_id, access_token_id=None, is_guest=False, device_id=None, app_service=None
	121	user_id,
	122	access_token_id=None,
	123	is_guest=False,
	124	shadow_banned=False,
	125	device_id=None,
	126	app_service=None,
110	127	):
111	128	"""
112	129	Create a new ``Requester`` object

116	133	access_token_id (int\|None): ID of the access token used for this
117	134	request, or None if it came via the appservice API or similar
118	135	is_guest (bool): True if the user making this request is a guest user
	136	shadow_banned (bool): True if the user making this request is shadow-banned.
119	137	device_id (str\|None): device_id which was set at authentication time
120	138	app_service (ApplicationService\|None): the AS requesting on behalf of the user
121	139

124	142	"""
125	143	if not isinstance(user_id, UserID):
126	144	user_id = UserID.from_string(user_id)
127		return Requester(user_id, access_token_id, is_guest, device_id, app_service)
	145	return Requester(
	146	user_id, access_token_id, is_guest, shadow_banned, device_id, app_service
	147	)
128	148
129	149
130	150	def get_domain_from_id(string):

508	528
509	529
510	530	@attr.s(slots=True)
511		class ReadReceipt(object):
	531	class ReadReceipt:
512	532	"""Information about a read-receipt"""
513	533
514	534	room_id = attr.ib()

+13

-3

synapse/util/__init__.py less more

24	24
25	25	logger = logging.getLogger(__name__)
26	26
27		# Create a custom encoder to reduce the whitespace produced by JSON encoding.
28		json_encoder = json.JSONEncoder(separators=(",", ":"))
	27
	28	def _reject_invalid_json(val):
	29	"""Do not allow Infinity, -Infinity, or NaN values in JSON."""
	30	raise ValueError("Invalid JSON value: '%s'" % val)
	31
	32
	33	# Create a custom encoder to reduce the whitespace produced by JSON encoding and
	34	# ensure that valid JSON is produced.
	35	json_encoder = json.JSONEncoder(allow_nan=False, separators=(",", ":"))
	36
	37	# Create a custom decoder to reject Python extensions to JSON.
	38	json_decoder = json.JSONDecoder(parse_constant=_reject_invalid_json)
29	39
30	40
31	41	def unwrapFirstError(failure):

35	45
36	46
37	47	@attr.s
38		class Clock(object):
	48	class Clock:
39	49	"""
40	50	A Clock wraps a Twisted reactor and provides utilities on top of it.
41	51

+12

-12

synapse/util/async_helpers.py less more

19	19	from typing import Dict, Sequence, Set, Union
20	20
21	21	import attr
	22	from typing_extensions import ContextManager
22	23
23	24	from twisted.internet import defer
24	25	from twisted.internet.defer import CancelledError

34	35	logger = logging.getLogger(__name__)
35	36
36	37
37		class ObservableDeferred(object):
	38	class ObservableDeferred:
38	39	"""Wraps a deferred object so that we can add observer deferreds. These
39	40	observer deferreds do not affect the callback chain of the original
40	41	deferred.

186	187	).addErrback(unwrapFirstError)
187	188
188	189
189		class Linearizer(object):
	190	class Linearizer:
190	191	"""Limits concurrent access to resources based on a key. Useful to ensure
191	192	only a few things happen at a time on a given resource.
192	193

336	337	return new_defer
337	338
338	339
339		class ReadWriteLock(object):
340		"""A deferred style read write lock.
	340	class ReadWriteLock:
	341	"""An async read write lock.
341	342
342	343	Example:
343	344
344		with (yield read_write_lock.read("test_key")):
	345	with await read_write_lock.read("test_key"):
345	346	# do some work
346	347	"""
347	348

364	365	# Latest writer queued
365	366	self.key_to_current_writer = {} # type: Dict[str, defer.Deferred]
366	367
367		@defer.inlineCallbacks
368		def read(self, key):
	368	async def read(self, key: str) -> ContextManager:
369	369	new_defer = defer.Deferred()
370	370
371	371	curr_readers = self.key_to_current_readers.setdefault(key, set())

375	375
376	376	# We wait for the latest writer to finish writing. We can safely ignore
377	377	# any existing readers... as they're readers.
378		yield make_deferred_yieldable(curr_writer)
	378	if curr_writer:
	379	await make_deferred_yieldable(curr_writer)
379	380
380	381	@contextmanager
381	382	def _ctx_manager():

387	388
388	389	return _ctx_manager()
389	390
390		@defer.inlineCallbacks
391		def write(self, key):
	391	async def write(self, key: str) -> ContextManager:
392	392	new_defer = defer.Deferred()
393	393
394	394	curr_readers = self.key_to_current_readers.get(key, set())

404	404	curr_readers.clear()
405	405	self.key_to_current_writer[key] = new_defer
406	406
407		yield make_deferred_yieldable(defer.gatherResults(to_wait_on))
	407	await make_deferred_yieldable(defer.gatherResults(to_wait_on))
408	408
409	409	@contextmanager
410	410	def _ctx_manager():

501	501
502	502
503	503	@attr.s(slots=True, frozen=True)
504		class DoneAwaitable(object):
	504	class DoneAwaitable:
505	505	"""Simple awaitable that returns the provided value.
506	506	"""
507	507

+1

-1

synapse/util/caches/__init__.py less more

42	42
43	43
44	44	@attr.s
45		class CacheMetric(object):
	45	class CacheMetric:
46	46
47	47	_cache = attr.ib()
48	48	_cache_type = attr.ib(type=str)

+41

-65

synapse/util/caches/descriptors.py less more

17	17	import inspect
18	18	import logging
19	19	import threading
20		from typing import Any, Tuple, Union, cast
	20	from typing import Any, Callable, Generic, Optional, Tuple, TypeVar, Union, cast
21	21	from weakref import WeakValueDictionary
22	22
23	23	from prometheus_client import Gauge
24		from typing_extensions import Protocol
25	24
26	25	from twisted.internet import defer
27	26

37	36
38	37	CacheKey = Union[Tuple, Any]
39	38
40
41		class _CachedFunction(Protocol):
	39	F = TypeVar("F", bound=Callable[..., Any])
	40
	41
	42	class _CachedFunction(Generic[F]):
42	43	invalidate = None # type: Any
43	44	invalidate_all = None # type: Any
44	45	invalidate_many = None # type: Any

46	47	cache = None # type: Any
47	48	num_args = None # type: Any
48	49
49		def __name__(self):
50		...
	50	__name__ = None # type: str
	51
	52	# Note: This function signature is actually fiddled with by the synapse mypy
	53	# plugin to a) make it a bound method, and b) remove any `cache_context` arg.
	54	__call__ = None # type: F
51	55
52	56
53	57	cache_pending_metric = Gauge(

59	63	_CacheSentinel = object()
60	64
61	65
62		class CacheEntry(object):
	66	class CacheEntry:
63	67	__slots__ = ["deferred", "callbacks", "invalidated"]
64	68
65	69	def __init__(self, deferred, callbacks):

75	79	self.callbacks.clear()
76	80
77	81
78		class Cache(object):
	82	class Cache:
79	83	__slots__ = (
80	84	"cache",
81	85	"name",

122	126
123	127	self.name = name
124	128	self.keylen = keylen
125		self.thread = None
	129	self.thread = None # type: Optional[threading.Thread]
126	130	self.metrics = register_cache(
127	131	"cache",
128	132	name,

283	287	self._pending_deferred_cache.clear()
284	288
285	289
286		class _CacheDescriptorBase(object):
287		def __init__(
288		self, orig: _CachedFunction, num_args, inlineCallbacks, cache_context=False
289		):
	290	class _CacheDescriptorBase:
	291	def __init__(self, orig: _CachedFunction, num_args, cache_context=False):
290	292	self.orig = orig
291
292		if inlineCallbacks:
293		self.function_to_call = defer.inlineCallbacks(orig)
294		else:
295		self.function_to_call = orig
296	293
297	294	arg_spec = inspect.getfullargspec(orig)
298	295	all_args = arg_spec.args

363	360	invalidated) by adding a special "cache_context" argument to the function
364	361	and passing that as a kwarg to all caches called. For example::
365	362
366		@cachedInlineCallbacks(cache_context=True)
	363	@cached(cache_context=True)
367	364	def foo(self, key, cache_context):
368	365	r1 = yield self.bar1(key, on_invalidate=cache_context.invalidate)
369	366	r2 = yield self.bar2(key, on_invalidate=cache_context.invalidate)

381	378	max_entries=1000,
382	379	num_args=None,
383	380	tree=False,
384		inlineCallbacks=False,
385	381	cache_context=False,
386	382	iterable=False,
387	383	):
388	384
389		super(CacheDescriptor, self).__init__(
390		orig,
391		num_args=num_args,
392		inlineCallbacks=inlineCallbacks,
393		cache_context=cache_context,
394		)
	385	super().__init__(orig, num_args=num_args, cache_context=cache_context)
395	386
396	387	self.max_entries = max_entries
397	388	self.tree = tree

464	455	observer = defer.succeed(cached_result_d)
465	456
466	457	except KeyError:
467		ret = defer.maybeDeferred(
468		preserve_fn(self.function_to_call), obj, args, *kwargs
469		)
	458	ret = defer.maybeDeferred(preserve_fn(self.orig), obj, args, *kwargs)
470	459
471	460	def onErr(f):
472	461	cache.invalidate(cache_key)

509	498	of results.
510	499	"""
511	500
512		def __init__(
513		self, orig, cached_method_name, list_name, num_args=None, inlineCallbacks=False
514		):
	501	def __init__(self, orig, cached_method_name, list_name, num_args=None):
515	502	"""
516	503	Args:
517	504	orig (function)

520	507	num_args (int): number of positional arguments (excluding ``self``,
521	508	but including list_name) to use as cache keys. Defaults to all
522	509	named args of the function.
523		inlineCallbacks (bool): Whether orig is a generator that should
524		be wrapped by defer.inlineCallbacks
525	510	"""
526		super(CacheListDescriptor, self).__init__(
527		orig, num_args=num_args, inlineCallbacks=inlineCallbacks
528		)
	511	super().__init__(orig, num_args=num_args)
529	512
530	513	self.list_name = list_name
531	514

630	613
631	614	cached_defers.append(
632	615	defer.maybeDeferred(
633		preserve_fn(self.function_to_call), **args_to_call
	616	preserve_fn(self.orig), **args_to_call
634	617	).addCallbacks(complete_all, errback)
635	618	)
636	619

682	665
683	666
684	667	def cached(
685		max_entries=1000, num_args=None, tree=False, cache_context=False, iterable=False
686		):
687		return lambda orig: CacheDescriptor(
	668	max_entries: int = 1000,
	669	num_args: Optional[int] = None,
	670	tree: bool = False,
	671	cache_context: bool = False,
	672	iterable: bool = False,
	673	) -> Callable[[F], _CachedFunction[F]]:
	674	func = lambda orig: CacheDescriptor(
688	675	orig,
689	676	max_entries=max_entries,
690	677	num_args=num_args,

693	680	iterable=iterable,
694	681	)
695	682
696
697		def cachedInlineCallbacks(
698		max_entries=1000, num_args=None, tree=False, cache_context=False, iterable=False
699		):
700		return lambda orig: CacheDescriptor(
701		orig,
702		max_entries=max_entries,
703		num_args=num_args,
704		tree=tree,
705		inlineCallbacks=True,
706		cache_context=cache_context,
707		iterable=iterable,
708		)
709
710
711		def cachedList(cached_method_name, list_name, num_args=None, inlineCallbacks=False):
	683	return cast(Callable[[F], _CachedFunction[F]], func)
	684
	685
	686	def cachedList(
	687	cached_method_name: str, list_name: str, num_args: Optional[int] = None
	688	) -> Callable[[F], _CachedFunction[F]]:
712	689	"""Creates a descriptor that wraps a function in a `CacheListDescriptor`.
713	690
714	691	Used to do batch lookups for an already created cache. A single argument

718	695	cache.
719	696
720	697	Args:
721		cached_method_name (str): The name of the single-item lookup method.
	698	cached_method_name: The name of the single-item lookup method.
722	699	This is only used to find the cache to use.
723		list_name (str): The name of the argument that is the list to use to
	700	list_name: The name of the argument that is the list to use to
724	701	do batch lookups in the cache.
725		num_args (int): Number of arguments to use as the key in the cache
	702	num_args: Number of arguments to use as the key in the cache
726	703	(including list_name). Defaults to all named parameters.
727		inlineCallbacks (bool): Should the function be wrapped in an
728		`defer.inlineCallbacks`?
729	704
730	705	Example:
731	706
732		class Example(object):
	707	class Example:
733	708	@cached(num_args=2)
734	709	def do_something(self, first_arg):
735	710	...

738	713	def batch_do_something(self, first_arg, second_args):
739	714	...
740	715	"""
741		return lambda orig: CacheListDescriptor(
	716	func = lambda orig: CacheListDescriptor(
742	717	orig,
743	718	cached_method_name=cached_method_name,
744	719	list_name=list_name,
745	720	num_args=num_args,
746		inlineCallbacks=inlineCallbacks,
747	721	)
	722
	723	return cast(Callable[[F], _CachedFunction[F]], func)

+2

-2

synapse/util/caches/dictionary_cache.py less more

39	39	return len(self.value)
40	40
41	41
42		class DictionaryCache(object):
	42	class DictionaryCache:
43	43	"""Caches key -> dictionary lookups, supporting caching partial dicts, i.e.
44	44	fetching a subset of dictionary keys for a particular key.
45	45	"""

52	52	self.thread = None
53	53	# caches_by_name[name] = self.cache
54	54
55		class Sentinel(object):
	55	class Sentinel:
56	56	__slots__ = []
57	57
58	58	self.sentinel = Sentinel()

+2

-2

synapse/util/caches/expiringcache.py less more

25	25	SENTINEL = object()
26	26
27	27
28		class ExpiringCache(object):
	28	class ExpiringCache:
29	29	def __init__(
30	30	self,
31	31	cache_name,

189	189	return False
190	190
191	191
192		class _CacheEntry(object):
	192	class _CacheEntry:
193	193	__slots__ = ["time", "value"]
194	194
195	195	def __init__(self, time, value):

+2

-2

synapse/util/caches/lrucache.py less more

29	29	yield m
30	30
31	31
32		class _Node(object):
	32	class _Node:
33	33	__slots__ = ["prev_node", "next_node", "key", "value", "callbacks"]
34	34
35	35	def __init__(self, prev_node, next_node, key, value, callbacks=set()):

40	40	self.callbacks = callbacks
41	41
42	42
43		class LruCache(object):
	43	class LruCache:
44	44	"""
45	45	Least-recently-used cache.
46	46	Supports del_multi only if cache_type=TreeCache

+1

-1

synapse/util/caches/response_cache.py less more

22	22	logger = logging.getLogger(__name__)
23	23
24	24
25		class ResponseCache(object):
	25	class ResponseCache:
26	26	"""
27	27	This caches a deferred response. Until the deferred completes it will be
28	28	returned from the cache. This means that if the client retries the request

+2

-2

synapse/util/caches/treecache.py less more

2	2	SENTINEL = object()
3	3
4	4
5		class TreeCache(object):
	5	class TreeCache:
6	6	"""
7	7	Tree-based backing store for LruCache. Allows subtrees of data to be deleted
8	8	efficiently.

88	88	yield d
89	89
90	90
91		class _Entry(object):
	91	class _Entry:
92	92	__slots__ = ["value"]
93	93
94	94	def __init__(self, value):

+2

-2

synapse/util/caches/ttlcache.py less more

25	25	SENTINEL = object()
26	26
27	27
28		class TTLCache(object):
	28	class TTLCache:
29	29	"""A key/value cache implementation where each entry has its own TTL"""
30	30
31	31	def __init__(self, cache_name, timer=time.time):

153	153
154	154
155	155	@attr.s(frozen=True, slots=True)
156		class _CacheEntry(object):
	156	class _CacheEntry:
157	157	"""TTLCache entry"""
158	158
159	159	# expiry_time is the first attribute, so that entries are sorted by expiry.

+2

-2

synapse/util/distributor.py less more

33	33	distributor.fire("user_joined_room", user=user, room_id=room_id)
34	34
35	35
36		class Distributor(object):
	36	class Distributor:
37	37	"""A central dispatch point for loosely-connected pieces of code to
38	38	register, observe, and fire signals.
39	39

102	102	return succeed(result)
103	103
104	104
105		class Signal(object):
	105	class Signal:
106	106	"""A Signal is a dispatch point that stores a list of callables as
107	107	observers of it.
108	108

+1

-1

synapse/util/file_consumer.py less more

19	19	from synapse.logging.context import make_deferred_yieldable, run_in_background
20	20
21	21
22		class BackgroundFileConsumer(object):
	22	class BackgroundFileConsumer:
23	23	"""A consumer that writes to a file like object. Supports both push
24	24	and pull producers
25	25

+1

-1

synapse/util/jsonobject.py less more

13	13	# limitations under the License.
14	14
15	15
16		class JsonEncodedObject(object):
	16	class JsonEncodedObject:
17	17	""" A common base class for defining protocol units that are represented
18	18	as JSON.
19	19

+1

-1

synapse/util/metrics.py less more

92	92	return wrapper
93	93
94	94
95		class Measure(object):
	95	class Measure:
96	96	__slots__ = [
97	97	"clock",
98	98	"name",

+2

-2

synapse/util/ratelimitutils.py less more

28	28	logger = logging.getLogger(__name__)
29	29
30	30
31		class FederationRateLimiter(object):
	31	class FederationRateLimiter:
32	32	def __init__(self, clock, config):
33	33	"""
34	34	Args:

59	59	return self.ratelimiters[host].ratelimit()
60	60
61	61
62		class _PerHostRatelimiter(object):
	62	class _PerHostRatelimiter:
63	63	def __init__(self, clock, config):
64	64	"""
65	65	Args:

+1

-1

synapse/util/retryutils.py less more

113	113	)
114	114
115	115
116		class RetryDestinationLimiter(object):
	116	class RetryDestinationLimiter:
117	117	def __init__(
118	118	self,
119	119	destination,

+1

-3

synapse/util/stringutils.py less more

23	23	_string_with_symbols = string.digits + string.ascii_letters + ".,;:^&*-_+=#~@"
24	24
25	25	# https://matrix.org/docs/spec/client_server/r0.6.0#post-matrix-client-r0-register-email-requesttoken
26		# Note: The : character is allowed here for older clients, but will be removed in a
27		# future release. Context: https://github.com/matrix-org/synapse/issues/6766
28		client_secret_regex = re.compile(r"^[0-9a-zA-Z\.\=\_\-\:]+$")
	26	client_secret_regex = re.compile(r"^[0-9a-zA-Z\.\=\_\-]+$")
29	27
30	28	# random_string and random_string_with_symbols are used for a range of things,
31	29	# some cryptographically important, some less so. We use SystemRandom to make sure

+2

-2

synapse/util/wheel_timer.py less more

13	13	# limitations under the License.
14	14
15	15
16		class _Entry(object):
	16	class _Entry:
17	17	__slots__ = ["end_key", "queue"]
18	18
19	19	def __init__(self, end_key):

21	21	self.queue = []
22	22
23	23
24		class WheelTimer(object):
	24	class WheelTimer:
25	25	"""Stores arbitrary objects that will be returned after their timers have
26	26	expired.
27	27	"""

+8

-0

sytest-blacklist less more

35	35
36	36	# Blacklisted until https://github.com/matrix-org/synapse/pull/6486 lands
37	37	Can upload self-signing keys
	38
	39	# Blacklisted until MSC2753 is implemented
	40	Local users can peek into world_readable rooms by room ID
	41	We can't peek into rooms with shared history_visibility
	42	We can't peek into rooms with invited history_visibility
	43	We can't peek into rooms with joined history_visibility
	44	Local users can peek by room alias
	45	Peeked rooms only turn up in the sync for the device who peeked them

+1

-1

tests/api/test_auth.py less more

35	35	from tests.utils import mock_getRawHeaders, setup_test_homeserver
36	36
37	37
38		class TestHandlers(object):
	38	class TestHandlers:
39	39	def __init__(self, hs):
40	40	self.auth_handler = synapse.handlers.auth.AuthHandler(hs)
41	41

+24

-12

tests/api/test_filtering.py less more

368	368	@defer.inlineCallbacks
369	369	def test_filter_presence_match(self):
370	370	user_filter_json = {"presence": {"types": ["m.*"]}}
371		filter_id = yield self.datastore.add_user_filter(
372		user_localpart=user_localpart, user_filter=user_filter_json
	371	filter_id = yield defer.ensureDeferred(
	372	self.datastore.add_user_filter(
	373	user_localpart=user_localpart, user_filter=user_filter_json
	374	)
373	375	)
374	376	event = MockEvent(sender="@foo:bar", type="m.profile")
375	377	events = [event]

387	389	def test_filter_presence_no_match(self):
388	390	user_filter_json = {"presence": {"types": ["m.*"]}}
389	391
390		filter_id = yield self.datastore.add_user_filter(
391		user_localpart=user_localpart + "2", user_filter=user_filter_json
	392	filter_id = yield defer.ensureDeferred(
	393	self.datastore.add_user_filter(
	394	user_localpart=user_localpart + "2", user_filter=user_filter_json
	395	)
392	396	)
393	397	event = MockEvent(
394	398	event_id="$asdasd:localhost",

409	413	@defer.inlineCallbacks
410	414	def test_filter_room_state_match(self):
411	415	user_filter_json = {"room": {"state": {"types": ["m.*"]}}}
412		filter_id = yield self.datastore.add_user_filter(
413		user_localpart=user_localpart, user_filter=user_filter_json
	416	filter_id = yield defer.ensureDeferred(
	417	self.datastore.add_user_filter(
	418	user_localpart=user_localpart, user_filter=user_filter_json
	419	)
414	420	)
415	421	event = MockEvent(sender="@foo:bar", type="m.room.topic", room_id="!foo:bar")
416	422	events = [event]

427	433	@defer.inlineCallbacks
428	434	def test_filter_room_state_no_match(self):
429	435	user_filter_json = {"room": {"state": {"types": ["m.*"]}}}
430		filter_id = yield self.datastore.add_user_filter(
431		user_localpart=user_localpart, user_filter=user_filter_json
	436	filter_id = yield defer.ensureDeferred(
	437	self.datastore.add_user_filter(
	438	user_localpart=user_localpart, user_filter=user_filter_json
	439	)
432	440	)
433	441	event = MockEvent(
434	442	sender="@foo:bar", type="org.matrix.custom.event", room_id="!foo:bar"

464	472	def test_add_filter(self):
465	473	user_filter_json = {"room": {"state": {"types": ["m.*"]}}}
466	474
467		filter_id = yield self.filtering.add_user_filter(
468		user_localpart=user_localpart, user_filter=user_filter_json
	475	filter_id = yield defer.ensureDeferred(
	476	self.filtering.add_user_filter(
	477	user_localpart=user_localpart, user_filter=user_filter_json
	478	)
469	479	)
470	480
471	481	self.assertEquals(filter_id, 0)

484	494	def test_get_filter(self):
485	495	user_filter_json = {"room": {"state": {"types": ["m.*"]}}}
486	496
487		filter_id = yield self.datastore.add_user_filter(
488		user_localpart=user_localpart, user_filter=user_filter_json
	497	filter_id = yield defer.ensureDeferred(
	498	self.datastore.add_user_filter(
	499	user_localpart=user_localpart, user_filter=user_filter_json
	500	)
489	501	)
490	502
491	503	filter = yield defer.ensureDeferred(

+82

-0

tests/config/test_base.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import os.path
	16	import tempfile
	17
	18	from synapse.config import ConfigError
	19	from synapse.util.stringutils import random_string
	20
	21	from tests import unittest
	22
	23
	24	class BaseConfigTestCase(unittest.HomeserverTestCase):
	25	def prepare(self, reactor, clock, hs):
	26	self.hs = hs
	27
	28	def test_loading_missing_templates(self):
	29	# Use a temporary directory that exists on the system, but that isn't likely to
	30	# contain template files
	31	with tempfile.TemporaryDirectory() as tmp_dir:
	32	# Attempt to load an HTML template from our custom template directory
	33	template = self.hs.config.read_templates(["sso_error.html"], tmp_dir)[0]
	34
	35	# If no errors, we should've gotten the default template instead
	36
	37	# Render the template
	38	a_random_string = random_string(5)
	39	html_content = template.render({"error_description": a_random_string})
	40
	41	# Check that our string exists in the template
	42	self.assertIn(
	43	a_random_string,
	44	html_content,
	45	"Template file did not contain our test string",
	46	)
	47
	48	def test_loading_custom_templates(self):
	49	# Use a temporary directory that exists on the system
	50	with tempfile.TemporaryDirectory() as tmp_dir:
	51	# Create a temporary bogus template file
	52	with tempfile.NamedTemporaryFile(dir=tmp_dir) as tmp_template:
	53	# Get temporary file's filename
	54	template_filename = os.path.basename(tmp_template.name)
	55
	56	# Write a custom HTML template
	57	contents = b"{{ test_variable }}"
	58	tmp_template.write(contents)
	59	tmp_template.flush()
	60
	61	# Attempt to load the template from our custom template directory
	62	template = (
	63	self.hs.config.read_templates([template_filename], tmp_dir)
	64	)[0]
	65
	66	# Render the template
	67	a_random_string = random_string(5)
	68	html_content = template.render({"test_variable": a_random_string})
	69
	70	# Check that our string exists in the template
	71	self.assertIn(
	72	a_random_string,
	73	html_content,
	74	"Template file did not contain our test string",
	75	)
	76
	77	def test_loading_template_from_nonexistent_custom_directory(self):
	78	with self.assertRaises(ConfigError):
	79	self.hs.config.read_templates(
	80	["some_filename.html"], "a_nonexistent_directory"
	81	)

+3

-3

tests/crypto/test_keyring.py less more

42	42	from tests.test_utils import make_awaitable
43	43
44	44
45		class MockPerspectiveServer(object):
	45	class MockPerspectiveServer:
46	46	def __init__(self):
47	47	self.server_name = "mock_server"
48	48	self.key = signedjson.key.generate_signing_key(0)

189	189
190	190	# should fail immediately on an unsigned object
191	191	d = _verify_json_for_server(kr, "server9", {}, 0, "test unsigned")
192		self.failureResultOf(d, SynapseError)
	192	self.get_failure(d, SynapseError)
193	193
194	194	# should succeed on a signed object
195	195	d = _verify_json_for_server(kr, "server9", json1, 500, "test signed")

220	220
221	221	# should fail immediately on an unsigned object
222	222	d = _verify_json_for_server(kr, "server9", {}, 0, "test unsigned")
223		self.failureResultOf(d, SynapseError)
	223	self.get_failure(d, SynapseError)
224	224
225	225	# should fail on a signed object with a non-zero minimum_valid_until_ms,
226	226	# as it tries to refetch the keys and fails.

+27

-19

tests/federation/test_complexity.py less more

14	14
15	15	from mock import Mock
16	16
17		from twisted.internet import defer
18
19	17	from synapse.api.errors import Codes, SynapseError
20	18	from synapse.rest import admin
21	19	from synapse.rest.client.v1 import login, room

59	57
60	58	# Artificially raise the complexity
61	59	store = self.hs.get_datastore()
62		store.get_current_state_event_counts = lambda x: defer.succeed(500 * 1.23)
	60	store.get_current_state_event_counts = lambda x: make_awaitable(500 * 1.23)
63	61
64	62	# Get the room complexity again -- make sure it's our artificial value
65	63	request, channel = self.make_request(

78	76	fed_transport = self.hs.get_federation_transport_client()
79	77
80	78	# Mock out some things, because we don't want to test the whole join
81		fed_transport.client.get_json = Mock(return_value=make_awaitable({"v1": 9999}))
82		handler.federation_handler.do_invite_join = Mock(
83		return_value=make_awaitable(("", 1))
	79	fed_transport.client.get_json = Mock(
	80	side_effect=lambda args, *kwargs: make_awaitable({"v1": 9999})
	81	)
	82	handler.federation_handler.do_invite_join = Mock(
	83	side_effect=lambda args, *kwargs: make_awaitable(("", 1))
84	84	)
85	85
86	86	d = handler._remote_join(

109	109	fed_transport = self.hs.get_federation_transport_client()
110	110
111	111	# Mock out some things, because we don't want to test the whole join
112		fed_transport.client.get_json = Mock(return_value=make_awaitable({"v1": 9999}))
113		handler.federation_handler.do_invite_join = Mock(
114		return_value=make_awaitable(("", 1))
	112	fed_transport.client.get_json = Mock(
	113	side_effect=lambda args, *kwargs: make_awaitable({"v1": 9999})
	114	)
	115	handler.federation_handler.do_invite_join = Mock(
	116	side_effect=lambda args, *kwargs: make_awaitable(("", 1))
115	117	)
116	118
117	119	d = handler._remote_join(

147	149	fed_transport = self.hs.get_federation_transport_client()
148	150
149	151	# Mock out some things, because we don't want to test the whole join
150		fed_transport.client.get_json = Mock(return_value=make_awaitable(None))
151		handler.federation_handler.do_invite_join = Mock(
152		return_value=make_awaitable(("", 1))
	152	fed_transport.client.get_json = Mock(
	153	side_effect=lambda args, *kwargs: make_awaitable(None)
	154	)
	155	handler.federation_handler.do_invite_join = Mock(
	156	side_effect=lambda args, *kwargs: make_awaitable(("", 1))
153	157	)
154	158
155	159	# Artificially raise the complexity
156		self.hs.get_datastore().get_current_state_event_counts = lambda x: defer.succeed(
	160	self.hs.get_datastore().get_current_state_event_counts = lambda x: make_awaitable(
157	161	600
158	162	)
159	163

203	207	fed_transport = self.hs.get_federation_transport_client()
204	208
205	209	# Mock out some things, because we don't want to test the whole join
206		fed_transport.client.get_json = Mock(return_value=make_awaitable({"v1": 9999}))
207		handler.federation_handler.do_invite_join = Mock(
208		return_value=make_awaitable(("", 1))
	210	fed_transport.client.get_json = Mock(
	211	side_effect=lambda args, *kwargs: make_awaitable({"v1": 9999})
	212	)
	213	handler.federation_handler.do_invite_join = Mock(
	214	side_effect=lambda args, *kwargs: make_awaitable(("", 1))
209	215	)
210	216
211	217	d = handler._remote_join(

233	239	fed_transport = self.hs.get_federation_transport_client()
234	240
235	241	# Mock out some things, because we don't want to test the whole join
236		fed_transport.client.get_json = Mock(return_value=make_awaitable({"v1": 9999}))
237		handler.federation_handler.do_invite_join = Mock(
238		return_value=make_awaitable(("", 1))
	242	fed_transport.client.get_json = Mock(
	243	side_effect=lambda args, *kwargs: make_awaitable({"v1": 9999})
	244	)
	245	handler.federation_handler.do_invite_join = Mock(
	246	side_effect=lambda args, *kwargs: make_awaitable(("", 1))
239	247	)
240	248
241	249	d = handler._remote_join(

+33

-0

tests/federation/test_federation_server.py less more

14	14	# limitations under the License.
15	15	import logging
16	16
	17	from parameterized import parameterized
	18
17	19	from synapse.events import make_event_from_dict
18	20	from synapse.federation.federation_server import server_matches_acl_event
19	21	from synapse.rest import admin
20	22	from synapse.rest.client.v1 import login, room
21	23
22	24	from tests import unittest
	25
	26
	27	class FederationServerTests(unittest.FederatingHomeserverTestCase):
	28
	29	servlets = [
	30	admin.register_servlets,
	31	room.register_servlets,
	32	login.register_servlets,
	33	]
	34
	35	@parameterized.expand([(b"",), (b"foo",), (b'{"limit": Infinity}',)])
	36	def test_bad_request(self, query_content):
	37	"""
	38	Querying with bad data returns a reasonable error code.
	39	"""
	40	u1 = self.register_user("u1", "pass")
	41	u1_token = self.login("u1", "pass")
	42
	43	room_1 = self.helper.create_room_as(u1, tok=u1_token)
	44	self.inject_room_member(room_1, "@user:other.example.com", "join")
	45
	46	"/get_missing_events/(?P<room_id>[^/]*)/?"
	47
	48	request, channel = self.make_request(
	49	"POST",
	50	"/_matrix/federation/v1/get_missing_events/%s" % (room_1,),
	51	query_content,
	52	)
	53	self.render(request)
	54	self.assertEquals(400, channel.code, channel.result)
	55	self.assertEqual(channel.json_body["errcode"], "M_NOT_JSON")
23	56
24	57
25	58	class ServerACLsTestCase(unittest.TestCase):

+1

-1

tests/federation/transport/test_server.py less more

25	25
26	26	class RoomDirectoryFederationTests(unittest.HomeserverTestCase):
27	27	def prepare(self, reactor, clock, homeserver):
28		class Authenticator(object):
	28	class Authenticator:
29	29	def authenticate_request(self, request, content):
30	30	return defer.succeed("otherserver.nottld")
31	31

+16

-15

tests/handlers/test_auth.py less more

23	23	from synapse.handlers.auth import AuthHandler
24	24
25	25	from tests import unittest
	26	from tests.test_utils import make_awaitable
26	27	from tests.utils import setup_test_homeserver
27	28
28	29
29		class AuthHandlers(object):
	30	class AuthHandlers:
30	31	def __init__(self, hs):
31	32	self.auth_handler = AuthHandler(hs)
32	33

141	142	def test_mau_limits_exceeded_large(self):
142	143	self.auth_blocking._limit_usage_by_mau = True
143	144	self.hs.get_datastore().get_monthly_active_count = Mock(
144		return_value=defer.succeed(self.large_number_of_users)
	145	side_effect=lambda: make_awaitable(self.large_number_of_users)
145	146	)
146	147
147	148	with self.assertRaises(ResourceLimitError):

152	153	)
153	154
154	155	self.hs.get_datastore().get_monthly_active_count = Mock(
155		return_value=defer.succeed(self.large_number_of_users)
	156	side_effect=lambda: make_awaitable(self.large_number_of_users)
156	157	)
157	158	with self.assertRaises(ResourceLimitError):
158	159	yield defer.ensureDeferred(

167	168
168	169	# If not in monthly active cohort
169	170	self.hs.get_datastore().get_monthly_active_count = Mock(
170		return_value=defer.succeed(self.auth_blocking._max_mau_value)
	171	side_effect=lambda: make_awaitable(self.auth_blocking._max_mau_value)
171	172	)
172	173	with self.assertRaises(ResourceLimitError):
173	174	yield defer.ensureDeferred(

177	178	)
178	179
179	180	self.hs.get_datastore().get_monthly_active_count = Mock(
180		return_value=defer.succeed(self.auth_blocking._max_mau_value)
	181	side_effect=lambda: make_awaitable(self.auth_blocking._max_mau_value)
181	182	)
182	183	with self.assertRaises(ResourceLimitError):
183	184	yield defer.ensureDeferred(

187	188	)
188	189	# If in monthly active cohort
189	190	self.hs.get_datastore().user_last_seen_monthly_active = Mock(
190		return_value=defer.succeed(self.hs.get_clock().time_msec())
191		)
192		self.hs.get_datastore().get_monthly_active_count = Mock(
193		return_value=defer.succeed(self.auth_blocking._max_mau_value)
	191	side_effect=lambda user_id: make_awaitable(self.hs.get_clock().time_msec())
	192	)
	193	self.hs.get_datastore().get_monthly_active_count = Mock(
	194	side_effect=lambda: make_awaitable(self.auth_blocking._max_mau_value)
194	195	)
195	196	yield defer.ensureDeferred(
196	197	self.auth_handler.get_access_token_for_user_id(

198	199	)
199	200	)
200	201	self.hs.get_datastore().user_last_seen_monthly_active = Mock(
201		return_value=defer.succeed(self.hs.get_clock().time_msec())
202		)
203		self.hs.get_datastore().get_monthly_active_count = Mock(
204		return_value=defer.succeed(self.auth_blocking._max_mau_value)
	202	side_effect=lambda user_id: make_awaitable(self.hs.get_clock().time_msec())
	203	)
	204	self.hs.get_datastore().get_monthly_active_count = Mock(
	205	side_effect=lambda: make_awaitable(self.auth_blocking._max_mau_value)
205	206	)
206	207	yield defer.ensureDeferred(
207	208	self.auth_handler.validate_short_term_login_token_and_get_user_id(

214	215	self.auth_blocking._limit_usage_by_mau = True
215	216
216	217	self.hs.get_datastore().get_monthly_active_count = Mock(
217		return_value=defer.succeed(self.small_number_of_users)
	218	side_effect=lambda: make_awaitable(self.small_number_of_users)
218	219	)
219	220	# Ensure does not raise exception
220	221	yield defer.ensureDeferred(

224	225	)
225	226
226	227	self.hs.get_datastore().get_monthly_active_count = Mock(
227		return_value=defer.succeed(self.small_number_of_users)
	228	side_effect=lambda: make_awaitable(self.small_number_of_users)
228	229	)
229	230	yield defer.ensureDeferred(
230	231	self.auth_handler.validate_short_term_login_token_and_get_user_id(

+54

-5

tests/handlers/test_oidc.py less more

74	74	COOKIE_NAME = b"oidc_session"
75	75	COOKIE_PATH = "/_synapse/oidc"
76	76
77		MockedMappingProvider = Mock(OidcMappingProvider)
	77
	78	class TestMappingProvider(OidcMappingProvider):
	79	@staticmethod
	80	def parse_config(config):
	81	return
	82
	83	def get_remote_user_id(self, userinfo):
	84	return userinfo["sub"]
	85
	86	async def map_user_attributes(self, userinfo, token):
	87	return {"localpart": userinfo["username"], "display_name": None}
78	88
79	89
80	90	def simple_async_mock(return_value=None, raises=None):

122	132	oidc_config["issuer"] = ISSUER
123	133	oidc_config["scopes"] = SCOPES
124	134	oidc_config["user_mapping_provider"] = {
125		"module": __name__ + ".MockedMappingProvider"
	135	"module": __name__ + ".TestMappingProvider",
126	136	}
127	137	config["oidc_config"] = oidc_config
128	138

373	383	self.handler._fetch_userinfo = simple_async_mock(return_value=userinfo)
374	384	self.handler._map_userinfo_to_user = simple_async_mock(return_value=user_id)
375	385	self.handler._auth_handler.complete_sso_login = simple_async_mock()
376		request = Mock(spec=["args", "getCookie", "addCookie"])
	386	request = Mock(
	387	spec=["args", "getCookie", "addCookie", "requestHeaders", "getClientIP"]
	388	)
377	389
378	390	code = "code"
379	391	state = "state"
380	392	nonce = "nonce"
381	393	client_redirect_url = "http://client/redirect"
	394	user_agent = "Browser"
	395	ip_address = "10.0.0.1"
382	396	session = self.handler._generate_oidc_session_token(
383	397	state=state,
384	398	nonce=nonce,

391	405	request.args[b"code"] = [code.encode("utf-8")]
392	406	request.args[b"state"] = [state.encode("utf-8")]
393	407
	408	request.requestHeaders = Mock(spec=["getRawHeaders"])
	409	request.requestHeaders.getRawHeaders.return_value = [user_agent.encode("ascii")]
	410	request.getClientIP.return_value = ip_address
	411
394	412	yield defer.ensureDeferred(self.handler.handle_oidc_callback(request))
395	413
396	414	self.handler._auth_handler.complete_sso_login.assert_called_once_with(

398	416	)
399	417	self.handler._exchange_code.assert_called_once_with(code)
400	418	self.handler._parse_id_token.assert_called_once_with(token, nonce=nonce)
401		self.handler._map_userinfo_to_user.assert_called_once_with(userinfo, token)
	419	self.handler._map_userinfo_to_user.assert_called_once_with(
	420	userinfo, token, user_agent, ip_address
	421	)
402	422	self.handler._fetch_userinfo.assert_not_called()
403	423	self.handler._render_error.assert_not_called()
404	424

430	450	)
431	451	self.handler._exchange_code.assert_called_once_with(code)
432	452	self.handler._parse_id_token.assert_not_called()
433		self.handler._map_userinfo_to_user.assert_called_once_with(userinfo, token)
	453	self.handler._map_userinfo_to_user.assert_called_once_with(
	454	userinfo, token, user_agent, ip_address
	455	)
434	456	self.handler._fetch_userinfo.assert_called_once_with(token)
435	457	self.handler._render_error.assert_not_called()
436	458

567	589	with self.assertRaises(OidcError) as exc:
568	590	yield defer.ensureDeferred(self.handler._exchange_code(code))
569	591	self.assertEqual(exc.exception.error, "some_error")
	592
	593	def test_map_userinfo_to_user(self):
	594	"""Ensure that mapping the userinfo returned from a provider to an MXID works properly."""
	595	userinfo = {
	596	"sub": "test_user",
	597	"username": "test_user",
	598	}
	599	# The token doesn't matter with the default user mapping provider.
	600	token = {}
	601	mxid = self.get_success(
	602	self.handler._map_userinfo_to_user(
	603	userinfo, token, "user-agent", "10.10.10.10"
	604	)
	605	)
	606	self.assertEqual(mxid, "@test_user:test")
	607
	608	# Some providers return an integer ID.
	609	userinfo = {
	610	"sub": 1234,
	611	"username": "test_user_2",
	612	}
	613	mxid = self.get_success(
	614	self.handler._map_userinfo_to_user(
	615	userinfo, token, "user-agent", "10.10.10.10"
	616	)
	617	)
	618	self.assertEqual(mxid, "@test_user_2:test")

+1

-1

tests/handlers/test_presence.py less more

18	18	from signedjson.key import generate_signing_key
19	19
20	20	from synapse.api.constants import EventTypes, Membership, PresenceState
	21	from synapse.api.presence import UserPresenceState
21	22	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
22	23	from synapse.events.builder import EventBuilder
23	24	from synapse.handlers.presence import (

31	32	handle_update,
32	33	)
33	34	from synapse.rest.client.v1 import room
34		from synapse.storage.presence import UserPresenceState
35	35	from synapse.types import UserID, get_domain_from_id
36	36
37	37	from tests import unittest

+52

-20

tests/handlers/test_profile.py less more

27	27	from tests.utils import setup_test_homeserver
28	28
29	29
30		class ProfileHandlers(object):
	30	class ProfileHandlers:
31	31	def __init__(self, hs):
32	32	self.profile_handler = MasterProfileHandler(hs)
33	33

63	63	self.bob = UserID.from_string("@4567:test")
64	64	self.alice = UserID.from_string("@alice:remote")
65	65
66		yield self.store.create_profile(self.frank.localpart)
	66	yield defer.ensureDeferred(self.store.create_profile(self.frank.localpart))
67	67
68	68	self.handler = hs.get_profile_handler()
69	69	self.hs = hs
70	70
71	71	@defer.inlineCallbacks
72	72	def test_get_my_name(self):
73		yield self.store.set_profile_displayname(self.frank.localpart, "Frank")
	73	yield defer.ensureDeferred(
	74	self.store.set_profile_displayname(self.frank.localpart, "Frank")
	75	)
74	76
75	77	displayname = yield defer.ensureDeferred(
76	78	self.handler.get_displayname(self.frank)

103	105	)
104	106
105	107	self.assertEquals(
106		(yield self.store.get_profile_displayname(self.frank.localpart)), "Frank",
	108	(
	109	yield defer.ensureDeferred(
	110	self.store.get_profile_displayname(self.frank.localpart)
	111	)
	112	),
	113	"Frank",
107	114	)
108	115
109	116	@defer.inlineCallbacks

111	118	self.hs.config.enable_set_displayname = False
112	119
113	120	# Setting displayname for the first time is allowed
114		yield self.store.set_profile_displayname(self.frank.localpart, "Frank")
115
116		self.assertEquals(
117		(yield self.store.get_profile_displayname(self.frank.localpart)), "Frank",
	121	yield defer.ensureDeferred(
	122	self.store.set_profile_displayname(self.frank.localpart, "Frank")
	123	)
	124
	125	self.assertEquals(
	126	(
	127	yield defer.ensureDeferred(
	128	self.store.get_profile_displayname(self.frank.localpart)
	129	)
	130	),
	131	"Frank",
118	132	)
119	133
120	134	# Setting displayname a second time is forbidden

156	170
157	171	@defer.inlineCallbacks
158	172	def test_incoming_fed_query(self):
159		yield self.store.create_profile("caroline")
160		yield self.store.set_profile_displayname("caroline", "Caroline")
	173	yield defer.ensureDeferred(self.store.create_profile("caroline"))
	174	yield defer.ensureDeferred(
	175	self.store.set_profile_displayname("caroline", "Caroline")
	176	)
161	177
162	178	response = yield defer.ensureDeferred(
163	179	self.query_handlers["profile"](

169	185
170	186	@defer.inlineCallbacks
171	187	def test_get_my_avatar(self):
172		yield self.store.set_profile_avatar_url(
173		self.frank.localpart, "http://my.server/me.png"
	188	yield defer.ensureDeferred(
	189	self.store.set_profile_avatar_url(
	190	self.frank.localpart, "http://my.server/me.png"
	191	)
174	192	)
175	193	avatar_url = yield defer.ensureDeferred(self.handler.get_avatar_url(self.frank))
176	194

187	205	)
188	206
189	207	self.assertEquals(
190		(yield self.store.get_profile_avatar_url(self.frank.localpart)),
	208	(
	209	yield defer.ensureDeferred(
	210	self.store.get_profile_avatar_url(self.frank.localpart)
	211	)
	212	),
191	213	"http://my.server/pic.gif",
192	214	)
193	215

201	223	)
202	224
203	225	self.assertEquals(
204		(yield self.store.get_profile_avatar_url(self.frank.localpart)),
	226	(
	227	yield defer.ensureDeferred(
	228	self.store.get_profile_avatar_url(self.frank.localpart)
	229	)
	230	),
205	231	"http://my.server/me.png",
206	232	)
207	233

210	236	self.hs.config.enable_set_avatar_url = False
211	237
212	238	# Setting displayname for the first time is allowed
213		yield self.store.set_profile_avatar_url(
214		self.frank.localpart, "http://my.server/me.png"
215		)
216
217		self.assertEquals(
218		(yield self.store.get_profile_avatar_url(self.frank.localpart)),
	239	yield defer.ensureDeferred(
	240	self.store.set_profile_avatar_url(
	241	self.frank.localpart, "http://my.server/me.png"
	242	)
	243	)
	244
	245	self.assertEquals(
	246	(
	247	yield defer.ensureDeferred(
	248	self.store.get_profile_avatar_url(self.frank.localpart)
	249	)
	250	),
219	251	"http://my.server/me.png",
220	252	)
221	253

+56

-8

tests/handlers/test_register.py less more

14	14
15	15	from mock import Mock
16	16
17		from twisted.internet import defer
18
	17	from synapse.api.auth import Auth
19	18	from synapse.api.constants import UserTypes
20	19	from synapse.api.errors import Codes, ResourceLimitError, SynapseError
21	20	from synapse.handlers.register import RegistrationHandler
	21	from synapse.spam_checker_api import RegistrationBehaviour
22	22	from synapse.types import RoomAlias, UserID, create_requester
23	23
24	24	from tests.test_utils import make_awaitable
25	25	from tests.unittest import override_config
	26	from tests.utils import mock_getRawHeaders
26	27
27	28	from .. import unittest
28	29
29	30
30		class RegistrationHandlers(object):
	31	class RegistrationHandlers:
31	32	def __init__(self, hs):
32	33	self.registration_handler = RegistrationHandler(hs)
33	34

98	99	def test_get_or_create_user_mau_not_blocked(self):
99	100	self.hs.config.limit_usage_by_mau = True
100	101	self.store.count_monthly_users = Mock(
101		return_value=defer.succeed(self.hs.config.max_mau_value - 1)
	102	side_effect=lambda: make_awaitable(self.hs.config.max_mau_value - 1)
102	103	)
103	104	# Ensure does not throw exception
104	105	self.get_success(self.get_or_create_user(self.requester, "c", "User"))

106	107	def test_get_or_create_user_mau_blocked(self):
107	108	self.hs.config.limit_usage_by_mau = True
108	109	self.store.get_monthly_active_count = Mock(
109		return_value=defer.succeed(self.lots_of_users)
	110	side_effect=lambda: make_awaitable(self.lots_of_users)
110	111	)
111	112	self.get_failure(
112	113	self.get_or_create_user(self.requester, "b", "display_name"),

114	115	)
115	116
116	117	self.store.get_monthly_active_count = Mock(
117		return_value=defer.succeed(self.hs.config.max_mau_value)
	118	side_effect=lambda: make_awaitable(self.hs.config.max_mau_value)
118	119	)
119	120	self.get_failure(
120	121	self.get_or_create_user(self.requester, "b", "display_name"),

124	125	def test_register_mau_blocked(self):
125	126	self.hs.config.limit_usage_by_mau = True
126	127	self.store.get_monthly_active_count = Mock(
127		return_value=defer.succeed(self.lots_of_users)
	128	side_effect=lambda: make_awaitable(self.lots_of_users)
128	129	)
129	130	self.get_failure(
130	131	self.handler.register_user(localpart="local_part"), ResourceLimitError
131	132	)
132	133
133	134	self.store.get_monthly_active_count = Mock(
134		return_value=defer.succeed(self.hs.config.max_mau_value)
	135	side_effect=lambda: make_awaitable(self.hs.config.max_mau_value)
135	136	)
136	137	self.get_failure(
137	138	self.handler.register_user(localpart="local_part"), ResourceLimitError

474	475	self.handler.register_user(localpart=invalid_user_id), SynapseError
475	476	)
476	477
	478	def test_spam_checker_deny(self):
	479	"""A spam checker can deny registration, which results in an error."""
	480
	481	class DenyAll:
	482	def check_registration_for_spam(
	483	self, email_threepid, username, request_info
	484	):
	485	return RegistrationBehaviour.DENY
	486
	487	# Configure a spam checker that denies all users.
	488	spam_checker = self.hs.get_spam_checker()
	489	spam_checker.spam_checkers = [DenyAll()]
	490
	491	self.get_failure(self.handler.register_user(localpart="user"), SynapseError)
	492
	493	def test_spam_checker_shadow_ban(self):
	494	"""A spam checker can choose to shadow-ban a user, which allows registration to succeed."""
	495
	496	class BanAll:
	497	def check_registration_for_spam(
	498	self, email_threepid, username, request_info
	499	):
	500	return RegistrationBehaviour.SHADOW_BAN
	501
	502	# Configure a spam checker that denies all users.
	503	spam_checker = self.hs.get_spam_checker()
	504	spam_checker.spam_checkers = [BanAll()]
	505
	506	user_id = self.get_success(self.handler.register_user(localpart="user"))
	507
	508	# Get an access token.
	509	token = self.macaroon_generator.generate_access_token(user_id)
	510	self.get_success(
	511	self.store.add_access_token_to_user(
	512	user_id=user_id, token=token, device_id=None, valid_until_ms=None
	513	)
	514	)
	515
	516	# Ensure the user was marked as shadow-banned.
	517	request = Mock(args={})
	518	request.args[b"access_token"] = [token.encode("ascii")]
	519	request.requestHeaders.getRawHeaders = mock_getRawHeaders()
	520	auth = Auth(self.hs)
	521	requester = self.get_success(auth.get_user_by_req(request))
	522
	523	self.assertTrue(requester.shadow_banned)
	524
477	525	async def get_or_create_user(
478	526	self, requester, localpart, displayname, password_hash=None
479	527	):

+3

-3

tests/handlers/test_stats.py less more

80	80	)
81	81	)
82	82
83		def get_all_room_state(self):
84		return self.store.db_pool.simple_select_list(
	83	async def get_all_room_state(self):
	84	return await self.store.db_pool.simple_select_list(
85	85	"room_stats_state", None, retcols=("name", "topic", "canonical_alias")
86	86	)
87	87

255	255	# self.handler.notify_new_event()
256	256
257	257	# We need to let the delta processor advance…
258		self.pump(10 * 60)
	258	self.reactor.advance(10 * 60)
259	259
260	260	# Get the slices! There should be two -- day 1, and day 2.
261	261	r = self.get_success(self.store.get_statistics_for_subject("room", room_1, 0))

+26

-10

tests/handlers/test_typing.py less more

20	20	from twisted.internet import defer
21	21
22	22	from synapse.api.errors import AuthError
23		from synapse.types import UserID
	23	from synapse.types import UserID, create_requester
24	24
25	25	from tests import unittest
26	26	from tests.test_utils import make_awaitable

143	143
144	144	self.datastore.get_users_in_room = get_users_in_room
145	145
146		self.datastore.get_user_directory_stream_pos.return_value = (
	146	self.datastore.get_user_directory_stream_pos.side_effect = (
147	147	# we deliberately return a non-None stream pos to avoid doing an initial_spam
148		defer.succeed(1)
	148	lambda: make_awaitable(1)
149	149	)
150	150
151	151	self.datastore.get_current_state_deltas.return_value = (0, None)

154	154	self.datastore.get_new_device_msgs_for_remote = lambda args, *kargs: make_awaitable(
155	155	([], 0)
156	156	)
157		self.datastore.delete_device_msgs_for_remote = lambda args, *kargs: None
158		self.datastore.set_received_txn_response = lambda args, *kwargs: defer.succeed(
	157	self.datastore.delete_device_msgs_for_remote = lambda args, *kargs: make_awaitable(
	158	None
	159	)
	160	self.datastore.set_received_txn_response = lambda args, *kwargs: make_awaitable(
159	161	None
160	162	)
161	163

166	168
167	169	self.get_success(
168	170	self.handler.started_typing(
169		target_user=U_APPLE, auth_user=U_APPLE, room_id=ROOM_ID, timeout=20000
	171	target_user=U_APPLE,
	172	requester=create_requester(U_APPLE),
	173	room_id=ROOM_ID,
	174	timeout=20000,
170	175	)
171	176	)
172	177

193	198
194	199	self.get_success(
195	200	self.handler.started_typing(
196		target_user=U_APPLE, auth_user=U_APPLE, room_id=ROOM_ID, timeout=20000
	201	target_user=U_APPLE,
	202	requester=create_requester(U_APPLE),
	203	room_id=ROOM_ID,
	204	timeout=20000,
197	205	)
198	206	)
199	207

268	276
269	277	self.get_success(
270	278	self.handler.stopped_typing(
271		target_user=U_APPLE, auth_user=U_APPLE, room_id=ROOM_ID
	279	target_user=U_APPLE,
	280	requester=create_requester(U_APPLE),
	281	room_id=ROOM_ID,
272	282	)
273	283	)
274	284

308	318
309	319	self.get_success(
310	320	self.handler.started_typing(
311		target_user=U_APPLE, auth_user=U_APPLE, room_id=ROOM_ID, timeout=10000
	321	target_user=U_APPLE,
	322	requester=create_requester(U_APPLE),
	323	room_id=ROOM_ID,
	324	timeout=10000,
312	325	)
313	326	)
314	327

347	360
348	361	self.get_success(
349	362	self.handler.started_typing(
350		target_user=U_APPLE, auth_user=U_APPLE, room_id=ROOM_ID, timeout=10000
	363	target_user=U_APPLE,
	364	requester=create_requester(U_APPLE),
	365	room_id=ROOM_ID,
	366	timeout=10000,
351	367	)
352	368	)
353	369

+3

-3

tests/handlers/test_user_directory.py less more

237	237
238	238	def test_spam_checker(self):
239	239	"""
240		A user which fails to the spam checks will not appear in search results.
	240	A user which fails the spam checks will not appear in search results.
241	241	"""
242	242	u1 = self.register_user("user1", "pass")
243	243	u1_token = self.login(u1, "pass")

268	268	# Configure a spam checker that does not filter any users.
269	269	spam_checker = self.hs.get_spam_checker()
270	270
271		class AllowAll(object):
	271	class AllowAll:
272	272	def check_username_for_spam(self, user_profile):
273	273	# Allow all users.
274	274	return False

281	281	self.assertEqual(len(s["results"]), 1)
282	282
283	283	# Configure a spam checker that filters all users.
284		class BlockAll(object):
	284	class BlockAll:
285	285	def check_username_for_spam(self, user_profile):
286	286	# All users are spammy.
287	287	return True

+1

-1

tests/http/__init__.py less more

132	132
133	133
134	134	@implementer(IOpenSSLServerConnectionCreator)
135		class TestServerTLSConnectionFactory(object):
	135	class TestServerTLSConnectionFactory:
136	136	"""An SSL connection creator which returns connections which present a certificate
137	137	signed by our test CA."""
138	138

+19

-7

tests/http/federation/test_matrix_federation_agent.py less more

971	971	def test_well_known_cache(self):
972	972	self.reactor.lookups["testserv"] = "1.2.3.4"
973	973
974		fetch_d = self.well_known_resolver.get_well_known(b"testserv")
	974	fetch_d = defer.ensureDeferred(
	975	self.well_known_resolver.get_well_known(b"testserv")
	976	)
975	977
976	978	# there should be an attempt to connect on port 443 for the .well-known
977	979	clients = self.reactor.tcpClients

994	996	well_known_server.loseConnection()
995	997
996	998	# repeat the request: it should hit the cache
997		fetch_d = self.well_known_resolver.get_well_known(b"testserv")
	999	fetch_d = defer.ensureDeferred(
	1000	self.well_known_resolver.get_well_known(b"testserv")
	1001	)
998	1002	r = self.successResultOf(fetch_d)
999	1003	self.assertEqual(r.delegated_server, b"target-server")
1000	1004

1002	1006	self.reactor.pump((1000.0,))
1003	1007
1004	1008	# now it should connect again
1005		fetch_d = self.well_known_resolver.get_well_known(b"testserv")
	1009	fetch_d = defer.ensureDeferred(
	1010	self.well_known_resolver.get_well_known(b"testserv")
	1011	)
1006	1012
1007	1013	self.assertEqual(len(clients), 1)
1008	1014	(host, port, client_factory, _timeout, _bindAddress) = clients.pop(0)

1025	1031
1026	1032	self.reactor.lookups["testserv"] = "1.2.3.4"
1027	1033
1028		fetch_d = self.well_known_resolver.get_well_known(b"testserv")
	1034	fetch_d = defer.ensureDeferred(
	1035	self.well_known_resolver.get_well_known(b"testserv")
	1036	)
1029	1037
1030	1038	# there should be an attempt to connect on port 443 for the .well-known
1031	1039	clients = self.reactor.tcpClients

1051	1059	# another lookup.
1052	1060	self.reactor.pump((900.0,))
1053	1061
1054		fetch_d = self.well_known_resolver.get_well_known(b"testserv")
	1062	fetch_d = defer.ensureDeferred(
	1063	self.well_known_resolver.get_well_known(b"testserv")
	1064	)
1055	1065
1056	1066	# The resolver may retry a few times, so fonx all requests that come along
1057	1067	attempts = 0

1081	1091	self.reactor.pump((10000.0,))
1082	1092
1083	1093	# Repated the request, this time it should fail if the lookup fails.
1084		fetch_d = self.well_known_resolver.get_well_known(b"testserv")
	1094	fetch_d = defer.ensureDeferred(
	1095	self.well_known_resolver.get_well_known(b"testserv")
	1096	)
1085	1097
1086	1098	clients = self.reactor.tcpClients
1087	1099	(host, port, client_factory, _timeout, _bindAddress) = clients.pop(0)

1251	1263
1252	1264
1253	1265	@implementer(IPolicyForHTTPS)
1254		class TrustingTLSPolicyForHTTPS(object):
	1266	class TrustingTLSPolicyForHTTPS:
1255	1267	"""An IPolicyForHTTPS which checks that the certificate belongs to the
1256	1268	right server, but doesn't check the certificate chain."""
1257	1269

+49

-1

tests/http/test_fedclient.py less more

15	15	from mock import Mock
16	16
17	17	from netaddr import IPSet
	18	from parameterized import parameterized
18	19
19	20	from twisted.internet import defer
20	21	from twisted.internet.defer import TimeoutError

507	508	self.assertFalse(conn.disconnecting)
508	509
509	510	# wait for a while
510		self.pump(120)
	511	self.reactor.advance(120)
511	512
512	513	self.assertTrue(conn.disconnecting)
	514
	515	@parameterized.expand([(b"",), (b"foo",), (b'{"a": Infinity}',)])
	516	def test_json_error(self, return_value):
	517	"""
	518	Test what happens if invalid JSON is returned from the remote endpoint.
	519	"""
	520
	521	test_d = defer.ensureDeferred(self.cl.get_json("testserv:8008", "foo/bar"))
	522
	523	self.pump()
	524
	525	# Nothing happened yet
	526	self.assertNoResult(test_d)
	527
	528	# Make sure treq is trying to connect
	529	clients = self.reactor.tcpClients
	530	self.assertEqual(len(clients), 1)
	531	(host, port, factory, _timeout, _bindAddress) = clients[0]
	532	self.assertEqual(host, "1.2.3.4")
	533	self.assertEqual(port, 8008)
	534
	535	# complete the connection and wire it up to a fake transport
	536	protocol = factory.buildProtocol(None)
	537	transport = StringTransport()
	538	protocol.makeConnection(transport)
	539
	540	# that should have made it send the request to the transport
	541	self.assertRegex(transport.value(), b"^GET /foo/bar")
	542	self.assertRegex(transport.value(), b"Host: testserv:8008")
	543
	544	# Deferred is still without a result
	545	self.assertNoResult(test_d)
	546
	547	# Send it the HTTP response
	548	protocol.dataReceived(
	549	b"HTTP/1.1 200 OK\r\n"
	550	b"Server: Fake\r\n"
	551	b"Content-Type: application/json\r\n"
	552	b"Content-Length: %i\r\n"
	553	b"\r\n"
	554	b"%s" % (len(return_value), return_value)
	555	)
	556
	557	self.pump()
	558
	559	f = self.failureResultOf(test_d)
	560	self.assertIsInstance(f.value, ValueError)

+80

-0

tests/http/test_servlet.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	import json
	15	from io import BytesIO
	16
	17	from mock import Mock
	18
	19	from synapse.api.errors import SynapseError
	20	from synapse.http.servlet import (
	21	parse_json_object_from_request,
	22	parse_json_value_from_request,
	23	)
	24
	25	from tests import unittest
	26
	27
	28	def make_request(content):
	29	"""Make an object that acts enough like a request."""
	30	request = Mock(spec=["content"])
	31
	32	if isinstance(content, dict):
	33	content = json.dumps(content).encode("utf8")
	34
	35	request.content = BytesIO(content)
	36	return request
	37
	38
	39	class TestServletUtils(unittest.TestCase):
	40	def test_parse_json_value(self):
	41	"""Basic tests for parse_json_value_from_request."""
	42	# Test round-tripping.
	43	obj = {"foo": 1}
	44	result = parse_json_value_from_request(make_request(obj))
	45	self.assertEqual(result, obj)
	46
	47	# Results don't have to be objects.
	48	result = parse_json_value_from_request(make_request(b'["foo"]'))
	49	self.assertEqual(result, ["foo"])
	50
	51	# Test empty.
	52	with self.assertRaises(SynapseError):
	53	parse_json_value_from_request(make_request(b""))
	54
	55	result = parse_json_value_from_request(make_request(b""), allow_empty_body=True)
	56	self.assertIsNone(result)
	57
	58	# Invalid UTF-8.
	59	with self.assertRaises(SynapseError):
	60	parse_json_value_from_request(make_request(b"\xFF\x00"))
	61
	62	# Invalid JSON.
	63	with self.assertRaises(SynapseError):
	64	parse_json_value_from_request(make_request(b"foo"))
	65
	66	with self.assertRaises(SynapseError):
	67	parse_json_value_from_request(make_request(b'{"foo": Infinity}'))
	68
	69	def test_parse_json_object(self):
	70	"""Basic tests for parse_json_object_from_request."""
	71	# Test empty.
	72	result = parse_json_object_from_request(
	73	make_request(b""), allow_empty_body=True
	74	)
	75	self.assertEqual(result, {})
	76
	77	# Test not an object
	78	with self.assertRaises(SynapseError):
	79	parse_json_object_from_request(make_request(b'["foo"]'))

+2

-2

tests/logging/test_structured.py less more

28	28	from tests.unittest import DEBUG, HomeserverTestCase
29	29
30	30
31		class FakeBeginner(object):
	31	class FakeBeginner:
32	32	def beginLoggingTo(self, observers, **kwargs):
33	33	self.observers = observers
34	34
35	35
36		class StructuredLoggingTestBase(object):
	36	class StructuredLoggingTestBase:
37	37	"""
38	38	Test base that registers a cleanup handler to reset the stdlib log handler
39	39	to 'unset'.

+1

-1

tests/module_api/test_api.py less more

34	34	# Check that the new user exists with all provided attributes
35	35	self.assertEqual(user_id, "@bob:test")
36	36	self.assertTrue(access_token)
37		self.assertTrue(self.store.get_user_by_id(user_id))
	37	self.assertTrue(self.get_success(self.store.get_user_by_id(user_id)))
38	38
39	39	# Check that the email was assigned
40	40	emails = self.get_success(self.store.user_get_threepids(user_id))

+2

-2

tests/push/test_email.py less more

26	26
27	27
28	28	@attr.s
29		class _User(object):
	29	class _User:
30	30	"Helper wrapper for user ID and access token"
31	31	id = attr.ib()
32	32	token = attr.ib()

169	169	last_stream_ordering = pushers[0]["last_stream_ordering"]
170	170
171	171	# Advance time a bit, so the pusher will register something has happened
172		self.pump(100)
	172	self.pump(10)
173	173
174	174	# It hasn't succeeded yet, so the stream ordering shouldn't have moved
175	175	pushers = self.get_success(

+6

-4

tests/replication/slave/storage/test_events.py less more

159	159	self.check(
160	160	"get_unread_event_push_actions_by_room_for_user",
161	161	[ROOM_ID, USER_ID_2, event1.event_id],
162		{"highlight_count": 0, "notify_count": 0},
	162	{"highlight_count": 0, "unread_count": 0, "notify_count": 0},
163	163	)
164	164
165	165	self.persist(

172	172	self.check(
173	173	"get_unread_event_push_actions_by_room_for_user",
174	174	[ROOM_ID, USER_ID_2, event1.event_id],
175		{"highlight_count": 0, "notify_count": 1},
	175	{"highlight_count": 0, "unread_count": 0, "notify_count": 1},
176	176	)
177	177
178	178	self.persist(

187	187	self.check(
188	188	"get_unread_event_push_actions_by_room_for_user",
189	189	[ROOM_ID, USER_ID_2, event1.event_id],
190		{"highlight_count": 1, "notify_count": 2},
	190	{"highlight_count": 1, "unread_count": 0, "notify_count": 2},
191	191	)
192	192
193	193	def test_get_rooms_for_user_with_stream_ordering(self):

367	367
368	368	self.get_success(
369	369	self.master_store.add_push_actions_to_staging(
370		event.event_id, {user_id: actions for user_id, actions in push_actions}
	370	event.event_id,
	371	{user_id: actions for user_id, actions in push_actions},
	372	False,
371	373	)
372	374	)
373	375	return event, context

+2

-2

tests/replication/test_federation_sender_shard.py less more

19	19	from synapse.events.builder import EventBuilderFactory
20	20	from synapse.rest.admin import register_servlets_for_client_rest_resource
21	21	from synapse.rest.client.v1 import login, room
22		from synapse.types import UserID
	22	from synapse.types import UserID, create_requester
23	23
24	24	from tests.replication._base import BaseMultiWorkerStreamTestCase
25	25	from tests.test_utils import make_awaitable

174	174	self.get_success(
175	175	typing_handler.started_typing(
176	176	target_user=UserID.from_string(user),
177		auth_user=UserID.from_string(user),
	177	requester=create_requester(user),
178	178	room_id=room,
179	179	timeout=20000,
180	180	)

+4

-5

tests/rest/admin/test_user.py less more

19	19
20	20	from mock import Mock
21	21
22		from twisted.internet import defer
23
24	22	import synapse.rest.admin
25	23	from synapse.api.constants import UserTypes
26	24	from synapse.api.errors import HttpResponseException, ResourceLimitError

28	26	from synapse.rest.client.v2_alpha import sync
29	27
30	28	from tests import unittest
	29	from tests.test_utils import make_awaitable
31	30	from tests.unittest import override_config
32	31
33	32

337	336
338	337	# Set monthly active users to the limit
339	338	store.get_monthly_active_count = Mock(
340		return_value=defer.succeed(self.hs.config.max_mau_value)
	339	side_effect=lambda: make_awaitable(self.hs.config.max_mau_value)
341	340	)
342	341	# Check that the blocking of monthly active users is working as expected
343	342	# The registration of a new user fails due to the limit

591	590
592	591	# Set monthly active users to the limit
593	592	self.store.get_monthly_active_count = Mock(
594		return_value=defer.succeed(self.hs.config.max_mau_value)
	593	side_effect=lambda: make_awaitable(self.hs.config.max_mau_value)
595	594	)
596	595	# Check that the blocking of monthly active users is working as expected
597	596	# The registration of a new user fails due to the limit

631	630
632	631	# Set monthly active users to the limit
633	632	self.store.get_monthly_active_count = Mock(
634		return_value=defer.succeed(self.hs.config.max_mau_value)
	633	side_effect=lambda: make_awaitable(self.hs.config.max_mau_value)
635	634	)
636	635	# Check that the blocking of monthly active users is working as expected
637	636	# The registration of a new user fails due to the limit

+76

-38

tests/rest/client/test_retention.py less more

44	44	}
45	45
46	46	self.hs = self.setup_test_homeserver(config=config)
	47
47	48	return self.hs
48	49
49	50	def prepare(self, reactor, clock, homeserver):
50	51	self.user_id = self.register_user("user", "password")
51	52	self.token = self.login("user", "password")
52	53
53		def test_retention_state_event(self):
54		"""Tests that the server configuration can limit the values a user can set to the
55		room's retention policy.
56		"""
57		room_id = self.helper.create_room_as(self.user_id, tok=self.token)
58
59		self.helper.send_state(
60		room_id=room_id,
61		event_type=EventTypes.Retention,
62		body={"max_lifetime": one_day_ms * 4},
63		tok=self.token,
64		expect_code=400,
65		)
66
67		self.helper.send_state(
68		room_id=room_id,
69		event_type=EventTypes.Retention,
70		body={"max_lifetime": one_hour_ms},
71		tok=self.token,
72		expect_code=400,
73		)
	54	self.store = self.hs.get_datastore()
	55	self.serializer = self.hs.get_event_client_serializer()
	56	self.clock = self.hs.get_clock()
74	57
75	58	def test_retention_event_purged_with_state_event(self):
76	59	"""Tests that expired events are correctly purged when the room's retention policy

89	72
90	73	self._test_retention_event_purged(room_id, one_day_ms * 1.5)
91	74
	75	def test_retention_event_purged_with_state_event_outside_allowed(self):
	76	"""Tests that the server configuration can override the policy for a room when
	77	running the purge jobs.
	78	"""
	79	room_id = self.helper.create_room_as(self.user_id, tok=self.token)
	80
	81	# Set a max_lifetime higher than the maximum allowed value.
	82	self.helper.send_state(
	83	room_id=room_id,
	84	event_type=EventTypes.Retention,
	85	body={"max_lifetime": one_day_ms * 4},
	86	tok=self.token,
	87	)
	88
	89	# Check that the event is purged after waiting for the maximum allowed duration
	90	# instead of the one specified in the room's policy.
	91	self._test_retention_event_purged(room_id, one_day_ms * 1.5)
	92
	93	# Set a max_lifetime lower than the minimum allowed value.
	94	self.helper.send_state(
	95	room_id=room_id,
	96	event_type=EventTypes.Retention,
	97	body={"max_lifetime": one_hour_ms},
	98	tok=self.token,
	99	)
	100
	101	# Check that the event is purged after waiting for the minimum allowed duration
	102	# instead of the one specified in the room's policy.
	103	self._test_retention_event_purged(room_id, one_day_ms * 0.5)
	104
92	105	def test_retention_event_purged_without_state_event(self):
93	106	"""Tests that expired events are correctly purged when the room's retention policy
94	107	is defined by the server's configuration's default retention policy.

139	152	# That event should be the second, not outdated event.
140	153	self.assertEqual(filtered_events[0].event_id, valid_event_id, filtered_events)
141	154
142		def _test_retention_event_purged(self, room_id, increment):
	155	def _test_retention_event_purged(self, room_id: str, increment: float):
	156	"""Run the following test scenario to test the message retention policy support:
	157
	158	1. Send event 1
	159	2. Increment time by `increment`
	160	3. Send event 2
	161	4. Increment time by `increment`
	162	5. Check that event 1 has been purged
	163	6. Check that event 2 has not been purged
	164	7. Check that state events that were sent before event 1 aren't purged.
	165	The main reason for sending a second event is because currently Synapse won't
	166	purge the latest message in a room because it would otherwise result in a lack of
	167	forward extremities for this room. It's also a good thing to ensure the purge jobs
	168	aren't too greedy and purge messages they shouldn't.
	169
	170	Args:
	171	room_id: The ID of the room to test retention in.
	172	increment: The number of milliseconds to advance the clock each time. Must be
	173	defined so that events in the room aren't purged if they are `increment`
	174	old but are purged if they are `increment * 2` old.
	175	"""
143	176	# Get the create event to, later, check that we can still access it.
144	177	message_handler = self.hs.get_message_handler()
145	178	create_event = self.get_success(
146	179	message_handler.get_room_data(
147		self.user_id, room_id, EventTypes.Create, state_key="", is_guest=False
	180	self.user_id, room_id, EventTypes.Create, state_key=""
148	181	)
149	182	)
150	183

155	188	expired_event_id = resp.get("event_id")
156	189
157	190	# Check that we can retrieve the event.
158		expired_event = self.get_event(room_id, expired_event_id)
	191	expired_event = self.get_event(expired_event_id)
159	192	self.assertEqual(
160	193	expired_event.get("content", {}).get("body"), "1", expired_event
161	194	)

173	206	# one should still be kept.
174	207	self.reactor.advance(increment / 1000)
175	208
176		# Check that the event has been purged from the database.
177		self.get_event(room_id, expired_event_id, expected_code=404)
178
179		# Check that the event that hasn't been purged can still be retrieved.
180		valid_event = self.get_event(room_id, valid_event_id)
	209	# Check that the first event has been purged from the database, i.e. that we
	210	# can't retrieve it anymore, because it has expired.
	211	self.get_event(expired_event_id, expect_none=True)
	212
	213	# Check that the event that hasn't expired can still be retrieved.
	214	valid_event = self.get_event(valid_event_id)
181	215	self.assertEqual(valid_event.get("content", {}).get("body"), "2", valid_event)
182	216
183	217	# Check that we can still access state events that were sent before the event that
184	218	# has been purged.
185	219	self.get_event(room_id, create_event.event_id)
186	220
187		def get_event(self, room_id, event_id, expected_code=200):
188		url = "/_matrix/client/r0/rooms/%s/event/%s" % (room_id, event_id)
189
190		request, channel = self.make_request("GET", url, access_token=self.token)
191		self.render(request)
192
193		self.assertEqual(channel.code, expected_code, channel.result)
194
195		return channel.json_body
	221	def get_event(self, event_id, expect_none=False):
	222	event = self.get_success(self.store.get_event(event_id, allow_none=True))
	223
	224	if expect_none:
	225	self.assertIsNone(event)
	226	return {}
	227
	228	self.assertIsNotNone(event)
	229
	230	time_now = self.clock.time_msec()
	231	serialized = self.get_success(self.serializer.serialize_event(event, time_now))
	232
	233	return serialized
196	234
197	235
198	236	class RetentionNoDefaultPolicyTestCase(unittest.HomeserverTestCase):

+312

-0

tests/rest/client/test_shadow_banned.py less more

	0	# Copyright 2020 The Matrix.org Foundation C.I.C.
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14	from mock import Mock, patch
	15
	16	import synapse.rest.admin
	17	from synapse.api.constants import EventTypes
	18	from synapse.rest.client.v1 import directory, login, profile, room
	19	from synapse.rest.client.v2_alpha import room_upgrade_rest_servlet
	20
	21	from tests import unittest
	22
	23
	24	class _ShadowBannedBase(unittest.HomeserverTestCase):
	25	def prepare(self, reactor, clock, homeserver):
	26	# Create two users, one of which is shadow-banned.
	27	self.banned_user_id = self.register_user("banned", "test")
	28	self.banned_access_token = self.login("banned", "test")
	29
	30	self.store = self.hs.get_datastore()
	31
	32	self.get_success(
	33	self.store.db_pool.simple_update(
	34	table="users",
	35	keyvalues={"name": self.banned_user_id},
	36	updatevalues={"shadow_banned": True},
	37	desc="shadow_ban",
	38	)
	39	)
	40
	41	self.other_user_id = self.register_user("otheruser", "pass")
	42	self.other_access_token = self.login("otheruser", "pass")
	43
	44
	45	# To avoid the tests timing out don't add a delay to "annoy the requester".
	46	@patch("random.randint", new=lambda a, b: 0)
	47	class RoomTestCase(_ShadowBannedBase):
	48	servlets = [
	49	synapse.rest.admin.register_servlets_for_client_rest_resource,
	50	directory.register_servlets,
	51	login.register_servlets,
	52	room.register_servlets,
	53	room_upgrade_rest_servlet.register_servlets,
	54	]
	55
	56	def test_invite(self):
	57	"""Invites from shadow-banned users don't actually get sent."""
	58
	59	# The create works fine.
	60	room_id = self.helper.create_room_as(
	61	self.banned_user_id, tok=self.banned_access_token
	62	)
	63
	64	# Inviting the user completes successfully.
	65	self.helper.invite(
	66	room=room_id,
	67	src=self.banned_user_id,
	68	tok=self.banned_access_token,
	69	targ=self.other_user_id,
	70	)
	71
	72	# But the user wasn't actually invited.
	73	invited_rooms = self.get_success(
	74	self.store.get_invited_rooms_for_local_user(self.other_user_id)
	75	)
	76	self.assertEqual(invited_rooms, [])
	77
	78	def test_invite_3pid(self):
	79	"""Ensure that a 3PID invite does not attempt to contact the identity server."""
	80	identity_handler = self.hs.get_handlers().identity_handler
	81	identity_handler.lookup_3pid = Mock(
	82	side_effect=AssertionError("This should not get called")
	83	)
	84
	85	# The create works fine.
	86	room_id = self.helper.create_room_as(
	87	self.banned_user_id, tok=self.banned_access_token
	88	)
	89
	90	# Inviting the user completes successfully.
	91	request, channel = self.make_request(
	92	"POST",
	93	"/rooms/%s/invite" % (room_id,),
	94	{"id_server": "test", "medium": "email", "address": "test@test.test"},
	95	access_token=self.banned_access_token,
	96	)
	97	self.render(request)
	98	self.assertEquals(200, channel.code, channel.result)
	99
	100	# This should have raised an error earlier, but double check this wasn't called.
	101	identity_handler.lookup_3pid.assert_not_called()
	102
	103	def test_create_room(self):
	104	"""Invitations during a room creation should be discarded, but the room still gets created."""
	105	# The room creation is successful.
	106	request, channel = self.make_request(
	107	"POST",
	108	"/_matrix/client/r0/createRoom",
	109	{"visibility": "public", "invite": [self.other_user_id]},
	110	access_token=self.banned_access_token,
	111	)
	112	self.render(request)
	113	self.assertEquals(200, channel.code, channel.result)
	114	room_id = channel.json_body["room_id"]
	115
	116	# But the user wasn't actually invited.
	117	invited_rooms = self.get_success(
	118	self.store.get_invited_rooms_for_local_user(self.other_user_id)
	119	)
	120	self.assertEqual(invited_rooms, [])
	121
	122	# Since a real room was created, the other user should be able to join it.
	123	self.helper.join(room_id, self.other_user_id, tok=self.other_access_token)
	124
	125	# Both users should be in the room.
	126	users = self.get_success(self.store.get_users_in_room(room_id))
	127	self.assertCountEqual(users, ["@banned:test", "@otheruser:test"])
	128
	129	def test_message(self):
	130	"""Messages from shadow-banned users don't actually get sent."""
	131
	132	room_id = self.helper.create_room_as(
	133	self.other_user_id, tok=self.other_access_token
	134	)
	135
	136	# The user should be in the room.
	137	self.helper.join(room_id, self.banned_user_id, tok=self.banned_access_token)
	138
	139	# Sending a message should complete successfully.
	140	result = self.helper.send_event(
	141	room_id=room_id,
	142	type=EventTypes.Message,
	143	content={"msgtype": "m.text", "body": "with right label"},
	144	tok=self.banned_access_token,
	145	)
	146	self.assertIn("event_id", result)
	147	event_id = result["event_id"]
	148
	149	latest_events = self.get_success(
	150	self.store.get_latest_event_ids_in_room(room_id)
	151	)
	152	self.assertNotIn(event_id, latest_events)
	153
	154	def test_upgrade(self):
	155	"""A room upgrade should fail, but look like it succeeded."""
	156
	157	# The create works fine.
	158	room_id = self.helper.create_room_as(
	159	self.banned_user_id, tok=self.banned_access_token
	160	)
	161
	162	request, channel = self.make_request(
	163	"POST",
	164	"/_matrix/client/r0/rooms/%s/upgrade" % (room_id,),
	165	{"new_version": "6"},
	166	access_token=self.banned_access_token,
	167	)
	168	self.render(request)
	169	self.assertEquals(200, channel.code, channel.result)
	170	# A new room_id should be returned.
	171	self.assertIn("replacement_room", channel.json_body)
	172
	173	new_room_id = channel.json_body["replacement_room"]
	174
	175	# It doesn't really matter what API we use here, we just want to assert
	176	# that the room doesn't exist.
	177	summary = self.get_success(self.store.get_room_summary(new_room_id))
	178	# The summary should be empty since the room doesn't exist.
	179	self.assertEqual(summary, {})
	180
	181	def test_typing(self):
	182	"""Typing notifications should not be propagated into the room."""
	183	# The create works fine.
	184	room_id = self.helper.create_room_as(
	185	self.banned_user_id, tok=self.banned_access_token
	186	)
	187
	188	request, channel = self.make_request(
	189	"PUT",
	190	"/rooms/%s/typing/%s" % (room_id, self.banned_user_id),
	191	{"typing": True, "timeout": 30000},
	192	access_token=self.banned_access_token,
	193	)
	194	self.render(request)
	195	self.assertEquals(200, channel.code)
	196
	197	# There should be no typing events.
	198	event_source = self.hs.get_event_sources().sources["typing"]
	199	self.assertEquals(event_source.get_current_key(), 0)
	200
	201	# The other user can join and send typing events.
	202	self.helper.join(room_id, self.other_user_id, tok=self.other_access_token)
	203
	204	request, channel = self.make_request(
	205	"PUT",
	206	"/rooms/%s/typing/%s" % (room_id, self.other_user_id),
	207	{"typing": True, "timeout": 30000},
	208	access_token=self.other_access_token,
	209	)
	210	self.render(request)
	211	self.assertEquals(200, channel.code)
	212
	213	# These appear in the room.
	214	self.assertEquals(event_source.get_current_key(), 1)
	215	events = self.get_success(
	216	event_source.get_new_events(from_key=0, room_ids=[room_id])
	217	)
	218	self.assertEquals(
	219	events[0],
	220	[
	221	{
	222	"type": "m.typing",
	223	"room_id": room_id,
	224	"content": {"user_ids": [self.other_user_id]},
	225	}
	226	],
	227	)
	228
	229
	230	# To avoid the tests timing out don't add a delay to "annoy the requester".
	231	@patch("random.randint", new=lambda a, b: 0)
	232	class ProfileTestCase(_ShadowBannedBase):
	233	servlets = [
	234	synapse.rest.admin.register_servlets_for_client_rest_resource,
	235	login.register_servlets,
	236	profile.register_servlets,
	237	room.register_servlets,
	238	]
	239
	240	def test_displayname(self):
	241	"""Profile changes should succeed, but don't end up in a room."""
	242	original_display_name = "banned"
	243	new_display_name = "new name"
	244
	245	# Join a room.
	246	room_id = self.helper.create_room_as(
	247	self.banned_user_id, tok=self.banned_access_token
	248	)
	249
	250	# The update should succeed.
	251	request, channel = self.make_request(
	252	"PUT",
	253	"/_matrix/client/r0/profile/%s/displayname" % (self.banned_user_id,),
	254	{"displayname": new_display_name},
	255	access_token=self.banned_access_token,
	256	)
	257	self.render(request)
	258	self.assertEquals(200, channel.code, channel.result)
	259	self.assertEqual(channel.json_body, {})
	260
	261	# The user's display name should be updated.
	262	request, channel = self.make_request(
	263	"GET", "/profile/%s/displayname" % (self.banned_user_id,)
	264	)
	265	self.render(request)
	266	self.assertEqual(channel.code, 200, channel.result)
	267	self.assertEqual(channel.json_body["displayname"], new_display_name)
	268
	269	# But the display name in the room should not be.
	270	message_handler = self.hs.get_message_handler()
	271	event = self.get_success(
	272	message_handler.get_room_data(
	273	self.banned_user_id, room_id, "m.room.member", self.banned_user_id,
	274	)
	275	)
	276	self.assertEqual(
	277	event.content, {"membership": "join", "displayname": original_display_name}
	278	)
	279
	280	def test_room_displayname(self):
	281	"""Changes to state events for a room should be processed, but not end up in the room."""
	282	original_display_name = "banned"
	283	new_display_name = "new name"
	284
	285	# Join a room.
	286	room_id = self.helper.create_room_as(
	287	self.banned_user_id, tok=self.banned_access_token
	288	)
	289
	290	# The update should succeed.
	291	request, channel = self.make_request(
	292	"PUT",
	293	"/_matrix/client/r0/rooms/%s/state/m.room.member/%s"
	294	% (room_id, self.banned_user_id),
	295	{"membership": "join", "displayname": new_display_name},
	296	access_token=self.banned_access_token,
	297	)
	298	self.render(request)
	299	self.assertEquals(200, channel.code, channel.result)
	300	self.assertIn("event_id", channel.json_body)
	301
	302	# The display name in the room should not be changed.
	303	message_handler = self.hs.get_message_handler()
	304	event = self.get_success(
	305	message_handler.get_room_data(
	306	self.banned_user_id, room_id, "m.room.member", self.banned_user_id,
	307	)
	308	)
	309	self.assertEqual(
	310	event.content, {"membership": "join", "displayname": original_display_name}
	311	)

+1

-1

tests/rest/client/third_party_rules.py less more

18	18	from tests import unittest
19	19
20	20
21		class ThirdPartyRulesTestModule(object):
	21	class ThirdPartyRulesTestModule:
22	22	def __init__(self, config):
23	23	pass
24	24

+5

-11

tests/rest/client/v1/test_login.py less more

61	61	"identifier": {"type": "m.id.user", "user": "kermit" + str(i)},
62	62	"password": "monkey",
63	63	}
64		request_data = json.dumps(params)
65		request, channel = self.make_request(b"POST", LOGIN_URL, request_data)
	64	request, channel = self.make_request(b"POST", LOGIN_URL, params)
66	65	self.render(request)
67	66
68	67	if i == 5:

75	74	# than 1min.
76	75	self.assertTrue(retry_after_ms < 6000)
77	76
78		self.reactor.advance(retry_after_ms / 1000.0)
	77	self.reactor.advance(retry_after_ms / 1000.0 + 1.0)
79	78
80	79	params = {
81	80	"type": "m.login.password",
82	81	"identifier": {"type": "m.id.user", "user": "kermit" + str(i)},
83	82	"password": "monkey",
84	83	}
85		request_data = json.dumps(params)
86	84	request, channel = self.make_request(b"POST", LOGIN_URL, params)
87	85	self.render(request)
88	86

110	108	"identifier": {"type": "m.id.user", "user": "kermit"},
111	109	"password": "monkey",
112	110	}
113		request_data = json.dumps(params)
114		request, channel = self.make_request(b"POST", LOGIN_URL, request_data)
	111	request, channel = self.make_request(b"POST", LOGIN_URL, params)
115	112	self.render(request)
116	113
117	114	if i == 5:

131	128	"identifier": {"type": "m.id.user", "user": "kermit"},
132	129	"password": "monkey",
133	130	}
134		request_data = json.dumps(params)
135	131	request, channel = self.make_request(b"POST", LOGIN_URL, params)
136	132	self.render(request)
137	133

159	155	"identifier": {"type": "m.id.user", "user": "kermit"},
160	156	"password": "notamonkey",
161	157	}
162		request_data = json.dumps(params)
163		request, channel = self.make_request(b"POST", LOGIN_URL, request_data)
	158	request, channel = self.make_request(b"POST", LOGIN_URL, params)
164	159	self.render(request)
165	160
166	161	if i == 5:

173	168	# than 1min.
174	169	self.assertTrue(retry_after_ms < 6000)
175	170
176		self.reactor.advance(retry_after_ms / 1000.0)
	171	self.reactor.advance(retry_after_ms / 1000.0 + 1.0)
177	172
178	173	params = {
179	174	"type": "m.login.password",
180	175	"identifier": {"type": "m.id.user", "user": "kermit"},
181	176	"password": "notamonkey",
182	177	}
183		request_data = json.dumps(params)
184	178	request, channel = self.make_request(b"POST", LOGIN_URL, params)
185	179	self.render(request)
186	180

+14

-13

tests/rest/client/v1/test_rooms.py less more

683	683	]
684	684
685	685	@unittest.override_config(
686		{"rc_joins": {"local": {"per_second": 3, "burst_count": 3}}}
	686	{"rc_joins": {"local": {"per_second": 0.5, "burst_count": 3}}}
687	687	)
688	688	def test_join_local_ratelimit(self):
689	689	"""Tests that local joins are actually rate-limited."""
690		for i in range(5):
	690	for i in range(3):
691	691	self.helper.create_room_as(self.user_id)
692	692
693	693	self.helper.create_room_as(self.user_id, expect_code=429)
694	694
695	695	@unittest.override_config(
696		{"rc_joins": {"local": {"per_second": 3, "burst_count": 3}}}
	696	{"rc_joins": {"local": {"per_second": 0.5, "burst_count": 3}}}
697	697	)
698	698	def test_join_local_ratelimit_profile_change(self):
699	699	"""Tests that sending a profile update into all of the user's joined rooms isn't
700	700	rate-limited by the rate-limiter on joins."""
701	701
702		# Create and join more rooms than the rate-limiting config allows in a second.
	702	# Create and join as many rooms as the rate-limiting config allows in a second.
703	703	room_ids = [
704	704	self.helper.create_room_as(self.user_id),
705	705	self.helper.create_room_as(self.user_id),
706	706	self.helper.create_room_as(self.user_id),
707	707	]
708		self.reactor.advance(1)
709		room_ids = room_ids + [
710		self.helper.create_room_as(self.user_id),
711		self.helper.create_room_as(self.user_id),
712		self.helper.create_room_as(self.user_id),
713		]
	708	# Let some time for the rate-limiter to forget about our multi-join.
	709	self.reactor.advance(2)
	710	# Add one to make sure we're joined to more rooms than the config allows us to
	711	# join in a second.
	712	room_ids.append(self.helper.create_room_as(self.user_id))
714	713
715	714	# Create a profile for the user, since it hasn't been done on registration.
716	715	store = self.hs.get_datastore()
717		store.create_profile(UserID.from_string(self.user_id).localpart)
	716	self.get_success(
	717	store.create_profile(UserID.from_string(self.user_id).localpart)
	718	)
718	719
719	720	# Update the display name for the user.
720	721	path = "/_matrix/client/r0/profile/%s/displayname" % self.user_id

737	738	self.assertEquals(channel.json_body["displayname"], "John Doe")
738	739
739	740	@unittest.override_config(
740		{"rc_joins": {"local": {"per_second": 3, "burst_count": 3}}}
	741	{"rc_joins": {"local": {"per_second": 0.5, "burst_count": 3}}}
741	742	)
742	743	def test_join_local_ratelimit_idempotent(self):
743	744	"""Tests that the room join endpoints remain idempotent despite rate-limiting

753	754	for path in paths_to_test:
754	755	# Make sure we send more requests than the rate-limiting config would allow
755	756	# if all of these requests ended up joining the user to a room.
756		for i in range(6):
	757	for i in range(4):
757	758	request, channel = self.make_request("POST", path % room_id, {})
758	759	self.render(request)
759	760	self.assertEquals(channel.code, 200)

+1

-1

tests/rest/client/v1/utils.py less more

29	29
30	30
31	31	@attr.s
32		class RestHelper(object):
	32	class RestHelper:
33	33	"""Contains extra helper functions to quickly and clearly perform a given
34	34	REST action, which isn't the focus of the test.
35	35	"""

+6

-2

tests/rest/client/v2_alpha/test_filter.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14
	15	from twisted.internet import defer
14	16
15	17	from synapse.api.errors import Codes
16	18	from synapse.rest.client.v2_alpha import filter

72	74	self.assertEquals(channel.json_body["errcode"], Codes.FORBIDDEN)
73	75
74	76	def test_get_filter(self):
75		filter_id = self.filtering.add_user_filter(
76		user_localpart="apple", user_filter=self.EXAMPLE_FILTER
	77	filter_id = defer.ensureDeferred(
	78	self.filtering.add_user_filter(
	79	user_localpart="apple", user_filter=self.EXAMPLE_FILTER
	80	)
77	81	)
78	82	self.reactor.advance(1)
79	83	filter_id = filter_id.result

+2

-2

tests/rest/client/v2_alpha/test_register.py less more

159	159	else:
160	160	self.assertEquals(channel.result["code"], b"200", channel.result)
161	161
162		self.reactor.advance(retry_after_ms / 1000.0)
	162	self.reactor.advance(retry_after_ms / 1000.0 + 1.0)
163	163
164	164	request, channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
165	165	self.render(request)

185	185	else:
186	186	self.assertEquals(channel.result["code"], b"200", channel.result)
187	187
188		self.reactor.advance(retry_after_ms / 1000.0)
	188	self.reactor.advance(retry_after_ms / 1000.0 + 1.0)
189	189
190	190	request, channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
191	191	self.render(request)

+138

-0

tests/rest/client/v2_alpha/test_shared_rooms.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 Half-Shot
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	import synapse.rest.admin
	15	from synapse.rest.client.v1 import login, room
	16	from synapse.rest.client.v2_alpha import shared_rooms
	17
	18	from tests import unittest
	19
	20
	21	class UserSharedRoomsTest(unittest.HomeserverTestCase):
	22	"""
	23	Tests the UserSharedRoomsServlet.
	24	"""
	25
	26	servlets = [
	27	login.register_servlets,
	28	synapse.rest.admin.register_servlets_for_client_rest_resource,
	29	room.register_servlets,
	30	shared_rooms.register_servlets,
	31	]
	32
	33	def make_homeserver(self, reactor, clock):
	34	config = self.default_config()
	35	config["update_user_directory"] = True
	36	return self.setup_test_homeserver(config=config)
	37
	38	def prepare(self, reactor, clock, hs):
	39	self.store = hs.get_datastore()
	40	self.handler = hs.get_user_directory_handler()
	41
	42	def _get_shared_rooms(self, token, other_user):
	43	request, channel = self.make_request(
	44	"GET",
	45	"/_matrix/client/unstable/uk.half-shot.msc2666/user/shared_rooms/%s"
	46	% other_user,
	47	access_token=token,
	48	)
	49	self.render(request)
	50	return request, channel
	51
	52	def test_shared_room_list_public(self):
	53	"""
	54	A room should show up in the shared list of rooms between two users
	55	if it is public.
	56	"""
	57	u1 = self.register_user("user1", "pass")
	58	u1_token = self.login(u1, "pass")
	59	u2 = self.register_user("user2", "pass")
	60	u2_token = self.login(u2, "pass")
	61
	62	room = self.helper.create_room_as(u1, is_public=True, tok=u1_token)
	63	self.helper.invite(room, src=u1, targ=u2, tok=u1_token)
	64	self.helper.join(room, user=u2, tok=u2_token)
	65
	66	request, channel = self._get_shared_rooms(u1_token, u2)
	67	self.assertEquals(200, channel.code, channel.result)
	68	self.assertEquals(len(channel.json_body["joined"]), 1)
	69	self.assertEquals(channel.json_body["joined"][0], room)
	70
	71	def test_shared_room_list_private(self):
	72	"""
	73	A room should show up in the shared list of rooms between two users
	74	if it is private.
	75	"""
	76	u1 = self.register_user("user1", "pass")
	77	u1_token = self.login(u1, "pass")
	78	u2 = self.register_user("user2", "pass")
	79	u2_token = self.login(u2, "pass")
	80
	81	room = self.helper.create_room_as(u1, is_public=False, tok=u1_token)
	82	self.helper.invite(room, src=u1, targ=u2, tok=u1_token)
	83	self.helper.join(room, user=u2, tok=u2_token)
	84
	85	request, channel = self._get_shared_rooms(u1_token, u2)
	86	self.assertEquals(200, channel.code, channel.result)
	87	self.assertEquals(len(channel.json_body["joined"]), 1)
	88	self.assertEquals(channel.json_body["joined"][0], room)
	89
	90	def test_shared_room_list_mixed(self):
	91	"""
	92	The shared room list between two users should contain both public and private
	93	rooms.
	94	"""
	95	u1 = self.register_user("user1", "pass")
	96	u1_token = self.login(u1, "pass")
	97	u2 = self.register_user("user2", "pass")
	98	u2_token = self.login(u2, "pass")
	99
	100	room_public = self.helper.create_room_as(u1, is_public=True, tok=u1_token)
	101	room_private = self.helper.create_room_as(u2, is_public=False, tok=u2_token)
	102	self.helper.invite(room_public, src=u1, targ=u2, tok=u1_token)
	103	self.helper.invite(room_private, src=u2, targ=u1, tok=u2_token)
	104	self.helper.join(room_public, user=u2, tok=u2_token)
	105	self.helper.join(room_private, user=u1, tok=u1_token)
	106
	107	request, channel = self._get_shared_rooms(u1_token, u2)
	108	self.assertEquals(200, channel.code, channel.result)
	109	self.assertEquals(len(channel.json_body["joined"]), 2)
	110	self.assertTrue(room_public in channel.json_body["joined"])
	111	self.assertTrue(room_private in channel.json_body["joined"])
	112
	113	def test_shared_room_list_after_leave(self):
	114	"""
	115	A room should no longer be considered shared if the other
	116	user has left it.
	117	"""
	118	u1 = self.register_user("user1", "pass")
	119	u1_token = self.login(u1, "pass")
	120	u2 = self.register_user("user2", "pass")
	121	u2_token = self.login(u2, "pass")
	122
	123	room = self.helper.create_room_as(u1, is_public=True, tok=u1_token)
	124	self.helper.invite(room, src=u1, targ=u2, tok=u1_token)
	125	self.helper.join(room, user=u2, tok=u2_token)
	126
	127	# Assert user directory is not empty
	128	request, channel = self._get_shared_rooms(u1_token, u2)
	129	self.assertEquals(200, channel.code, channel.result)
	130	self.assertEquals(len(channel.json_body["joined"]), 1)
	131	self.assertEquals(channel.json_body["joined"][0], room)
	132
	133	self.helper.leave(room, user=u1, tok=u1_token)
	134
	135	request, channel = self._get_shared_rooms(u2_token, u1)
	136	self.assertEquals(200, channel.code, channel.result)
	137	self.assertEquals(len(channel.json_body["joined"]), 0)

+155

-2

tests/rest/client/v2_alpha/test_sync.py less more

15	15	import json
16	16
17	17	import synapse.rest.admin
18		from synapse.api.constants import EventContentFields, EventTypes
	18	from synapse.api.constants import EventContentFields, EventTypes, RelationTypes
19	19	from synapse.rest.client.v1 import login, room
20		from synapse.rest.client.v2_alpha import sync
	20	from synapse.rest.client.v2_alpha import read_marker, sync
21	21
22	22	from tests import unittest
23	23	from tests.server import TimedOutException

323	323	"GET", sync_url % (access_token, next_batch)
324	324	)
325	325	self.assertRaises(TimedOutException, self.render, request)
	326
	327
	328	class UnreadMessagesTestCase(unittest.HomeserverTestCase):
	329	servlets = [
	330	synapse.rest.admin.register_servlets,
	331	login.register_servlets,
	332	read_marker.register_servlets,
	333	room.register_servlets,
	334	sync.register_servlets,
	335	]
	336
	337	def prepare(self, reactor, clock, hs):
	338	self.url = "/sync?since=%s"
	339	self.next_batch = "s0"
	340
	341	# Register the first user (used to check the unread counts).
	342	self.user_id = self.register_user("kermit", "monkey")
	343	self.tok = self.login("kermit", "monkey")
	344
	345	# Create the room we'll check unread counts for.
	346	self.room_id = self.helper.create_room_as(self.user_id, tok=self.tok)
	347
	348	# Register the second user (used to send events to the room).
	349	self.user2 = self.register_user("kermit2", "monkey")
	350	self.tok2 = self.login("kermit2", "monkey")
	351
	352	# Change the power levels of the room so that the second user can send state
	353	# events.
	354	self.helper.send_state(
	355	self.room_id,
	356	EventTypes.PowerLevels,
	357	{
	358	"users": {self.user_id: 100, self.user2: 100},
	359	"users_default": 0,
	360	"events": {
	361	"m.room.name": 50,
	362	"m.room.power_levels": 100,
	363	"m.room.history_visibility": 100,
	364	"m.room.canonical_alias": 50,
	365	"m.room.avatar": 50,
	366	"m.room.tombstone": 100,
	367	"m.room.server_acl": 100,
	368	"m.room.encryption": 100,
	369	},
	370	"events_default": 0,
	371	"state_default": 50,
	372	"ban": 50,
	373	"kick": 50,
	374	"redact": 50,
	375	"invite": 0,
	376	},
	377	tok=self.tok,
	378	)
	379
	380	def test_unread_counts(self):
	381	"""Tests that /sync returns the right value for the unread count (MSC2654)."""
	382
	383	# Check that our own messages don't increase the unread count.
	384	self.helper.send(self.room_id, "hello", tok=self.tok)
	385	self._check_unread_count(0)
	386
	387	# Join the new user and check that this doesn't increase the unread count.
	388	self.helper.join(room=self.room_id, user=self.user2, tok=self.tok2)
	389	self._check_unread_count(0)
	390
	391	# Check that the new user sending a message increases our unread count.
	392	res = self.helper.send(self.room_id, "hello", tok=self.tok2)
	393	self._check_unread_count(1)
	394
	395	# Send a read receipt to tell the server we've read the latest event.
	396	body = json.dumps({"m.read": res["event_id"]}).encode("utf8")
	397	request, channel = self.make_request(
	398	"POST",
	399	"/rooms/%s/read_markers" % self.room_id,
	400	body,
	401	access_token=self.tok,
	402	)
	403	self.render(request)
	404	self.assertEqual(channel.code, 200, channel.json_body)
	405
	406	# Check that the unread counter is back to 0.
	407	self._check_unread_count(0)
	408
	409	# Check that room name changes increase the unread counter.
	410	self.helper.send_state(
	411	self.room_id, "m.room.name", {"name": "my super room"}, tok=self.tok2,
	412	)
	413	self._check_unread_count(1)
	414
	415	# Check that room topic changes increase the unread counter.
	416	self.helper.send_state(
	417	self.room_id, "m.room.topic", {"topic": "welcome!!!"}, tok=self.tok2,
	418	)
	419	self._check_unread_count(2)
	420
	421	# Check that encrypted messages increase the unread counter.
	422	self.helper.send_event(self.room_id, EventTypes.Encrypted, {}, tok=self.tok2)
	423	self._check_unread_count(3)
	424
	425	# Check that custom events with a body increase the unread counter.
	426	self.helper.send_event(
	427	self.room_id, "org.matrix.custom_type", {"body": "hello"}, tok=self.tok2,
	428	)
	429	self._check_unread_count(4)
	430
	431	# Check that edits don't increase the unread counter.
	432	self.helper.send_event(
	433	room_id=self.room_id,
	434	type=EventTypes.Message,
	435	content={
	436	"body": "hello",
	437	"msgtype": "m.text",
	438	"m.relates_to": {"rel_type": RelationTypes.REPLACE},
	439	},
	440	tok=self.tok2,
	441	)
	442	self._check_unread_count(4)
	443
	444	# Check that notices don't increase the unread counter.
	445	self.helper.send_event(
	446	room_id=self.room_id,
	447	type=EventTypes.Message,
	448	content={"body": "hello", "msgtype": "m.notice"},
	449	tok=self.tok2,
	450	)
	451	self._check_unread_count(4)
	452
	453	# Check that tombstone events changes increase the unread counter.
	454	self.helper.send_state(
	455	self.room_id,
	456	EventTypes.Tombstone,
	457	{"replacement_room": "!someroom:test"},
	458	tok=self.tok2,
	459	)
	460	self._check_unread_count(5)
	461
	462	def _check_unread_count(self, expected_count: True):
	463	"""Syncs and compares the unread count with the expected value."""
	464
	465	request, channel = self.make_request(
	466	"GET", self.url % self.next_batch, access_token=self.tok,
	467	)
	468	self.render(request)
	469
	470	self.assertEqual(channel.code, 200, channel.json_body)
	471
	472	room_entry = channel.json_body["rooms"]["join"][self.room_id]
	473	self.assertEqual(
	474	room_entry["org.matrix.msc2654.unread_count"], expected_count, room_entry,
	475	)
	476
	477	# Store the next batch for the next request.
	478	self.next_batch = channel.json_body["next_batch"]

+3

-3

tests/rest/media/v1/test_url_preview.py less more

31	31
32	32
33	33	@attr.s
34		class FakeResponse(object):
	34	class FakeResponse:
35	35	version = attr.ib()
36	36	code = attr.ib()
37	37	phrase = attr.ib()

42	42	@property
43	43	def request(self):
44	44	@attr.s
45		class FakeTransport(object):
	45	class FakeTransport:
46	46	absoluteURI = self.absoluteURI
47	47
48	48	return FakeTransport()

110	110
111	111	self.lookups = {}
112	112
113		class Resolver(object):
	113	class Resolver:
114	114	def resolveHostName(
115	115	_self,
116	116	resolutionReceiver,

+3

-3

tests/server.py less more

34	34
35	35
36	36	@attr.s
37		class FakeChannel(object):
	37	class FakeChannel:
38	38	"""
39	39	A fake Twisted Web Channel (the part that interfaces with the
40	40	wire).

241	241	lookups = self.lookups = {}
242	242
243	243	@implementer(IResolverSimple)
244		class FakeResolver(object):
	244	class FakeResolver:
245	245	def getHostByName(self, name, timeout=None):
246	246	if name not in lookups:
247	247	return fail(DNSLookupError("OH NO: unknown %s" % (name,)))

370	370
371	371
372	372	@attr.s(cmp=False)
373		class FakeTransport(object):
	373	class FakeTransport:
374	374	"""
375	375	A twisted.internet.interfaces.ITransport implementation which sends all its data
376	376	straight into an IProtocol object: it exists to connect two IProtocols together.

+9

-7

tests/server_notices/test_resource_limits_server_notices.py less more

66	66	raise Exception("Failed to find reference to ResourceLimitsServerNotices")
67	67
68	68	self._rlsn._store.user_last_seen_monthly_active = Mock(
69		return_value=defer.succeed(1000)
	69	side_effect=lambda user_id: make_awaitable(1000)
70	70	)
71	71	self._rlsn._server_notices_manager.send_notice = Mock(
72	72	return_value=defer.succeed(Mock())

103	103	type=EventTypes.Message, content={"msgtype": ServerNoticeMsgType}
104	104	)
105	105	self._rlsn._store.get_events = Mock(
106		return_value=defer.succeed({"123": mock_event})
	106	return_value=make_awaitable({"123": mock_event})
107	107	)
108	108	self.get_success(self._rlsn.maybe_send_server_notice_to_user(self.user_id))
109	109	# Would be better to check the content, but once == remove blocking event

121	121	type=EventTypes.Message, content={"msgtype": ServerNoticeMsgType}
122	122	)
123	123	self._rlsn._store.get_events = Mock(
124		return_value=defer.succeed({"123": mock_event})
	124	return_value=make_awaitable({"123": mock_event})
125	125	)
126	126
127	127	self.get_success(self._rlsn.maybe_send_server_notice_to_user(self.user_id))

157	157	"""
158	158	self._rlsn._auth.check_auth_blocking = Mock(return_value=defer.succeed(None))
159	159	self._rlsn._store.user_last_seen_monthly_active = Mock(
160		return_value=defer.succeed(None)
	160	side_effect=lambda user_id: make_awaitable(None)
161	161	)
162	162	self.get_success(self._rlsn.maybe_send_server_notice_to_user(self.user_id))
163	163

216	216	type=EventTypes.Message, content={"msgtype": ServerNoticeMsgType}
217	217	)
218	218	self._rlsn._store.get_events = Mock(
219		return_value=defer.succeed({"123": mock_event})
	219	return_value=make_awaitable({"123": mock_event})
220	220	)
221	221	self.get_success(self._rlsn.maybe_send_server_notice_to_user(self.user_id))
222	222

260	260	self.user_id = "@user_id:test"
261	261
262	262	def test_server_notice_only_sent_once(self):
263		self.store.get_monthly_active_count = Mock(return_value=defer.succeed(1000))
	263	self.store.get_monthly_active_count = Mock(
	264	side_effect=lambda: make_awaitable(1000)
	265	)
264	266
265	267	self.store.user_last_seen_monthly_active = Mock(
266		return_value=defer.succeed(1000)
	268	side_effect=lambda user_id: make_awaitable(1000)
267	269	)
268	270
269	271	# Call the function multiple times to ensure we only send the notice once

+2

-2

tests/state/test_v2.py less more

48	48	return defer.succeed(None)
49	49
50	50
51		class FakeEvent(object):
	51	class FakeEvent:
52	52	"""A fake event we use as a convenience.
53	53
54	54	NOTE: Again as a convenience we use "node_ids" rather than event_ids to

594	594
595	595
596	596	@attr.s
597		class TestStateResolutionStore(object):
	597	class TestStateResolutionStore:
598	598	event_map = attr.ib()
599	599
600	600	def get_events(self, event_ids, allow_rejected=False):

+9

-9

tests/storage/test__base.py less more

98	98	class CacheDecoratorTestCase(unittest.HomeserverTestCase):
99	99	@defer.inlineCallbacks
100	100	def test_passthrough(self):
101		class A(object):
	101	class A:
102	102	@cached()
103	103	def func(self, key):
104	104	return key

112	112	def test_hit(self):
113	113	callcount = [0]
114	114
115		class A(object):
	115	class A:
116	116	@cached()
117	117	def func(self, key):
118	118	callcount[0] += 1

130	130	def test_invalidate(self):
131	131	callcount = [0]
132	132
133		class A(object):
	133	class A:
134	134	@cached()
135	135	def func(self, key):
136	136	callcount[0] += 1

148	148	self.assertEquals(callcount[0], 2)
149	149
150	150	def test_invalidate_missing(self):
151		class A(object):
	151	class A:
152	152	@cached()
153	153	def func(self, key):
154	154	return key

159	159	def test_max_entries(self):
160	160	callcount = [0]
161	161
162		class A(object):
	162	class A:
163	163	@cached(max_entries=10)
164	164	def func(self, key):
165	165	callcount[0] += 1

186	186
187	187	d = defer.succeed(123)
188	188
189		class A(object):
	189	class A:
190	190	@cached()
191	191	def func(self, key):
192	192	callcount[0] += 1

204	204	callcount = [0]
205	205	callcount2 = [0]
206	206
207		class A(object):
	207	class A:
208	208	@cached()
209	209	def func(self, key):
210	210	callcount[0] += 1

237	237	callcount = [0]
238	238	callcount2 = [0]
239	239
240		class A(object):
	240	class A:
241	241	@cached(max_entries=2)
242	242	def func(self, key):
243	243	callcount[0] += 1

274	274	callcount = [0]
275	275	callcount2 = [0]
276	276
277		class A(object):
	277	class A:
278	278	@cached()
279	279	def func(self, key):
280	280	callcount[0] += 1

+32

-11

tests/storage/test_appservice.py less more

30	30	)
31	31
32	32	from tests import unittest
	33	from tests.test_utils import make_awaitable
33	34	from tests.utils import setup_test_homeserver
34	35
35	36

206	207	@defer.inlineCallbacks
207	208	def test_set_appservices_state_down(self):
208	209	service = Mock(id=self.as_list[1]["id"])
209		yield self.store.set_appservice_state(service, ApplicationServiceState.DOWN)
	210	yield defer.ensureDeferred(
	211	self.store.set_appservice_state(service, ApplicationServiceState.DOWN)
	212	)
210	213	rows = yield self.db_pool.runQuery(
211	214	self.engine.convert_param_style(
212	215	"SELECT as_id FROM application_services_state WHERE state=?"

218	221	@defer.inlineCallbacks
219	222	def test_set_appservices_state_multiple_up(self):
220	223	service = Mock(id=self.as_list[1]["id"])
221		yield self.store.set_appservice_state(service, ApplicationServiceState.UP)
222		yield self.store.set_appservice_state(service, ApplicationServiceState.DOWN)
223		yield self.store.set_appservice_state(service, ApplicationServiceState.UP)
	224	yield defer.ensureDeferred(
	225	self.store.set_appservice_state(service, ApplicationServiceState.UP)
	226	)
	227	yield defer.ensureDeferred(
	228	self.store.set_appservice_state(service, ApplicationServiceState.DOWN)
	229	)
	230	yield defer.ensureDeferred(
	231	self.store.set_appservice_state(service, ApplicationServiceState.UP)
	232	)
224	233	rows = yield self.db_pool.runQuery(
225	234	self.engine.convert_param_style(
226	235	"SELECT as_id FROM application_services_state WHERE state=?"

233	242	def test_create_appservice_txn_first(self):
234	243	service = Mock(id=self.as_list[0]["id"])
235	244	events = [Mock(event_id="e1"), Mock(event_id="e2")]
236		txn = yield self.store.create_appservice_txn(service, events)
	245	txn = yield defer.ensureDeferred(
	246	self.store.create_appservice_txn(service, events)
	247	)
237	248	self.assertEquals(txn.id, 1)
238	249	self.assertEquals(txn.events, events)
239	250	self.assertEquals(txn.service, service)

245	256	yield self._set_last_txn(service.id, 9643) # AS is falling behind
246	257	yield self._insert_txn(service.id, 9644, events)
247	258	yield self._insert_txn(service.id, 9645, events)
248		txn = yield self.store.create_appservice_txn(service, events)
	259	txn = yield defer.ensureDeferred(
	260	self.store.create_appservice_txn(service, events)
	261	)
249	262	self.assertEquals(txn.id, 9646)
250	263	self.assertEquals(txn.events, events)
251	264	self.assertEquals(txn.service, service)

255	268	service = Mock(id=self.as_list[0]["id"])
256	269	events = [Mock(event_id="e1"), Mock(event_id="e2")]
257	270	yield self._set_last_txn(service.id, 9643)
258		txn = yield self.store.create_appservice_txn(service, events)
	271	txn = yield defer.ensureDeferred(
	272	self.store.create_appservice_txn(service, events)
	273	)
259	274	self.assertEquals(txn.id, 9644)
260	275	self.assertEquals(txn.events, events)
261	276	self.assertEquals(txn.service, service)

276	291	yield self._insert_txn(self.as_list[2]["id"], 10, events)
277	292	yield self._insert_txn(self.as_list[3]["id"], 9643, events)
278	293
279		txn = yield self.store.create_appservice_txn(service, events)
	294	txn = yield defer.ensureDeferred(
	295	self.store.create_appservice_txn(service, events)
	296	)
280	297	self.assertEquals(txn.id, 9644)
281	298	self.assertEquals(txn.events, events)
282	299	self.assertEquals(txn.service, service)

288	305	txn_id = 1
289	306
290	307	yield self._insert_txn(service.id, txn_id, events)
291		yield self.store.complete_appservice_txn(txn_id=txn_id, service=service)
	308	yield defer.ensureDeferred(
	309	self.store.complete_appservice_txn(txn_id=txn_id, service=service)
	310	)
292	311
293	312	res = yield self.db_pool.runQuery(
294	313	self.engine.convert_param_style(

314	333	txn_id = 5
315	334	yield self._set_last_txn(service.id, 4)
316	335	yield self._insert_txn(service.id, txn_id, events)
317		yield self.store.complete_appservice_txn(txn_id=txn_id, service=service)
	336	yield defer.ensureDeferred(
	337	self.store.complete_appservice_txn(txn_id=txn_id, service=service)
	338	)
318	339
319	340	res = yield self.db_pool.runQuery(
320	341	self.engine.convert_param_style(

348	369	other_events = [Mock(event_id="e5"), Mock(event_id="e6")]
349	370
350	371	# we aren't testing store._base stuff here, so mock this out
351		self.store.get_events_as_list = Mock(return_value=defer.succeed(events))
	372	self.store.get_events_as_list = Mock(return_value=make_awaitable(events))
352	373
353	374	yield self._insert_txn(self.as_list[1]["id"], 9, other_events)
354	375	yield self._insert_txn(service.id, 10, events)

+5

-9

tests/storage/test_background_update.py less more

0	0	from mock import Mock
1
2		from twisted.internet import defer
3	1
4	2	from synapse.storage.background_updates import BackgroundUpdater
5	3

37	35	)
38	36
39	37	# first step: make a bit of progress
40		@defer.inlineCallbacks
41		def update(progress, count):
42		yield self.clock.sleep((count * duration_ms) / 1000)
	38	async def update(progress, count):
	39	await self.clock.sleep((count * duration_ms) / 1000)
43	40	progress = {"my_key": progress["my_key"] + 1}
44		yield store.db_pool.runInteraction(
	41	await store.db_pool.runInteraction(
45	42	"update_progress",
46	43	self.updates._background_update_progress_txn,
47	44	"test_update",

66	63
67	64	# second step: complete the update
68	65	# we should now get run with a much bigger number of items to update
69		@defer.inlineCallbacks
70		def update(progress, count):
	66	async def update(progress, count):
71	67	self.assertEqual(progress, {"my_key": 2})
72	68	self.assertAlmostEqual(
73	69	count, target_background_update_duration_ms / duration_ms, places=0,
74	70	)
75		yield self.updates._end_background_update("test_update")
	71	await self.updates._end_background_update("test_update")
76	72	return count
77	73
78	74	self.update_handler.side_effect = update

+47

-29

tests/storage/test_base.py less more

65	65	def test_insert_1col(self):
66	66	self.mock_txn.rowcount = 1
67	67
68		yield self.datastore.db_pool.simple_insert(
69		table="tablename", values={"columname": "Value"}
	68	yield defer.ensureDeferred(
	69	self.datastore.db_pool.simple_insert(
	70	table="tablename", values={"columname": "Value"}
	71	)
70	72	)
71	73
72	74	self.mock_txn.execute.assert_called_with(

77	79	def test_insert_3cols(self):
78	80	self.mock_txn.rowcount = 1
79	81
80		yield self.datastore.db_pool.simple_insert(
81		table="tablename",
82		# Use OrderedDict() so we can assert on the SQL generated
83		values=OrderedDict([("colA", 1), ("colB", 2), ("colC", 3)]),
	82	yield defer.ensureDeferred(
	83	self.datastore.db_pool.simple_insert(
	84	table="tablename",
	85	# Use OrderedDict() so we can assert on the SQL generated
	86	values=OrderedDict([("colA", 1), ("colB", 2), ("colC", 3)]),
	87	)
84	88	)
85	89
86	90	self.mock_txn.execute.assert_called_with(

92	96	self.mock_txn.rowcount = 1
93	97	self.mock_txn.__iter__ = Mock(return_value=iter([("Value",)]))
94	98
95		value = yield self.datastore.db_pool.simple_select_one_onecol(
96		table="tablename", keyvalues={"keycol": "TheKey"}, retcol="retcol"
	99	value = yield defer.ensureDeferred(
	100	self.datastore.db_pool.simple_select_one_onecol(
	101	table="tablename", keyvalues={"keycol": "TheKey"}, retcol="retcol"
	102	)
97	103	)
98	104
99	105	self.assertEquals("Value", value)

106	112	self.mock_txn.rowcount = 1
107	113	self.mock_txn.fetchone.return_value = (1, 2, 3)
108	114
109		ret = yield self.datastore.db_pool.simple_select_one(
110		table="tablename",
111		keyvalues={"keycol": "TheKey"},
112		retcols=["colA", "colB", "colC"],
	115	ret = yield defer.ensureDeferred(
	116	self.datastore.db_pool.simple_select_one(
	117	table="tablename",
	118	keyvalues={"keycol": "TheKey"},
	119	retcols=["colA", "colB", "colC"],
	120	)
113	121	)
114	122
115	123	self.assertEquals({"colA": 1, "colB": 2, "colC": 3}, ret)

122	130	self.mock_txn.rowcount = 0
123	131	self.mock_txn.fetchone.return_value = None
124	132
125		ret = yield self.datastore.db_pool.simple_select_one(
126		table="tablename",
127		keyvalues={"keycol": "Not here"},
128		retcols=["colA"],
129		allow_none=True,
	133	ret = yield defer.ensureDeferred(
	134	self.datastore.db_pool.simple_select_one(
	135	table="tablename",
	136	keyvalues={"keycol": "Not here"},
	137	retcols=["colA"],
	138	allow_none=True,
	139	)
130	140	)
131	141
132	142	self.assertFalse(ret)

137	147	self.mock_txn.__iter__ = Mock(return_value=iter([(1,), (2,), (3,)]))
138	148	self.mock_txn.description = (("colA", None, None, None, None, None, None),)
139	149
140		ret = yield self.datastore.db_pool.simple_select_list(
141		table="tablename", keyvalues={"keycol": "A set"}, retcols=["colA"]
	150	ret = yield defer.ensureDeferred(
	151	self.datastore.db_pool.simple_select_list(
	152	table="tablename", keyvalues={"keycol": "A set"}, retcols=["colA"]
	153	)
142	154	)
143	155
144	156	self.assertEquals([{"colA": 1}, {"colA": 2}, {"colA": 3}], ret)

150	162	def test_update_one_1col(self):
151	163	self.mock_txn.rowcount = 1
152	164
153		yield self.datastore.db_pool.simple_update_one(
154		table="tablename",
155		keyvalues={"keycol": "TheKey"},
156		updatevalues={"columnname": "New Value"},
	165	yield defer.ensureDeferred(
	166	self.datastore.db_pool.simple_update_one(
	167	table="tablename",
	168	keyvalues={"keycol": "TheKey"},
	169	updatevalues={"columnname": "New Value"},
	170	)
157	171	)
158	172
159	173	self.mock_txn.execute.assert_called_with(

165	179	def test_update_one_4cols(self):
166	180	self.mock_txn.rowcount = 1
167	181
168		yield self.datastore.db_pool.simple_update_one(
169		table="tablename",
170		keyvalues=OrderedDict([("colA", 1), ("colB", 2)]),
171		updatevalues=OrderedDict([("colC", 3), ("colD", 4)]),
	182	yield defer.ensureDeferred(
	183	self.datastore.db_pool.simple_update_one(
	184	table="tablename",
	185	keyvalues=OrderedDict([("colA", 1), ("colB", 2)]),
	186	updatevalues=OrderedDict([("colC", 3), ("colD", 4)]),
	187	)
172	188	)
173	189
174	190	self.mock_txn.execute.assert_called_with(

180	196	def test_delete_one(self):
181	197	self.mock_txn.rowcount = 1
182	198
183		yield self.datastore.db_pool.simple_delete_one(
184		table="tablename", keyvalues={"keycol": "Go away"}
	199	yield defer.ensureDeferred(
	200	self.datastore.db_pool.simple_delete_one(
	201	table="tablename", keyvalues={"keycol": "Go away"}
	202	)
185	203	)
186	204
187	205	self.mock_txn.execute.assert_called_with(

+5

-4

tests/storage/test_cleanup_extrems.py less more

37	37
38	38	# Create a test user and room
39	39	self.user = UserID("alice", "test")
40		self.requester = Requester(self.user, None, False, None, None)
	40	self.requester = Requester(self.user, None, False, False, None, None)
41	41	info, _ = self.get_success(self.room_creator.create_room(self.requester, {}))
42	42	self.room_id = info["room_id"]
43	43

259	259	# Create a test user and room
260	260	self.user = UserID.from_string(self.register_user("user1", "password"))
261	261	self.token1 = self.login("user1", "password")
262		self.requester = Requester(self.user, None, False, None, None)
	262	self.requester = Requester(self.user, None, False, False, None, None)
263	263	info, _ = self.get_success(self.room_creator.create_room(self.requester, {}))
264	264	self.room_id = info["room_id"]
265	265	self.event_creator = homeserver.get_event_creation_handler()

270	270
271	271	# Pump the reactor repeatedly so that the background updates have a
272	272	# chance to run.
273		self.pump(10 * 60)
	273	self.pump(20)
274	274
275	275	latest_event_ids = self.get_success(
276	276	self.store.get_latest_event_ids_in_room(self.room_id)

352	352	self.event_creator_handler._rooms_to_exclude_from_dummy_event_insertion[
353	353	"3"
354	354	] = 300000
	355
355	356	self.event_creator_handler._expire_rooms_to_exclude_from_dummy_event_insertion()
356	357	# All entries within time frame
357	358	self.assertEqual(

361	362	3,
362	363	)
363	364	# Oldest room to expire
364		self.pump(1)
	365	self.pump(1.01)
365	366	self.event_creator_handler._expire_rooms_to_exclude_from_dummy_event_insertion()
366	367	self.assertEqual(
367	368	len(

+2

-3

tests/storage/test_client_ips.py less more

15	15
16	16	from mock import Mock
17	17
18		from twisted.internet import defer
19
20	18	import synapse.rest.admin
21	19	from synapse.http.site import XForwardedForRequest
22	20	from synapse.rest.client.v1 import login
23	21
24	22	from tests import unittest
	23	from tests.test_utils import make_awaitable
25	24	from tests.unittest import override_config
26	25
27	26

154	153	user_id = "@user:server"
155	154
156	155	self.store.get_monthly_active_count = Mock(
157		return_value=defer.succeed(lots_of_users)
	156	side_effect=lambda: make_awaitable(lots_of_users)
158	157	)
159	158	self.get_success(
160	159	self.store.insert_client_ip(

+4

-4

tests/storage/test_devices.py less more

37	37	self.store.store_device("user_id", "device_id", "display_name")
38	38	)
39	39
40		res = yield self.store.get_device("user_id", "device_id")
	40	res = yield defer.ensureDeferred(self.store.get_device("user_id", "device_id"))
41	41	self.assertDictContainsSubset(
42	42	{
43	43	"user_id": "user_id",

110	110	self.store.store_device("user_id", "device_id", "display_name 1")
111	111	)
112	112
113		res = yield self.store.get_device("user_id", "device_id")
	113	res = yield defer.ensureDeferred(self.store.get_device("user_id", "device_id"))
114	114	self.assertEqual("display_name 1", res["display_name"])
115	115
116	116	# do a no-op first
117	117	yield defer.ensureDeferred(self.store.update_device("user_id", "device_id"))
118		res = yield self.store.get_device("user_id", "device_id")
	118	res = yield defer.ensureDeferred(self.store.get_device("user_id", "device_id"))
119	119	self.assertEqual("display_name 1", res["display_name"])
120	120
121	121	# do the update

126	126	)
127	127
128	128	# check it worked
129		res = yield self.store.get_device("user_id", "device_id")
	129	res = yield defer.ensureDeferred(self.store.get_device("user_id", "device_id"))
130	130	self.assertEqual("display_name 2", res["display_name"])
131	131
132	132	@defer.inlineCallbacks

+5

-1

tests/storage/test_directory.py less more

41	41
42	42	self.assertEquals(
43	43	["#my-room:test"],
44		(yield self.store.get_aliases_for_room(self.room.to_string())),
	44	(
	45	yield defer.ensureDeferred(
	46	self.store.get_aliases_for_room(self.room.to_string())
	47	)
	48	),
45	49	)
46	50
47	51	@defer.inlineCallbacks

+29

-11

tests/storage/test_end_to_end_keys.py less more

31	31
32	32	yield defer.ensureDeferred(self.store.store_device("user", "device", None))
33	33
34		yield self.store.set_e2e_device_keys("user", "device", now, json)
	34	yield defer.ensureDeferred(
	35	self.store.set_e2e_device_keys("user", "device", now, json)
	36	)
35	37
36	38	res = yield defer.ensureDeferred(
37		self.store.get_e2e_device_keys((("user", "device"),))
	39	self.store.get_e2e_device_keys_for_cs_api((("user", "device"),))
38	40	)
39	41	self.assertIn("user", res)
40	42	self.assertIn("device", res["user"])

48	50
49	51	yield defer.ensureDeferred(self.store.store_device("user", "device", None))
50	52
51		changed = yield self.store.set_e2e_device_keys("user", "device", now, json)
	53	changed = yield defer.ensureDeferred(
	54	self.store.set_e2e_device_keys("user", "device", now, json)
	55	)
52	56	self.assertTrue(changed)
53	57
54	58	# If we try to upload the same key then we should be told nothing
55	59	# changed
56		changed = yield self.store.set_e2e_device_keys("user", "device", now, json)
	60	changed = yield defer.ensureDeferred(
	61	self.store.set_e2e_device_keys("user", "device", now, json)
	62	)
57	63	self.assertFalse(changed)
58	64
59	65	@defer.inlineCallbacks

61	67	now = 1470174257070
62	68	json = {"key": "value"}
63	69
64		yield self.store.set_e2e_device_keys("user", "device", now, json)
	70	yield defer.ensureDeferred(
	71	self.store.set_e2e_device_keys("user", "device", now, json)
	72	)
65	73	yield defer.ensureDeferred(
66	74	self.store.store_device("user", "device", "display_name")
67	75	)
68	76
69	77	res = yield defer.ensureDeferred(
70		self.store.get_e2e_device_keys((("user", "device"),))
	78	self.store.get_e2e_device_keys_for_cs_api((("user", "device"),))
71	79	)
72	80	self.assertIn("user", res)
73	81	self.assertIn("device", res["user"])

85	93	yield defer.ensureDeferred(self.store.store_device("user2", "device1", None))
86	94	yield defer.ensureDeferred(self.store.store_device("user2", "device2", None))
87	95
88		yield self.store.set_e2e_device_keys("user1", "device1", now, {"key": "json11"})
89		yield self.store.set_e2e_device_keys("user1", "device2", now, {"key": "json12"})
90		yield self.store.set_e2e_device_keys("user2", "device1", now, {"key": "json21"})
91		yield self.store.set_e2e_device_keys("user2", "device2", now, {"key": "json22"})
	96	yield defer.ensureDeferred(
	97	self.store.set_e2e_device_keys("user1", "device1", now, {"key": "json11"})
	98	)
	99	yield defer.ensureDeferred(
	100	self.store.set_e2e_device_keys("user1", "device2", now, {"key": "json12"})
	101	)
	102	yield defer.ensureDeferred(
	103	self.store.set_e2e_device_keys("user2", "device1", now, {"key": "json21"})
	104	)
	105	yield defer.ensureDeferred(
	106	self.store.set_e2e_device_keys("user2", "device2", now, {"key": "json22"})
	107	)
92	108
93	109	res = yield defer.ensureDeferred(
94		self.store.get_e2e_device_keys((("user1", "device1"), ("user2", "device2")))
	110	self.store.get_e2e_device_keys_for_cs_api(
	111	(("user1", "device1"), ("user2", "device2"))
	112	)
95	113	)
96	114	self.assertIn("user1", res)
97	115	self.assertIn("device1", res["user1"])

+1

-1

tests/storage/test_event_metrics.py less more

26	26	room_creator = self.hs.get_room_creation_handler()
27	27
28	28	user = UserID("alice", "test")
29		requester = Requester(user, None, False, None, None)
	29	requester = Requester(user, None, False, False, None, None)
30	30
31	31	# Real events, forward extremities
32	32	events = [(3, 2), (6, 2), (4, 6)]

+81

-45

tests/storage/test_event_push_actions.py less more

59	59
60	60	@defer.inlineCallbacks
61	61	def _assert_counts(noitf_count, highlight_count):
62		counts = yield self.store.db_pool.runInteraction(
63		"", self.store._get_unread_counts_by_pos_txn, room_id, user_id, 0
	62	counts = yield defer.ensureDeferred(
	63	self.store.db_pool.runInteraction(
	64	"", self.store._get_unread_counts_by_pos_txn, room_id, user_id, 0
	65	)
64	66	)
65	67	self.assertEquals(
66	68	counts,
67		{"notify_count": noitf_count, "highlight_count": highlight_count},
	69	{
	70	"notify_count": noitf_count,
	71	"unread_count": 0, # Unread counts are tested in the sync tests.
	72	"highlight_count": highlight_count,
	73	},
68	74	)
69	75
70	76	@defer.inlineCallbacks

77	83
78	84	yield defer.ensureDeferred(
79	85	self.store.add_push_actions_to_staging(
80		event.event_id, {user_id: action}
81		)
82		)
83		yield self.store.db_pool.runInteraction(
84		"",
85		self.persist_events_store._set_push_actions_for_event_and_users_txn,
86		[(event, None)],
87		[(event, None)],
	86	event.event_id, {user_id: action}, False,
	87	)
	88	)
	89	yield defer.ensureDeferred(
	90	self.store.db_pool.runInteraction(
	91	"",
	92	self.persist_events_store._set_push_actions_for_event_and_users_txn,
	93	[(event, None)],
	94	[(event, None)],
	95	)
88	96	)
89	97
90	98	def _rotate(stream):
91		return self.store.db_pool.runInteraction(
92		"", self.store._rotate_notifs_before_txn, stream
	99	return defer.ensureDeferred(
	100	self.store.db_pool.runInteraction(
	101	"", self.store._rotate_notifs_before_txn, stream
	102	)
93	103	)
94	104
95	105	def _mark_read(stream, depth):
96		return self.store.db_pool.runInteraction(
97		"",
98		self.store._remove_old_push_actions_before_txn,
99		room_id,
100		user_id,
101		stream,
	106	return defer.ensureDeferred(
	107	self.store.db_pool.runInteraction(
	108	"",
	109	self.store._remove_old_push_actions_before_txn,
	110	room_id,
	111	user_id,
	112	stream,
	113	)
102	114	)
103	115
104	116	yield _assert_counts(0, 0)

122	134	yield _inject_actions(6, PlAIN_NOTIF)
123	135	yield _rotate(7)
124	136
125		yield self.store.db_pool.simple_delete(
126		table="event_push_actions", keyvalues={"1": 1}, desc=""
	137	yield defer.ensureDeferred(
	138	self.store.db_pool.simple_delete(
	139	table="event_push_actions", keyvalues={"1": 1}, desc=""
	140	)
127	141	)
128	142
129	143	yield _assert_counts(1, 0)

141	155	@defer.inlineCallbacks
142	156	def test_find_first_stream_ordering_after_ts(self):
143	157	def add_event(so, ts):
144		return self.store.db_pool.simple_insert(
145		"events",
146		{
147		"stream_ordering": so,
148		"received_ts": ts,
149		"event_id": "event%i" % so,
150		"type": "",
151		"room_id": "",
152		"content": "",
153		"processed": True,
154		"outlier": False,
155		"topological_ordering": 0,
156		"depth": 0,
157		},
	158	return defer.ensureDeferred(
	159	self.store.db_pool.simple_insert(
	160	"events",
	161	{
	162	"stream_ordering": so,
	163	"received_ts": ts,
	164	"event_id": "event%i" % so,
	165	"type": "",
	166	"room_id": "",
	167	"content": "",
	168	"processed": True,
	169	"outlier": False,
	170	"topological_ordering": 0,
	171	"depth": 0,
	172	},
	173	)
158	174	)
159	175
160	176	# start with the base case where there are no events in the table
161		r = yield self.store.find_first_stream_ordering_after_ts(11)
	177	r = yield defer.ensureDeferred(
	178	self.store.find_first_stream_ordering_after_ts(11)
	179	)
162	180	self.assertEqual(r, 0)
163	181
164	182	# now with one event
165	183	yield add_event(2, 10)
166		r = yield self.store.find_first_stream_ordering_after_ts(9)
	184	r = yield defer.ensureDeferred(
	185	self.store.find_first_stream_ordering_after_ts(9)
	186	)
167	187	self.assertEqual(r, 2)
168		r = yield self.store.find_first_stream_ordering_after_ts(10)
	188	r = yield defer.ensureDeferred(
	189	self.store.find_first_stream_ordering_after_ts(10)
	190	)
169	191	self.assertEqual(r, 2)
170		r = yield self.store.find_first_stream_ordering_after_ts(11)
	192	r = yield defer.ensureDeferred(
	193	self.store.find_first_stream_ordering_after_ts(11)
	194	)
171	195	self.assertEqual(r, 3)
172	196
173	197	# add a bunch of dummy events to the events table

180	204	):
181	205	yield add_event(stream_ordering, ts)
182	206
183		r = yield self.store.find_first_stream_ordering_after_ts(110)
	207	r = yield defer.ensureDeferred(
	208	self.store.find_first_stream_ordering_after_ts(110)
	209	)
184	210	self.assertEqual(r, 3, "First event after 110ms should be 3, was %i" % r)
185	211
186	212	# 4 and 5 are both after 120: we want 4 rather than 5
187		r = yield self.store.find_first_stream_ordering_after_ts(120)
	213	r = yield defer.ensureDeferred(
	214	self.store.find_first_stream_ordering_after_ts(120)
	215	)
188	216	self.assertEqual(r, 4, "First event after 120ms should be 4, was %i" % r)
189	217
190		r = yield self.store.find_first_stream_ordering_after_ts(129)
	218	r = yield defer.ensureDeferred(
	219	self.store.find_first_stream_ordering_after_ts(129)
	220	)
191	221	self.assertEqual(r, 10, "First event after 129ms should be 10, was %i" % r)
192	222
193	223	# check we can get the last event
194		r = yield self.store.find_first_stream_ordering_after_ts(140)
	224	r = yield defer.ensureDeferred(
	225	self.store.find_first_stream_ordering_after_ts(140)
	226	)
195	227	self.assertEqual(r, 20, "First event after 14ms should be 20, was %i" % r)
196	228
197	229	# off the end
198		r = yield self.store.find_first_stream_ordering_after_ts(160)
	230	r = yield defer.ensureDeferred(
	231	self.store.find_first_stream_ordering_after_ts(160)
	232	)
199	233	self.assertEqual(r, 21)
200	234
201	235	# check we can find an event at ordering zero
202	236	yield add_event(0, 5)
203		r = yield self.store.find_first_stream_ordering_after_ts(1)
	237	r = yield defer.ensureDeferred(
	238	self.store.find_first_stream_ordering_after_ts(1)
	239	)
204	240	self.assertEqual(r, 0)

+196

-9

tests/storage/test_id_generators.py less more

57	57	return self.get_success(self.db_pool.runWithConnection(_create))
58	58
59	59	def _insert_rows(self, instance_name: str, number: int):
	60	"""Insert N rows as the given instance, inserting with stream IDs pulled
	61	from the postgres sequence.
	62	"""
	63
60	64	def _insert(txn):
61	65	for _ in range(number):
62	66	txn.execute(

64	68	(instance_name,),
65	69	)
66	70
67		self.get_success(self.db_pool.runInteraction("test_single_instance", _insert))
	71	self.get_success(self.db_pool.runInteraction("_insert_rows", _insert))
	72
	73	def _insert_row_with_id(self, instance_name: str, stream_id: int):
	74	"""Insert one row as the given instance with given stream_id, updating
	75	the postgres sequence position to match.
	76	"""
	77
	78	def _insert(txn):
	79	txn.execute(
	80	"INSERT INTO foobar VALUES (?, ?)", (stream_id, instance_name,),
	81	)
	82	txn.execute("SELECT setval('foobar_seq', ?)", (stream_id,))
	83
	84	self.get_success(self.db_pool.runInteraction("_insert_row_with_id", _insert))
68	85
69	86	def test_empty(self):
70	87	"""Test an ID generator against an empty database gives sensible

87	104	id_gen = self._create_id_generator()
88	105
89	106	self.assertEqual(id_gen.get_positions(), {"master": 7})
90		self.assertEqual(id_gen.get_current_token("master"), 7)
	107	self.assertEqual(id_gen.get_current_token_for_writer("master"), 7)
91	108
92	109	# Try allocating a new ID gen and check that we only see position
93	110	# advanced after we leave the context manager.

97	114	self.assertEqual(stream_id, 8)
98	115
99	116	self.assertEqual(id_gen.get_positions(), {"master": 7})
100		self.assertEqual(id_gen.get_current_token("master"), 7)
	117	self.assertEqual(id_gen.get_current_token_for_writer("master"), 7)
101	118
102	119	self.get_success(_get_next_async())
103	120
104	121	self.assertEqual(id_gen.get_positions(), {"master": 8})
105		self.assertEqual(id_gen.get_current_token("master"), 8)
	122	self.assertEqual(id_gen.get_current_token_for_writer("master"), 8)
106	123
107	124	def test_multi_instance(self):
108	125	"""Test that reads and writes from multiple processes are handled

115	132	second_id_gen = self._create_id_generator("second")
116	133
117	134	self.assertEqual(first_id_gen.get_positions(), {"first": 3, "second": 7})
118		self.assertEqual(first_id_gen.get_current_token("first"), 3)
119		self.assertEqual(first_id_gen.get_current_token("second"), 7)
	135	self.assertEqual(first_id_gen.get_current_token_for_writer("first"), 3)
	136	self.assertEqual(first_id_gen.get_current_token_for_writer("second"), 7)
120	137
121	138	# Try allocating a new ID gen and check that we only see position
122	139	# advanced after we leave the context manager.

165	182	id_gen = self._create_id_generator()
166	183
167	184	self.assertEqual(id_gen.get_positions(), {"master": 7})
168		self.assertEqual(id_gen.get_current_token("master"), 7)
	185	self.assertEqual(id_gen.get_current_token_for_writer("master"), 7)
169	186
170	187	# Try allocating a new ID gen and check that we only see position
171	188	# advanced after we leave the context manager.

175	192	self.assertEqual(stream_id, 8)
176	193
177	194	self.assertEqual(id_gen.get_positions(), {"master": 7})
178		self.assertEqual(id_gen.get_current_token("master"), 7)
	195	self.assertEqual(id_gen.get_current_token_for_writer("master"), 7)
179	196
180	197	self.get_success(self.db_pool.runInteraction("test", _get_next_txn))
181	198
182	199	self.assertEqual(id_gen.get_positions(), {"master": 8})
183		self.assertEqual(id_gen.get_current_token("master"), 8)
	200	self.assertEqual(id_gen.get_current_token_for_writer("master"), 8)
	201
	202	def test_get_persisted_upto_position(self):
	203	"""Test that `get_persisted_upto_position` correctly tracks updates to
	204	positions.
	205	"""
	206
	207	# The following tests are a bit cheeky in that we notify about new
	208	# positions via `advance` without actually advancing the postgres
	209	# sequence.
	210
	211	self._insert_row_with_id("first", 3)
	212	self._insert_row_with_id("second", 5)
	213
	214	id_gen = self._create_id_generator("first")
	215
	216	self.assertEqual(id_gen.get_positions(), {"first": 3, "second": 5})
	217
	218	# Min is 3 and there is a gap between 5, so we expect it to be 3.
	219	self.assertEqual(id_gen.get_persisted_upto_position(), 3)
	220
	221	# We advance "first" straight to 6. Min is now 5 but there is no gap so
	222	# we expect it to be 6
	223	id_gen.advance("first", 6)
	224	self.assertEqual(id_gen.get_persisted_upto_position(), 6)
	225
	226	# No gap, so we expect 7.
	227	id_gen.advance("second", 7)
	228	self.assertEqual(id_gen.get_persisted_upto_position(), 7)
	229
	230	# We haven't seen 8 yet, so we expect 7 still.
	231	id_gen.advance("second", 9)
	232	self.assertEqual(id_gen.get_persisted_upto_position(), 7)
	233
	234	# Now that we've seen 7, 8 and 9 we can got straight to 9.
	235	id_gen.advance("first", 8)
	236	self.assertEqual(id_gen.get_persisted_upto_position(), 9)
	237
	238	# Jump forward with gaps. The minimum is 11, even though we haven't seen
	239	# 10 we know that everything before 11 must be persisted.
	240	id_gen.advance("first", 11)
	241	id_gen.advance("second", 15)
	242	self.assertEqual(id_gen.get_persisted_upto_position(), 11)
	243
	244	def test_get_persisted_upto_position_get_next(self):
	245	"""Test that `get_persisted_upto_position` correctly tracks updates to
	246	positions when `get_next` is called.
	247	"""
	248
	249	self._insert_row_with_id("first", 3)
	250	self._insert_row_with_id("second", 5)
	251
	252	id_gen = self._create_id_generator("first")
	253
	254	self.assertEqual(id_gen.get_positions(), {"first": 3, "second": 5})
	255
	256	self.assertEqual(id_gen.get_persisted_upto_position(), 3)
	257	with self.get_success(id_gen.get_next()) as stream_id:
	258	self.assertEqual(stream_id, 6)
	259	self.assertEqual(id_gen.get_persisted_upto_position(), 3)
	260
	261	self.assertEqual(id_gen.get_persisted_upto_position(), 6)
	262
	263	# We assume that so long as `get_next` does correctly advance the
	264	# `persisted_upto_position` in this case, then it will be correct in the
	265	# other cases that are tested above (since they'll hit the same code).
	266
	267
	268	class BackwardsMultiWriterIdGeneratorTestCase(HomeserverTestCase):
	269	"""Tests MultiWriterIdGenerator that produce negative stream IDs.
	270	"""
	271
	272	if not USE_POSTGRES_FOR_TESTS:
	273	skip = "Requires Postgres"
	274
	275	def prepare(self, reactor, clock, hs):
	276	self.store = hs.get_datastore()
	277	self.db_pool = self.store.db_pool # type: DatabasePool
	278
	279	self.get_success(self.db_pool.runInteraction("_setup_db", self._setup_db))
	280
	281	def _setup_db(self, txn):
	282	txn.execute("CREATE SEQUENCE foobar_seq")
	283	txn.execute(
	284	"""
	285	CREATE TABLE foobar (
	286	stream_id BIGINT NOT NULL,
	287	instance_name TEXT NOT NULL,
	288	data TEXT
	289	);
	290	"""
	291	)
	292
	293	def _create_id_generator(self, instance_name="master") -> MultiWriterIdGenerator:
	294	def _create(conn):
	295	return MultiWriterIdGenerator(
	296	conn,
	297	self.db_pool,
	298	instance_name=instance_name,
	299	table="foobar",
	300	instance_column="instance_name",
	301	id_column="stream_id",
	302	sequence_name="foobar_seq",
	303	positive=False,
	304	)
	305
	306	return self.get_success(self.db_pool.runWithConnection(_create))
	307
	308	def _insert_row(self, instance_name: str, stream_id: int):
	309	"""Insert one row as the given instance with given stream_id.
	310	"""
	311
	312	def _insert(txn):
	313	txn.execute(
	314	"INSERT INTO foobar VALUES (?, ?)", (stream_id, instance_name,),
	315	)
	316
	317	self.get_success(self.db_pool.runInteraction("_insert_row", _insert))
	318
	319	def test_single_instance(self):
	320	"""Test that reads and writes from a single process are handled
	321	correctly.
	322	"""
	323	id_gen = self._create_id_generator()
	324
	325	with self.get_success(id_gen.get_next()) as stream_id:
	326	self._insert_row("master", stream_id)
	327
	328	self.assertEqual(id_gen.get_positions(), {"master": -1})
	329	self.assertEqual(id_gen.get_current_token_for_writer("master"), -1)
	330	self.assertEqual(id_gen.get_persisted_upto_position(), -1)
	331
	332	with self.get_success(id_gen.get_next_mult(3)) as stream_ids:
	333	for stream_id in stream_ids:
	334	self._insert_row("master", stream_id)
	335
	336	self.assertEqual(id_gen.get_positions(), {"master": -4})
	337	self.assertEqual(id_gen.get_current_token_for_writer("master"), -4)
	338	self.assertEqual(id_gen.get_persisted_upto_position(), -4)
	339
	340	# Test loading from DB by creating a second ID gen
	341	second_id_gen = self._create_id_generator()
	342
	343	self.assertEqual(second_id_gen.get_positions(), {"master": -4})
	344	self.assertEqual(second_id_gen.get_current_token_for_writer("master"), -4)
	345	self.assertEqual(second_id_gen.get_persisted_upto_position(), -4)
	346
	347	def test_multiple_instance(self):
	348	"""Tests that having multiple instances that get advanced over
	349	federation works corretly.
	350	"""
	351	id_gen_1 = self._create_id_generator("first")
	352	id_gen_2 = self._create_id_generator("second")
	353
	354	with self.get_success(id_gen_1.get_next()) as stream_id:
	355	self._insert_row("first", stream_id)
	356	id_gen_2.advance("first", stream_id)
	357
	358	self.assertEqual(id_gen_1.get_positions(), {"first": -1})
	359	self.assertEqual(id_gen_2.get_positions(), {"first": -1})
	360	self.assertEqual(id_gen_1.get_persisted_upto_position(), -1)
	361	self.assertEqual(id_gen_2.get_persisted_upto_position(), -1)
	362
	363	with self.get_success(id_gen_2.get_next()) as stream_id:
	364	self._insert_row("second", stream_id)
	365	id_gen_1.advance("second", stream_id)
	366
	367	self.assertEqual(id_gen_1.get_positions(), {"first": -1, "second": -2})
	368	self.assertEqual(id_gen_2.get_positions(), {"first": -1, "second": -2})
	369	self.assertEqual(id_gen_1.get_persisted_upto_position(), -2)
	370	self.assertEqual(id_gen_2.get_persisted_upto_position(), -2)

+9

-5

tests/storage/test_main.py less more

33	33
34	34	@defer.inlineCallbacks
35	35	def test_get_users_paginate(self):
36		yield self.store.register_user(self.user.to_string(), "pass")
37		yield self.store.create_profile(self.user.localpart)
38		yield self.store.set_profile_displayname(self.user.localpart, self.displayname)
	36	yield defer.ensureDeferred(
	37	self.store.register_user(self.user.to_string(), "pass")
	38	)
	39	yield defer.ensureDeferred(self.store.create_profile(self.user.localpart))
	40	yield defer.ensureDeferred(
	41	self.store.set_profile_displayname(self.user.localpart, self.displayname)
	42	)
39	43
40		users, total = yield self.store.get_users_paginate(
41		0, 10, name="bc", guests=False
	44	users, total = yield defer.ensureDeferred(
	45	self.store.get_users_paginate(0, 10, name="bc", guests=False)
42	46	)
43	47
44	48	self.assertEquals(1, total)

+20

-7

tests/storage/test_profile.py less more

32	32
33	33	@defer.inlineCallbacks
34	34	def test_displayname(self):
35		yield self.store.create_profile(self.u_frank.localpart)
	35	yield defer.ensureDeferred(self.store.create_profile(self.u_frank.localpart))
36	36
37		yield self.store.set_profile_displayname(self.u_frank.localpart, "Frank")
	37	yield defer.ensureDeferred(
	38	self.store.set_profile_displayname(self.u_frank.localpart, "Frank")
	39	)
38	40
39	41	self.assertEquals(
40		"Frank", (yield self.store.get_profile_displayname(self.u_frank.localpart))
	42	"Frank",
	43	(
	44	yield defer.ensureDeferred(
	45	self.store.get_profile_displayname(self.u_frank.localpart)
	46	)
	47	),
41	48	)
42	49
43	50	@defer.inlineCallbacks
44	51	def test_avatar_url(self):
45		yield self.store.create_profile(self.u_frank.localpart)
	52	yield defer.ensureDeferred(self.store.create_profile(self.u_frank.localpart))
46	53
47		yield self.store.set_profile_avatar_url(
48		self.u_frank.localpart, "http://my.site/here"
	54	yield defer.ensureDeferred(
	55	self.store.set_profile_avatar_url(
	56	self.u_frank.localpart, "http://my.site/here"
	57	)
49	58	)
50	59
51	60	self.assertEquals(
52	61	"http://my.site/here",
53		(yield self.store.get_profile_avatar_url(self.u_frank.localpart)),
	62	(
	63	yield defer.ensureDeferred(
	64	self.store.get_profile_avatar_url(self.u_frank.localpart)
	65	)
	66	),
54	67	)

+16

-33

tests/storage/test_purge.py less more

14	14
15	15	from twisted.internet import defer
16	16
	17	from synapse.api.errors import NotFoundError
17	18	from synapse.rest.client.v1 import room
18	19
19	20	from tests.unittest import HomeserverTestCase

45	46	storage = self.hs.get_storage()
46	47
47	48	# Get the topological token
48		event = store.get_topological_token_for_event(last["event_id"])
49		self.pump()
50		event = self.successResultOf(event)
	49	event = self.get_success(
	50	store.get_topological_token_for_event(last["event_id"])
	51	)
51	52
52	53	# Purge everything before this topological token
53		purge = defer.ensureDeferred(
54		storage.purge_events.purge_history(self.room_id, event, True)
55		)
56		self.pump()
57		self.assertEqual(self.successResultOf(purge), None)
58
59		# Try and get the events
60		get_first = store.get_event(first["event_id"])
61		get_second = store.get_event(second["event_id"])
62		get_third = store.get_event(third["event_id"])
63		get_last = store.get_event(last["event_id"])
64		self.pump()
	54	self.get_success(storage.purge_events.purge_history(self.room_id, event, True))
65	55
66	56	# 1-3 should fail and last will succeed, meaning that 1-3 are deleted
67	57	# and last is not.
68		self.failureResultOf(get_first)
69		self.failureResultOf(get_second)
70		self.failureResultOf(get_third)
71		self.successResultOf(get_last)
	58	self.get_failure(store.get_event(first["event_id"]), NotFoundError)
	59	self.get_failure(store.get_event(second["event_id"]), NotFoundError)
	60	self.get_failure(store.get_event(third["event_id"]), NotFoundError)
	61	self.get_success(store.get_event(last["event_id"]))
72	62
73	63	def test_purge_wont_delete_extrems(self):
74	64	"""

83	73	storage = self.hs.get_datastore()
84	74
85	75	# Set the topological token higher than it should be
86		event = storage.get_topological_token_for_event(last["event_id"])
87		self.pump()
88		event = self.successResultOf(event)
	76	event = self.get_success(
	77	storage.get_topological_token_for_event(last["event_id"])
	78	)
89	79	event = "t{}-{}".format(
90	80	*list(map(lambda x: x + 1, map(int, event[1:].split("-"))))
91	81	)

97	87	self.assertIn("greater than forward", f.value.args[0])
98	88
99	89	# Try and get the events
100		get_first = storage.get_event(first["event_id"])
101		get_second = storage.get_event(second["event_id"])
102		get_third = storage.get_event(third["event_id"])
103		get_last = storage.get_event(last["event_id"])
104		self.pump()
105
106		# Nothing is deleted.
107		self.successResultOf(get_first)
108		self.successResultOf(get_second)
109		self.successResultOf(get_third)
110		self.successResultOf(get_last)
	90	self.get_success(storage.get_event(first["event_id"]))
	91	self.get_success(storage.get_event(second["event_id"]))
	92	self.get_success(storage.get_event(third["event_id"]))
	93	self.get_success(storage.get_event(last["event_id"]))

+60

-17

tests/storage/test_registration.py less more

16	16	from twisted.internet import defer
17	17
18	18	from synapse.api.constants import UserTypes
	19	from synapse.api.errors import ThreepidValidationError
19	20
20	21	from tests import unittest
21	22	from tests.utils import setup_test_homeserver

35	36
36	37	@defer.inlineCallbacks
37	38	def test_register(self):
38		yield self.store.register_user(self.user_id, self.pwhash)
	39	yield defer.ensureDeferred(self.store.register_user(self.user_id, self.pwhash))
39	40
40	41	self.assertEquals(
41	42	{

51	52	"user_type": None,
52	53	"deactivated": 0,
53	54	},
54		(yield self.store.get_user_by_id(self.user_id)),
	55	(yield defer.ensureDeferred(self.store.get_user_by_id(self.user_id))),
55	56	)
56	57
57	58	@defer.inlineCallbacks
58	59	def test_add_tokens(self):
59		yield self.store.register_user(self.user_id, self.pwhash)
	60	yield defer.ensureDeferred(self.store.register_user(self.user_id, self.pwhash))
60	61	yield defer.ensureDeferred(
61	62	self.store.add_access_token_to_user(
62	63	self.user_id, self.tokens[1], self.device_id, valid_until_ms=None
63	64	)
64	65	)
65	66
66		result = yield self.store.get_user_by_access_token(self.tokens[1])
	67	result = yield defer.ensureDeferred(
	68	self.store.get_user_by_access_token(self.tokens[1])
	69	)
67	70
68	71	self.assertDictContainsSubset(
69	72	{"name": self.user_id, "device_id": self.device_id}, result

74	77	@defer.inlineCallbacks
75	78	def test_user_delete_access_tokens(self):
76	79	# add some tokens
77		yield self.store.register_user(self.user_id, self.pwhash)
	80	yield defer.ensureDeferred(self.store.register_user(self.user_id, self.pwhash))
78	81	yield defer.ensureDeferred(
79	82	self.store.add_access_token_to_user(
80	83	self.user_id, self.tokens[0], device_id=None, valid_until_ms=None

87	90	)
88	91
89	92	# now delete some
90		yield self.store.user_delete_access_tokens(
91		self.user_id, device_id=self.device_id
	93	yield defer.ensureDeferred(
	94	self.store.user_delete_access_tokens(self.user_id, device_id=self.device_id)
92	95	)
93	96
94	97	# check they were deleted
95		user = yield self.store.get_user_by_access_token(self.tokens[1])
	98	user = yield defer.ensureDeferred(
	99	self.store.get_user_by_access_token(self.tokens[1])
	100	)
96	101	self.assertIsNone(user, "access token was not deleted by device_id")
97	102
98	103	# check the one not associated with the device was not deleted
99		user = yield self.store.get_user_by_access_token(self.tokens[0])
	104	user = yield defer.ensureDeferred(
	105	self.store.get_user_by_access_token(self.tokens[0])
	106	)
100	107	self.assertEqual(self.user_id, user["name"])
101	108
102	109	# now delete the rest
103		yield self.store.user_delete_access_tokens(self.user_id)
	110	yield defer.ensureDeferred(self.store.user_delete_access_tokens(self.user_id))
104	111
105		user = yield self.store.get_user_by_access_token(self.tokens[0])
	112	user = yield defer.ensureDeferred(
	113	self.store.get_user_by_access_token(self.tokens[0])
	114	)
106	115	self.assertIsNone(user, "access token was not deleted without device_id")
107	116
108	117	@defer.inlineCallbacks

110	119	TEST_USER = "@test:test"
111	120	SUPPORT_USER = "@support:test"
112	121
113		res = yield self.store.is_support_user(None)
	122	res = yield defer.ensureDeferred(self.store.is_support_user(None))
114	123	self.assertFalse(res)
115		yield self.store.register_user(user_id=TEST_USER, password_hash=None)
116		res = yield self.store.is_support_user(TEST_USER)
	124	yield defer.ensureDeferred(
	125	self.store.register_user(user_id=TEST_USER, password_hash=None)
	126	)
	127	res = yield defer.ensureDeferred(self.store.is_support_user(TEST_USER))
117	128	self.assertFalse(res)
118	129
119		yield self.store.register_user(
120		user_id=SUPPORT_USER, password_hash=None, user_type=UserTypes.SUPPORT
	130	yield defer.ensureDeferred(
	131	self.store.register_user(
	132	user_id=SUPPORT_USER, password_hash=None, user_type=UserTypes.SUPPORT
	133	)
121	134	)
122		res = yield self.store.is_support_user(SUPPORT_USER)
	135	res = yield defer.ensureDeferred(self.store.is_support_user(SUPPORT_USER))
123	136	self.assertTrue(res)
	137
	138	@defer.inlineCallbacks
	139	def test_3pid_inhibit_invalid_validation_session_error(self):
	140	"""Tests that enabling the configuration option to inhibit 3PID errors on
	141	/requestToken also inhibits validation errors caused by an unknown session ID.
	142	"""
	143
	144	# Check that, with the config setting set to false (the default value), a
	145	# validation error is caused by the unknown session ID.
	146	try:
	147	yield defer.ensureDeferred(
	148	self.store.validate_threepid_session(
	149	"fake_sid", "fake_client_secret", "fake_token", 0,
	150	)
	151	)
	152	except ThreepidValidationError as e:
	153	self.assertEquals(e.msg, "Unknown session_id", e)
	154
	155	# Set the config setting to true.
	156	self.store._ignore_unknown_session_error = True
	157
	158	# Check that now the validation error is caused by the token not matching.
	159	try:
	160	yield defer.ensureDeferred(
	161	self.store.validate_threepid_session(
	162	"fake_sid", "fake_client_secret", "fake_token", 0,
	163	)
	164	)
	165	except ThreepidValidationError as e:
	166	self.assertEquals(e.msg, "Validation token not found or has expired", e)

+16

-4

tests/storage/test_room.py less more

53	53	"creator": self.u_creator.to_string(),
54	54	"is_public": True,
55	55	},
56		(yield self.store.get_room(self.room.to_string())),
	56	(yield defer.ensureDeferred(self.store.get_room(self.room.to_string()))),
57	57	)
58	58
59	59	@defer.inlineCallbacks
60	60	def test_get_room_unknown_room(self):
61		self.assertIsNone((yield self.store.get_room("!uknown:test")),)
	61	self.assertIsNone(
	62	(yield defer.ensureDeferred(self.store.get_room("!uknown:test")))
	63	)
62	64
63	65	@defer.inlineCallbacks
64	66	def test_get_room_with_stats(self):

68	70	"creator": self.u_creator.to_string(),
69	71	"public": True,
70	72	},
71		(yield self.store.get_room_with_stats(self.room.to_string())),
	73	(
	74	yield defer.ensureDeferred(
	75	self.store.get_room_with_stats(self.room.to_string())
	76	)
	77	),
72	78	)
73	79
74	80	@defer.inlineCallbacks
75	81	def test_get_room_with_stats_unknown_room(self):
76		self.assertIsNone((yield self.store.get_room_with_stats("!uknown:test")),)
	82	self.assertIsNone(
	83	(
	84	yield defer.ensureDeferred(
	85	self.store.get_room_with_stats("!uknown:test")
	86	)
	87	),
	88	)
77	89
78	90
79	91	class RoomEventsStoreTestCase(unittest.TestCase):

+4

-4

tests/storage/test_roommember.py less more

86	86	self.inject_room_member(self.room, self.u_bob, Membership.JOIN)
87	87	self.inject_room_member(self.room, self.u_charlie.to_string(), Membership.JOIN)
88	88
89		self.pump(20)
	89	self.pump()
90	90
91	91	self.assertTrue("_known_servers_count" not in self.store.__dict__.keys())
92	92

100	100	# Initialises to 1 -- itself
101	101	self.assertEqual(self.store._known_servers_count, 1)
102	102
103		self.pump(20)
	103	self.pump()
104	104
105	105	# No rooms have been joined, so technically the SQL returns 0, but it
106	106	# will still say it knows about itself.

110	110	self.inject_room_member(self.room, self.u_bob, Membership.JOIN)
111	111	self.inject_room_member(self.room, self.u_charlie.to_string(), Membership.JOIN)
112	112
113		self.pump(20)
	113	self.pump(1)
114	114
115	115	# It now knows about Charlie's server.
116	116	self.assertEqual(self.store._known_servers_count, 2)

186	186
187	187	# Now let's create a room, which will insert a membership
188	188	user = UserID("alice", "test")
189		requester = Requester(user, None, False, None, None)
	189	requester = Requester(user, None, False, False, None, None)
190	190	self.get_success(self.room_creator.create_room(requester, {}))
191	191
192	192	# Register the background update to run again.

+12

-4

tests/storage/test_user_directory.py less more

30	30
31	31	# alice and bob are both in !room_id. bobby is not but shares
32	32	# a homeserver with alice.
33		yield self.store.update_profile_in_user_dir(ALICE, "alice", None)
34		yield self.store.update_profile_in_user_dir(BOB, "bob", None)
35		yield self.store.update_profile_in_user_dir(BOBBY, "bobby", None)
36		yield self.store.add_users_in_public_rooms("!room:id", (ALICE, BOB))
	33	yield defer.ensureDeferred(
	34	self.store.update_profile_in_user_dir(ALICE, "alice", None)
	35	)
	36	yield defer.ensureDeferred(
	37	self.store.update_profile_in_user_dir(BOB, "bob", None)
	38	)
	39	yield defer.ensureDeferred(
	40	self.store.update_profile_in_user_dir(BOBBY, "bobby", None)
	41	)
	42	yield defer.ensureDeferred(
	43	self.store.add_users_in_public_rooms("!room:id", (ALICE, BOB))
	44	)
37	45
38	46	@defer.inlineCallbacks
39	47	def test_search_user_dir(self):

+25

-33

tests/test_federation.py less more

14	14
15	15	from mock import Mock
16	16
17		from twisted.internet.defer import ensureDeferred, maybeDeferred, succeed
18
	17	from twisted.internet.defer import succeed
	18
	19	from synapse.api.errors import FederationError
19	20	from synapse.events import make_event_from_dict
20	21	from synapse.logging.context import LoggingContext
21	22	from synapse.types import Requester, UserID

41	42	)
42	43
43	44	user_id = UserID("us", "test")
44		our_user = Requester(user_id, None, False, None, None)
	45	our_user = Requester(user_id, None, False, False, None, None)
45	46	room_creator = self.homeserver.get_room_creation_handler()
46		room_deferred = ensureDeferred(
	47	self.room_id = self.get_success(
47	48	room_creator.create_room(
48	49	our_user, room_creator._presets_dict["public_chat"], ratelimit=False
49	50	)
50		)
51		self.reactor.advance(0.1)
52		self.room_id = self.successResultOf(room_deferred)[0]["room_id"]
	51	)[0]["room_id"]
53	52
54	53	self.store = self.homeserver.get_datastore()
55	54
56	55	# Figure out what the most recent event is
57		most_recent = self.successResultOf(
58		maybeDeferred(
59		self.homeserver.get_datastore().get_latest_event_ids_in_room,
60		self.room_id,
61		)
	56	most_recent = self.get_success(
	57	self.homeserver.get_datastore().get_latest_event_ids_in_room(self.room_id)
62	58	)[0]
63	59
64	60	join_event = make_event_from_dict(

88	84	)
89	85
90	86	# Send the join, it should return None (which is not an error)
91		d = ensureDeferred(
92		self.handler.on_receive_pdu(
93		"test.serv", join_event, sent_to_us_directly=True
94		)
95		)
96		self.reactor.advance(1)
97		self.assertEqual(self.successResultOf(d), None)
	87	self.assertEqual(
	88	self.get_success(
	89	self.handler.on_receive_pdu(
	90	"test.serv", join_event, sent_to_us_directly=True
	91	)
	92	),
	93	None,
	94	)
98	95
99	96	# Make sure we actually joined the room
100	97	self.assertEqual(
101		self.successResultOf(
102		maybeDeferred(self.store.get_latest_event_ids_in_room, self.room_id)
103		)[0],
	98	self.get_success(self.store.get_latest_event_ids_in_room(self.room_id))[0],
104	99	"$join:test.serv",
105	100	)
106	101

118	113	self.http_client.post_json = post_json
119	114
120	115	# Figure out what the most recent event is
121		most_recent = self.successResultOf(
122		maybeDeferred(self.store.get_latest_event_ids_in_room, self.room_id)
	116	most_recent = self.get_success(
	117	self.store.get_latest_event_ids_in_room(self.room_id)
123	118	)[0]
124	119
125	120	# Now lie about an event

139	134	)
140	135
141	136	with LoggingContext(request="lying_event"):
142		d = ensureDeferred(
	137	failure = self.get_failure(
143	138	self.handler.on_receive_pdu(
144	139	"test.serv", lying_event, sent_to_us_directly=True
145		)
146		)
147
148		# Step the reactor, so the database fetches come back
149		self.reactor.advance(1)
	140	),
	141	FederationError,
	142	)
150	143
151	144	# on_receive_pdu should throw an error
152		failure = self.failureResultOf(d)
153	145	self.assertEqual(
154	146	failure.value.args[0],
155	147	(

159	151	)
160	152
161	153	# Make sure the invalid event isn't there
162		extrem = maybeDeferred(self.store.get_latest_event_ids_in_room, self.room_id)
163		self.assertEqual(self.successResultOf(extrem)[0], "$join:test.serv")
	154	extrem = self.get_success(self.store.get_latest_event_ids_in_room(self.room_id))
	155	self.assertEqual(extrem[0], "$join:test.serv")
164	156
165	157	def test_retry_device_list_resync(self):
166	158	"""Tests that device lists are marked as stale if they couldn't be synced, and

+0

-1

tests/test_server.py less more

177	177
178	178	self.assertEqual(channel.result["code"], b"200")
179	179	self.assertNotIn("body", channel.result)
180		self.assertEqual(channel.headers.getRawHeaders(b"Content-Length"), [b"15"])
181	180
182	181
183	182	class OptionsResourceTests(unittest.TestCase):

+51

-45

tests/test_state.py less more

70	70	return event
71	71
72	72
73		class StateGroupStore(object):
	73	class StateGroupStore:
74	74	def __init__(self):
75	75	self._event_to_state_group = {}
76	76	self._group_to_state = {}

79	79
80	80	self._next_group = 1
81	81
82		def get_state_groups_ids(self, room_id, event_ids):
	82	async def get_state_groups_ids(self, room_id, event_ids):
83	83	groups = {}
84	84	for event_id in event_ids:
85	85	group = self._event_to_state_group.get(event_id)
86	86	if group:
87	87	groups[group] = self._group_to_state[group]
88	88
89		return defer.succeed(groups)
90
91		def store_state_group(
	89	return groups
	90
	91	async def store_state_group(
92	92	self, event_id, room_id, prev_group, delta_ids, current_state_ids
93	93	):
94	94	state_group = self._next_group

96	96
97	97	self._group_to_state[state_group] = dict(current_state_ids)
98	98
99		return defer.succeed(state_group)
100
101		def get_events(self, event_ids, **kwargs):
102		return defer.succeed(
103		{
104		e_id: self._event_id_to_event[e_id]
105		for e_id in event_ids
106		if e_id in self._event_id_to_event
107		}
108		)
109
110		def get_state_group_delta(self, name):
111		return defer.succeed((None, None))
	99	return state_group
	100
	101	async def get_events(self, event_ids, **kwargs):
	102	return {
	103	e_id: self._event_id_to_event[e_id]
	104	for e_id in event_ids
	105	if e_id in self._event_id_to_event
	106	}
	107
	108	async def get_state_group_delta(self, name):
	109	return (None, None)
112	110
113	111	def register_events(self, events):
114	112	for e in events:

120	118	def register_event_id_state_group(self, event_id, state_group):
121	119	self._event_to_state_group[event_id] = state_group
122	120
123		def get_room_version_id(self, room_id):
124		return defer.succeed(RoomVersions.V1.identifier)
	121	async def get_room_version_id(self, room_id):
	122	return RoomVersions.V1.identifier
125	123
126	124
127	125	class DictObj(dict):

130	128	self.__dict__ = self
131	129
132	130
133		class Graph(object):
	131	class Graph:
134	132	def __init__(self, nodes, edges):
135	133	events = {}
136	134	clobbered = set(events.keys())

475	473	create_event(type="test2", state_key=""),
476	474	]
477	475
478		group_name = yield self.store.store_state_group(
479		prev_event_id,
480		event.room_id,
481		None,
482		None,
483		{(e.type, e.state_key): e.event_id for e in old_state},
	476	group_name = yield defer.ensureDeferred(
	477	self.store.store_state_group(
	478	prev_event_id,
	479	event.room_id,
	480	None,
	481	None,
	482	{(e.type, e.state_key): e.event_id for e in old_state},
	483	)
484	484	)
485	485	self.store.register_event_id_state_group(prev_event_id, group_name)
486	486

507	507	create_event(type="test2", state_key=""),
508	508	]
509	509
510		group_name = yield self.store.store_state_group(
511		prev_event_id,
512		event.room_id,
513		None,
514		None,
515		{(e.type, e.state_key): e.event_id for e in old_state},
	510	group_name = yield defer.ensureDeferred(
	511	self.store.store_state_group(
	512	prev_event_id,
	513	event.room_id,
	514	None,
	515	None,
	516	{(e.type, e.state_key): e.event_id for e in old_state},
	517	)
516	518	)
517	519	self.store.register_event_id_state_group(prev_event_id, group_name)
518	520

690	692	def _get_context(
691	693	self, event, prev_event_id_1, old_state_1, prev_event_id_2, old_state_2
692	694	):
693		sg1 = yield self.store.store_state_group(
694		prev_event_id_1,
695		event.room_id,
696		None,
697		None,
698		{(e.type, e.state_key): e.event_id for e in old_state_1},
	695	sg1 = yield defer.ensureDeferred(
	696	self.store.store_state_group(
	697	prev_event_id_1,
	698	event.room_id,
	699	None,
	700	None,
	701	{(e.type, e.state_key): e.event_id for e in old_state_1},
	702	)
699	703	)
700	704	self.store.register_event_id_state_group(prev_event_id_1, sg1)
701	705
702		sg2 = yield self.store.store_state_group(
703		prev_event_id_2,
704		event.room_id,
705		None,
706		None,
707		{(e.type, e.state_key): e.event_id for e in old_state_2},
	706	sg2 = yield defer.ensureDeferred(
	707	self.store.store_state_group(
	708	prev_event_id_2,
	709	event.room_id,
	710	None,
	711	None,
	712	{(e.type, e.state_key): e.event_id for e in old_state_2},
	713	)
708	714	)
709	715	self.store.register_event_id_state_group(prev_event_id_2, sg2)
710	716

+3

-4

tests/test_utils/event_injection.py less more

12	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
15		from typing import Optional, Tuple
	15	from typing import List, Optional, Tuple
16	16
17	17	import synapse.server
18	18	from synapse.api.constants import EventTypes
19	19	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
20	20	from synapse.events import EventBase
21	21	from synapse.events.snapshot import EventContext
22		from synapse.types import Collection
23	22
24	23	"""
25	24	Utility functions for poking events into the storage of the server under test.

57	56	async def inject_event(
58	57	hs: synapse.server.HomeServer,
59	58	room_version: Optional[str] = None,
60		prev_event_ids: Optional[Collection[str]] = None,
	59	prev_event_ids: Optional[List[str]] = None,
61	60	**kwargs
62	61	) -> EventBase:
63	62	"""Inject a generic event into a room

79	78	async def create_event(
80	79	hs: synapse.server.HomeServer,
81	80	room_version: Optional[str] = None,
82		prev_event_ids: Optional[Collection[str]] = None,
	81	prev_event_ids: Optional[List[str]] = None,
83	82	**kwargs
84	83	) -> Tuple[EventBase, EventContext]:
85	84	if room_version is None:

+4

-3

tests/test_visibility.py less more

36	36	self.hs = yield setup_test_homeserver(self.addCleanup)
37	37	self.event_creation_handler = self.hs.get_event_creation_handler()
38	38	self.event_builder_factory = self.hs.get_event_builder_factory()
39		self.store = self.hs.get_datastore()
40	39	self.storage = self.hs.get_storage()
41	40
42	41	yield defer.ensureDeferred(create_room(self.hs, TEST_ROOM_ID, "@someone:ROOM"))

98	97	events_to_filter.append(evt)
99	98
100	99	# the erasey user gets erased
101		yield self.hs.get_datastore().mark_user_erased("@erased:local_hs")
	100	yield defer.ensureDeferred(
	101	self.hs.get_datastore().mark_user_erased("@erased:local_hs")
	102	)
102	103
103	104	# ... and the filtering happens.
104	105	filtered = yield defer.ensureDeferred(

292	293	test_large_room.skip = "Disabled by default because it's slow"
293	294
294	295
295		class _TestStore(object):
	296	class _TestStore:
296	297	"""Implements a few methods of the DataStore, so that we can test
297	298	filter_events_for_server
298	299

+7

-3

tests/unittest.py less more

249	249
250	250	async def get_user_by_req(request, allow_guest=False, rights="access"):
251	251	return create_requester(
252		UserID.from_string(self.helper.auth_user_id), 1, False, None
	252	UserID.from_string(self.helper.auth_user_id),
	253	1,
	254	False,
	255	False,
	256	None,
253	257	)
254	258
255	259	self.hs.get_auth().get_user_by_req = get_user_by_req

539	543	"""
540	544	event_creator = self.hs.get_event_creation_handler()
541	545	secrets = self.hs.get_secrets()
542		requester = Requester(user, None, False, None, None)
	546	requester = Requester(user, None, False, False, None, None)
543	547
544	548	event, context = self.get_success(
545	549	event_creator.create_event(

609	613	"""
610	614
611	615	def prepare(self, reactor, clock, homeserver):
612		class Authenticator(object):
	616	class Authenticator:
613	617	def authenticate_request(self, request, content):
614	618	return succeed("other.example.com")
615	619

+16

-16

tests/util/caches/test_descriptors.py less more

87	87	class DescriptorTestCase(unittest.TestCase):
88	88	@defer.inlineCallbacks
89	89	def test_cache(self):
90		class Cls(object):
	90	class Cls:
91	91	def __init__(self):
92	92	self.mock = mock.Mock()
93	93

121	121	def test_cache_num_args(self):
122	122	"""Only the first num_args arguments should matter to the cache"""
123	123
124		class Cls(object):
	124	class Cls:
125	125	def __init__(self):
126	126	self.mock = mock.Mock()
127	127

155	155	"""If the wrapped function throws synchronously, things should continue to work
156	156	"""
157	157
158		class Cls(object):
	158	class Cls:
159	159	@cached()
160	160	def fn(self, arg1):
161	161	raise SynapseError(100, "mai spoon iz too big!!1")

179	179
180	180	complete_lookup = defer.Deferred()
181	181
182		class Cls(object):
	182	class Cls:
183	183	@descriptors.cached()
184	184	def fn(self, arg1):
185	185	@defer.inlineCallbacks

222	222	"""Check that the cache sets and restores logcontexts correctly when
223	223	the lookup function throws an exception"""
224	224
225		class Cls(object):
	225	class Cls:
226	226	@descriptors.cached()
227	227	def fn(self, arg1):
228	228	@defer.inlineCallbacks

262	262
263	263	@defer.inlineCallbacks
264	264	def test_cache_default_args(self):
265		class Cls(object):
	265	class Cls:
266	266	def __init__(self):
267	267	self.mock = mock.Mock()
268	268

299	299	obj.mock.assert_not_called()
300	300
301	301	def test_cache_iterable(self):
302		class Cls(object):
	302	class Cls:
303	303	def __init__(self):
304	304	self.mock = mock.Mock()
305	305

335	335	"""If the wrapped function throws synchronously, things should continue to work
336	336	"""
337	337
338		class Cls(object):
	338	class Cls:
339	339	@descriptors.cached(iterable=True)
340	340	def fn(self, arg1):
341	341	raise SynapseError(100, "mai spoon iz too big!!1")

357	357	class CachedListDescriptorTestCase(unittest.TestCase):
358	358	@defer.inlineCallbacks
359	359	def test_cache(self):
360		class Cls(object):
	360	class Cls:
361	361	def __init__(self):
362	362	self.mock = mock.Mock()
363	363

365	365	def fn(self, arg1, arg2):
366	366	pass
367	367
368		@descriptors.cachedList("fn", "args1", inlineCallbacks=True)
369		def list_fn(self, args1, arg2):
	368	@descriptors.cachedList("fn", "args1")
	369	async def list_fn(self, args1, arg2):
370	370	assert current_context().request == "c1"
371	371	# we want this to behave like an asynchronous function
372		yield run_on_reactor()
	372	await run_on_reactor()
373	373	assert current_context().request == "c1"
374	374	return self.mock(args1, arg2)
375	375

407	407	def test_invalidate(self):
408	408	"""Make sure that invalidation callbacks are called."""
409	409
410		class Cls(object):
	410	class Cls:
411	411	def __init__(self):
412	412	self.mock = mock.Mock()
413	413

415	415	def fn(self, arg1, arg2):
416	416	pass
417	417
418		@descriptors.cachedList("fn", "args1", inlineCallbacks=True)
419		def list_fn(self, args1, arg2):
	418	@descriptors.cachedList("fn", "args1")
	419	async def list_fn(self, args1, arg2):
420	420	# we want this to behave like an asynchronous function
421		yield run_on_reactor()
	421	await run_on_reactor()
422	422	return self.mock(args1, arg2)
423	423
424	424	obj = Cls()

+2

-2

tests/util/test_file_consumer.py less more

111	111	self.assertTrue(string_file.closed)
112	112
113	113
114		class DummyPullProducer(object):
	114	class DummyPullProducer:
115	115	def __init__(self):
116	116	self.consumer = None
117	117	self.deferred = defer.Deferred()

133	133	return d
134	134
135	135
136		class BlockingStringWrite(object):
	136	class BlockingStringWrite:
137	137	def __init__(self):
138	138	self.buffer = ""
139	139	self.closed = False

+1

-1

tests/util/test_retryutils.py less more

90	90	#
91	91	# one more go, with success
92	92	#
93		self.pump(MIN_RETRY_INTERVAL * RETRY_MULTIPLIER * 2.0)
	93	self.reactor.advance(MIN_RETRY_INTERVAL * RETRY_MULTIPLIER * 2.0)
94	94	limiter = self.get_success(get_retry_limiter("test_dest", self.clock, store))
95	95
96	96	self.pump(1)

+4

-2

tests/util/test_rwlock.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
	15	from twisted.internet import defer
15	16
16	17	from synapse.util.async_helpers import ReadWriteLock
17	18

42	43	rwlock.read(key), # 5
43	44	rwlock.write(key), # 6
44	45	]
	46	ds = [defer.ensureDeferred(d) for d in ds]
45	47
46	48	self._assert_called_before_not_after(ds, 2)
47	49

72	74	with ds[6].result:
73	75	pass
74	76
75		d = rwlock.write(key)
	77	d = defer.ensureDeferred(rwlock.write(key))
76	78	self.assertTrue(d.called)
77	79	with d.result:
78	80	pass
79	81
80		d = rwlock.read(key)
	82	d = defer.ensureDeferred(rwlock.read(key))
81	83	self.assertTrue(d.called)
82	84	with d.result:
83	85	pass

+0

-3

tests/util/test_stringutils.py less more

27	27	"_--something==_",
28	28	"...--==-18913",
29	29	"8Dj2odd-e9asd.cd==_--ddas-secret-",
30		# We temporarily allow : characters: https://github.com/matrix-org/synapse/issues/6766
31		# To be removed in a future release
32		"SECRET:1234567890",
33	30	]
34	31
35	32	bad = [

+3

-3

tests/utils.py less more

471	471	self.callbacks.append((method, path_pattern, callback))
472	472
473	473
474		class MockKey(object):
	474	class MockKey:
475	475	alg = "mock_alg"
476	476	version = "mock_version"
477	477	signature = b"\x9a\x87$"

490	490	return b"<fake_encoded_key>"
491	491
492	492
493		class MockClock(object):
	493	class MockClock:
494	494	now = 1000
495	495
496	496	def __init__(self):

567	567	)
568	568
569	569
570		class DeferredMockCallable(object):
	570	class DeferredMockCallable:
571	571	"""A callable instance that stores a set of pending call expectations and
572	572	return values for them. It allows a unit test to assert that the given set
573	573	of function calls are eventually made, by awaiting on them to be called.

+2

-50

tox.ini less more

168	168	skip_install = True
169	169	deps =
170	170	{[base]deps}
171		mypy==0.750
	171	mypy==0.782
172	172	mypy-zope
173		env =
174		MYPYPATH = stubs/
175	173	extras = all
176		commands = mypy \
177		synapse/api \
178		synapse/appservice \
179		synapse/config \
180		synapse/event_auth.py \
181		synapse/events/builder.py \
182		synapse/events/spamcheck.py \
183		synapse/federation \
184		synapse/handlers/auth.py \
185		synapse/handlers/cas_handler.py \
186		synapse/handlers/directory.py \
187		synapse/handlers/federation.py \
188		synapse/handlers/identity.py \
189		synapse/handlers/message.py \
190		synapse/handlers/oidc_handler.py \
191		synapse/handlers/presence.py \
192		synapse/handlers/room_member.py \
193		synapse/handlers/room_member_worker.py \
194		synapse/handlers/saml_handler.py \
195		synapse/handlers/sync.py \
196		synapse/handlers/ui_auth \
197		synapse/http/server.py \
198		synapse/http/site.py \
199		synapse/logging/ \
200		synapse/metrics \
201		synapse/module_api \
202		synapse/notifier.py \
203		synapse/push/pusherpool.py \
204		synapse/push/push_rule_evaluator.py \
205		synapse/replication \
206		synapse/rest \
207		synapse/server.py \
208		synapse/server_notices \
209		synapse/spam_checker_api \
210		synapse/storage/databases/main/ui_auth.py \
211		synapse/storage/database.py \
212		synapse/storage/engines \
213		synapse/storage/state.py \
214		synapse/storage/util \
215		synapse/streams \
216		synapse/types.py \
217		synapse/util/caches/stream_change_cache.py \
218		synapse/util/metrics.py \
219		tests/replication \
220		tests/test_utils \
221		tests/rest/client/v2_alpha/test_auth.py \
222		tests/util/test_stream_change_cache.py
	174	commands = mypy
223	175
224	176	# To find all folders that pass mypy you run:
225	177	#