Commit upstream/1.19.0 - matrix-synapse

+2

-4

.circleci/config.yml less more

3	3	machine: true
4	4	steps:
5	5	- checkout
6		- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 .
	6	- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} .
7	7	- run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
8	8	- run: docker push matrixdotorg/synapse:${CIRCLE_TAG}
9		- run: docker push matrixdotorg/synapse:${CIRCLE_TAG}-py3
10	9	dockerhubuploadlatest:
11	10	machine: true
12	11	steps:
13	12	- checkout
14		- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest -t matrixdotorg/synapse:latest-py3 .
	13	- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest .
15	14	- run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
16	15	- run: docker push matrixdotorg/synapse:latest
17		- run: docker push matrixdotorg/synapse:latest-py3
18	16
19	17	workflows:
20	18	version: 2

+2

-2

.github/ISSUE_TEMPLATE/BUG_REPORT.md less more

3	3
4	4	---
5	5
	6	<!--
	7
6	8	THIS IS NOT A SUPPORT CHANNEL!
7	9	IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER,
8	10	please ask in #synapse:matrix.org (using a matrix.org account if necessary)
9
10		<!--
11	11
12	12	If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/
13	13

+74

-0

CHANGES.md less more

	0	Synapse 1.19.0 (2020-08-17)
	1	===========================
	2
	3	No significant changes since 1.19.0rc1.
	4
	5	Removal warning
	6	---------------
	7
	8	As outlined in the [previous release](https://github.com/matrix-org/synapse/releases/tag/v1.18.0), we are no longer publishing Docker images with the `-py3` tag suffix. On top of that, we have also removed the `latest-py3` tag. Please see [the announcement in the upgrade notes for 1.18.0](https://github.com/matrix-org/synapse/blob/develop/UPGRADE.rst#upgrading-to-v1180).
	9
	10
	11	Synapse 1.19.0rc1 (2020-08-13)
	12	==============================
	13
	14	Features
	15	--------
	16
	17	- Add option to allow server admins to join rooms which fail complexity checks. Contributed by @lugino-emeritus. ([\#7902](https://github.com/matrix-org/synapse/issues/7902))
	18	- Add an option to purge room or not with delete room admin endpoint (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). Contributed by @dklimpel. ([\#7964](https://github.com/matrix-org/synapse/issues/7964))
	19	- Add rate limiting to users joining rooms. ([\#8008](https://github.com/matrix-org/synapse/issues/8008))
	20	- Add a `/health` endpoint to every configured HTTP listener that can be used as a health check endpoint by load balancers. ([\#8048](https://github.com/matrix-org/synapse/issues/8048))
	21	- Allow login to be blocked based on the values of SAML attributes. ([\#8052](https://github.com/matrix-org/synapse/issues/8052))
	22	- Allow guest access to the `GET /_matrix/client/r0/rooms/{room_id}/members` endpoint, according to MSC2689. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#7314](https://github.com/matrix-org/synapse/issues/7314))
	23
	24
	25	Bugfixes
	26	--------
	27
	28	- Fix a bug introduced in Synapse v1.7.2 which caused inaccurate membership counts in the room directory. ([\#7977](https://github.com/matrix-org/synapse/issues/7977))
	29	- Fix a long standing bug: 'Duplicate key value violates unique constraint "event_relations_id"' when message retention is configured. ([\#7978](https://github.com/matrix-org/synapse/issues/7978))
	30	- Fix "no create event in auth events" when trying to reject invitation after inviter leaves. Bug introduced in Synapse v1.10.0. ([\#7980](https://github.com/matrix-org/synapse/issues/7980))
	31	- Fix various comments and minor discrepencies in server notices code. ([\#7996](https://github.com/matrix-org/synapse/issues/7996))
	32	- Fix a long standing bug where HTTP HEAD requests resulted in a 400 error. ([\#7999](https://github.com/matrix-org/synapse/issues/7999))
	33	- Fix a long-standing bug which caused two copies of some log lines to be written when synctl was used along with a MemoryHandler logger. ([\#8011](https://github.com/matrix-org/synapse/issues/8011), [\#8012](https://github.com/matrix-org/synapse/issues/8012))
	34
	35
	36	Updates to the Docker image
	37	---------------------------
	38
	39	- We no longer publish Docker images with the `-py3` tag suffix, as [announced in the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/UPGRADE.rst#upgrading-to-v1180). ([\#8056](https://github.com/matrix-org/synapse/issues/8056))
	40
	41
	42	Improved Documentation
	43	----------------------
	44
	45	- Document how to set up a client .well-known file and fix several pieces of outdated documentation. ([\#7899](https://github.com/matrix-org/synapse/issues/7899))
	46	- Improve workers docs. ([\#7990](https://github.com/matrix-org/synapse/issues/7990), [\#8000](https://github.com/matrix-org/synapse/issues/8000))
	47	- Fix typo in `docs/workers.md`. ([\#7992](https://github.com/matrix-org/synapse/issues/7992))
	48	- Add documentation for how to undo a room shutdown. ([\#7998](https://github.com/matrix-org/synapse/issues/7998), [\#8010](https://github.com/matrix-org/synapse/issues/8010))
	49
	50
	51	Internal Changes
	52	----------------
	53
	54	- Reduce the amount of whitespace in JSON stored and sent in responses. Contributed by David Vo. ([\#7372](https://github.com/matrix-org/synapse/issues/7372))
	55	- Switch to the JSON implementation from the standard library and bump the minimum version of the canonicaljson library to 1.2.0. ([\#7936](https://github.com/matrix-org/synapse/issues/7936), [\#7979](https://github.com/matrix-org/synapse/issues/7979))
	56	- Convert various parts of the codebase to async/await. ([\#7947](https://github.com/matrix-org/synapse/issues/7947), [\#7948](https://github.com/matrix-org/synapse/issues/7948), [\#7949](https://github.com/matrix-org/synapse/issues/7949), [\#7951](https://github.com/matrix-org/synapse/issues/7951), [\#7963](https://github.com/matrix-org/synapse/issues/7963), [\#7973](https://github.com/matrix-org/synapse/issues/7973), [\#7975](https://github.com/matrix-org/synapse/issues/7975), [\#7976](https://github.com/matrix-org/synapse/issues/7976), [\#7981](https://github.com/matrix-org/synapse/issues/7981), [\#7987](https://github.com/matrix-org/synapse/issues/7987), [\#7989](https://github.com/matrix-org/synapse/issues/7989), [\#8003](https://github.com/matrix-org/synapse/issues/8003), [\#8014](https://github.com/matrix-org/synapse/issues/8014), [\#8016](https://github.com/matrix-org/synapse/issues/8016), [\#8027](https://github.com/matrix-org/synapse/issues/8027), [\#8031](https://github.com/matrix-org/synapse/issues/8031), [\#8032](https://github.com/matrix-org/synapse/issues/8032), [\#8035](https://github.com/matrix-org/synapse/issues/8035), [\#8042](https://github.com/matrix-org/synapse/issues/8042), [\#8044](https://github.com/matrix-org/synapse/issues/8044), [\#8045](https://github.com/matrix-org/synapse/issues/8045), [\#8061](https://github.com/matrix-org/synapse/issues/8061), [\#8062](https://github.com/matrix-org/synapse/issues/8062), [\#8063](https://github.com/matrix-org/synapse/issues/8063), [\#8066](https://github.com/matrix-org/synapse/issues/8066), [\#8069](https://github.com/matrix-org/synapse/issues/8069), [\#8070](https://github.com/matrix-org/synapse/issues/8070))
	57	- Move some database-related log lines from the default logger to the database/transaction loggers. ([\#7952](https://github.com/matrix-org/synapse/issues/7952))
	58	- Add a script to detect source code files using non-unix line terminators. ([\#7965](https://github.com/matrix-org/synapse/issues/7965), [\#7970](https://github.com/matrix-org/synapse/issues/7970))
	59	- Log the SAML session ID during creation. ([\#7971](https://github.com/matrix-org/synapse/issues/7971))
	60	- Implement new experimental push rules for some users. ([\#7997](https://github.com/matrix-org/synapse/issues/7997))
	61	- Remove redundant and unreliable signature check for v1 Identity Service lookup responses. ([\#8001](https://github.com/matrix-org/synapse/issues/8001))
	62	- Improve the performance of the register endpoint. ([\#8009](https://github.com/matrix-org/synapse/issues/8009))
	63	- Reduce less useful output in the newsfragment CI step. Add a link to the changelog section of the contributing guide on error. ([\#8024](https://github.com/matrix-org/synapse/issues/8024))
	64	- Rename storage layer objects to be more sensible. ([\#8033](https://github.com/matrix-org/synapse/issues/8033))
	65	- Change the default log config to reduce disk I/O and storage for new servers. ([\#8040](https://github.com/matrix-org/synapse/issues/8040))
	66	- Add an assertion on `prev_events` in `create_new_client_event`. ([\#8041](https://github.com/matrix-org/synapse/issues/8041))
	67	- Add a comment to `ServerContextFactory` about the use of `SSLv23_METHOD`. ([\#8043](https://github.com/matrix-org/synapse/issues/8043))
	68	- Log `OPTIONS` requests at `DEBUG` rather than `INFO` level to reduce amount logged at `INFO`. ([\#8049](https://github.com/matrix-org/synapse/issues/8049))
	69	- Reduce amount of outbound request logging at `INFO` level. ([\#8050](https://github.com/matrix-org/synapse/issues/8050))
	70	- It is no longer necessary to explicitly define `filters` in the logging configuration. (Continuing to do so is redundant but harmless.) ([\#8051](https://github.com/matrix-org/synapse/issues/8051))
	71	- Add and improve type hints. ([\#8058](https://github.com/matrix-org/synapse/issues/8058), [\#8064](https://github.com/matrix-org/synapse/issues/8064), [\#8060](https://github.com/matrix-org/synapse/issues/8060), [\#8067](https://github.com/matrix-org/synapse/issues/8067))
	72
	73
0	74	Synapse 1.18.0 (2020-07-30)
1	75	===========================
2	76

+97

-16

INSTALL.md less more

0	0	- [Choosing your server name](#choosing-your-server-name)
	1	- [Picking a database engine](#picking-a-database-engine)
1	2	- [Installing Synapse](#installing-synapse)
2	3	- [Installing from source](#installing-from-source)
3	4	- [Platform-Specific Instructions](#platform-specific-instructions)
4	5	- [Prebuilt packages](#prebuilt-packages)
5	6	- [Setting up Synapse](#setting-up-synapse)
6	7	- [TLS certificates](#tls-certificates)
	8	- [Client Well-Known URI](#client-well-known-uri)
7	9	- [Email](#email)
8	10	- [Registering a user](#registering-a-user)
9	11	- [Setting up a TURN server](#setting-up-a-turn-server)

25	27	that your email address is probably `user@example.com` rather than
26	28	`user@email.example.com`) - but doing so may require more advanced setup: see
27	29	[Setting up Federation](docs/federate.md).
	30
	31	# Picking a database engine
	32
	33	Synapse offers two database engines:
	34	* [PostgreSQL](https://www.postgresql.org)
	35	* [SQLite](https://sqlite.org/)
	36
	37	Almost all installations should opt to use PostgreSQL. Advantages include:
	38
	39	* significant performance improvements due to the superior threading and
	40	caching model, smarter query optimiser
	41	* allowing the DB to be run on separate hardware
	42
	43	For information on how to install and use PostgreSQL, please see
	44	[docs/postgres.md](docs/postgres.md)
	45
	46	By default Synapse uses SQLite and in doing so trades performance for convenience.
	47	SQLite is only recommended in Synapse for testing purposes or for servers with
	48	light workloads.
28	49
29	50	# Installing Synapse
30	51

233	254
234	255	There is an offical synapse image available at
235	256	https://hub.docker.com/r/matrixdotorg/synapse which can be used with
236		the docker-compose file available at [contrib/docker](contrib/docker). Further information on
237		this including configuration options is available in the README on
238		hub.docker.com.
	257	the docker-compose file available at [contrib/docker](contrib/docker). Further
	258	information on this including configuration options is available in the README
	259	on hub.docker.com.
239	260
240	261	Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a
241	262	Dockerfile to automate a synapse server in a single Docker image, at

243	264
244	265	Slavi Pantaleev has created an Ansible playbook,
245	266	which installs the offical Docker image of Matrix Synapse
246		along with many other Matrix-related services (Postgres database, riot-web, coturn, mxisd, SSL support, etc.).
	267	along with many other Matrix-related services (Postgres database, Element, coturn,
	268	ma1sd, SSL support, etc.).
247	269	For more details, see
248	270	https://github.com/spantaleev/matrix-docker-ansible-deploy
249	271

276	298	/usr/share/keyrings/matrix-org-archive-keyring.gpg`) is
277	299	`AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`.
278	300
279		#### Downstream Debian/Ubuntu packages
280
281		For `buster` and `sid`, Synapse is available in the Debian repositories and
282		it should be possible to install it with simply:
	301	#### Downstream Debian packages
	302
	303	We do not recommend using the packages from the default Debian `buster`
	304	repository at this time, as they are old and suffer from known security
	305	vulnerabilities. You can install the latest version of Synapse from
	306	[our repository](#matrixorg-packages) or from `buster-backports`. Please
	307	see the [Debian documentation](https://backports.debian.org/Instructions/)
	308	for information on how to use backports.
	309
	310	If you are using Debian `sid` or testing, Synapse is available in the default
	311	repositories and it should be possible to install it simply with:
283	312
284	313	```
285	314	sudo apt install matrix-synapse
286	315	```
287	316
288		There is also a version of `matrix-synapse` in `stretch-backports`. Please see
289		the [Debian documentation on
290		backports](https://backports.debian.org/Instructions/) for information on how
291		to use them.
292
293		We do not recommend using the packages in downstream Ubuntu at this time, as
294		they are old and suffer from known security vulnerabilities.
	317	#### Downstream Ubuntu packages
	318
	319	We do not recommend using the packages in the default Ubuntu repository
	320	at this time, as they are old and suffer from known security vulnerabilities.
	321	The latest version of Synapse can be installed from [our repository](#matrixorg-packages).
295	322
296	323	### Fedora
297	324

418	445	For a more detailed guide to configuring your server for federation, see
419	446	[federate.md](docs/federate.md).
420	447
	448	## Client Well-Known URI
	449
	450	Setting up the client Well-Known URI is optional but if you set it up, it will
	451	allow users to enter their full username (e.g. `@user:<server_name>`) into clients
	452	which support well-known lookup to automatically configure the homeserver and
	453	identity server URLs. This is useful so that users don't have to memorize or think
	454	about the actual homeserver URL you are using.
	455
	456	The URL `https://<server_name>/.well-known/matrix/client` should return JSON in
	457	the following format.
	458
	459	```
	460	{
	461	"m.homeserver": {
	462	"base_url": "https://<matrix.example.com>"
	463	}
	464	}
	465	```
	466
	467	It can optionally contain identity server information as well.
	468
	469	```
	470	{
	471	"m.homeserver": {
	472	"base_url": "https://<matrix.example.com>"
	473	},
	474	"m.identity_server": {
	475	"base_url": "https://<identity.example.com>"
	476	}
	477	}
	478	```
	479
	480	To work in browser based clients, the file must be served with the appropriate
	481	Cross-Origin Resource Sharing (CORS) headers. A recommended value would be
	482	`Access-Control-Allow-Origin: *` which would allow all browser based clients to
	483	view it.
	484
	485	In nginx this would be something like:
	486	```
	487	location /.well-known/matrix/client {
	488	return 200 '{"m.homeserver": {"base_url": "https://<matrix.example.com>"}}';
	489	add_header Content-Type application/json;
	490	add_header Access-Control-Allow-Origin *;
	491	}
	492	```
	493
	494	You should also ensure the `public_baseurl` option in `homeserver.yaml` is set
	495	correctly. `public_baseurl` should be set to the URL that clients will use to
	496	connect to your server. This is the same URL you put for the `m.homeserver`
	497	`base_url` above.
	498
	499	```
	500	public_baseurl: "https://<matrix.example.com>"
	501	```
421	502
422	503	## Email
423	504

436	517
437	518	## Registering a user
438	519
439		The easiest way to create a new user is to do so from a client like [Riot](https://riot.im).
	520	The easiest way to create a new user is to do so from a client like [Element](https://element.io/).
440	521
441	522	Alternatively you can do so from the command line if you have installed via pip.
442	523

+7

-36

README.rst less more

44	44	- Eventually-consistent cryptographically secure synchronisation of room
45	45	state across a global open network of federated servers and services
46	46	- Sending and receiving extensible messages in a room with (optional)
47		end-to-end encryption[1]
	47	end-to-end encryption
48	48	- Inviting, joining, leaving, kicking, banning room members
49	49	- Managing user accounts (registration, login, logout)
50	50	- Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers,

81	81
82	82	Thanks for using Matrix!
83	83
84		[1] End-to-end encryption is currently in beta: `blog post <https://matrix.org/blog/2016/11/21/matrixs-olm-end-to-end-encryption-security-assessment-released-and-implemented-cross-platform-on-riot-at-last>`_.
85
86
87	84	Support
88	85	=======
89	86

114	111	general, you will need to enable TLS support before you can successfully
115	112	connect from a client: see `<INSTALL.md#tls-certificates>`_.
116	113
117		An easy way to get started is to login or register via Riot at
118		https://riot.im/app/#/login or https://riot.im/app/#/register respectively.
	114	An easy way to get started is to login or register via Element at
	115	https://app.element.io/#/login or https://app.element.io/#/register respectively.
119	116	You will need to change the server you are logging into from ``matrix.org``
120	117	and instead specify a Homeserver URL of ``https://<server_name>:8448``
121	118	(or just ``https://<server_name>`` if you are using a reverse proxy).
122		(Leave the identity server as the default - see `Identity servers`_.)
123	119	If you prefer to use another client, refer to our
124	120	`client breakdown <https://matrix.org/docs/projects/clients-matrix>`_.
125	121

136	132	recommended to also set up CAPTCHA - see `<docs/CAPTCHA_SETUP.md>`_.)
137	133
138	134	Once ``enable_registration`` is set to ``true``, it is possible to register a
139		user via `riot.im <https://riot.im/app/#/register>`_ or other Matrix clients.
	135	user via a Matrix client.
140	136
141	137	Your new user name will be formed partly from the ``server_name``, and partly
142	138	from a localpart you specify when you create the account. Your name will take

181	177	versions of synapse.
182	178
183	179	.. _UPGRADE.rst: UPGRADE.rst
184
185
186		Using PostgreSQL
187		================
188
189		Synapse offers two database engines:
190		* `PostgreSQL <https://www.postgresql.org>`_
191		* `SQLite <https://sqlite.org/>`_
192
193		Almost all installations should opt to use PostgreSQL. Advantages include:
194
195		* significant performance improvements due to the superior threading and
196		caching model, smarter query optimiser
197		* allowing the DB to be run on separate hardware
198		* allowing basic active/backup high-availability with a "hot spare" synapse
199		pointing at the same DB master, as well as enabling DB replication in
200		synapse itself.
201
202		For information on how to install and use PostgreSQL, please see
203		`docs/postgres.md <docs/postgres.md>`_.
204
205		By default Synapse uses SQLite and in doing so trades performance for convenience.
206		SQLite is only recommended in Synapse for testing purposes or for servers with
207		light workloads.
208	180
209	181	.. _reverse-proxy:
210	182

254	226	Password reset
255	227	==============
256	228
257		If a user has registered an email address to their account using an identity
258		server, they can request a password-reset token via clients such as Riot.
259
260		A manual password reset can be done via direct database access as follows.
	229	Users can reset their password through their client. Alternatively, a server admin
	230	can reset a users password using the `admin API <docs/admin_api/user_admin_api.rst#reset-password>`_
	231	or by directly editing the database as shown below.
261	232
262	233	First calculate the hash of the new password::
263	234

+9

-7

contrib/cmdclient/console.py less more

608	608
609	609	@defer.inlineCallbacks
610	610	def _do_event_stream(self, timeout):
611		res = yield self.http_client.get_json(
612		self._url() + "/events",
613		{
614		"access_token": self._tok(),
615		"timeout": str(timeout),
616		"from": self.event_stream_token,
617		},
	611	res = yield defer.ensureDeferred(
	612	self.http_client.get_json(
	613	self._url() + "/events",
	614	{
	615	"access_token": self._tok(),
	616	"timeout": str(timeout),
	617	"from": self.event_stream_token,
	618	},
	619	)
618	620	)
619	621	print(json.dumps(res, indent=4))
620	622

+10

-0

debian/changelog less more

	0	matrix-synapse-py3 (1.19.0) stable; urgency=medium
	1
	2	[ Synapse Packaging team ]
	3	* New synapse release 1.19.0.
	4
	5	[ Aaron Raimist ]
	6	* Fix outdated documentation for SYNAPSE_CACHE_FACTOR
	7
	8	-- Synapse Packaging team <packages@matrix.org> Mon, 17 Aug 2020 14:06:42 +0100
	9
0	10	matrix-synapse-py3 (1.18.0) stable; urgency=medium
1	11
2	12	* New synapse release 1.18.0.

+1

-1

debian/matrix-synapse.default less more

0	0	# Specify environment variables used when running Synapse
1		# SYNAPSE_CACHE_FACTOR=1 (default)
	1	# SYNAPSE_CACHE_FACTOR=0.5 (default)

+14

-13

debian/synctl.ronn less more

45	45	## ENVIRONMENT
46	46
47	47	* `SYNAPSE_CACHE_FACTOR`:
48		Synapse's architecture is quite RAM hungry currently - a lot of
49		recent room data and metadata is deliberately cached in RAM in
50		order to speed up common requests. This will be improved in
51		future, but for now the easiest way to either reduce the RAM usage
52		(at the risk of slowing things down) is to set the
53		SYNAPSE_CACHE_FACTOR environment variable. Roughly speaking, a
54		SYNAPSE_CACHE_FACTOR of 1.0 will max out at around 3-4GB of
55		resident memory - this is what we currently run the matrix.org
56		on. The default setting is currently 0.1, which is probably around
57		a ~700MB footprint. You can dial it down further to 0.02 if
58		desired, which targets roughly ~512MB. Conversely you can dial it
59		up if you need performance for lots of users and have a box with a
60		lot of RAM.
	48	Synapse's architecture is quite RAM hungry currently - we deliberately
	49	cache a lot of recent room data and metadata in RAM in order to speed up
	50	common requests. We'll improve this in the future, but for now the easiest
	51	way to either reduce the RAM usage (at the risk of slowing things down)
	52	is to set the almost-undocumented ``SYNAPSE_CACHE_FACTOR`` environment
	53	variable. The default is 0.5, which can be decreased to reduce RAM usage
	54	in memory constrained enviroments, or increased if performance starts to
	55	degrade.
	56
	57	However, degraded performance due to a low cache factor, common on
	58	machines with slow disks, often leads to explosions in memory use due
	59	backlogged requests. In this case, reducing the cache factor will make
	60	things worse. Instead, try increasing it drastically. 2.0 is a good
	61	starting value.
61	62
62	63	## COPYRIGHT
63	64

+0

-6

docker/conf/log.config less more

3	3	precise:
4	4	format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
5	5
6		filters:
7		context:
8		(): synapse.logging.context.LoggingContextFilter
9		request: ""
10
11	6	handlers:
12	7	console:
13	8	class: logging.StreamHandler
14	9	formatter: precise
15		filters: [context]
16	10
17	11	loggers:
18	12	synapse.storage.SQL:

+11

-0

docs/.sample_config_header.yaml less more

9	9	# homeserver.yaml. Instead, if you are starting from scratch, please generate
10	10	# a fresh config using Synapse by following the instructions in INSTALL.md.
11	11
	12	# Configuration options that take a time period can be set using a number
	13	# followed by a letter. Letters have the following meanings:
	14	# s = second
	15	# m = minute
	16	# h = hour
	17	# d = day
	18	# w = week
	19	# y = year
	20	# For example, setting redaction_retention_period: 5m would remove redacted
	21	# messages from the database after 5 minutes, rather than 5 months.
	22
12	23	################################################################################
13	24

+9

-4

docs/admin_api/rooms.md less more

368	368	If `block` is `True` it prevents new joins to the old room.
369	369
370	370	This API will remove all trace of the old room from your database after removing
371		all local users.
	371	all local users. If `purge` is `true` (the default), all traces of the old room will
	372	be removed from your database after removing all local users. If you do not want
	373	this to happen, set `purge` to `false`.
372	374	Depending on the amount of history being purged a call to the API may take
373	375	several minutes or longer.
374	376

387	389	"new_room_user_id": "@someuser:example.com",
388	390	"room_name": "Content Violation Notification",
389	391	"message": "Bad Room has been shutdown due to content violations on this server. Please review our Terms of Service.",
390		"block": true
	392	"block": true,
	393	"purge": true
391	394	}
392	395	```
393	396

429	432	`new_room_user_id` in the new room. Ideally this will clearly convey why the
430	433	original room was shut down. Defaults to `Sharing illegal content on this server
431	434	is not permitted and rooms in violation will be blocked.`
432		* `block` - Optional. If set to `true`, this room will be added to a blocking list, preventing future attempts to
433		join the room. Defaults to `false`.
	435	* `block` - Optional. If set to `true`, this room will be added to a blocking list, preventing
	436	future attempts to join the room. Defaults to `false`.
	437	* `purge` - Optional. If set to `true`, it will remove all traces of the room from your database.
	438	Defaults to `true`.
434	439
435	440	The JSON body must not be empty. The body must be at least `{}`.
436	441

+28

-1

docs/admin_api/shutdown_room.md less more

32	32	* `message` - Optional. A string containing the first message that will be sent as
33	33	`new_room_user_id` in the new room. Ideally this will clearly convey why the
34	34	original room was shut down.
35
	35
36	36	If not specified, the default value of `room_name` is "Content Violation
37	37	Notification". The default value of `message` is "Sharing illegal content on
38	38	othis server is not permitted and rooms in violation will be blocked."

71	71	"new_room_id": "!newroomid:example.com",
72	72	},
73	73	```
	74
	75	## Undoing room shutdowns
	76
	77	Note: This guide may be outdated by the time you read it. By nature of room shutdowns being performed at the database level,
	78	the structure can and does change without notice.
	79
	80	First, it's important to understand that a room shutdown is very destructive. Undoing a shutdown is not as simple as pretending it
	81	never happened - work has to be done to move forward instead of resetting the past. In fact, in some cases it might not be possible
	82	to recover at all:
	83
	84	* If the room was invite-only, your users will need to be re-invited.
	85	* If the room no longer has any members at all, it'll be impossible to rejoin.
	86	* The first user to rejoin will have to do so via an alias on a different server.
	87
	88	With all that being said, if you still want to try and recover the room:
	89
	90	1. For safety reasons, shut down Synapse.
	91	2. In the database, run `DELETE FROM blocked_rooms WHERE room_id = '!example:example.org';`
	92	* For caution: it's recommended to run this in a transaction: `BEGIN; DELETE ...;`, verify you got 1 result, then `COMMIT;`.
	93	* The room ID is the same one supplied to the shutdown room API, not the Content Violation room.
	94	3. Restart Synapse.
	95
	96	You will have to manually handle, if you so choose, the following:
	97
	98	* Aliases that would have been redirected to the Content Violation room.
	99	* Users that would have been booted from the room (and will have been force-joined to the Content Violation room).
	100	* Removal of the Content Violation room if desired.

+1

-1

docs/metrics-howto.md less more

26	26	different thread to Synapse. This can make it more resilient to
27	27	heavy load meaning metrics cannot be retrieved, and can be exposed
28	28	to just internal networks easier. The served metrics are available
29		over HTTP only, and will be available at `/`.
	29	over HTTP only, and will be available at `/_synapse/metrics`.
30	30
31	31	Add a new listener to homeserver.yaml:
32	32

+3

-0

docs/postgres.md less more

187	187
188	188	It is safe to at any time kill the port script and restart it.
189	189
	190	Note that the database may take up significantly more (25% - 100% more)
	191	space on disk after porting to Postgres.
	192
190	193	### Using the port script
191	194
192	195	Firstly, shut down the currently running synapse server and copy its

+7

-0

docs/reverse_proxy.md less more

138	138	Having done so, you can then use `https://matrix.example.com` (instead
139	139	of `https://matrix.example.com:8448`) as the "Custom server" when
140	140	connecting to Synapse from a client.
	141
	142
	143	## Health check endpoint
	144
	145	Synapse exposes a health check endpoint for use by reverse proxies.
	146	Each configured HTTP listener has a `/health` endpoint which always returns
	147	200 OK (and doesn't get logged).

+38

-18

docs/sample_config.yaml less more

8	8	# It is not intended to be copied and used as the basis for a real
9	9	# homeserver.yaml. Instead, if you are starting from scratch, please generate
10	10	# a fresh config using Synapse by following the instructions in INSTALL.md.
	11
	12	# Configuration options that take a time period can be set using a number
	13	# followed by a letter. Letters have the following meanings:
	14	# s = second
	15	# m = minute
	16	# h = hour
	17	# d = day
	18	# w = week
	19	# y = year
	20	# For example, setting redaction_retention_period: 5m would remove redacted
	21	# messages from the database after 5 minutes, rather than 5 months.
11	22
12	23	################################################################################
13	24

312	323	# override the error which is returned when the room is too complex.
313	324	#
314	325	#complexity_error: "This room is too complex."
	326
	327	# allow server admins to join complex rooms. Default is false.
	328	#
	329	#admins_can_join: true
315	330
316	331	# Whether to require a user to be in the room to add an alias to it.
317	332	# Defaults to 'true'.

730	745	# - one for ratelimiting redactions by room admins. If this is not explicitly
731	746	# set then it uses the same ratelimiting as per rc_message. This is useful
732	747	# to allow room admins to deal with abuse quickly.
	748	# - two for ratelimiting number of rooms a user can join, "local" for when
	749	# users are joining rooms the server is already in (this is cheap) vs
	750	# "remote" for when users are trying to join rooms not on the server (which
	751	# can be more expensive)
733	752	#
734	753	# The defaults are as shown below.
735	754	#

755	774	#rc_admin_redaction:
756	775	# per_second: 1
757	776	# burst_count: 50
	777	#
	778	#rc_joins:
	779	# local:
	780	# per_second: 0.1
	781	# burst_count: 3
	782	# remote:
	783	# per_second: 0.01
	784	# burst_count: 3
758	785
759	786
760	787	# Ratelimiting settings for incoming federation

1144	1171	#
1145	1172	#default_identity_server: https://matrix.org
1146	1173
1147		# The list of identity servers trusted to verify third party
1148		# identifiers by this server.
1149		#
1150		# Also defines the ID server which will be called when an account is
1151		# deactivated (one will be picked arbitrarily).
1152		#
1153		# Note: This option is deprecated. Since v0.99.4, Synapse has tracked which identity
1154		# server a 3PID has been bound to. For 3PIDs bound before then, Synapse runs a
1155		# background migration script, informing itself that the identity server all of its
1156		# 3PIDs have been bound to is likely one of the below.
1157		#
1158		# As of Synapse v1.4.0, all other functionality of this option has been deprecated, and
1159		# it is now solely used for the purposes of the background migration script, and can be
1160		# removed once it has run.
1161		#trusted_third_party_id_servers:
1162		# - matrix.org
1163		# - vector.im
1164
1165	1174	# Handle threepid (email/phone etc) registration and password resets through a set of
1166	1175	# trusted identity servers. Note that this allows the configured identity server to
1167	1176	# reset passwords for accounts!

1566	1575	# The default is 'uid'.
1567	1576	#
1568	1577	#grandfathered_mxid_source_attribute: upn
	1578
	1579	# It is possible to configure Synapse to only allow logins if SAML attributes
	1580	# match particular values. The requirements can be listed under
	1581	# `attribute_requirements` as shown below. All of the listed attributes must
	1582	# match for the login to be permitted.
	1583	#
	1584	#attribute_requirements:
	1585	# - attribute: userGroup
	1586	# value: "staff"
	1587	# - attribute: department
	1588	# value: "sales"
1569	1589
1570	1590	# Directory in which Synapse will try to find the template files below.
1571	1591	# If not set, default templates from within the Synapse package will be used.

+35

-11

docs/sample_log_config.yaml less more

10	10	precise:
11	11	format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
12	12
13		filters:
14		context:
15		(): synapse.logging.context.LoggingContextFilter
16		request: ""
17
18	13	handlers:
19	14	file:
20		class: logging.handlers.RotatingFileHandler
	15	class: logging.handlers.TimedRotatingFileHandler
21	16	formatter: precise
22	17	filename: /var/log/matrix-synapse/homeserver.log
23		maxBytes: 104857600
24		backupCount: 10
25		filters: [context]
	18	when: midnight
	19	backupCount: 3 # Does not include the current log file.
26	20	encoding: utf8
	21
	22	# Default to buffering writes to log file for efficiency. This means that
	23	# will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
	24	# logs will still be flushed immediately.
	25	buffer:
	26	class: logging.handlers.MemoryHandler
	27	target: file
	28	# The capacity is the number of log lines that are buffered before
	29	# being written to disk. Increasing this will lead to better
	30	# performance, at the expensive of it taking longer for log lines to
	31	# be written to disk.
	32	capacity: 10
	33	flushLevel: 30 # Flush for WARNING logs as well
	34
	35	# A handler that writes logs to stderr. Unused by default, but can be used
	36	# instead of "buffer" and "file" in the logger handlers.
27	37	console:
28	38	class: logging.StreamHandler
29	39	formatter: precise
30		filters: [context]
31	40
32	41	loggers:
33	42	synapse.storage.SQL:

35	44	# information such as access tokens.
36	45	level: INFO
37	46
	47	twisted:
	48	# We send the twisted logging directly to the file handler,
	49	# to work around https://github.com/matrix-org/synapse/issues/3471
	50	# when using "buffer" logger. Use "console" to log to stderr instead.
	51	handlers: [file]
	52	propagate: false
	53
38	54	root:
39	55	level: INFO
40		handlers: [file, console]
	56
	57	# Write logs to the `buffer` handler, which will buffer them together in memory,
	58	# then write them to a file.
	59	#
	60	# Replace "buffer" with "console" to log to stderr instead. (Note that you'll
	61	# also need to update the configuation for the `twisted` logger above, in
	62	# this case.)
	63	#
	64	handlers: [buffer]
41	65
42	66	disable_existing_loggers: false

+1

-1

docs/systemd-with-workers/workers/federation_reader.yaml less more

0	0	worker_app: synapse.app.federation_reader
	1	worker_name: federation_reader1
1	2
2	3	worker_replication_host: 127.0.0.1
3		worker_replication_port: 9092
4	4	worker_replication_http_port: 9093
5	5
6	6	worker_listeners:

+1

-1

docs/user_directory.md less more

6	6
7	7	The directory info is stored in various tables, which can (typically after
8	8	DB corruption) get stale or out of sync. If this happens, for now the
9		solution to fix it is to execute the SQL [here](../synapse/storage/data_stores/main/schema/delta/53/user_dir_populate.sql)
	9	solution to fix it is to execute the SQL [here](../synapse/storage/databases/main/schema/delta/53/user_dir_populate.sql)
10	10	and then restart synapse. This should then start a background task to
11	11	flush the current tables and regenerate the directory.

+73

-36

docs/workers.md less more

0	0	# Scaling synapse via workers
1	1
2		For small instances it recommended to run Synapse in monolith mode (the
3		default). For larger instances where performance is a concern it can be helpful
4		to split out functionality into multiple separate python processes. These
5		processes are called 'workers', and are (eventually) intended to scale
6		horizontally independently.
	2	For small instances it recommended to run Synapse in the default monolith mode.
	3	For larger instances where performance is a concern it can be helpful to split
	4	out functionality into multiple separate python processes. These processes are
	5	called 'workers', and are (eventually) intended to scale horizontally
	6	independently.
7	7
8	8	Synapse's worker support is under active development and subject to change as
9	9	we attempt to rapidly scale ever larger Synapse instances. However we are

22	22	feeds streams of newly written data between processes so they can be kept in
23	23	sync with the database state.
24	24
25		Additionally, processes may make HTTP requests to each other. Typically this is
26		used for operations which need to wait for a reply - such as sending an event.
27
28		As of Synapse v1.13.0, it is possible to configure Synapse to send replication
29		via a [Redis pub/sub channel](https://redis.io/topics/pubsub), and is now the
30		recommended way of configuring replication. This is an alternative to the old
31		direct TCP connections to the main process: rather than all the workers
32		connecting to the main process, all the workers and the main process connect to
33		Redis, which relays replication commands between processes. This can give a
34		significant cpu saving on the main process and will be a prerequisite for
35		upcoming performance improvements.
36
37		(See the [Architectural diagram](#architectural-diagram) section at the end for
38		a visualisation of what this looks like)
	25	When configured to do so, Synapse uses a
	26	[Redis pub/sub channel](https://redis.io/topics/pubsub) to send the replication
	27	stream between all configured Synapse processes. Additionally, processes may
	28	make HTTP requests to each other, primarily for operations which need to wait
	29	for a reply ─ such as sending an event.
	30
	31	Redis support was added in v1.13.0 with it becoming the recommended method in
	32	v1.18.0. It replaced the old direct TCP connections (which is deprecated as of
	33	v1.18.0) to the main process. With Redis, rather than all the workers connecting
	34	to the main process, all the workers and the main process connect to Redis,
	35	which relays replication commands between processes. This can give a significant
	36	cpu saving on the main process and will be a prerequisite for upcoming
	37	performance improvements.
	38
	39	See the [Architectural diagram](#architectural-diagram) section at the end for
	40	a visualisation of what this looks like.
39	41
40	42
41	43	## Setting up workers
42	44
43	45	A Redis server is required to manage the communication between the processes.
44		(The older direct TCP connections are now deprecated.) The Redis server
45		should be installed following the normal procedure for your distribution (e.g.
46		`apt install redis-server` on Debian). It is safe to use an existing Redis
47		deployment if you have one.
	46	The Redis server should be installed following the normal procedure for your
	47	distribution (e.g. `apt install redis-server` on Debian). It is safe to use an
	48	existing Redis deployment if you have one.
48	49
49	50	Once installed, check that Redis is running and accessible from the host running
50	51	Synapse, for example by executing `echo PING \| nc -q1 localhost 6379` and seeing

64	65
65	66	To make effective use of the workers, you will need to configure an HTTP
66	67	reverse-proxy such as nginx or haproxy, which will direct incoming requests to
67		the correct worker, or to the main synapse instance. See [reverse_proxy.md](reverse_proxy.md)
68		for information on setting up a reverse proxy.
69
70		To enable workers you should create a configuration file for each worker
71		process. Each worker configuration file inherits the configuration of the shared
72		homeserver configuration file. You can then override configuration specific to
73		that worker, e.g. the HTTP listener that it provides (if any); logging
74		configuration; etc. You should minimise the number of overrides though to
75		maintain a usable config.
76
77		Next you need to add both a HTTP replication listener and redis config to the
78		shared Synapse configuration file (`homeserver.yaml`). For example:
	68	the correct worker, or to the main synapse instance. See
	69	[reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse
	70	proxy.
	71
	72	When using workers, each worker process has its own configuration file which
	73	contains settings specific to that worker, such as the HTTP listener that it
	74	provides (if any), logging configuration, etc.
	75
	76	Normally, the worker processes are configured to read from a shared
	77	configuration file as well as the worker-specific configuration files. This
	78	makes it easier to keep common configuration settings synchronised across all
	79	the processes.
	80
	81	The main process is somewhat special in this respect: it does not normally
	82	need its own configuration file and can take all of its configuration from the
	83	shared configuration file.
	84
	85
	86	### Shared configuration
	87
	88	Normally, only a couple of changes are needed to make an existing configuration
	89	file suitable for use with workers. First, you need to enable an "HTTP replication
	90	listener" for the main process; and secondly, you need to enable redis-based
	91	replication. For example:
	92
79	93
80	94	```yaml
81	95	# extend the existing `listeners` section. This defines the ports that the

97	111	Under no circumstances should the replication listener be exposed to the
98	112	public internet; it has no authentication and is unencrypted.
99	113
	114
	115	### Worker configuration
	116
100	117	In the config file for each worker, you must specify the type of worker
101	118	application (`worker_app`), and you should specify a unqiue name for the worker
102	119	(`worker_name`). The currently available worker applications are listed below.

134	151
135	152	Obviously you should configure your reverse-proxy to route the relevant
136	153	endpoints to the worker (`localhost:8083` in the above example).
	154
	155
	156	### Running Synapse with workers
137	157
138	158	Finally, you need to start your worker processes. This can be done with either
139	159	`synctl` or your distribution's preferred service manager such as `systemd`. We

277	297	host: localhost
278	298	port: 8034
279	299
280		streams_writers:
	300	stream_writers:
281	301	events: event_persister1
282	302	```
283	303

395	415	that handles sending out push notifications for new events. The intention is for
396	416	all these to be folded into the `generic_worker` app and to use config to define
397	417	which processes handle the various proccessing such as push notifications.
	418
	419
	420	## Migration from old config
	421
	422	There are two main independent changes that have been made: introducing Redis
	423	support and merging apps into `synapse.app.generic_worker`. Both these changes
	424	are backwards compatible and so no changes to the config are required, however
	425	server admins are encouraged to plan to migrate to Redis as the old style direct
	426	TCP replication config is deprecated.
	427
	428	To migrate to Redis add the `redis` config as above, and optionally remove the
	429	TCP `replication` listener from master and `worker_replication_port` from worker
	430	config.
	431
	432	To migrate apps to use `synapse.app.generic_worker` simply update the
	433	`worker_app` option in the worker configs, and where worker are started (e.g.
	434	in systemd service files, but not required for synctl).
398	435
399	436
400	437	## Architectural diagram

+3

-0

mypy.ini less more

80	80
81	81	[mypy-rust_python_jaeger_reporter.*]
82	82	ignore_missing_imports = True
	83
	84	[mypy-nacl.*]
	85	ignore_missing_imports = True

+39

-39

scripts/synapse_port_db less more

34	34	make_deferred_yieldable,
35	35	run_in_background,
36	36	)
37		from synapse.storage.data_stores.main.client_ips import ClientIpBackgroundUpdateStore
38		from synapse.storage.data_stores.main.deviceinbox import (
39		DeviceInboxBackgroundUpdateStore,
40		)
41		from synapse.storage.data_stores.main.devices import DeviceBackgroundUpdateStore
42		from synapse.storage.data_stores.main.events_bg_updates import (
	37	from synapse.storage.database import DatabasePool, make_conn
	38	from synapse.storage.databases.main.client_ips import ClientIpBackgroundUpdateStore
	39	from synapse.storage.databases.main.deviceinbox import DeviceInboxBackgroundUpdateStore
	40	from synapse.storage.databases.main.devices import DeviceBackgroundUpdateStore
	41	from synapse.storage.databases.main.events_bg_updates import (
43	42	EventsBackgroundUpdatesStore,
44	43	)
45		from synapse.storage.data_stores.main.media_repository import (
	44	from synapse.storage.databases.main.media_repository import (
46	45	MediaRepositoryBackgroundUpdateStore,
47	46	)
48		from synapse.storage.data_stores.main.registration import (
	47	from synapse.storage.databases.main.registration import (
49	48	RegistrationBackgroundUpdateStore,
50	49	find_max_generated_user_id_localpart,
51	50	)
52		from synapse.storage.data_stores.main.room import RoomBackgroundUpdateStore
53		from synapse.storage.data_stores.main.roommember import RoomMemberBackgroundUpdateStore
54		from synapse.storage.data_stores.main.search import SearchBackgroundUpdateStore
55		from synapse.storage.data_stores.main.state import MainStateBackgroundUpdateStore
56		from synapse.storage.data_stores.main.stats import StatsStore
57		from synapse.storage.data_stores.main.user_directory import (
	51	from synapse.storage.databases.main.room import RoomBackgroundUpdateStore
	52	from synapse.storage.databases.main.roommember import RoomMemberBackgroundUpdateStore
	53	from synapse.storage.databases.main.search import SearchBackgroundUpdateStore
	54	from synapse.storage.databases.main.state import MainStateBackgroundUpdateStore
	55	from synapse.storage.databases.main.stats import StatsStore
	56	from synapse.storage.databases.main.user_directory import (
58	57	UserDirectoryBackgroundUpdateStore,
59	58	)
60		from synapse.storage.data_stores.state.bg_updates import StateBackgroundUpdateStore
61		from synapse.storage.database import Database, make_conn
	59	from synapse.storage.databases.state.bg_updates import StateBackgroundUpdateStore
62	60	from synapse.storage.engines import create_engine
63	61	from synapse.storage.prepare_database import prepare_database
64	62	from synapse.util import Clock

174	172	StatsStore,
175	173	):
176	174	def execute(self, f, args, *kwargs):
177		return self.db.runInteraction(f.__name__, f, args, *kwargs)
	175	return self.db_pool.runInteraction(f.__name__, f, args, *kwargs)
178	176
179	177	def execute_sql(self, sql, *args):
180	178	def r(txn):
181	179	txn.execute(sql, args)
182	180	return txn.fetchall()
183	181
184		return self.db.runInteraction("execute_sql", r)
	182	return self.db_pool.runInteraction("execute_sql", r)
185	183
186	184	def insert_many_txn(self, txn, table, headers, rows):
187	185	sql = "INSERT INTO %s (%s) VALUES (%s)" % (

226	224	async def setup_table(self, table):
227	225	if table in APPEND_ONLY_TABLES:
228	226	# It's safe to just carry on inserting.
229		row = await self.postgres_store.db.simple_select_one(
	227	row = await self.postgres_store.db_pool.simple_select_one(
230	228	table="port_from_sqlite3",
231	229	keyvalues={"table_name": table},
232	230	retcols=("forward_rowid", "backward_rowid"),

243	241	) = await self._setup_sent_transactions()
244	242	backward_chunk = 0
245	243	else:
246		await self.postgres_store.db.simple_insert(
	244	await self.postgres_store.db_pool.simple_insert(
247	245	table="port_from_sqlite3",
248	246	values={
249	247	"table_name": table,

273	271
274	272	await self.postgres_store.execute(delete_all)
275	273
276		await self.postgres_store.db.simple_insert(
	274	await self.postgres_store.db_pool.simple_insert(
277	275	table="port_from_sqlite3",
278	276	values={"table_name": table, "forward_rowid": 1, "backward_rowid": 0},
279	277	)

317	315	if table == "user_directory_stream_pos":
318	316	# We need to make sure there is a single row, `(X, null), as that is
319	317	# what synapse expects to be there.
320		await self.postgres_store.db.simple_insert(
	318	await self.postgres_store.db_pool.simple_insert(
321	319	table=table, values={"stream_id": None}
322	320	)
323	321	self.progress.update(table, table_size) # Mark table as done

358	356
359	357	return headers, forward_rows, backward_rows
360	358
361		headers, frows, brows = await self.sqlite_store.db.runInteraction(
	359	headers, frows, brows = await self.sqlite_store.db_pool.runInteraction(
362	360	"select", r
363	361	)
364	362

374	372	def insert(txn):
375	373	self.postgres_store.insert_many_txn(txn, table, headers[1:], rows)
376	374
377		self.postgres_store.db.simple_update_one_txn(
	375	self.postgres_store.db_pool.simple_update_one_txn(
378	376	txn,
379	377	table="port_from_sqlite3",
380	378	keyvalues={"table_name": table},

412	410
413	411	return headers, rows
414	412
415		headers, rows = await self.sqlite_store.db.runInteraction("select", r)
	413	headers, rows = await self.sqlite_store.db_pool.runInteraction("select", r)
416	414
417	415	if rows:
418	416	forward_chunk = rows[-1][0] + 1

450	448	],
451	449	)
452	450
453		self.postgres_store.db.simple_update_one_txn(
	451	self.postgres_store.db_pool.simple_update_one_txn(
454	452	txn,
455	453	table="port_from_sqlite3",
456	454	keyvalues={"table_name": "event_search"},

493	491	db_conn, allow_outdated_version=allow_outdated_version
494	492	)
495	493	prepare_database(db_conn, engine, config=self.hs_config)
496		store = Store(Database(hs, db_config, engine), db_conn, hs)
	494	store = Store(DatabasePool(hs, db_config, engine), db_conn, hs)
497	495	db_conn.commit()
498	496
499	497	return store

501	499	async def run_background_updates_on_postgres(self):
502	500	# Manually apply all background updates on the PostgreSQL database.
503	501	postgres_ready = (
504		await self.postgres_store.db.updates.has_completed_background_updates()
	502	await self.postgres_store.db_pool.updates.has_completed_background_updates()
505	503	)
506	504
507	505	if not postgres_ready:

510	508	self.progress.set_state("Running background updates on PostgreSQL")
511	509
512	510	while not postgres_ready:
513		await self.postgres_store.db.updates.do_next_background_update(100)
	511	await self.postgres_store.db_pool.updates.do_next_background_update(100)
514	512	postgres_ready = await (
515		self.postgres_store.db.updates.has_completed_background_updates()
	513	self.postgres_store.db_pool.updates.has_completed_background_updates()
516	514	)
517	515
518	516	async def run(self):

533	531
534	532	# Check if all background updates are done, abort if not.
535	533	updates_complete = (
536		await self.sqlite_store.db.updates.has_completed_background_updates()
	534	await self.sqlite_store.db_pool.updates.has_completed_background_updates()
537	535	)
538	536	if not updates_complete:
539	537	end_error = (

575	573	)
576	574
577	575	try:
578		await self.postgres_store.db.runInteraction("alter_table", alter_table)
	576	await self.postgres_store.db_pool.runInteraction(
	577	"alter_table", alter_table
	578	)
579	579	except Exception:
580	580	# On Error Resume Next
581	581	pass
582	582
583		await self.postgres_store.db.runInteraction(
	583	await self.postgres_store.db_pool.runInteraction(
584	584	"create_port_table", create_port_table
585	585	)
586	586
587	587	# Step 2. Get tables.
588	588	self.progress.set_state("Fetching tables")
589		sqlite_tables = await self.sqlite_store.db.simple_select_onecol(
	589	sqlite_tables = await self.sqlite_store.db_pool.simple_select_onecol(
590	590	table="sqlite_master", keyvalues={"type": "table"}, retcol="name"
591	591	)
592	592
593		postgres_tables = await self.postgres_store.db.simple_select_onecol(
	593	postgres_tables = await self.postgres_store.db_pool.simple_select_onecol(
594	594	table="information_schema.tables",
595	595	keyvalues={},
596	596	retcol="distinct table_name",

691	691
692	692	return headers, [r for r in rows if r[ts_ind] < yesterday]
693	693
694		headers, rows = await self.sqlite_store.db.runInteraction("select", r)
	694	headers, rows = await self.sqlite_store.db_pool.runInteraction("select", r)
695	695
696	696	rows = self._convert_rows("sent_transactions", headers, rows)
697	697

724	724	next_chunk = await self.sqlite_store.execute(get_start_id)
725	725	next_chunk = max(max_inserted_rowid + 1, next_chunk)
726	726
727		await self.postgres_store.db.simple_insert(
	727	await self.postgres_store.db_pool.simple_insert(
728	728	table="port_from_sqlite3",
729	729	values={
730	730	"table_name": "sent_transactions",

793	793	next_id = curr_id + 1
794	794	txn.execute("ALTER SEQUENCE state_group_id_seq RESTART WITH %s", (next_id,))
795	795
796		return self.postgres_store.db.runInteraction("setup_state_group_id_seq", r)
	796	return self.postgres_store.db_pool.runInteraction("setup_state_group_id_seq", r)
797	797
798	798	def _setup_user_id_seq(self):
799	799	def r(txn):
800	800	next_id = find_max_generated_user_id_localpart(txn) + 1
801	801	txn.execute("ALTER SEQUENCE user_id_seq RESTART WITH %s", (next_id,))
802	802
803		return self.postgres_store.db.runInteraction("setup_user_id_seq", r)
	803	return self.postgres_store.db_pool.runInteraction("setup_user_id_seq", r)
804	804
805	805
806	806	##############################################

+12

-1

scripts-dev/check-newsfragment less more

1	1	#
2	2	# A script which checks that an appropriate news file has been added on this
3	3	# branch.
	4
	5	echo -e "+++ \033[32mChecking newsfragment\033[m"
4	6
5	7	set -e
6	8

15	17	if ! git diff --quiet FETCH_HEAD... -- debian; then
16	18	if git diff --quiet FETCH_HEAD... -- debian/changelog; then
17	19	echo "Updates to debian directory, but no update to the changelog." >&2
	20	echo "!! Please see the contributing guide for help writing your changelog entry:" >&2
	21	echo "https://github.com/matrix-org/synapse/blob/develop/CONTRIBUTING.md#debian-changelog" >&2
18	22	exit 1
19	23	fi
20	24	fi

25	29	exit 0
26	30	fi
27	31
28		tox -qe check-newsfragment
	32	# Print a link to the contributing guide if the user makes a mistake
	33	CONTRIBUTING_GUIDE_TEXT="!! Please see the contributing guide for help writing your changelog entry:
	34	https://github.com/matrix-org/synapse/blob/develop/CONTRIBUTING.md#changelog"
	35
	36	# If check-newsfragment returns a non-zero exit code, print the contributing guide and exit
	37	tox -qe check-newsfragment \|\| (echo -e "$CONTRIBUTING_GUIDE_TEXT" >&2 && exit 1)
29	38
30	39	echo
31	40	echo "--------------------------"

37	46	lastchar=`tr -d '\n' < $f \| tail -c 1`
38	47	if [ $lastchar != '.' -a $lastchar != '!' ]; then
39	48	echo -e "\e[31mERROR: newsfragment $f does not end with a '.' or '!'\e[39m" >&2
	49	echo -e "$CONTRIBUTING_GUIDE_TEXT" >&2
40	50	exit 1
41	51	fi
42	52

46	56
47	57	if [[ -n "$pr" && "$matched" -eq 0 ]]; then
48	58	echo -e "\e[31mERROR: Did not find a news fragment with the right number: expected changelog.d/$pr.*.\e[39m" >&2
	59	echo -e "$CONTRIBUTING_GUIDE_TEXT" >&2
49	60	exit 1
50	61	fi

+34

-0

scripts-dev/check_line_terminators.sh less more

	0	#!/bin/bash
	1	#
	2	# Copyright 2020 The Matrix.org Foundation C.I.C.
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15	#
	16	# This script checks that line terminators in all repository files (excluding
	17	# those in the .git directory) feature unix line terminators.
	18	#
	19	# Usage:
	20	#
	21	# ./check_line_terminators.sh
	22	#
	23	# The script will emit exit code 1 if any files that do not use unix line
	24	# terminators are found, 0 otherwise.
	25
	26	# cd to the root of the repository
	27	cd `dirname $0`/..
	28
	29	# Find and print files with non-unix line terminators
	30	if find . -path './.git/*' -prune -o -type f -print0 \| xargs -0 grep -I -l $'\r$'; then
	31	echo -e '\e[31mERROR: found files with CRLF line endings. See above.\e[39m'
	32	exit 1
	33	fi

+2

-2

scripts-dev/update_database less more

39	39	config.server_name, reactor=reactor, config=config, **kwargs
40	40	)
41	41
42		self.version_string = "Synapse/"+get_version_string(synapse)
	42	self.version_string = "Synapse/" + get_version_string(synapse)
43	43
44	44
45	45	if __name__ == "__main__":

85	85	store = hs.get_datastore()
86	86
87	87	async def run_background_updates():
88		await store.db.updates.run_background_updates(sleep=False)
	88	await store.db_pool.updates.run_background_updates(sleep=False)
89	89	# Stop the reactor to exit the script once every background update is run.
90	90	reactor.stop()
91	91

+13

-1

synapse/__init__.py less more

16	16	""" This is a reference implementation of a Matrix homeserver.
17	17	"""
18	18
	19	import json
19	20	import os
20	21	import sys
21	22

24	25	print("Synapse requires Python 3.5 or above.")
25	26	sys.exit(1)
26	27
	28	# Twisted and canonicaljson will fail to import when this file is executed to
	29	# get the __version__ during a fresh install. That's OK and subsequent calls to
	30	# actually start Synapse will import these libraries fine.
27	31	try:
28	32	from twisted.internet import protocol
29	33	from twisted.internet.protocol import Factory

35	39	except ImportError:
36	40	pass
37	41
38		__version__ = "1.18.0"
	42	# Use the standard library json implementation instead of simplejson.
	43	try:
	44	from canonicaljson import set_json_library
	45
	46	set_json_library(json)
	47	except ImportError:
	48	pass
	49
	50	__version__ = "1.19.0"
39	51
40	52	if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
41	53	# We import here so that we don't have to install a bunch of deps when

+58

-69

synapse/api/auth.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
15		from typing import Optional
	15	from typing import List, Optional, Tuple
16	16
17	17	import pymacaroons
18	18	from netaddr import IPAddress
19	19
20		from twisted.internet import defer
21	20	from twisted.web.server import Request
22	21
23	22	import synapse.types

79	78	self._track_appservice_user_ips = hs.config.track_appservice_user_ips
80	79	self._macaroon_secret_key = hs.config.macaroon_secret_key
81	80
82		@defer.inlineCallbacks
83		def check_from_context(self, room_version: str, event, context, do_sig_check=True):
84		prev_state_ids = yield context.get_prev_state_ids()
85		auth_events_ids = yield self.compute_auth_events(
	81	async def check_from_context(
	82	self, room_version: str, event, context, do_sig_check=True
	83	):
	84	prev_state_ids = await context.get_prev_state_ids()
	85	auth_events_ids = self.compute_auth_events(
86	86	event, prev_state_ids, for_verification=True
87	87	)
88		auth_events = yield self.store.get_events(auth_events_ids)
	88	auth_events = await self.store.get_events(auth_events_ids)
89	89	auth_events = {(e.type, e.state_key): e for e in auth_events.values()}
90	90
91	91	room_version_obj = KNOWN_ROOM_VERSIONS[room_version]

93	93	room_version_obj, event, auth_events=auth_events, do_sig_check=do_sig_check
94	94	)
95	95
96		@defer.inlineCallbacks
97		def check_user_in_room(
	96	async def check_user_in_room(
98	97	self,
99	98	room_id: str,
100	99	user_id: str,
101	100	current_state: Optional[StateMap[EventBase]] = None,
102	101	allow_departed_users: bool = False,
103		):
	102	) -> EventBase:
104	103	"""Check if the user is in the room, or was at some point.
105	104	Args:
106	105	room_id: The room to check.

118	117	Raises:
119	118	AuthError if the user is/was not in the room.
120	119	Returns:
121		Deferred[Optional[EventBase]]:
122		Membership event for the user if the user was in the
123		room. This will be the join event if they are currently joined to
124		the room. This will be the leave event if they have left the room.
	120	Membership event for the user if the user was in the
	121	room. This will be the join event if they are currently joined to
	122	the room. This will be the leave event if they have left the room.
125	123	"""
126	124	if current_state:
127	125	member = current_state.get((EventTypes.Member, user_id), None)
128	126	else:
129		member = yield defer.ensureDeferred(
130		self.state.get_current_state(
131		room_id=room_id, event_type=EventTypes.Member, state_key=user_id
132		)
	127	member = await self.state.get_current_state(
	128	room_id=room_id, event_type=EventTypes.Member, state_key=user_id
133	129	)
134		membership = member.membership if member else None
135
136		if membership == Membership.JOIN:
137		return member
138
139		# XXX this looks totally bogus. Why do we not allow users who have been banned,
140		# or those who were members previously and have been re-invited?
141		if allow_departed_users and membership == Membership.LEAVE:
142		forgot = yield self.store.did_forget(user_id, room_id)
143		if not forgot:
	130
	131	if member:
	132	membership = member.membership
	133
	134	if membership == Membership.JOIN:
144	135	return member
145	136
	137	# XXX this looks totally bogus. Why do we not allow users who have been banned,
	138	# or those who were members previously and have been re-invited?
	139	if allow_departed_users and membership == Membership.LEAVE:
	140	forgot = await self.store.did_forget(user_id, room_id)
	141	if not forgot:
	142	return member
	143
146	144	raise AuthError(403, "User %s not in room %s" % (user_id, room_id))
147	145
148		@defer.inlineCallbacks
149		def check_host_in_room(self, room_id, host):
	146	async def check_host_in_room(self, room_id, host):
150	147	with Measure(self.clock, "check_host_in_room"):
151		latest_event_ids = yield self.store.is_host_joined(room_id, host)
	148	latest_event_ids = await self.store.is_host_joined(room_id, host)
152	149	return latest_event_ids
153	150
154	151	def can_federate(self, event, auth_events):

159	156	def get_public_keys(self, invite_event):
160	157	return event_auth.get_public_keys(invite_event)
161	158
162		@defer.inlineCallbacks
163		def get_user_by_req(
	159	async def get_user_by_req(
164	160	self,
165	161	request: Request,
166	162	allow_guest: bool = False,
167	163	rights: str = "access",
168	164	allow_expired: bool = False,
169		):
	165	) -> synapse.types.Requester:
170	166	""" Get a registered user's ID.
171	167
172	168	Args:

179	175	/login will deliver access tokens regardless of expiration.
180	176
181	177	Returns:
182		defer.Deferred: resolves to a `synapse.types.Requester` object
	178	Resolves to the requester
183	179	Raises:
184	180	InvalidClientCredentialsError if no user by that token exists or the token
185	181	is invalid.

193	189
194	190	access_token = self.get_access_token_from_request(request)
195	191
196		user_id, app_service = yield self._get_appservice_user_id(request)
	192	user_id, app_service = await self._get_appservice_user_id(request)
197	193	if user_id:
198	194	request.authenticated_entity = user_id
199	195	opentracing.set_tag("authenticated_entity", user_id)
200	196	opentracing.set_tag("appservice_id", app_service.id)
201	197
202	198	if ip_addr and self._track_appservice_user_ips:
203		yield self.store.insert_client_ip(
	199	await self.store.insert_client_ip(
204	200	user_id=user_id,
205	201	access_token=access_token,
206	202	ip=ip_addr,

210	206
211	207	return synapse.types.create_requester(user_id, app_service=app_service)
212	208
213		user_info = yield self.get_user_by_access_token(
	209	user_info = await self.get_user_by_access_token(
214	210	access_token, rights, allow_expired=allow_expired
215	211	)
216	212	user = user_info["user"]

220	216	# Deny the request if the user account has expired.
221	217	if self._account_validity.enabled and not allow_expired:
222	218	user_id = user.to_string()
223		expiration_ts = yield self.store.get_expiration_ts_for_user(user_id)
	219	expiration_ts = await self.store.get_expiration_ts_for_user(user_id)
224	220	if (
225	221	expiration_ts is not None
226	222	and self.clock.time_msec() >= expiration_ts

234	230	device_id = user_info.get("device_id")
235	231
236	232	if user and access_token and ip_addr:
237		yield self.store.insert_client_ip(
	233	await self.store.insert_client_ip(
238	234	user_id=user.to_string(),
239	235	access_token=access_token,
240	236	ip=ip_addr,

260	256	except KeyError:
261	257	raise MissingClientTokenError()
262	258
263		@defer.inlineCallbacks
264		def _get_appservice_user_id(self, request):
	259	async def _get_appservice_user_id(self, request):
265	260	app_service = self.store.get_app_service_by_token(
266	261	self.get_access_token_from_request(request)
267	262	)

282	277
283	278	if not app_service.is_interested_in_user(user_id):
284	279	raise AuthError(403, "Application service cannot masquerade as this user.")
285		if not (yield self.store.get_user_by_id(user_id)):
	280	if not (await self.store.get_user_by_id(user_id)):
286	281	raise AuthError(403, "Application service has not registered this user")
287	282	return user_id, app_service
288	283
289		@defer.inlineCallbacks
290		def get_user_by_access_token(
	284	async def get_user_by_access_token(
291	285	self, token: str, rights: str = "access", allow_expired: bool = False,
292		):
	286	) -> dict:
293	287	""" Validate access token and get user_id from it
294	288
295	289	Args:

299	293	allow_expired: If False, raises an InvalidClientTokenError
300	294	if the token is expired
301	295	Returns:
302		Deferred[dict]: dict that includes:
	296	dict that includes:
303	297	`user` (UserID)
304	298	`is_guest` (bool)
305	299	`token_id` (int\|None): access token id. May be None if guest

313	307
314	308	if rights == "access":
315	309	# first look in the database
316		r = yield self._look_up_user_by_access_token(token)
	310	r = await self._look_up_user_by_access_token(token)
317	311	if r:
318	312	valid_until_ms = r["valid_until_ms"]
319	313	if (

351	345	# It would of course be much easier to store guest access
352	346	# tokens in the database as well, but that would break existing
353	347	# guest tokens.
354		stored_user = yield self.store.get_user_by_id(user_id)
	348	stored_user = await self.store.get_user_by_id(user_id)
355	349	if not stored_user:
356	350	raise InvalidClientTokenError("Unknown user_id %s" % user_id)
357	351	if not stored_user["is_guest"]:

481	475	now = self.hs.get_clock().time_msec()
482	476	return now < expiry
483	477
484		@defer.inlineCallbacks
485		def _look_up_user_by_access_token(self, token):
486		ret = yield self.store.get_user_by_access_token(token)
	478	async def _look_up_user_by_access_token(self, token):
	479	ret = await self.store.get_user_by_access_token(token)
487	480	if not ret:
488	481	return None
489	482

506	499	logger.warning("Unrecognised appservice access token.")
507	500	raise InvalidClientTokenError()
508	501	request.authenticated_entity = service.sender
509		return defer.succeed(service)
	502	return service
510	503
511	504	async def is_server_admin(self, user: UserID) -> bool:
512	505	""" Check if the given user is a local server admin.

521	514
522	515	def compute_auth_events(
523	516	self, event, current_state_ids: StateMap[str], for_verification: bool = False,
524		):
	517	) -> List[str]:
525	518	"""Given an event and current state return the list of event IDs used
526	519	to auth an event.
527	520

529	522	should be added to the event's `auth_events`.
530	523
531	524	Returns:
532		defer.Deferred(list[str]): List of event IDs.
	525	List of event IDs.
533	526	"""
534	527
535	528	if event.type == EventTypes.Create:
536		return defer.succeed([])
	529	return []
537	530
538	531	# Currently we ignore the `for_verification` flag even though there are
539	532	# some situations where we can drop particular auth events when adding

552	545	if auth_ev_id:
553	546	auth_ids.append(auth_ev_id)
554	547
555		return defer.succeed(auth_ids)
	548	return auth_ids
556	549
557	550	async def check_can_change_room_list(self, room_id: str, user: UserID):
558	551	"""Determine whether the user is allowed to edit the room's entry in the

635	628
636	629	return query_params[0].decode("ascii")
637	630
638		@defer.inlineCallbacks
639		def check_user_in_room_or_world_readable(
	631	async def check_user_in_room_or_world_readable(
640	632	self, room_id: str, user_id: str, allow_departed_users: bool = False
641		):
	633	) -> Tuple[str, Optional[str]]:
642	634	"""Checks that the user is or was in the room or the room is world
643	635	readable. If it isn't then an exception is raised.
644	636

649	641	members but have now departed
650	642
651	643	Returns:
652		Deferred[tuple[str, str\|None]]: Resolves to the current membership of
653		the user in the room and the membership event ID of the user. If
654		the user is not in the room and never has been, then
655		`(Membership.JOIN, None)` is returned.
	644	Resolves to the current membership of the user in the room and the
	645	membership event ID of the user. If the user is not in the room and
	646	never has been, then `(Membership.JOIN, None)` is returned.
656	647	"""
657	648
658	649	try:

661	652	# * The user is a non-guest user, and was ever in the room
662	653	# * The user is a guest user, and has joined the room
663	654	# else it will throw.
664		member_event = yield self.check_user_in_room(
	655	member_event = await self.check_user_in_room(
665	656	room_id, user_id, allow_departed_users=allow_departed_users
666	657	)
667	658	return member_event.membership, member_event.event_id
668	659	except AuthError:
669		visibility = yield defer.ensureDeferred(
670		self.state.get_current_state(
671		room_id, EventTypes.RoomHistoryVisibility, ""
672		)
	660	visibility = await self.state.get_current_state(
	661	room_id, EventTypes.RoomHistoryVisibility, ""
673	662	)
674	663	if (
675	664	visibility

+5

-8

synapse/api/auth_blocking.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
16
17		from twisted.internet import defer
18	16
19	17	from synapse.api.constants import LimitBlockingTypes, UserTypes
20	18	from synapse.api.errors import Codes, ResourceLimitError

35	33	self._limit_usage_by_mau = hs.config.limit_usage_by_mau
36	34	self._mau_limits_reserved_threepids = hs.config.mau_limits_reserved_threepids
37	35
38		@defer.inlineCallbacks
39		def check_auth_blocking(self, user_id=None, threepid=None, user_type=None):
	36	async def check_auth_blocking(self, user_id=None, threepid=None, user_type=None):
40	37	"""Checks if the user should be rejected for some external reason,
41	38	such as monthly active user limiting or global disable flag
42	39

59	56	if user_id is not None:
60	57	if user_id == self._server_notices_mxid:
61	58	return
62		if (yield self.store.is_support_user(user_id)):
	59	if await self.store.is_support_user(user_id):
63	60	return
64	61
65	62	if self._hs_disabled:

75	72
76	73	# If the user is already part of the MAU cohort or a trial user
77	74	if user_id:
78		timestamp = yield self.store.user_last_seen_monthly_active(user_id)
	75	timestamp = await self.store.user_last_seen_monthly_active(user_id)
79	76	if timestamp:
80	77	return
81	78
82		is_trial = yield self.store.is_trial_user(user_id)
	79	is_trial = await self.store.is_trial_user(user_id)
83	80	if is_trial:
84	81	return
85	82	elif threepid:

92	89	# allow registration. Support users are excluded from MAU checks.
93	90	return
94	91	# Else if there is no room in the MAU bucket, bail
95		current_mau = yield self.store.get_monthly_active_count()
	92	current_mau = await self.store.get_monthly_active_count()
96	93	if current_mau >= self._max_mau_value:
97	94	raise ResourceLimitError(
98	95	403,

+3

-1

synapse/api/errors.py less more

237	237	(This indicates we should return a 401 with 'result' as the body)
238	238
239	239	Attributes:
	240	session_id: The ID of the ongoing interactive auth session.
240	241	result: the server response to the request, which should be
241	242	passed back to the client
242	243	"""
243	244
244		def __init__(self, result: "JsonDict"):
	245	def __init__(self, session_id: str, result: "JsonDict"):
245	246	super(InteractiveAuthIncompleteError, self).__init__(
246	247	"Interactive auth not yet complete"
247	248	)
	249	self.session_id = session_id
248	250	self.result = result
249	251
250	252

+2

-5

synapse/api/filtering.py less more

19	19	import jsonschema
20	20	from canonicaljson import json
21	21	from jsonschema import FormatChecker
22
23		from twisted.internet import defer
24	22
25	23	from synapse.api.constants import EventContentFields
26	24	from synapse.api.errors import SynapseError

136	134	super(Filtering, self).__init__()
137	135	self.store = hs.get_datastore()
138	136
139		@defer.inlineCallbacks
140		def get_user_filter(self, user_localpart, filter_id):
141		result = yield self.store.get_user_filter(user_localpart, filter_id)
	137	async def get_user_filter(self, user_localpart, filter_id):
	138	result = await self.store.get_user_filter(user_localpart, filter_id)
142	139	return FilterCollection(result)
143	140
144	141	def add_user_filter(self, user_localpart, user_filter):

+4

-14

synapse/app/_base.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14
15	14	import gc
16	15	import logging
17	16	import os

21	20	import traceback
22	21	from typing import Iterable
23	22
24		from daemonize import Daemonize
25	23	from typing_extensions import NoReturn
26	24
27	25	from twisted.internet import defer, error, reactor

33	31	from synapse.crypto import context_factory
34	32	from synapse.logging.context import PreserveLoggingContext
35	33	from synapse.util.async_helpers import Linearizer
	34	from synapse.util.daemonize import daemonize_process
36	35	from synapse.util.rlimit import change_resource_limit
37	36	from synapse.util.versionstring import get_version_string
38	37

128	127	if print_pidfile:
129	128	print(pid_file)
130	129
131		daemon = Daemonize(
132		app=appname,
133		pid=pid_file,
134		action=run,
135		auto_close_fds=False,
136		verbose=True,
137		logger=logger,
138		)
139		daemon.start()
140		else:
141		run()
	130	daemonize_process(pid_file, logger)
	131	run()
142	132
143	133
144	134	def quit_with_error(error_string: str) -> NoReturn:

277	267
278	268	# It is now safe to start your Synapse.
279	269	hs.start_listening(listeners)
280		hs.get_datastore().db.start_profiling()
	270	hs.get_datastore().db_pool.start_profiling()
281	271	hs.get_pusherpool().start()
282	272
283	273	setup_sentry(hs)

+18

-12

synapse/app/generic_worker.py less more

122	122	from synapse.rest.client.v2_alpha.keys import KeyChangesServlet, KeyQueryServlet
123	123	from synapse.rest.client.v2_alpha.register import RegisterRestServlet
124	124	from synapse.rest.client.versions import VersionsRestServlet
	125	from synapse.rest.health import HealthResource
125	126	from synapse.rest.key.v2 import KeyApiV2Resource
126		from synapse.server import HomeServer
127		from synapse.storage.data_stores.main.censor_events import CensorEventsStore
128		from synapse.storage.data_stores.main.media_repository import MediaRepositoryStore
129		from synapse.storage.data_stores.main.monthly_active_users import (
	127	from synapse.server import HomeServer, cache_in_self
	128	from synapse.storage.databases.main.censor_events import CensorEventsStore
	129	from synapse.storage.databases.main.media_repository import MediaRepositoryStore
	130	from synapse.storage.databases.main.monthly_active_users import (
130	131	MonthlyActiveUsersWorkerStore,
131	132	)
132		from synapse.storage.data_stores.main.presence import UserPresenceState
133		from synapse.storage.data_stores.main.search import SearchWorkerStore
134		from synapse.storage.data_stores.main.ui_auth import UIAuthWorkerStore
135		from synapse.storage.data_stores.main.user_directory import UserDirectoryStore
	133	from synapse.storage.databases.main.presence import UserPresenceState
	134	from synapse.storage.databases.main.search import SearchWorkerStore
	135	from synapse.storage.databases.main.ui_auth import UIAuthWorkerStore
	136	from synapse.storage.databases.main.user_directory import UserDirectoryStore
136	137	from synapse.types import ReadReceipt
137	138	from synapse.util.async_helpers import Linearizer
138	139	from synapse.util.httpresourcetree import create_resource_tree

492	493	site_tag = listener_config.http_options.tag
493	494	if site_tag is None:
494	495	site_tag = port
495		resources = {}
	496
	497	# We always include a health resource.
	498	resources = {"/health": HealthResource()}
	499
496	500	for res in listener_config.http_options.resources:
497	501	for name in res.names:
498	502	if name == "metrics":

627	631
628	632	self.get_tcp_replication().start_replication(self)
629	633
630		def remove_pusher(self, app_id, push_key, user_id):
	634	async def remove_pusher(self, app_id, push_key, user_id):
631	635	self.get_tcp_replication().send_remove_pusher(app_id, push_key, user_id)
632	636
633		def build_replication_data_handler(self):
	637	@cache_in_self
	638	def get_replication_data_handler(self):
634	639	return GenericWorkerReplicationHandler(self)
635	640
636		def build_presence_handler(self):
	641	@cache_in_self
	642	def get_presence_handler(self):
637	643	return GenericWorkerPresence(self)
638	644
639	645

+13

-11

synapse/app/homeserver.py less more

67	67	from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
68	68	from synapse.rest import ClientRestResource
69	69	from synapse.rest.admin import AdminRestResource
	70	from synapse.rest.health import HealthResource
70	71	from synapse.rest.key.v2 import KeyApiV2Resource
71	72	from synapse.rest.well_known import WellKnownResource
72	73	from synapse.server import HomeServer

97	98	if site_tag is None:
98	99	site_tag = port
99	100
100		resources = {}
	101	# We always include a health resource.
	102	resources = {"/health": HealthResource()}
	103
101	104	for res in listener_config.http_options.resources:
102	105	for name in res.names:
103	106	if name == "openid" and "federation" in res.names:

379	382
380	383	hs.setup_master()
381	384
382		@defer.inlineCallbacks
383		def do_acme():
	385	async def do_acme() -> bool:
384	386	"""
385	387	Reprovision an ACME certificate, if it's required.
386	388
387	389	Returns:
388		Deferred[bool]: Whether the cert has been updated.
	390	Whether the cert has been updated.
389	391	"""
390	392	acme = hs.get_acme_handler()
391	393

404	406	provision = True
405	407
406	408	if provision:
407		yield acme.provision_certificate()
	409	await acme.provision_certificate()
408	410
409	411	return provision
410	412

414	416	Provision a certificate from ACME, if required, and reload the TLS
415	417	certificate if it's renewed.
416	418	"""
417		reprovisioned = yield do_acme()
	419	reprovisioned = yield defer.ensureDeferred(do_acme())
418	420	if reprovisioned:
419	421	_base.refresh_certificate(hs)
420	422

426	428	acme = hs.get_acme_handler()
427	429	# Start up the webservices which we will respond to ACME
428	430	# challenges with, and then provision.
429		yield acme.start_listening()
430		yield do_acme()
	431	yield defer.ensureDeferred(acme.start_listening())
	432	yield defer.ensureDeferred(do_acme())
431	433
432	434	# Check if it needs to be reprovisioned every day.
433	435	hs.get_clock().looping_call(reprovision_acme, 24 * 60 * 60 * 1000)

441	443
442	444	_base.start(hs, config.listeners)
443	445
444		hs.get_datastore().db.updates.start_doing_background_updates()
	446	hs.get_datastore().db_pool.updates.start_doing_background_updates()
445	447	except Exception:
446	448	# Print the exception and bail out.
447	449	print("Error during startup:", file=sys.stderr)

551	553	#
552	554
553	555	# This only reports info about the main database.
554		stats["database_engine"] = hs.get_datastore().db.engine.module.__name__
555		stats["database_server_version"] = hs.get_datastore().db.engine.server_version
	556	stats["database_engine"] = hs.get_datastore().db_pool.engine.module.__name__
	557	stats["database_server_version"] = hs.get_datastore().db_pool.engine.server_version
556	558
557	559	logger.info("Reporting stats to %s: %s" % (hs.config.report_stats_endpoint, stats))
558	560	try:

+15

-20

synapse/appservice/__init__.py less more

14	14	import logging
15	15	import re
16	16
17		from twisted.internet import defer
18
19	17	from synapse.api.constants import EventTypes
20	18	from synapse.types import GroupID, get_domain_from_id
21		from synapse.util.caches.descriptors import cachedInlineCallbacks
	19	from synapse.util.caches.descriptors import cached
22	20
23	21	logger = logging.getLogger(__name__)
24	22

42	40	Args:
43	41	as_api(ApplicationServiceApi): The API to use to send.
44	42	Returns:
45		A Deferred which resolves to True if the transaction was sent.
	43	An Awaitable which resolves to True if the transaction was sent.
46	44	"""
47	45	return as_api.push_bulk(
48	46	service=self.service, events=self.events, txn_id=self.id

171	169	return regex_obj["exclusive"]
172	170	return False
173	171
174		@defer.inlineCallbacks
175		def _matches_user(self, event, store):
	172	async def _matches_user(self, event, store):
176	173	if not event:
177	174	return False
178	175

187	184	if not store:
188	185	return False
189	186
190		does_match = yield self._matches_user_in_member_list(event.room_id, store)
	187	does_match = await self._matches_user_in_member_list(event.room_id, store)
191	188	return does_match
192	189
193		@cachedInlineCallbacks(num_args=1, cache_context=True)
194		def _matches_user_in_member_list(self, room_id, store, cache_context):
195		member_list = yield store.get_users_in_room(
	190	@cached(num_args=1, cache_context=True)
	191	async def _matches_user_in_member_list(self, room_id, store, cache_context):
	192	member_list = await store.get_users_in_room(
196	193	room_id, on_invalidate=cache_context.invalidate
197	194	)
198	195

207	204	return self.is_interested_in_room(event.room_id)
208	205	return False
209	206
210		@defer.inlineCallbacks
211		def _matches_aliases(self, event, store):
	207	async def _matches_aliases(self, event, store):
212	208	if not store or not event:
213	209	return False
214	210
215		alias_list = yield store.get_aliases_for_room(event.room_id)
	211	alias_list = await store.get_aliases_for_room(event.room_id)
216	212	for alias in alias_list:
217	213	if self.is_interested_in_alias(alias):
218	214	return True
219	215	return False
220	216
221		@defer.inlineCallbacks
222		def is_interested(self, event, store=None):
	217	async def is_interested(self, event, store=None) -> bool:
223	218	"""Check if this service is interested in this event.
224	219
225	220	Args:
226	221	event(Event): The event to check.
227	222	store(DataStore)
228	223	Returns:
229		bool: True if this service would like to know about this event.
	224	True if this service would like to know about this event.
230	225	"""
231	226	# Do cheap checks first
232	227	if self._matches_room_id(event):
233	228	return True
234	229
235		if (yield self._matches_aliases(event, store)):
236		return True
237
238		if (yield self._matches_user(event, store)):
	230	if await self._matches_aliases(event, store):
	231	return True
	232
	233	if await self._matches_user(event, store):
239	234	return True
240	235
241	236	return False

+10

-15

synapse/appservice/api.py less more

92	92	hs, "as_protocol_meta", timeout_ms=HOUR_IN_MS
93	93	)
94	94
95		@defer.inlineCallbacks
96		def query_user(self, service, user_id):
	95	async def query_user(self, service, user_id):
97	96	if service.url is None:
98	97	return False
99	98	uri = service.url + ("/users/%s" % urllib.parse.quote(user_id))
100	99	try:
101		response = yield self.get_json(uri, {"access_token": service.hs_token})
	100	response = await self.get_json(uri, {"access_token": service.hs_token})
102	101	if response is not None: # just an empty json object
103	102	return True
104	103	except CodeMessageException as e:

109	108	logger.warning("query_user to %s threw exception %s", uri, ex)
110	109	return False
111	110
112		@defer.inlineCallbacks
113		def query_alias(self, service, alias):
	111	async def query_alias(self, service, alias):
114	112	if service.url is None:
115	113	return False
116	114	uri = service.url + ("/rooms/%s" % urllib.parse.quote(alias))
117		response = None
118		try:
119		response = yield self.get_json(uri, {"access_token": service.hs_token})
	115	try:
	116	response = await self.get_json(uri, {"access_token": service.hs_token})
120	117	if response is not None: # just an empty json object
121	118	return True
122	119	except CodeMessageException as e:

127	124	logger.warning("query_alias to %s threw exception %s", uri, ex)
128	125	return False
129	126
130		@defer.inlineCallbacks
131		def query_3pe(self, service, kind, protocol, fields):
	127	async def query_3pe(self, service, kind, protocol, fields):
132	128	if kind == ThirdPartyEntityKind.USER:
133	129	required_field = "userid"
134	130	elif kind == ThirdPartyEntityKind.LOCATION:

145	141	urllib.parse.quote(protocol),
146	142	)
147	143	try:
148		response = yield self.get_json(uri, fields)
	144	response = await self.get_json(uri, fields)
149	145	if not isinstance(response, list):
150	146	logger.warning(
151	147	"query_3pe to %s returned an invalid response %r", uri, response

178	174	urllib.parse.quote(protocol),
179	175	)
180	176	try:
181		info = yield self.get_json(uri, {})
	177	info = yield defer.ensureDeferred(self.get_json(uri, {}))
182	178
183	179	if not _is_valid_3pe_metadata(info):
184	180	logger.warning(

201	197	key = (service.id, protocol)
202	198	return self.protocol_meta_cache.wrap(key, _get)
203	199
204		@defer.inlineCallbacks
205		def push_bulk(self, service, events, txn_id=None):
	200	async def push_bulk(self, service, events, txn_id=None):
206	201	if service.url is None:
207	202	return True
208	203

217	212
218	213	uri = service.url + ("/transactions/%s" % urllib.parse.quote(txn_id))
219	214	try:
220		yield self.put_json(
	215	await self.put_json(
221	216	uri=uri,
222	217	json_body={"events": events},
223	218	args={"access_token": service.hs_token},

+22

-31

synapse/appservice/scheduler.py less more

49	49	"""
50	50	import logging
51	51
52		from twisted.internet import defer
53
54	52	from synapse.appservice import ApplicationServiceState
55	53	from synapse.logging.context import run_in_background
56	54	from synapse.metrics.background_process_metrics import run_as_background_process

72	70	self.txn_ctrl = _TransactionController(self.clock, self.store, self.as_api)
73	71	self.queuer = _ServiceQueuer(self.txn_ctrl, self.clock)
74	72
75		@defer.inlineCallbacks
76		def start(self):
	73	async def start(self):
77	74	logger.info("Starting appservice scheduler")
78	75
79	76	# check for any DOWN ASes and start recoverers for them.
80		services = yield self.store.get_appservices_by_state(
	77	services = await self.store.get_appservices_by_state(
81	78	ApplicationServiceState.DOWN
82	79	)
83	80

116	113	"as-sender-%s" % (service.id,), self._send_request, service
117	114	)
118	115
119		@defer.inlineCallbacks
120		def _send_request(self, service):
	116	async def _send_request(self, service):
121	117	# sanity-check: we shouldn't get here if this service already has a sender
122	118	# running.
123	119	assert service.id not in self.requests_in_flight

129	125	if not events:
130	126	return
131	127	try:
132		yield self.txn_ctrl.send(service, events)
	128	await self.txn_ctrl.send(service, events)
133	129	except Exception:
134	130	logger.exception("AS request failed")
135	131	finally:

161	157	# for UTs
162	158	self.RECOVERER_CLASS = _Recoverer
163	159
164		@defer.inlineCallbacks
165		def send(self, service, events):
166		try:
167		txn = yield self.store.create_appservice_txn(service=service, events=events)
168		service_is_up = yield self._is_service_up(service)
	160	async def send(self, service, events):
	161	try:
	162	txn = await self.store.create_appservice_txn(service=service, events=events)
	163	service_is_up = await self._is_service_up(service)
169	164	if service_is_up:
170		sent = yield txn.send(self.as_api)
	165	sent = await txn.send(self.as_api)
171	166	if sent:
172		yield txn.complete(self.store)
	167	await txn.complete(self.store)
173	168	else:
174	169	run_in_background(self._on_txn_fail, service)
175	170	except Exception:
176	171	logger.exception("Error creating appservice transaction")
177	172	run_in_background(self._on_txn_fail, service)
178	173
179		@defer.inlineCallbacks
180		def on_recovered(self, recoverer):
	174	async def on_recovered(self, recoverer):
181	175	logger.info(
182	176	"Successfully recovered application service AS ID %s", recoverer.service.id
183	177	)
184	178	self.recoverers.pop(recoverer.service.id)
185	179	logger.info("Remaining active recoverers: %s", len(self.recoverers))
186		yield self.store.set_appservice_state(
	180	await self.store.set_appservice_state(
187	181	recoverer.service, ApplicationServiceState.UP
188	182	)
189	183
190		@defer.inlineCallbacks
191		def _on_txn_fail(self, service):
192		try:
193		yield self.store.set_appservice_state(service, ApplicationServiceState.DOWN)
	184	async def _on_txn_fail(self, service):
	185	try:
	186	await self.store.set_appservice_state(service, ApplicationServiceState.DOWN)
194	187	self.start_recoverer(service)
195	188	except Exception:
196	189	logger.exception("Error starting AS recoverer")

210	203	recoverer.recover()
211	204	logger.info("Now %i active recoverers", len(self.recoverers))
212	205
213		@defer.inlineCallbacks
214		def _is_service_up(self, service):
215		state = yield self.store.get_appservice_state(service)
	206	async def _is_service_up(self, service):
	207	state = await self.store.get_appservice_state(service)
216	208	return state == ApplicationServiceState.UP or state is None
217	209
218	210

253	245	self.backoff_counter += 1
254	246	self.recover()
255	247
256		@defer.inlineCallbacks
257		def retry(self):
	248	async def retry(self):
258	249	logger.info("Starting retries on %s", self.service.id)
259	250	try:
260	251	while True:
261		txn = yield self.store.get_oldest_unsent_txn(self.service)
	252	txn = await self.store.get_oldest_unsent_txn(self.service)
262	253	if not txn:
263	254	# nothing left: we're done!
264		self.callback(self)
	255	await self.callback(self)
265	256	return
266	257
267	258	logger.info(
268	259	"Retrying transaction %s for AS ID %s", txn.id, txn.service.id
269	260	)
270		sent = yield txn.send(self.as_api)
	261	sent = await txn.send(self.as_api)
271	262	if not sent:
272	263	break
273	264
274		yield txn.complete(self.store)
	265	await txn.complete(self.store)
275	266
276	267	# reset the backoff counter and then process the next transaction
277	268	self.backoff_counter = 1

+49

-0

synapse/config/_util.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	from typing import Any, List
	15
	16	import jsonschema
	17
	18	from synapse.config._base import ConfigError
	19	from synapse.types import JsonDict
	20
	21
	22	def validate_config(json_schema: JsonDict, config: Any, config_path: List[str]) -> None:
	23	"""Validates a config setting against a JsonSchema definition
	24
	25	This can be used to validate a section of the config file against a schema
	26	definition. If the validation fails, a ConfigError is raised with a textual
	27	description of the problem.
	28
	29	Args:
	30	json_schema: the schema to validate against
	31	config: the configuration value to be validated
	32	config_path: the path within the config file. This will be used as a basis
	33	for the error message.
	34	"""
	35	try:
	36	jsonschema.validate(config, json_schema)
	37	except jsonschema.ValidationError as e:
	38	# copy `config_path` before modifying it.
	39	path = list(config_path)
	40	for p in list(e.path):
	41	if isinstance(p, int):
	42	path.append("<item %i>" % p)
	43	else:
	44	path.append(str(p))
	45
	46	raise ConfigError(
	47	"Unable to parse configuration: %s at %s" % (e.message, ".".join(path))
	48	)

+4

-1

synapse/config/database.py less more

99	99
100	100	self.name = name
101	101	self.config = db_config
102		self.data_stores = data_stores
	102
	103	# The `data_stores` config is actually talking about `databases` (we
	104	# changed the name).
	105	self.databases = data_stores
103	106
104	107
105	108	class DatabaseConfig(Config):

+51

-12

synapse/config/logger.py less more

54	54	format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - \
55	55	%(request)s - %(message)s'
56	56
57		filters:
58		context:
59		(): synapse.logging.context.LoggingContextFilter
60		request: ""
61
62	57	handlers:
63	58	file:
64		class: logging.handlers.RotatingFileHandler
	59	class: logging.handlers.TimedRotatingFileHandler
65	60	formatter: precise
66	61	filename: ${log_file}
67		maxBytes: 104857600
68		backupCount: 10
69		filters: [context]
	62	when: midnight
	63	backupCount: 3 # Does not include the current log file.
70	64	encoding: utf8
	65
	66	# Default to buffering writes to log file for efficiency. This means that
	67	# will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
	68	# logs will still be flushed immediately.
	69	buffer:
	70	class: logging.handlers.MemoryHandler
	71	target: file
	72	# The capacity is the number of log lines that are buffered before
	73	# being written to disk. Increasing this will lead to better
	74	# performance, at the expensive of it taking longer for log lines to
	75	# be written to disk.
	76	capacity: 10
	77	flushLevel: 30 # Flush for WARNING logs as well
	78
	79	# A handler that writes logs to stderr. Unused by default, but can be used
	80	# instead of "buffer" and "file" in the logger handlers.
71	81	console:
72	82	class: logging.StreamHandler
73	83	formatter: precise
74		filters: [context]
75	84
76	85	loggers:
77	86	synapse.storage.SQL:

79	88	# information such as access tokens.
80	89	level: INFO
81	90
	91	twisted:
	92	# We send the twisted logging directly to the file handler,
	93	# to work around https://github.com/matrix-org/synapse/issues/3471
	94	# when using "buffer" logger. Use "console" to log to stderr instead.
	95	handlers: [file]
	96	propagate: false
	97
82	98	root:
83	99	level: INFO
84		handlers: [file, console]
	100
	101	# Write logs to the `buffer` handler, which will buffer them together in memory,
	102	# then write them to a file.
	103	#
	104	# Replace "buffer" with "console" to log to stderr instead. (Note that you'll
	105	# also need to update the configuation for the `twisted` logger above, in
	106	# this case.)
	107	#
	108	handlers: [buffer]
85	109
86	110	disable_existing_loggers: false
87	111	"""

167	191
168	192	handler = logging.StreamHandler()
169	193	handler.setFormatter(formatter)
170		handler.addFilter(LoggingContextFilter(request=""))
171	194	logger.addHandler(handler)
172	195	else:
173	196	logging.config.dictConfig(log_config)
	197
	198	# We add a log record factory that runs all messages through the
	199	# LoggingContextFilter so that we get the context at the time we log
	200	# rather than when we write to a handler. This can be done in config using
	201	# filter options, but care must when using e.g. MemoryHandler to buffer
	202	# writes.
	203
	204	log_filter = LoggingContextFilter(request="")
	205	old_factory = logging.getLogRecordFactory()
	206
	207	def factory(args, *kwargs):
	208	record = old_factory(args, *kwargs)
	209	log_filter.filter(record)
	210	return record
	211
	212	logging.setLogRecordFactory(factory)
174	213
175	214	# Route Twisted's native logging through to the standard library logging
176	215	# system.

+21

-0

synapse/config/ratelimiting.py less more

92	92	if rc_admin_redaction:
93	93	self.rc_admin_redaction = RateLimitConfig(rc_admin_redaction)
94	94
	95	self.rc_joins_local = RateLimitConfig(
	96	config.get("rc_joins", {}).get("local", {}),
	97	defaults={"per_second": 0.1, "burst_count": 3},
	98	)
	99	self.rc_joins_remote = RateLimitConfig(
	100	config.get("rc_joins", {}).get("remote", {}),
	101	defaults={"per_second": 0.01, "burst_count": 3},
	102	)
	103
95	104	def generate_config_section(self, **kwargs):
96	105	return """\
97	106	## Ratelimiting ##

117	126	# - one for ratelimiting redactions by room admins. If this is not explicitly
118	127	# set then it uses the same ratelimiting as per rc_message. This is useful
119	128	# to allow room admins to deal with abuse quickly.
	129	# - two for ratelimiting number of rooms a user can join, "local" for when
	130	# users are joining rooms the server is already in (this is cheap) vs
	131	# "remote" for when users are trying to join rooms not on the server (which
	132	# can be more expensive)
120	133	#
121	134	# The defaults are as shown below.
122	135	#

142	155	#rc_admin_redaction:
143	156	# per_second: 1
144	157	# burst_count: 50
	158	#
	159	#rc_joins:
	160	# local:
	161	# per_second: 0.1
	162	# burst_count: 3
	163	# remote:
	164	# per_second: 0.01
	165	# burst_count: 3
145	166
146	167
147	168	# Ratelimiting settings for incoming federation

+0

-18

synapse/config/registration.py less more

332	332	#
333	333	#default_identity_server: https://matrix.org
334	334
335		# The list of identity servers trusted to verify third party
336		# identifiers by this server.
337		#
338		# Also defines the ID server which will be called when an account is
339		# deactivated (one will be picked arbitrarily).
340		#
341		# Note: This option is deprecated. Since v0.99.4, Synapse has tracked which identity
342		# server a 3PID has been bound to. For 3PIDs bound before then, Synapse runs a
343		# background migration script, informing itself that the identity server all of its
344		# 3PIDs have been bound to is likely one of the below.
345		#
346		# As of Synapse v1.4.0, all other functionality of this option has been deprecated, and
347		# it is now solely used for the purposes of the background migration script, and can be
348		# removed once it has run.
349		#trusted_third_party_id_servers:
350		# - matrix.org
351		# - vector.im
352
353	335	# Handle threepid (email/phone etc) registration and password resets through a set of
354	336	# trusted identity servers. Note that this allows the configured identity server to
355	337	# reset passwords for accounts!

+51

-1

synapse/config/saml2_config.py less more

14	14	# limitations under the License.
15	15
16	16	import logging
17
	17	from typing import Any, List
	18
	19	import attr
18	20	import jinja2
19	21	import pkg_resources
20	22

22	24	from synapse.util.module_loader import load_module, load_python_module
23	25
24	26	from ._base import Config, ConfigError
	27	from ._util import validate_config
25	28
26	29	logger = logging.getLogger(__name__)
27	30

78	81	raise ConfigError(e.message)
79	82
80	83	self.saml2_enabled = True
	84
	85	attribute_requirements = saml2_config.get("attribute_requirements") or []
	86	self.attribute_requirements = _parse_attribute_requirements_def(
	87	attribute_requirements
	88	)
81	89
82	90	self.saml2_grandfathered_mxid_source_attribute = saml2_config.get(
83	91	"grandfathered_mxid_source_attribute", "uid"

340	348	#
341	349	#grandfathered_mxid_source_attribute: upn
342	350
	351	# It is possible to configure Synapse to only allow logins if SAML attributes
	352	# match particular values. The requirements can be listed under
	353	# `attribute_requirements` as shown below. All of the listed attributes must
	354	# match for the login to be permitted.
	355	#
	356	#attribute_requirements:
	357	# - attribute: userGroup
	358	# value: "staff"
	359	# - attribute: department
	360	# value: "sales"
	361
343	362	# Directory in which Synapse will try to find the template files below.
344	363	# If not set, default templates from within the Synapse package will be used.
345	364	#

367	386	""" % {
368	387	"config_dir_path": config_dir_path
369	388	}
	389
	390
	391	@attr.s(frozen=True)
	392	class SamlAttributeRequirement:
	393	"""Object describing a single requirement for SAML attributes."""
	394
	395	attribute = attr.ib(type=str)
	396	value = attr.ib(type=str)
	397
	398	JSON_SCHEMA = {
	399	"type": "object",
	400	"properties": {"attribute": {"type": "string"}, "value": {"type": "string"}},
	401	"required": ["attribute", "value"],
	402	}
	403
	404
	405	ATTRIBUTE_REQUIREMENTS_SCHEMA = {
	406	"type": "array",
	407	"items": SamlAttributeRequirement.JSON_SCHEMA,
	408	}
	409
	410
	411	def _parse_attribute_requirements_def(
	412	attribute_requirements: Any,
	413	) -> List[SamlAttributeRequirement]:
	414	validate_config(
	415	ATTRIBUTE_REQUIREMENTS_SCHEMA,
	416	attribute_requirements,
	417	config_path=["saml2_config", "attribute_requirements"],
	418	)
	419	return [SamlAttributeRequirement(**x) for x in attribute_requirements]

+22

-0

synapse/config/server.py less more

438	438	validator=attr.validators.instance_of(str),
439	439	default=ROOM_COMPLEXITY_TOO_GREAT,
440	440	)
	441	admins_can_join = attr.ib(
	442	validator=attr.validators.instance_of(bool), default=False
	443	)
441	444
442	445	self.limit_remote_rooms = LimitRemoteRoomsConfig(
443	446	**(config.get("limit_remote_rooms") or {})

525	528	self.request_token_inhibit_3pid_errors = config.get(
526	529	"request_token_inhibit_3pid_errors", False,
527	530	)
	531
	532	# List of users trialing the new experimental default push rules. This setting is
	533	# not included in the sample configuration file on purpose as it's a temporary
	534	# hack, so that some users can trial the new defaults without impacting every
	535	# user on the homeserver.
	536	users_new_default_push_rules = (
	537	config.get("users_new_default_push_rules") or []
	538	) # type: list
	539	if not isinstance(users_new_default_push_rules, list):
	540	raise ConfigError("'users_new_default_push_rules' must be a list")
	541
	542	# Turn the list into a set to improve lookup speed.
	543	self.users_new_default_push_rules = set(
	544	users_new_default_push_rules
	545	) # type: set
528	546
529	547	def has_tls_listener(self) -> bool:
530	548	return any(listener.tls for listener in self.listeners)

892	910	#
893	911	#complexity_error: "This room is too complex."
894	912
	913	# allow server admins to join complex rooms. Default is false.
	914	#
	915	#admins_can_join: true
	916
895	917	# Whether to require a user to be in the room to add an alias to it.
896	918	# Defaults to 'true'.
897	919	#

+8

-0

synapse/crypto/context_factory.py less more

47	47	connections."""
48	48
49	49	def __init__(self, config):
	50	# TODO: once pyOpenSSL exposes TLS_METHOD and SSL_CTX_set_min_proto_version,
	51	# switch to those (see https://github.com/pyca/cryptography/issues/5379).
	52	#
	53	# note that, despite the confusing name, SSLv23_METHOD does not enforce SSLv2
	54	# or v3, but is a synonym for TLS_METHOD, which allows the client and server
	55	# to negotiate an appropriate version of TLS constrained by the version options
	56	# set with context.set_options.
	57	#
50	58	self._context = SSL.Context(SSL.SSLv23_METHOD)
51	59	self.configure_context(self._context, config)
52	60

+70

-83

synapse/crypto/keyring.py less more

222	222
223	223	return results
224	224
225		@defer.inlineCallbacks
226		def _start_key_lookups(self, verify_requests):
	225	async def _start_key_lookups(self, verify_requests):
227	226	"""Sets off the key fetches for each verify request
228	227
229	228	Once each fetch completes, verify_request.key_ready will be resolved.

244	243	server_to_request_ids.setdefault(server_name, set()).add(request_id)
245	244
246	245	# Wait for any previous lookups to complete before proceeding.
247		yield self.wait_for_previous_lookups(server_to_request_ids.keys())
	246	await self.wait_for_previous_lookups(server_to_request_ids.keys())
248	247
249	248	# take out a lock on each of the servers by sticking a Deferred in
250	249	# key_downloads

282	281	except Exception:
283	282	logger.exception("Error starting key lookups")
284	283
285		@defer.inlineCallbacks
286		def wait_for_previous_lookups(self, server_names):
	284	async def wait_for_previous_lookups(self, server_names) -> None:
287	285	"""Waits for any previous key lookups for the given servers to finish.
288	286
289	287	Args:
290	288	server_names (Iterable[str]): list of servers which we want to look up
291	289
292	290	Returns:
293		Deferred[None]: resolves once all key lookups for the given servers have
	291	Resolves once all key lookups for the given servers have
294	292	completed. Follows the synapse rules of logcontext preservation.
295	293	"""
296	294	loop_count = 1

308	306	loop_count,
309	307	)
310	308	with PreserveLoggingContext():
311		yield defer.DeferredList((w[1] for w in wait_on))
	309	await defer.DeferredList((w[1] for w in wait_on))
312	310
313	311	loop_count += 1
314	312

325	323
326	324	remaining_requests = {rq for rq in verify_requests if not rq.key_ready.called}
327	325
328		@defer.inlineCallbacks
329		def do_iterations():
330		with Measure(self.clock, "get_server_verify_keys"):
331		for f in self._key_fetchers:
332		if not remaining_requests:
333		return
334		yield self._attempt_key_fetches_with_fetcher(f, remaining_requests)
335
336		# look for any requests which weren't satisfied
	326	async def do_iterations():
	327	try:
	328	with Measure(self.clock, "get_server_verify_keys"):
	329	for f in self._key_fetchers:
	330	if not remaining_requests:
	331	return
	332	await self._attempt_key_fetches_with_fetcher(
	333	f, remaining_requests
	334	)
	335
	336	# look for any requests which weren't satisfied
	337	with PreserveLoggingContext():
	338	for verify_request in remaining_requests:
	339	verify_request.key_ready.errback(
	340	SynapseError(
	341	401,
	342	"No key for %s with ids in %s (min_validity %i)"
	343	% (
	344	verify_request.server_name,
	345	verify_request.key_ids,
	346	verify_request.minimum_valid_until_ts,
	347	),
	348	Codes.UNAUTHORIZED,
	349	)
	350	)
	351	except Exception as err:
	352	# we don't really expect to get here, because any errors should already
	353	# have been caught and logged. But if we do, let's log the error and make
	354	# sure that all of the deferreds are resolved.
	355	logger.error("Unexpected error in _get_server_verify_keys: %s", err)
337	356	with PreserveLoggingContext():
338	357	for verify_request in remaining_requests:
339		verify_request.key_ready.errback(
340		SynapseError(
341		401,
342		"No key for %s with ids in %s (min_validity %i)"
343		% (
344		verify_request.server_name,
345		verify_request.key_ids,
346		verify_request.minimum_valid_until_ts,
347		),
348		Codes.UNAUTHORIZED,
349		)
350		)
351
352		def on_err(err):
353		# we don't really expect to get here, because any errors should already
354		# have been caught and logged. But if we do, let's log the error and make
355		# sure that all of the deferreds are resolved.
356		logger.error("Unexpected error in _get_server_verify_keys: %s", err)
357		with PreserveLoggingContext():
358		for verify_request in remaining_requests:
359		if not verify_request.key_ready.called:
360		verify_request.key_ready.errback(err)
361
362		run_in_background(do_iterations).addErrback(on_err)
363
364		@defer.inlineCallbacks
365		def _attempt_key_fetches_with_fetcher(self, fetcher, remaining_requests):
	358	if not verify_request.key_ready.called:
	359	verify_request.key_ready.errback(err)
	360
	361	run_in_background(do_iterations)
	362
	363	async def _attempt_key_fetches_with_fetcher(self, fetcher, remaining_requests):
366	364	"""Use a key fetcher to attempt to satisfy some key requests
367	365
368	366	Args:

389	387	verify_request.minimum_valid_until_ts,
390	388	)
391	389
392		results = yield fetcher.get_keys(missing_keys)
	390	results = await fetcher.get_keys(missing_keys)
393	391
394	392	completed = []
395	393	for verify_request in remaining_requests:

422	420
423	421
424	422	class KeyFetcher(object):
425		def get_keys(self, keys_to_fetch):
	423	async def get_keys(self, keys_to_fetch):
426	424	"""
427	425	Args:
428	426	keys_to_fetch (dict[str, dict[str, int]]):

441	439	def __init__(self, hs):
442	440	self.store = hs.get_datastore()
443	441
444		@defer.inlineCallbacks
445		def get_keys(self, keys_to_fetch):
	442	async def get_keys(self, keys_to_fetch):
446	443	"""see KeyFetcher.get_keys"""
447	444
448	445	keys_to_fetch = (

451	448	for key_id in keys_for_server.keys()
452	449	)
453	450
454		res = yield self.store.get_server_verify_keys(keys_to_fetch)
	451	res = await self.store.get_server_verify_keys(keys_to_fetch)
455	452	keys = {}
456	453	for (server_name, key_id), key in res.items():
457	454	keys.setdefault(server_name, {})[key_id] = key

463	460	self.store = hs.get_datastore()
464	461	self.config = hs.get_config()
465	462
466		@defer.inlineCallbacks
467		def process_v2_response(self, from_server, response_json, time_added_ms):
	463	async def process_v2_response(self, from_server, response_json, time_added_ms):
468	464	"""Parse a 'Server Keys' structure from the result of a /key request
469	465
470	466	This is used to parse either the entirety of the response from

536	532
537	533	key_json_bytes = encode_canonical_json(response_json)
538	534
539		yield make_deferred_yieldable(
	535	await make_deferred_yieldable(
540	536	defer.gatherResults(
541	537	[
542	538	run_in_background(

566	562	self.client = hs.get_http_client()
567	563	self.key_servers = self.config.key_servers
568	564
569		@defer.inlineCallbacks
570		def get_keys(self, keys_to_fetch):
	565	async def get_keys(self, keys_to_fetch):
571	566	"""see KeyFetcher.get_keys"""
572	567
573		@defer.inlineCallbacks
574		def get_key(key_server):
	568	async def get_key(key_server):
575	569	try:
576		result = yield self.get_server_verify_key_v2_indirect(
	570	result = await self.get_server_verify_key_v2_indirect(
577	571	keys_to_fetch, key_server
578	572	)
579	573	return result

591	585
592	586	return {}
593	587
594		results = yield make_deferred_yieldable(
	588	results = await make_deferred_yieldable(
595	589	defer.gatherResults(
596	590	[run_in_background(get_key, server) for server in self.key_servers],
597	591	consumeErrors=True,

605	599
606	600	return union_of_keys
607	601
608		@defer.inlineCallbacks
609		def get_server_verify_key_v2_indirect(self, keys_to_fetch, key_server):
	602	async def get_server_verify_key_v2_indirect(self, keys_to_fetch, key_server):
610	603	"""
611	604	Args:
612	605	keys_to_fetch (dict[str, dict[str, int]]):

616	609	the keys
617	610
618	611	Returns:
619		Deferred[dict[str, dict[str, synapse.storage.keys.FetchKeyResult]]]: map
	612	dict[str, dict[str, synapse.storage.keys.FetchKeyResult]]: map
620	613	from server_name -> key_id -> FetchKeyResult
621	614
622	615	Raises:

631	624	)
632	625
633	626	try:
634		query_response = yield self.client.post_json(
	627	query_response = await self.client.post_json(
635	628	destination=perspective_name,
636	629	path="/_matrix/key/v2/query",
637	630	data={

667	660	try:
668	661	self._validate_perspectives_response(key_server, response)
669	662
670		processed_response = yield self.process_v2_response(
	663	processed_response = await self.process_v2_response(
671	664	perspective_name, response, time_added_ms=time_now_ms
672	665	)
673	666	except KeyLookupError as e:

686	679	)
687	680	keys.setdefault(server_name, {}).update(processed_response)
688	681
689		yield self.store.store_server_verify_keys(
	682	await self.store.store_server_verify_keys(
690	683	perspective_name, time_now_ms, added_keys
691	684	)
692	685

738	731	self.clock = hs.get_clock()
739	732	self.client = hs.get_http_client()
740	733
741		def get_keys(self, keys_to_fetch):
	734	async def get_keys(self, keys_to_fetch):
742	735	"""
743	736	Args:
744	737	keys_to_fetch (dict[str, iterable[str]]):
745	738	the keys to be fetched. server_name -> key_ids
746	739
747	740	Returns:
748		Deferred[dict[str, dict[str, synapse.storage.keys.FetchKeyResult\|None]]]:
	741	dict[str, dict[str, synapse.storage.keys.FetchKeyResult\|None]]:
749	742	map from server_name -> key_id -> FetchKeyResult
750	743	"""
751	744
752	745	results = {}
753	746
754		@defer.inlineCallbacks
755		def get_key(key_to_fetch_item):
	747	async def get_key(key_to_fetch_item):
756	748	server_name, key_ids = key_to_fetch_item
757	749	try:
758		keys = yield self.get_server_verify_key_v2_direct(server_name, key_ids)
	750	keys = await self.get_server_verify_key_v2_direct(server_name, key_ids)
759	751	results[server_name] = keys
760	752	except KeyLookupError as e:
761	753	logger.warning(

764	756	except Exception:
765	757	logger.exception("Error getting keys %s from %s", key_ids, server_name)
766	758
767		return yieldable_gather_results(get_key, keys_to_fetch.items()).addCallback(
768		lambda _: results
769		)
770
771		@defer.inlineCallbacks
772		def get_server_verify_key_v2_direct(self, server_name, key_ids):
	759	return await yieldable_gather_results(
	760	get_key, keys_to_fetch.items()
	761	).addCallback(lambda _: results)
	762
	763	async def get_server_verify_key_v2_direct(self, server_name, key_ids):
773	764	"""
774	765
775	766	Args:

791	782
792	783	time_now_ms = self.clock.time_msec()
793	784	try:
794		response = yield self.client.get_json(
	785	response = await self.client.get_json(
795	786	destination=server_name,
796	787	path="/_matrix/key/v2/server/"
797	788	+ urllib.parse.quote(requested_key_id),

822	813	% (server_name, response["server_name"])
823	814	)
824	815
825		response_keys = yield self.process_v2_response(
	816	response_keys = await self.process_v2_response(
826	817	from_server=server_name,
827	818	response_json=response,
828	819	time_added_ms=time_now_ms,
829	820	)
830		yield self.store.store_server_verify_keys(
	821	await self.store.store_server_verify_keys(
831	822	server_name,
832	823	time_now_ms,
833	824	((server_name, key_id, key) for key_id, key in response_keys.items()),

837	828	return keys
838	829
839	830
840		@defer.inlineCallbacks
841		def _handle_key_deferred(verify_request):
	831	async def _handle_key_deferred(verify_request) -> None:
842	832	"""Waits for the key to become available, and then performs a verification
843	833
844	834	Args:
845	835	verify_request (VerifyJsonRequest):
846
847		Returns:
848		Deferred[None]
849	836
850	837	Raises:
851	838	SynapseError if there was a problem performing the verification
852	839	"""
853	840	server_name = verify_request.server_name
854	841	with PreserveLoggingContext():
855		_, key_id, verify_key = yield verify_request.key_ready
	842	_, key_id, verify_key = await verify_request.key_ready
856	843
857	844	json_object = verify_request.json_object
858	845

+40

-39

synapse/events/builder.py less more

16	16	import attr
17	17	from nacl.signing import SigningKey
18	18
19		from twisted.internet import defer
20
	19	from synapse.api.auth import Auth
21	20	from synapse.api.constants import MAX_DEPTH
22	21	from synapse.api.errors import UnsupportedRoomVersionError
23	22	from synapse.api.room_versions import (

28	27	)
29	28	from synapse.crypto.event_signing import add_hashes_and_signatures
30	29	from synapse.events import EventBase, _EventInternalMetadata, make_event_from_dict
	30	from synapse.state import StateHandler
	31	from synapse.storage.databases.main import DataStore
31	32	from synapse.types import EventID, JsonDict
32	33	from synapse.util import Clock
33	34	from synapse.util.stringutils import random_string

43	44
44	45	Attributes:
45	46	room_version: Version of the target room
46		room_id (str)
47		type (str)
48		sender (str)
49		content (dict)
50		unsigned (dict)
51		internal_metadata (_EventInternalMetadata)
52
53		_state (StateHandler)
54		_auth (synapse.api.Auth)
55		_store (DataStore)
56		_clock (Clock)
57		_hostname (str): The hostname of the server creating the event
	47	room_id
	48	type
	49	sender
	50	content
	51	unsigned
	52	internal_metadata
	53
	54	_state
	55	_auth
	56	_store
	57	_clock
	58	_hostname: The hostname of the server creating the event
58	59	_signing_key: The signing key to use to sign the event as the server
59	60	"""
60	61
61		_state = attr.ib()
62		_auth = attr.ib()
63		_store = attr.ib()
64		_clock = attr.ib()
65		_hostname = attr.ib()
66		_signing_key = attr.ib()
	62	_state = attr.ib(type=StateHandler)
	63	_auth = attr.ib(type=Auth)
	64	_store = attr.ib(type=DataStore)
	65	_clock = attr.ib(type=Clock)
	66	_hostname = attr.ib(type=str)
	67	_signing_key = attr.ib(type=SigningKey)
67	68
68	69	room_version = attr.ib(type=RoomVersion)
69	70
70		room_id = attr.ib()
71		type = attr.ib()
72		sender = attr.ib()
73
74		content = attr.ib(default=attr.Factory(dict))
75		unsigned = attr.ib(default=attr.Factory(dict))
	71	room_id = attr.ib(type=str)
	72	type = attr.ib(type=str)
	73	sender = attr.ib(type=str)
	74
	75	content = attr.ib(default=attr.Factory(dict), type=JsonDict)
	76	unsigned = attr.ib(default=attr.Factory(dict), type=JsonDict)
76	77
77	78	# These only exist on a subset of events, so they raise AttributeError if
78	79	# someone tries to get them when they don't exist.
79		_state_key = attr.ib(default=None)
80		_redacts = attr.ib(default=None)
81		_origin_server_ts = attr.ib(default=None)
	80	_state_key = attr.ib(default=None, type=Optional[str])
	81	_redacts = attr.ib(default=None, type=Optional[str])
	82	_origin_server_ts = attr.ib(default=None, type=Optional[int])
82	83
83	84	internal_metadata = attr.ib(
84		default=attr.Factory(lambda: _EventInternalMetadata({}))
	85	default=attr.Factory(lambda: _EventInternalMetadata({})),
	86	type=_EventInternalMetadata,
85	87	)
86	88
87	89	@property

94	96	def is_state(self):
95	97	return self._state_key is not None
96	98
97		@defer.inlineCallbacks
98		def build(self, prev_event_ids):
	99	async def build(self, prev_event_ids):
99	100	"""Transform into a fully signed and hashed event
100	101
101	102	Args:
102	103	prev_event_ids (list[str]): The event IDs to use as the prev events
103	104
104	105	Returns:
105		Deferred[FrozenEvent]
	106	FrozenEvent
106	107	"""
107	108
108		state_ids = yield defer.ensureDeferred(
109		self._state.get_current_state_ids(self.room_id, prev_event_ids)
	109	state_ids = await self._state.get_current_state_ids(
	110	self.room_id, prev_event_ids
110	111	)
111		auth_ids = yield self._auth.compute_auth_events(self, state_ids)
	112	auth_ids = self._auth.compute_auth_events(self, state_ids)
112	113
113	114	format_version = self.room_version.event_format
114	115	if format_version == EventFormatVersions.V1:
115		auth_events = yield self._store.add_event_hashes(auth_ids)
116		prev_events = yield self._store.add_event_hashes(prev_event_ids)
	116	auth_events = await self._store.add_event_hashes(auth_ids)
	117	prev_events = await self._store.add_event_hashes(prev_event_ids)
117	118	else:
118	119	auth_events = auth_ids
119	120	prev_events = prev_event_ids
120	121
121		old_depth = yield self._store.get_max_depth_of(prev_event_ids)
	122	old_depth = await self._store.get_max_depth_of(prev_event_ids)
122	123	depth = old_depth + 1
123	124
124	125	# we cap depth of generated events, to ensure that they are not

+24

-26

synapse/events/snapshot.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14		from typing import Optional, Union
	14	from typing import TYPE_CHECKING, Optional, Union
15	15
16	16	import attr
17	17	from frozendict import frozendict
18	18
19		from twisted.internet import defer
20
21	19	from synapse.appservice import ApplicationService
	20	from synapse.events import EventBase
22	21	from synapse.logging.context import make_deferred_yieldable, run_in_background
23	22	from synapse.types import StateMap
	23
	24	if TYPE_CHECKING:
	25	from synapse.storage.databases.main import DataStore
24	26
25	27
26	28	@attr.s(slots=True)

128	130	delta_ids=delta_ids,
129	131	)
130	132
131		@defer.inlineCallbacks
132		def serialize(self, event, store):
	133	async def serialize(self, event: EventBase, store: "DataStore") -> dict:
133	134	"""Converts self to a type that can be serialized as JSON, and then
134	135	deserialized by `deserialize`
135	136

145	146	# the prev_state_ids, so if we're a state event we include the event
146	147	# id that we replaced in the state.
147	148	if event.is_state():
148		prev_state_ids = yield self.get_prev_state_ids()
	149	prev_state_ids = await self.get_prev_state_ids()
149	150	prev_state_id = prev_state_ids.get((event.type, event.state_key))
150	151	else:
151	152	prev_state_id = None

213	214
214	215	return self._state_group
215	216
216		@defer.inlineCallbacks
217		def get_current_state_ids(self):
	217	async def get_current_state_ids(self) -> Optional[StateMap[str]]:
218	218	"""
219	219	Gets the room state map, including this event - ie, the state in ``state_group``
220	220

223	223	``rejected`` is set.
224	224
225	225	Returns:
226		Deferred[dict[(str, str), str]\|None]: Returns None if state_group
227		is None, which happens when the associated event is an outlier.
228
229		Maps a (type, state_key) to the event ID of the state event matching
230		this tuple.
	226	Returns None if state_group is None, which happens when the associated
	227	event is an outlier.
	228
	229	Maps a (type, state_key) to the event ID of the state event matching
	230	this tuple.
231	231	"""
232	232	if self.rejected:
233	233	raise RuntimeError("Attempt to access state_ids of rejected event")
234	234
235		yield self._ensure_fetched()
	235	await self._ensure_fetched()
236	236	return self._current_state_ids
237	237
238		@defer.inlineCallbacks
239		def get_prev_state_ids(self):
	238	async def get_prev_state_ids(self):
240	239	"""
241	240	Gets the room state map, excluding this event.
242	241
243	242	For a non-state event, this will be the same as get_current_state_ids().
244	243
245	244	Returns:
246		Deferred[dict[(str, str), str]\|None]: Returns None if state_group
	245	dict[(str, str), str]\|None: Returns None if state_group
247	246	is None, which happens when the associated event is an outlier.
248	247	Maps a (type, state_key) to the event ID of the state event matching
249	248	this tuple.
250	249	"""
251		yield self._ensure_fetched()
	250	await self._ensure_fetched()
252	251	return self._prev_state_ids
253	252
254	253	def get_cached_current_state_ids(self):

268	267
269	268	return self._current_state_ids
270	269
271		def _ensure_fetched(self):
272		return defer.succeed(None)
	270	async def _ensure_fetched(self):
	271	return None
273	272
274	273
275	274	@attr.s(slots=True)

302	301	_event_state_key = attr.ib(default=None)
303	302	_fetching_state_deferred = attr.ib(default=None)
304	303
305		def _ensure_fetched(self):
	304	async def _ensure_fetched(self):
306	305	if not self._fetching_state_deferred:
307	306	self._fetching_state_deferred = run_in_background(self._fill_out_state)
308	307
309		return make_deferred_yieldable(self._fetching_state_deferred)
310
311		@defer.inlineCallbacks
312		def _fill_out_state(self):
	308	return await make_deferred_yieldable(self._fetching_state_deferred)
	309
	310	async def _fill_out_state(self):
313	311	"""Called to populate the _current_state_ids and _prev_state_ids
314	312	attributes by loading from the database.
315	313	"""
316	314	if self.state_group is None:
317	315	return
318	316
319		self._current_state_ids = yield self._storage.state.get_state_ids_for_group(
	317	self._current_state_ids = await self._storage.state.get_state_ids_for_group(
320	318	self.state_group
321	319	)
322	320	if self._event_state_key is not None:

+30

-25

synapse/events/third_party_rules.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		from twisted.internet import defer
	15	from synapse.events import EventBase
	16	from synapse.events.snapshot import EventContext
	17	from synapse.types import Requester
16	18
17	19
18	20	class ThirdPartyEventRules(object):

38	40	config=config, http_client=hs.get_simple_http_client()
39	41	)
40	42
41		@defer.inlineCallbacks
42		def check_event_allowed(self, event, context):
	43	async def check_event_allowed(
	44	self, event: EventBase, context: EventContext
	45	) -> bool:
43	46	"""Check if a provided event should be allowed in the given context.
44	47
45	48	Args:
46		event (synapse.events.EventBase): The event to be checked.
47		context (synapse.events.snapshot.EventContext): The context of the event.
	49	event: The event to be checked.
	50	context: The context of the event.
48	51
49	52	Returns:
50		defer.Deferred[bool]: True if the event should be allowed, False if not.
	53	True if the event should be allowed, False if not.
51	54	"""
52	55	if self.third_party_rules is None:
53	56	return True
54	57
55		prev_state_ids = yield context.get_prev_state_ids()
	58	prev_state_ids = await context.get_prev_state_ids()
56	59
57	60	# Retrieve the state events from the database.
58	61	state_events = {}
59	62	for key, event_id in prev_state_ids.items():
60		state_events[key] = yield self.store.get_event(event_id, allow_none=True)
	63	state_events[key] = await self.store.get_event(event_id, allow_none=True)
61	64
62		ret = yield self.third_party_rules.check_event_allowed(event, state_events)
	65	ret = await self.third_party_rules.check_event_allowed(event, state_events)
63	66	return ret
64	67
65		@defer.inlineCallbacks
66		def on_create_room(self, requester, config, is_requester_admin):
	68	async def on_create_room(
	69	self, requester: Requester, config: dict, is_requester_admin: bool
	70	) -> bool:
67	71	"""Intercept requests to create room to allow, deny or update the
68	72	request config.
69	73
70	74	Args:
71		requester (Requester)
72		config (dict): The creation config from the client.
73		is_requester_admin (bool): If the requester is an admin
	75	requester
	76	config: The creation config from the client.
	77	is_requester_admin: If the requester is an admin
74	78
75	79	Returns:
76		defer.Deferred[bool]: Whether room creation is allowed or denied.
	80	Whether room creation is allowed or denied.
77	81	"""
78	82
79	83	if self.third_party_rules is None:
80	84	return True
81	85
82		ret = yield self.third_party_rules.on_create_room(
	86	ret = await self.third_party_rules.on_create_room(
83	87	requester, config, is_requester_admin
84	88	)
85	89	return ret
86	90
87		@defer.inlineCallbacks
88		def check_threepid_can_be_invited(self, medium, address, room_id):
	91	async def check_threepid_can_be_invited(
	92	self, medium: str, address: str, room_id: str
	93	) -> bool:
89	94	"""Check if a provided 3PID can be invited in the given room.
90	95
91	96	Args:
92		medium (str): The 3PID's medium.
93		address (str): The 3PID's address.
94		room_id (str): The room we want to invite the threepid to.
	97	medium: The 3PID's medium.
	98	address: The 3PID's address.
	99	room_id: The room we want to invite the threepid to.
95	100
96	101	Returns:
97		defer.Deferred[bool], True if the 3PID can be invited, False if not.
	102	True if the 3PID can be invited, False if not.
98	103	"""
99	104
100	105	if self.third_party_rules is None:
101	106	return True
102	107
103		state_ids = yield self.store.get_filtered_current_state_ids(room_id)
104		room_state_events = yield self.store.get_events(state_ids.values())
	108	state_ids = await self.store.get_filtered_current_state_ids(room_id)
	109	room_state_events = await self.store.get_events(state_ids.values())
105	110
106	111	state_events = {}
107	112	for key, event_id in state_ids.items():
108	113	state_events[key] = room_state_events[event_id]
109	114
110		ret = yield self.third_party_rules.check_threepid_can_be_invited(
	115	ret = await self.third_party_rules.check_threepid_can_be_invited(
111	116	medium, address, state_events
112	117	)
113	118	return ret

+7

-8

synapse/events/utils.py less more

16	16	from typing import Any, Mapping, Union
17	17
18	18	from frozendict import frozendict
19
20		from twisted.internet import defer
21	19
22	20	from synapse.api.constants import EventTypes, RelationTypes
23	21	from synapse.api.errors import Codes, SynapseError

336	334	hs.config.experimental_msc1849_support_enabled
337	335	)
338	336
339		@defer.inlineCallbacks
340		def serialize_event(self, event, time_now, bundle_aggregations=True, **kwargs):
	337	async def serialize_event(
	338	self, event, time_now, bundle_aggregations=True, **kwargs
	339	):
341	340	"""Serializes a single event.
342	341
343	342	Args:

347	346	**kwargs: Arguments to pass to `serialize_event`
348	347
349	348	Returns:
350		Deferred[dict]: The serialized event
	349	dict: The serialized event
351	350	"""
352	351	# To handle the case of presence events and the like
353	352	if not isinstance(event, EventBase):

362	361	if not event.internal_metadata.is_redacted() and (
363	362	self.experimental_msc1849_support_enabled and bundle_aggregations
364	363	):
365		annotations = yield self.store.get_aggregation_groups_for_event(event_id)
366		references = yield self.store.get_relations_for_event(
	364	annotations = await self.store.get_aggregation_groups_for_event(event_id)
	365	references = await self.store.get_relations_for_event(
367	366	event_id, RelationTypes.REFERENCE, direction="f"
368	367	)
369	368

377	376
378	377	edit = None
379	378	if event.type == EventTypes.Message:
380		edit = yield self.store.get_applicable_edit(event_id)
	379	edit = await self.store.get_applicable_edit(event_id)
381	380
382	381	if edit:
383	382	# If there is an edit replace the content, preserving existing

+4

-4

synapse/federation/federation_client.py less more

134	134	and try the request anyway.
135	135
136	136	Returns:
137		a Deferred which will eventually yield a JSON object from the
	137	a Awaitable which will eventually yield a JSON object from the
138	138	response
139	139	"""
140	140	sent_queries_counter.labels(query_type).inc()

156	156	content (dict): The query content.
157	157
158	158	Returns:
159		a Deferred which will eventually yield a JSON object from the
	159	an Awaitable which will eventually yield a JSON object from the
160	160	response
161	161	"""
162	162	sent_queries_counter.labels("client_device_keys").inc()

179	179	content (dict): The query content.
180	180
181	181	Returns:
182		a Deferred which will eventually yield a JSON object from the
	182	an Awaitable which will eventually yield a JSON object from the
183	183	response
184	184	"""
185	185	sent_queries_counter.labels("client_one_time_keys").inc()

899	899	party instance
900	900
901	901	Returns:
902		Deferred[Dict[str, Any]]: The response from the remote server, or None if
	902	Awaitable[Dict[str, Any]]: The response from the remote server, or None if
903	903	`remote_server` is the same as the local server_name
904	904
905	905	Raises:

+6

-13

synapse/federation/sender/__init__.py less more

287	287	for destination in destinations:
288	288	self._get_per_destination_queue(destination).send_pdu(pdu, order)
289	289
290		@defer.inlineCallbacks
291		def send_read_receipt(self, receipt: ReadReceipt):
	290	async def send_read_receipt(self, receipt: ReadReceipt) -> None:
292	291	"""Send a RR to any other servers in the room
293	292
294	293	Args:

329	328	room_id = receipt.room_id
330	329
331	330	# Work out which remote servers should be poked and poke them.
332		domains = yield defer.ensureDeferred(
333		self.state.get_current_hosts_in_room(room_id)
334		)
	331	domains = await self.state.get_current_hosts_in_room(room_id)
335	332	domains = [
336	333	d
337	334	for d in domains

386	383	queue.flush_read_receipts_for_room(room_id)
387	384
388	385	@preserve_fn # the caller should not yield on this
389		@defer.inlineCallbacks
390		def send_presence(self, states: List[UserPresenceState]):
	386	async def send_presence(self, states: List[UserPresenceState]):
391	387	"""Send the new presence states to the appropriate destinations.
392	388
393	389	This actually queues up the presence states ready for sending and

422	418	if not states_map:
423	419	break
424	420
425		yield self._process_presence_inner(list(states_map.values()))
	421	await self._process_presence_inner(list(states_map.values()))
426	422	except Exception:
427	423	logger.exception("Error sending presence states to servers")
428	424	finally:

449	445	self._get_per_destination_queue(destination).send_presence(states)
450	446
451	447	@measure_func("txnqueue._process_presence")
452		@defer.inlineCallbacks
453		def _process_presence_inner(self, states: List[UserPresenceState]):
	448	async def _process_presence_inner(self, states: List[UserPresenceState]):
454	449	"""Given a list of states populate self.pending_presence_by_dest and
455	450	poke to send a new transaction to each destination
456	451	"""
457		hosts_and_states = yield defer.ensureDeferred(
458		get_interested_remotes(self.store, states, self.state)
459		)
	452	hosts_and_states = await get_interested_remotes(self.store, states, self.state)
460	453
461	454	for destinations, states in hosts_and_states:
462	455	for destination in destinations:

+5

-2

synapse/federation/sender/transaction_manager.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
15		from typing import TYPE_CHECKING, List
	15	from typing import TYPE_CHECKING, List, Tuple
16	16
17	17	from canonicaljson import json
18	18

53	53
54	54	@measure_func("_send_new_transaction")
55	55	async def send_new_transaction(
56		self, destination: str, pending_pdus: List[EventBase], pending_edus: List[Edu]
	56	self,
	57	destination: str,
	58	pending_pdus: List[Tuple[EventBase, int]],
	59	pending_edus: List[Edu],
57	60	):
58	61
59	62	# Make a transaction-sending opentracing span. This span follows on from

+56

-72

synapse/federation/transport/client.py less more

17	17	import urllib
18	18	from typing import Any, Dict, Optional
19	19
20		from twisted.internet import defer
21
22	20	from synapse.api.constants import Membership
23	21	from synapse.api.errors import Codes, HttpResponseException, SynapseError
24	22	from synapse.api.urls import (

50	48	event_id (str): The event we want the context at.
51	49
52	50	Returns:
53		Deferred: Results in a dict received from the remote homeserver.
	51	Awaitable: Results in a dict received from the remote homeserver.
54	52	"""
55	53	logger.debug("get_room_state_ids dest=%s, room=%s", destination, room_id)
56	54

74	72	giving up. None indicates no timeout.
75	73
76	74	Returns:
77		Deferred: Results in a dict received from the remote homeserver.
	75	Awaitable: Results in a dict received from the remote homeserver.
78	76	"""
79	77	logger.debug("get_pdu dest=%s, event_id=%s", destination, event_id)
80	78

95	93	limit (int)
96	94
97	95	Returns:
98		Deferred: Results in a dict received from the remote homeserver.
	96	Awaitable: Results in a dict received from the remote homeserver.
99	97	"""
100	98	logger.debug(
101	99	"backfill dest=%s, room_id=%s, event_tuples=%r, limit=%s",

117	115	destination, path=path, args=args, try_trailing_slash_on_400=True
118	116	)
119	117
120		@defer.inlineCallbacks
121		@log_function
122		def send_transaction(self, transaction, json_data_callback=None):
	118	@log_function
	119	async def send_transaction(self, transaction, json_data_callback=None):
123	120	""" Sends the given Transaction to its destination
124	121
125	122	Args:
126	123	transaction (Transaction)
127	124
128	125	Returns:
129		Deferred: Succeeds when we get a 2xx HTTP response. The result
	126	Succeeds when we get a 2xx HTTP response. The result
130	127	will be the decoded JSON body.
131	128
132	129	Fails with ``HTTPRequestException`` if we get an HTTP response

153	150
154	151	path = _create_v1_path("/send/%s", transaction.transaction_id)
155	152
156		response = yield self.client.put_json(
	153	response = await self.client.put_json(
157	154	transaction.destination,
158	155	path=path,
159	156	data=json_data,

165	162
166	163	return response
167	164
168		@defer.inlineCallbacks
169		@log_function
170		def make_query(
	165	@log_function
	166	async def make_query(
171	167	self, destination, query_type, args, retry_on_dns_fail, ignore_backoff=False
172	168	):
173	169	path = _create_v1_path("/query/%s", query_type)
174	170
175		content = yield self.client.get_json(
	171	content = await self.client.get_json(
176	172	destination=destination,
177	173	path=path,
178	174	args=args,

183	179
184	180	return content
185	181
186		@defer.inlineCallbacks
187		@log_function
188		def make_membership_event(self, destination, room_id, user_id, membership, params):
	182	@log_function
	183	async def make_membership_event(
	184	self, destination, room_id, user_id, membership, params
	185	):
189	186	"""Asks a remote server to build and sign us a membership event
190	187
191	188	Note that this does not append any events to any graphs.

199	196	request.
200	197
201	198	Returns:
202		Deferred: Succeeds when we get a 2xx HTTP response. The result
	199	Succeeds when we get a 2xx HTTP response. The result
203	200	will be the decoded JSON body (ie, the new event).
204	201
205	202	Fails with ``HTTPRequestException`` if we get an HTTP response

230	227	ignore_backoff = True
231	228	retry_on_dns_fail = True
232	229
233		content = yield self.client.get_json(
	230	content = await self.client.get_json(
234	231	destination=destination,
235	232	path=path,
236	233	args=params,

241	238
242	239	return content
243	240
244		@defer.inlineCallbacks
245		@log_function
246		def send_join_v1(self, destination, room_id, event_id, content):
	241	@log_function
	242	async def send_join_v1(self, destination, room_id, event_id, content):
247	243	path = _create_v1_path("/send_join/%s/%s", room_id, event_id)
248	244
249		response = yield self.client.put_json(
	245	response = await self.client.put_json(
250	246	destination=destination, path=path, data=content
251	247	)
252	248
253	249	return response
254	250
255		@defer.inlineCallbacks
256		@log_function
257		def send_join_v2(self, destination, room_id, event_id, content):
	251	@log_function
	252	async def send_join_v2(self, destination, room_id, event_id, content):
258	253	path = _create_v2_path("/send_join/%s/%s", room_id, event_id)
259	254
260		response = yield self.client.put_json(
	255	response = await self.client.put_json(
261	256	destination=destination, path=path, data=content
262	257	)
263	258
264	259	return response
265	260
266		@defer.inlineCallbacks
267		@log_function
268		def send_leave_v1(self, destination, room_id, event_id, content):
	261	@log_function
	262	async def send_leave_v1(self, destination, room_id, event_id, content):
269	263	path = _create_v1_path("/send_leave/%s/%s", room_id, event_id)
270	264
271		response = yield self.client.put_json(
	265	response = await self.client.put_json(
272	266	destination=destination,
273	267	path=path,
274	268	data=content,

281	275
282	276	return response
283	277
284		@defer.inlineCallbacks
285		@log_function
286		def send_leave_v2(self, destination, room_id, event_id, content):
	278	@log_function
	279	async def send_leave_v2(self, destination, room_id, event_id, content):
287	280	path = _create_v2_path("/send_leave/%s/%s", room_id, event_id)
288	281
289		response = yield self.client.put_json(
	282	response = await self.client.put_json(
290	283	destination=destination,
291	284	path=path,
292	285	data=content,

299	292
300	293	return response
301	294
302		@defer.inlineCallbacks
303		@log_function
304		def send_invite_v1(self, destination, room_id, event_id, content):
	295	@log_function
	296	async def send_invite_v1(self, destination, room_id, event_id, content):
305	297	path = _create_v1_path("/invite/%s/%s", room_id, event_id)
306	298
307		response = yield self.client.put_json(
	299	response = await self.client.put_json(
308	300	destination=destination, path=path, data=content, ignore_backoff=True
309	301	)
310	302
311	303	return response
312	304
313		@defer.inlineCallbacks
314		@log_function
315		def send_invite_v2(self, destination, room_id, event_id, content):
	305	@log_function
	306	async def send_invite_v2(self, destination, room_id, event_id, content):
316	307	path = _create_v2_path("/invite/%s/%s", room_id, event_id)
317	308
318		response = yield self.client.put_json(
	309	response = await self.client.put_json(
319	310	destination=destination, path=path, data=content, ignore_backoff=True
320	311	)
321	312
322	313	return response
323	314
324		@defer.inlineCallbacks
325		@log_function
326		def get_public_rooms(
	315	@log_function
	316	async def get_public_rooms(
327	317	self,
328	318	remote_server: str,
329	319	limit: Optional[int] = None,

354	344	data["filter"] = search_filter
355	345
356	346	try:
357		response = yield self.client.post_json(
	347	response = await self.client.post_json(
358	348	destination=remote_server, path=path, data=data, ignore_backoff=True
359	349	)
360	350	except HttpResponseException as e:

380	370	args["since"] = [since_token]
381	371
382	372	try:
383		response = yield self.client.get_json(
	373	response = await self.client.get_json(
384	374	destination=remote_server, path=path, args=args, ignore_backoff=True
385	375	)
386	376	except HttpResponseException as e:

395	385
396	386	return response
397	387
398		@defer.inlineCallbacks
399		@log_function
400		def exchange_third_party_invite(self, destination, room_id, event_dict):
	388	@log_function
	389	async def exchange_third_party_invite(self, destination, room_id, event_dict):
401	390	path = _create_v1_path("/exchange_third_party_invite/%s", room_id)
402	391
403		response = yield self.client.put_json(
	392	response = await self.client.put_json(
404	393	destination=destination, path=path, data=event_dict
405	394	)
406	395
407	396	return response
408	397
409		@defer.inlineCallbacks
410		@log_function
411		def get_event_auth(self, destination, room_id, event_id):
	398	@log_function
	399	async def get_event_auth(self, destination, room_id, event_id):
412	400	path = _create_v1_path("/event_auth/%s/%s", room_id, event_id)
413	401
414		content = yield self.client.get_json(destination=destination, path=path)
	402	content = await self.client.get_json(destination=destination, path=path)
415	403
416	404	return content
417	405
418		@defer.inlineCallbacks
419		@log_function
420		def query_client_keys(self, destination, query_content, timeout):
	406	@log_function
	407	async def query_client_keys(self, destination, query_content, timeout):
421	408	"""Query the device keys for a list of user ids hosted on a remote
422	409	server.
423	410

452	439	"""
453	440	path = _create_v1_path("/user/keys/query")
454	441
455		content = yield self.client.post_json(
	442	content = await self.client.post_json(
456	443	destination=destination, path=path, data=query_content, timeout=timeout
457	444	)
458	445	return content
459	446
460		@defer.inlineCallbacks
461		@log_function
462		def query_user_devices(self, destination, user_id, timeout):
	447	@log_function
	448	async def query_user_devices(self, destination, user_id, timeout):
463	449	"""Query the devices for a user id hosted on a remote server.
464	450
465	451	Response:

492	478	"""
493	479	path = _create_v1_path("/user/devices/%s", user_id)
494	480
495		content = yield self.client.get_json(
	481	content = await self.client.get_json(
496	482	destination=destination, path=path, timeout=timeout
497	483	)
498	484	return content
499	485
500		@defer.inlineCallbacks
501		@log_function
502		def claim_client_keys(self, destination, query_content, timeout):
	486	@log_function
	487	async def claim_client_keys(self, destination, query_content, timeout):
503	488	"""Claim one-time keys for a list of devices hosted on a remote server.
504	489
505	490	Request:

531	516
532	517	path = _create_v1_path("/user/keys/claim")
533	518
534		content = yield self.client.post_json(
	519	content = await self.client.post_json(
535	520	destination=destination, path=path, data=query_content, timeout=timeout
536	521	)
537	522	return content
538	523
539		@defer.inlineCallbacks
540		@log_function
541		def get_missing_events(
	524	@log_function
	525	async def get_missing_events(
542	526	self,
543	527	destination,
544	528	room_id,

550	534	):
551	535	path = _create_v1_path("/get_missing_events/%s", room_id)
552	536
553		content = yield self.client.post_json(
	537	content = await self.client.post_json(
554	538	destination=destination,
555	539	path=path,
556	540	data={

+11

-14

synapse/groups/attestations.py less more

40	40
41	41	from signedjson.sign import sign_json
42	42
43		from twisted.internet import defer
44
45	43	from synapse.api.errors import HttpResponseException, RequestSendFailed, SynapseError
46	44	from synapse.metrics.background_process_metrics import run_as_background_process
47	45	from synapse.types import get_domain_from_id

71	69	self.server_name = hs.hostname
72	70	self.signing_key = hs.signing_key
73	71
74		@defer.inlineCallbacks
75		def verify_attestation(self, attestation, group_id, user_id, server_name=None):
	72	async def verify_attestation(
	73	self, attestation, group_id, user_id, server_name=None
	74	):
76	75	"""Verifies that the given attestation matches the given parameters.
77	76
78	77	An optional server_name can be supplied to explicitly set which server's

101	100	if valid_until_ms < now:
102	101	raise SynapseError(400, "Attestation expired")
103	102
104		yield self.keyring.verify_json_for_server(
	103	await self.keyring.verify_json_for_server(
105	104	server_name, attestation, now, "Group attestation"
106	105	)
107	106

141	140	self._start_renew_attestations, 30 * 60 * 1000
142	141	)
143	142
144		@defer.inlineCallbacks
145		def on_renew_attestation(self, group_id, user_id, content):
	143	async def on_renew_attestation(self, group_id, user_id, content):
146	144	"""When a remote updates an attestation
147	145	"""
148	146	attestation = content["attestation"]

150	148	if not self.is_mine_id(group_id) and not self.is_mine_id(user_id):
151	149	raise SynapseError(400, "Neither user not group are on this server")
152	150
153		yield self.attestations.verify_attestation(
	151	await self.attestations.verify_attestation(
154	152	attestation, user_id=user_id, group_id=group_id
155	153	)
156	154
157		yield self.store.update_remote_attestion(group_id, user_id, attestation)
	155	await self.store.update_remote_attestion(group_id, user_id, attestation)
158	156
159	157	return {}
160	158

171	169	now + UPDATE_ATTESTATION_TIME_MS
172	170	)
173	171
174		@defer.inlineCallbacks
175		def _renew_attestation(group_user: Tuple[str, str]):
	172	async def _renew_attestation(group_user: Tuple[str, str]):
176	173	group_id, user_id = group_user
177	174	try:
178	175	if not self.is_mine_id(group_id):

185	182	user_id,
186	183	group_id,
187	184	)
188		yield self.store.remove_attestation_renewal(group_id, user_id)
	185	await self.store.remove_attestation_renewal(group_id, user_id)
189	186	return
190	187
191	188	attestation = self.attestations.create_attestation(group_id, user_id)
192	189
193		yield self.transport_client.renew_group_attestation(
	190	await self.transport_client.renew_group_attestation(
194	191	destination, group_id, user_id, content={"attestation": attestation}
195	192	)
196	193
197		yield self.store.update_attestation_renewal(
	194	await self.store.update_attestation_renewal(
198	195	group_id, user_id, attestation
199	196	)
200	197	except (RequestSendFailed, HttpResponseException) as e:

+4

-7

synapse/handlers/acme.py less more

16	16
17	17	import twisted
18	18	import twisted.internet.error
19		from twisted.internet import defer
20	19	from twisted.web import server, static
21	20	from twisted.web.resource import Resource
22	21

40	39	self.reactor = hs.get_reactor()
41	40	self._acme_domain = hs.config.acme_domain
42	41
43		@defer.inlineCallbacks
44		def start_listening(self):
	42	async def start_listening(self):
45	43	from synapse.handlers import acme_issuing_service
46	44
47	45	# Configure logging for txacme, if you need to debug

81	79	self._issuer._registered = False
82	80
83	81	try:
84		yield self._issuer._ensure_registered()
	82	await self._issuer._ensure_registered()
85	83	except Exception:
86	84	logger.error(ACME_REGISTER_FAIL_ERROR)
87	85	raise
88	86
89		@defer.inlineCallbacks
90		def provision_certificate(self):
	87	async def provision_certificate(self):
91	88
92	89	logger.warning("Reprovisioning %s", self._acme_domain)
93	90
94	91	try:
95		yield self._issuer.issue_cert(self._acme_domain)
	92	await self._issuer.issue_cert(self._acme_domain)
96	93	except Exception:
97	94	logger.exception("Fail!")
98	95	raise

+5

-5

synapse/handlers/appservice.py less more

26	26	event_processing_loop_room_count,
27	27	)
28	28	from synapse.metrics.background_process_metrics import run_as_background_process
29		from synapse.util import log_failure
30	29	from synapse.util.metrics import Measure
31	30
32	31	logger = logging.getLogger(__name__)

99	98
100	99	if not self.started_scheduler:
101	100
102		def start_scheduler():
103		return self.scheduler.start().addErrback(
104		log_failure, "Application Services Failure"
105		)
	101	async def start_scheduler():
	102	try:
	103	return await self.scheduler.start()
	104	except Exception:
	105	logger.error("Application Services Failure")
106	106
107	107	run_as_background_process("as_scheduler", start_scheduler)
108	108	self.started_scheduler = True

+12

-7

synapse/handlers/auth.py less more

161	161	request_body: Dict[str, Any],
162	162	clientip: str,
163	163	description: str,
164		) -> dict:
	164	) -> Tuple[dict, str]:
165	165	"""
166	166	Checks that the user is who they claim to be, via a UI auth.
167	167

182	182	describes the operation happening on their account.
183	183
184	184	Returns:
185		The parameters for this request (which may
	185	A tuple of (params, session_id).
	186
	187	'params' contains the parameters for this request (which may
186	188	have been given only in a previous call).
	189
	190	'session_id' is the ID of this session, either passed in by the
	191	client or assigned by this call
187	192
188	193	Raises:
189	194	InteractiveAuthIncompleteError if the client has not yet completed

206	211	flows = [[login_type] for login_type in self._supported_ui_auth_types]
207	212
208	213	try:
209		result, params, _ = await self.check_auth(
	214	result, params, session_id = await self.check_ui_auth(
210	215	flows, request, request_body, clientip, description
211	216	)
212	217	except LoginError:

229	234	if user_id != requester.user.to_string():
230	235	raise AuthError(403, "Invalid auth")
231	236
232		return params
	237	return params, session_id
233	238
234	239	def get_enabled_auth_types(self):
235	240	"""Return the enabled user-interactive authentication types

239	244	"""
240	245	return self.checkers.keys()
241	246
242		async def check_auth(
	247	async def check_ui_auth(
243	248	self,
244	249	flows: List[List[str]],
245	250	request: SynapseRequest,

362	367
363	368	if not authdict:
364	369	raise InteractiveAuthIncompleteError(
365		self._auth_dict_for_flows(flows, session.session_id)
	370	session.session_id, self._auth_dict_for_flows(flows, session.session_id)
366	371	)
367	372
368	373	# check auth type currently being presented

409	414	ret = self._auth_dict_for_flows(flows, session.session_id)
410	415	ret["completed"] = list(creds)
411	416	ret.update(errordict)
412		raise InteractiveAuthIncompleteError(ret)
	417	raise InteractiveAuthIncompleteError(session.session_id, ret)
413	418
414	419	async def add_oob_auth(
415	420	self, stagetype: str, authdict: Dict[str, Any], clientip: str

+0

-4

synapse/handlers/events.py less more

56	56	timeout=0,
57	57	as_client_event=True,
58	58	affect_presence=True,
59		only_keys=None,
60	59	room_id=None,
61	60	is_guest=False,
62	61	):
63	62	"""Fetches the events stream for a given user.
64
65		If `only_keys` is not None, events from keys will be sent down.
66	63	"""
67	64
68	65	if room_id:

92	89	auth_user,
93	90	pagin_config,
94	91	timeout,
95		only_keys=only_keys,
96	92	is_guest=is_guest,
97	93	explicit_room_id=room_id,
98	94	)

+3

-3

synapse/handlers/federation.py less more

70	70	)
71	71	from synapse.replication.http.membership import ReplicationUserJoinedLeftRoomRestServlet
72	72	from synapse.state import StateResolutionStore, resolve_events_with_store
73		from synapse.storage.data_stores.main.events_worker import EventRedactBehaviour
	73	from synapse.storage.databases.main.events_worker import EventRedactBehaviour
74	74	from synapse.types import JsonDict, StateMap, UserID, get_domain_from_id
75	75	from synapse.util.async_helpers import Linearizer, concurrently_execute
76	76	from synapse.util.distributor import user_joined_room

2063	2063
2064	2064	if not auth_events:
2065	2065	prev_state_ids = await context.get_prev_state_ids()
2066		auth_events_ids = await self.auth.compute_auth_events(
	2066	auth_events_ids = self.auth.compute_auth_events(
2067	2067	event, prev_state_ids, for_verification=True
2068	2068	)
2069	2069	auth_events_x = await self.store.get_events(auth_events_ids)

2469	2469	}
2470	2470
2471	2471	current_state_ids = await context.get_current_state_ids()
2472		current_state_ids = dict(current_state_ids)
	2472	current_state_ids = dict(current_state_ids) # type: ignore
2473	2473
2474	2474	current_state_ids.update(state_updates)
2475	2475

+18

-25

synapse/handlers/groups_local.py less more

22	22
23	23
24	24	def _create_rerouter(func_name):
25		"""Returns a function that looks at the group id and calls the function
	25	"""Returns an async function that looks at the group id and calls the function
26	26	on federation or the local group server if the group is local
27	27	"""
28	28
29		def f(self, group_id, args, *kwargs):
30		if self.is_mine_id(group_id):
31		return getattr(self.groups_server_handler, func_name)(
	29	async def f(self, group_id, args, *kwargs):
	30	if self.is_mine_id(group_id):
	31	return await getattr(self.groups_server_handler, func_name)(
32	32	group_id, args, *kwargs
33	33	)
34	34	else:
35	35	destination = get_domain_from_id(group_id)
36		d = getattr(self.transport_client, func_name)(
37		destination, group_id, args, *kwargs
38		)
39
40		# Capture errors returned by the remote homeserver and
41		# re-throw specific errors as SynapseErrors. This is so
42		# when the remote end responds with things like 403 Not
43		# In Group, we can communicate that to the client instead
44		# of a 500.
45		def http_response_errback(failure):
46		failure.trap(HttpResponseException)
47		e = failure.value
48		raise e.to_synapse_error()
49
50		def request_failed_errback(failure):
51		failure.trap(RequestSendFailed)
52		raise SynapseError(502, "Failed to contact group server")
53
54		d.addErrback(http_response_errback)
55		d.addErrback(request_failed_errback)
56		return d
	36
	37	try:
	38	return await getattr(self.transport_client, func_name)(
	39	destination, group_id, args, *kwargs
	40	)
	41	except HttpResponseException as e:
	42	# Capture errors returned by the remote homeserver and
	43	# re-throw specific errors as SynapseErrors. This is so
	44	# when the remote end responds with things like 403 Not
	45	# In Group, we can communicate that to the client instead
	46	# of a 500.
	47	raise e.to_synapse_error()
	48	except RequestSendFailed:
	49	raise SynapseError(502, "Failed to contact group server")
57	50
58	51	return f
59	52

+3

-31

synapse/handlers/identity.py less more

21	21	from typing import Awaitable, Callable, Dict, List, Optional, Tuple
22	22
23	23	from canonicaljson import json
24		from signedjson.key import decode_verify_key_bytes
25		from signedjson.sign import verify_signed_json
26		from unpaddedbase64 import decode_base64
27	24
28	25	from twisted.internet.error import TimeoutError
29	26
30	27	from synapse.api.errors import (
31		AuthError,
32	28	CodeMessageException,
33	29	Codes,
34	30	HttpResponseException,

627	623	)
628	624
629	625	if "mxid" in data:
630		if "signatures" not in data:
631		raise AuthError(401, "No signatures on 3pid binding")
632		await self._verify_any_signature(data, id_server)
	626	# note: we used to verify the identity server's signature here, but no longer
	627	# require or validate it. See the following for context:
	628	# https://github.com/matrix-org/synapse/issues/5253#issuecomment-666246950
633	629	return data["mxid"]
634	630	except TimeoutError:
635	631	raise SynapseError(500, "Timed out contacting identity server")

749	745	# Return the MXID if it's available, or None otherwise
750	746	mxid = lookup_results["mappings"].get(lookup_value)
751	747	return mxid
752
753		async def _verify_any_signature(self, data, server_hostname):
754		if server_hostname not in data["signatures"]:
755		raise AuthError(401, "No signature from server %s" % (server_hostname,))
756		for key_name, signature in data["signatures"][server_hostname].items():
757		try:
758		key_data = await self.blacklisting_http_client.get_json(
759		"%s%s/_matrix/identity/api/v1/pubkey/%s"
760		% (id_server_scheme, server_hostname, key_name)
761		)
762		except TimeoutError:
763		raise SynapseError(500, "Timed out contacting identity server")
764		if "public_key" not in key_data:
765		raise AuthError(
766		401, "No public key named %s from %s" % (key_name, server_hostname)
767		)
768		verify_signed_json(
769		data,
770		server_hostname,
771		decode_verify_key_bytes(
772		key_name, decode_base64(key_data["public_key"])
773		),
774		)
775		return
776	748
777	749	async def ask_id_server_for_third_party_invite(
778	750	self,

+2

-2

synapse/handlers/initial_sync.py less more

108	108
109	109	rooms_ret = []
110	110
111		now_token = await self.hs.get_event_sources().get_current_token()
	111	now_token = self.hs.get_event_sources().get_current_token()
112	112
113	113	presence_stream = self.hs.get_event_sources().sources["presence"]
114	114	pagination_config = PaginationConfig(from_token=now_token)

359	359	current_state.values(), time_now
360	360	)
361	361
362		now_token = await self.hs.get_event_sources().get_current_token()
	362	now_token = self.hs.get_event_sources().get_current_token()
363	363
364	364	limit = pagin_config.limit if pagin_config else None
365	365	if limit is None:

+24

-11

synapse/handlers/message.py less more

14	14	# See the License for the specific language governing permissions and
15	15	# limitations under the License.
16	16	import logging
17		from typing import TYPE_CHECKING, List, Optional, Tuple
	17	from typing import TYPE_CHECKING, Dict, List, Optional, Tuple
18	18
19	19	from canonicaljson import encode_canonical_json, json
20	20

44	44	from synapse.logging.context import run_in_background
45	45	from synapse.metrics.background_process_metrics import run_as_background_process
46	46	from synapse.replication.http.send_event import ReplicationSendEventRestServlet
47		from synapse.storage.data_stores.main.events_worker import EventRedactBehaviour
	47	from synapse.storage.databases.main.events_worker import EventRedactBehaviour
48	48	from synapse.storage.state import StateFilter
49	49	from synapse.types import (
50	50	Collection,

92	92
93	93	async def get_room_data(
94	94	self,
95		user_id: str = None,
96		room_id: str = None,
97		event_type: Optional[str] = None,
98		state_key: str = "",
99		is_guest: bool = False,
	95	user_id: str,
	96	room_id: str,
	97	event_type: str,
	98	state_key: str,
	99	is_guest: bool,
100	100	) -> dict:
101	101	""" Get data from a room.
102	102

406	406	#
407	407	# map from room id to time-of-last-attempt.
408	408	#
409		self._rooms_to_exclude_from_dummy_event_insertion = {} # type: dict[str, int]
	409	self._rooms_to_exclude_from_dummy_event_insertion = {} # type: Dict[str, int]
410	410
411	411	# we need to construct a ConsentURIBuilder here, as it checks that the necessary
412	412	# config options, but only if we have a configuration for which we are

706	706	async def create_and_send_nonmember_event(
707	707	self,
708	708	requester: Requester,
709		event_dict: EventBase,
	709	event_dict: dict,
710	710	ratelimit: bool = True,
711	711	txn_id: Optional[str] = None,
712	712	) -> Tuple[EventBase, int]:

766	766	)
767	767	else:
768	768	prev_event_ids = await self.store.get_prev_events_for_room(builder.room_id)
	769
	770	# we now ought to have some prev_events (unless it's a create event).
	771	#
	772	# do a quick sanity check here, rather than waiting until we've created the
	773	# event and then try to auth it (which fails with a somewhat confusing "No
	774	# create event in auth events")
	775	assert (
	776	builder.type == EventTypes.Create or len(prev_event_ids) > 0
	777	), "Attempting to create an event with no prev_events"
769	778
770	779	event = await builder.build(prev_event_ids=prev_event_ids)
771	780	context = await self.state.compute_event_context(event)

961	970	# Validate a newly added alias or newly added alt_aliases.
962	971
963	972	original_alias = None
964		original_alt_aliases = set()
	973	original_alt_aliases = [] # type: List[str]
965	974
966	975	original_event_id = event.unsigned.get("replaces_state")
967	976	if original_event_id:

1008	1017	return e.type == EventTypes.Member and e.sender == event.sender
1009	1018
1010	1019	current_state_ids = await context.get_current_state_ids()
	1020
	1021	# We know this event is not an outlier, so this must be
	1022	# non-None.
	1023	assert current_state_ids is not None
1011	1024
1012	1025	state_to_include_ids = [
1013	1026	e_id

1060	1073	raise SynapseError(400, "Cannot redact event from a different room")
1061	1074
1062	1075	prev_state_ids = await context.get_prev_state_ids()
1063		auth_events_ids = await self.auth.compute_auth_events(
	1076	auth_events_ids = self.auth.compute_auth_events(
1064	1077	event, prev_state_ids, for_verification=True
1065	1078	)
1066	1079	auth_events = await self.store.get_events(auth_events_ids)

+5

-3

synapse/handlers/oidc_handler.py less more

13	13	# limitations under the License.
14	14	import json
15	15	import logging
16		from typing import Dict, Generic, List, Optional, Tuple, TypeVar
	16	from typing import TYPE_CHECKING, Dict, Generic, List, Optional, Tuple, TypeVar
17	17	from urllib.parse import urlencode
18	18
19	19	import attr

38	38	from synapse.http.site import SynapseRequest
39	39	from synapse.logging.context import make_deferred_yieldable
40	40	from synapse.push.mailer import load_jinja2_templates
41		from synapse.server import HomeServer
42	41	from synapse.types import UserID, map_username_to_mxid_localpart
	42
	43	if TYPE_CHECKING:
	44	from synapse.server import HomeServer
43	45
44	46	logger = logging.getLogger(__name__)
45	47

90	92	"""Handles requests related to the OpenID Connect login flow.
91	93	"""
92	94
93		def __init__(self, hs: HomeServer):
	95	def __init__(self, hs: "HomeServer"):
94	96	self._callback_url = hs.config.oidc_callback_url # type: str
95	97	self._scopes = hs.config.oidc_scopes # type: List[str]
96	98	self._client_auth = ClientAuth(

+1

-1

synapse/handlers/pagination.py less more

308	308	room_token = pagin_config.from_token.room_key
309	309	else:
310	310	pagin_config.from_token = (
311		await self.hs.get_event_sources().get_current_token_for_pagination()
	311	self.hs.get_event_sources().get_current_token_for_pagination()
312	312	)
313	313	room_token = pagin_config.from_token.room_key
314	314

+2

-2

synapse/handlers/presence.py less more

37	37	from synapse.metrics import LaterGauge
38	38	from synapse.metrics.background_process_metrics import run_as_background_process
39	39	from synapse.state import StateHandler
40		from synapse.storage.data_stores.main import DataStore
	40	from synapse.storage.databases.main import DataStore
41	41	from synapse.storage.presence import UserPresenceState
42	42	from synapse.types import JsonDict, UserID, get_domain_from_id
43	43	from synapse.util.async_helpers import Linearizer

318	318	is some spurious presence changes that will self-correct.
319	319	"""
320	320	# If the DB pool has already terminated, don't try updating
321		if not self.store.db.is_running():
	321	if not self.store.db_pool.is_running():
322	322	return
323	323
324	324	logger.info(

+1

-1

synapse/handlers/register.py less more

547	547	address (str\|None): the IP address used to perform the registration.
548	548
549	549	Returns:
550		Deferred
	550	Awaitable
551	551	"""
552	552	if self.hs.config.worker_app:
553	553	return self._register_client(

+6

-6

synapse/handlers/room.py less more

21	21	import math
22	22	import string
23	23	from collections import OrderedDict
24		from typing import Optional, Tuple
	24	from typing import Awaitable, Optional, Tuple
25	25
26	26	from synapse.api.constants import (
27	27	EventTypes,

1040	1040	):
1041	1041	# We just ignore the key for now.
1042	1042
1043		to_key = await self.get_current_key()
	1043	to_key = self.get_current_key()
1044	1044
1045	1045	from_token = RoomStreamToken.parse(from_key)
1046	1046	if from_token.topological:

1080	1080
1081	1081	return (events, end_key)
1082	1082
1083		def get_current_key(self):
1084		return self.store.get_room_events_max_id()
1085
1086		def get_current_key_for_room(self, room_id):
	1083	def get_current_key(self) -> str:
	1084	return "s%d" % (self.store.get_room_max_stream_ordering(),)
	1085
	1086	def get_current_key_for_room(self, room_id: str) -> Awaitable[str]:
1087	1087	return self.store.get_room_events_max_id(room_id)
1088	1088
1089	1089

+69

-15

synapse/handlers/room_member.py less more

15	15	import abc
16	16	import logging
17	17	from http import HTTPStatus
18		from typing import Dict, Iterable, List, Optional, Tuple, Union
	18	from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Tuple, Union
19	19
20	20	from unpaddedbase64 import encode_base64
21	21
22	22	from synapse import types
23	23	from synapse.api.constants import MAX_DEPTH, EventTypes, Membership
24		from synapse.api.errors import AuthError, Codes, SynapseError
	24	from synapse.api.errors import AuthError, Codes, LimitExceededError, SynapseError
	25	from synapse.api.ratelimiting import Ratelimiter
25	26	from synapse.api.room_versions import EventFormatVersions
26	27	from synapse.crypto.event_signing import compute_event_reference_hash
27	28	from synapse.events import EventBase

35	36
36	37	from ._base import BaseHandler
37	38
	39	if TYPE_CHECKING:
	40	from synapse.server import HomeServer
	41
	42
38	43	logger = logging.getLogger(__name__)
39	44
40	45

46	51
47	52	__metaclass__ = abc.ABCMeta
48	53
49		def __init__(self, hs):
	54	def __init__(self, hs: "HomeServer"):
50	55	self.hs = hs
51	56	self.store = hs.get_datastore()
52	57	self.auth = hs.get_auth()

75	80	)
76	81	if self._is_on_event_persistence_instance:
77	82	self.persist_event_storage = hs.get_storage().persistence
	83
	84	self._join_rate_limiter_local = Ratelimiter(
	85	clock=self.clock,
	86	rate_hz=hs.config.ratelimiting.rc_joins_local.per_second,
	87	burst_count=hs.config.ratelimiting.rc_joins_local.burst_count,
	88	)
	89	self._join_rate_limiter_remote = Ratelimiter(
	90	clock=self.clock,
	91	rate_hz=hs.config.ratelimiting.rc_joins_remote.per_second,
	92	burst_count=hs.config.ratelimiting.rc_joins_remote.burst_count,
	93	)
78	94
79	95	# This is only used to get at ratelimit function, and
80	96	# maybe_kick_guest_users. It's fine there are multiple of these as

194	210	return duplicate.event_id, stream_id
195	211
196	212	stream_id = await self.event_creation_handler.handle_new_client_event(
197		requester, event, context, extra_users=[target], ratelimit=ratelimit
	213	requester, event, context, extra_users=[target], ratelimit=ratelimit,
198	214	)
199	215
200	216	prev_state_ids = await context.get_prev_state_ids()

440	456	# so don't really fit into the general auth process.
441	457	raise AuthError(403, "Guest access not allowed")
442	458
443		if not is_host_in_room:
	459	if is_host_in_room:
	460	time_now_s = self.clock.time()
	461	allowed, time_allowed = self._join_rate_limiter_local.can_do_action(
	462	requester.user.to_string(),
	463	)
	464
	465	if not allowed:
	466	raise LimitExceededError(
	467	retry_after_ms=int(1000 * (time_allowed - time_now_s))
	468	)
	469
	470	else:
	471	time_now_s = self.clock.time()
	472	allowed, time_allowed = self._join_rate_limiter_remote.can_do_action(
	473	requester.user.to_string(),
	474	)
	475
	476	if not allowed:
	477	raise LimitExceededError(
	478	retry_after_ms=int(1000 * (time_allowed - time_now_s))
	479	)
	480
444	481	inviter = await self._get_inviter(target.to_string(), room_id)
445	482	if inviter and not self.hs.is_mine(inviter):
446	483	remote_room_hosts.append(inviter.domain)

468	505	user_id=target.to_string(), room_id=room_id
469	506	) # type: Optional[RoomsForUser]
470	507	if not invite:
	508	logger.info(
	509	"%s sent a leave request to %s, but that is not an active room "
	510	"on this server, and there is no pending invite",
	511	target,
	512	room_id,
	513	)
	514
471	515	raise SynapseError(404, "Not a known room")
472	516
473	517	logger.info(
474	518	"%s rejects invite to %s from %s", target, room_id, invite.sender
475	519	)
476	520
477		if self.hs.is_mine_id(invite.sender):
478		# the inviter was on our server, but has now left. Carry on
479		# with the normal rejection codepath.
480		#
481		# This is a bit of a hack, because the room might still be
482		# active on other servers.
483		pass
484		else:
	521	if not self.hs.is_mine_id(invite.sender):
485	522	# send the rejection to the inviter's HS (with fallback to
486	523	# local event)
487	524	return await self.remote_reject_invite(
488	525	invite.event_id, txn_id, requester, content,
489	526	)
	527
	528	# the inviter was on our server, but has now left. Carry on
	529	# with the normal rejection codepath, which will also send the
	530	# rejection out to any other servers we believe are still in the room.
	531
	532	# thanks to overzealous cleaning up of event_forward_extremities in
	533	# `delete_old_current_state_events`, it's possible to end up with no
	534	# forward extremities here. If that happens, let's just hang the
	535	# rejection off the invite event.
	536	#
	537	# see: https://github.com/matrix-org/synapse/issues/7139
	538	if len(latest_event_ids) == 0:
	539	latest_event_ids = [invite.event_id]
490	540
491	541	return await self._local_membership_update(
492	542	requester=requester,

951	1001	if len(remote_room_hosts) == 0:
952	1002	raise SynapseError(404, "No known servers")
953	1003
954		if self.hs.config.limit_remote_rooms.enabled:
	1004	check_complexity = self.hs.config.limit_remote_rooms.enabled
	1005	if check_complexity and self.hs.config.limit_remote_rooms.admins_can_join:
	1006	check_complexity = not await self.auth.is_server_admin(user)
	1007
	1008	if check_complexity:
955	1009	# Fetch the room complexity
956	1010	too_complex = await self._is_remote_room_too_complex(
957	1011	room_id, remote_room_hosts

974	1028
975	1029	# Check the room we just joined wasn't too large, if we didn't fetch the
976	1030	# complexity of it before.
977		if self.hs.config.limit_remote_rooms.enabled:
	1031	if check_complexity:
978	1032	if too_complex is False:
979	1033	# We checked, and we're under the limit.
980	1034	return event_id, stream_id

+39

-6

synapse/handlers/saml_handler.py less more

13	13	# limitations under the License.
14	14	import logging
15	15	import re
16		from typing import Callable, Dict, Optional, Set, Tuple
	16	from typing import TYPE_CHECKING, Callable, Dict, Optional, Set, Tuple
17	17
18	18	import attr
19	19	import saml2
20	20	import saml2.response
21	21	from saml2.client import Saml2Client
22	22
23		from synapse.api.errors import SynapseError
	23	from synapse.api.errors import AuthError, SynapseError
24	24	from synapse.config import ConfigError
	25	from synapse.config.saml2_config import SamlAttributeRequirement
25	26	from synapse.http.servlet import parse_string
26	27	from synapse.http.site import SynapseRequest
27	28	from synapse.module_api import ModuleApi

33	34	from synapse.util.async_helpers import Linearizer
34	35	from synapse.util.iterutils import chunk_seq
35	36
	37	if TYPE_CHECKING:
	38	import synapse.server
	39
36	40	logger = logging.getLogger(__name__)
37	41
38	42

48	52
49	53
50	54	class SamlHandler:
51		def __init__(self, hs):
	55	def __init__(self, hs: "synapse.server.HomeServer"):
52	56	self._saml_client = Saml2Client(hs.config.saml2_sp_config)
53	57	self._auth = hs.get_auth()
54	58	self._auth_handler = hs.get_auth_handler()

61	65	self._grandfathered_mxid_source_attribute = (
62	66	hs.config.saml2_grandfathered_mxid_source_attribute
63	67	)
	68	self._saml2_attribute_requirements = hs.config.saml2.attribute_requirements
64	69
65	70	# plugin to do custom mapping from saml response to mxid
66	71	self._user_mapping_provider = hs.config.saml2_user_mapping_provider_class(

72	77	self._auth_provider_id = "saml"
73	78
74	79	# a map from saml session id to Saml2SessionData object
75		self._outstanding_requests_dict = {}
	80	self._outstanding_requests_dict = {} # type: Dict[str, Saml2SessionData]
76	81
77	82	# a lock on the mappings
78	83	self._mapping_lock = Linearizer(name="saml_mapping", clock=self._clock)

94	99	reqid, info = self._saml_client.prepare_for_authenticate(
95	100	relay_state=client_redirect_url
96	101	)
	102
	103	# Since SAML sessions timeout it is useful to log when they were created.
	104	logger.info("Initiating a new SAML session: %s" % (reqid,))
97	105
98	106	now = self._clock.time_msec()
99	107	self._outstanding_requests_dict[reqid] = Saml2SessionData(

161	169	saml2.BINDING_HTTP_POST,
162	170	outstanding=self._outstanding_requests_dict,
163	171	)
	172	except saml2.response.UnsolicitedResponse as e:
	173	# the pysaml2 library helpfully logs an ERROR here, but neglects to log
	174	# the session ID. I don't really want to put the full text of the exception
	175	# in the (user-visible) exception message, so let's log the exception here
	176	# so we can track down the session IDs later.
	177	logger.warning(str(e))
	178	raise SynapseError(400, "Unexpected SAML2 login.")
164	179	except Exception as e:
165		raise SynapseError(400, "Unable to parse SAML2 response: %s" % (e,))
	180	raise SynapseError(400, "Unable to parse SAML2 response: %s." % (e,))
166	181
167	182	if saml2_auth.not_signed:
168		raise SynapseError(400, "SAML2 response was not signed")
	183	raise SynapseError(400, "SAML2 response was not signed.")
169	184
170	185	logger.debug("SAML2 response: %s", saml2_auth.origxml)
171	186	for assertion in saml2_auth.assertions:

183	198	current_session = self._outstanding_requests_dict.pop(
184	199	saml2_auth.in_response_to, None
185	200	)
	201
	202	for requirement in self._saml2_attribute_requirements:
	203	_check_attribute_requirement(saml2_auth.ava, requirement)
186	204
187	205	remote_user_id = self._user_mapping_provider.get_remote_user_id(
188	206	saml2_auth, client_redirect_url

290	308	del self._outstanding_requests_dict[reqid]
291	309
292	310
	311	def _check_attribute_requirement(ava: dict, req: SamlAttributeRequirement):
	312	values = ava.get(req.attribute, [])
	313	for v in values:
	314	if v == req.value:
	315	return
	316
	317	logger.info(
	318	"SAML2 attribute %s did not match required value '%s' (was '%s')",
	319	req.attribute,
	320	req.value,
	321	values,
	322	)
	323	raise AuthError(403, "You are not authorized to log in here.")
	324
	325
293	326	DOT_REPLACE_PATTERN = re.compile(
294	327	("[^%s]" % (re.escape("".join(mxid_localpart_allowed_characters)),))
295	328	)

+1

-1

synapse/handlers/search.py less more

339	339	# If client has asked for "context" for each event (i.e. some surrounding
340	340	# events and state), fetch that
341	341	if event_context is not None:
342		now_token = await self.hs.get_event_sources().get_current_token()
	342	now_token = self.hs.get_event_sources().get_current_token()
343	343
344	344	contexts = {}
345	345	for event in allowed_events:

+1

-1

synapse/handlers/stats.py less more

231	231
232	232	if membership == prev_membership:
233	233	pass # noop
234		if membership == Membership.JOIN:
	234	elif membership == Membership.JOIN:
235	235	room_stats_delta["joined_members"] += 1
236	236	elif membership == Membership.INVITE:
237	237	room_stats_delta["invited_members"] += 1

+1

-1

synapse/handlers/sync.py less more

959	959	# this is due to some of the underlying streams not supporting the ability
960	960	# to query up to a given point.
961	961	# Always use the `now_token` in `SyncResultBuilder`
962		now_token = await self.event_sources.get_current_token()
	962	now_token = self.event_sources.get_current_token()
963	963
964	964	logger.debug(
965	965	"Calculating sync response for %r between %s and %s",

+41

-40

synapse/http/client.py less more

283	283	ip_blacklist=self._ip_blacklist,
284	284	)
285	285
286		@defer.inlineCallbacks
287		def request(self, method, uri, data=None, headers=None):
	286	async def request(self, method, uri, data=None, headers=None):
288	287	"""
289	288	Args:
290	289	method (str): HTTP method to use.

297	296	outgoing_requests_counter.labels(method).inc()
298	297
299	298	# log request but strip `access_token` (AS requests for example include this)
300		logger.info("Sending request %s %s", method, redact_uri(uri))
	299	logger.debug("Sending request %s %s", method, redact_uri(uri))
301	300
302	301	with start_active_span(
303	302	"outgoing-client-request",

329	328	self.hs.get_reactor(),
330	329	cancelled_to_request_timed_out_error,
331	330	)
332		response = yield make_deferred_yieldable(request_deferred)
	331	response = await make_deferred_yieldable(request_deferred)
333	332
334	333	incoming_responses_counter.labels(method, response.code).inc()
335	334	logger.info(

352	351	set_tag("error_reason", e.args[0])
353	352	raise
354	353
355		@defer.inlineCallbacks
356		def post_urlencoded_get_json(self, uri, args={}, headers=None):
	354	async def post_urlencoded_get_json(self, uri, args={}, headers=None):
357	355	"""
358	356	Args:
359	357	uri (str):

362	360	header name to a list of values for that header
363	361
364	362	Returns:
365		Deferred[object]: parsed json
	363	object: parsed json
366	364
367	365	Raises:
368	366	HttpResponseException: On a non-2xx HTTP response.

385	383	if headers:
386	384	actual_headers.update(headers)
387	385
388		response = yield self.request(
	386	response = await self.request(
389	387	"POST", uri, headers=Headers(actual_headers), data=query_bytes
390	388	)
391	389
392		body = yield make_deferred_yieldable(readBody(response))
	390	body = await make_deferred_yieldable(readBody(response))
393	391
394	392	if 200 <= response.code < 300:
395	393	return json.loads(body.decode("utf-8"))
396	394	else:
397		raise HttpResponseException(response.code, response.phrase, body)
398
399		@defer.inlineCallbacks
400		def post_json_get_json(self, uri, post_json, headers=None):
	395	raise HttpResponseException(
	396	response.code, response.phrase.decode("ascii", errors="replace"), body
	397	)
	398
	399	async def post_json_get_json(self, uri, post_json, headers=None):
401	400	"""
402	401
403	402	Args:

407	406	header name to a list of values for that header
408	407
409	408	Returns:
410		Deferred[object]: parsed json
	409	object: parsed json
411	410
412	411	Raises:
413	412	HttpResponseException: On a non-2xx HTTP response.

426	425	if headers:
427	426	actual_headers.update(headers)
428	427
429		response = yield self.request(
	428	response = await self.request(
430	429	"POST", uri, headers=Headers(actual_headers), data=json_str
431	430	)
432	431
433		body = yield make_deferred_yieldable(readBody(response))
	432	body = await make_deferred_yieldable(readBody(response))
434	433
435	434	if 200 <= response.code < 300:
436	435	return json.loads(body.decode("utf-8"))
437	436	else:
438		raise HttpResponseException(response.code, response.phrase, body)
439
440		@defer.inlineCallbacks
441		def get_json(self, uri, args={}, headers=None):
	437	raise HttpResponseException(
	438	response.code, response.phrase.decode("ascii", errors="replace"), body
	439	)
	440
	441	async def get_json(self, uri, args={}, headers=None):
442	442	""" Gets some json from the given URI.
443	443
444	444	Args:

450	450	headers (dict[str\|bytes, List[str\|bytes]]\|None): If not None, a map from
451	451	header name to a list of values for that header
452	452	Returns:
453		Deferred: Succeeds when we get any 2xx HTTP response, with the
	453	Succeeds when we get any 2xx HTTP response, with the
454	454	HTTP body as JSON.
455	455	Raises:
456	456	HttpResponseException On a non-2xx HTTP response.

461	461	if headers:
462	462	actual_headers.update(headers)
463	463
464		body = yield self.get_raw(uri, args, headers=headers)
	464	body = await self.get_raw(uri, args, headers=headers)
465	465	return json.loads(body.decode("utf-8"))
466	466
467		@defer.inlineCallbacks
468		def put_json(self, uri, json_body, args={}, headers=None):
	467	async def put_json(self, uri, json_body, args={}, headers=None):
469	468	""" Puts some json to the given URI.
470	469
471	470	Args:

478	477	headers (dict[str\|bytes, List[str\|bytes]]\|None): If not None, a map from
479	478	header name to a list of values for that header
480	479	Returns:
481		Deferred: Succeeds when we get any 2xx HTTP response, with the
	480	Succeeds when we get any 2xx HTTP response, with the
482	481	HTTP body as JSON.
483	482	Raises:
484	483	HttpResponseException On a non-2xx HTTP response.

499	498	if headers:
500	499	actual_headers.update(headers)
501	500
502		response = yield self.request(
	501	response = await self.request(
503	502	"PUT", uri, headers=Headers(actual_headers), data=json_str
504	503	)
505	504
506		body = yield make_deferred_yieldable(readBody(response))
	505	body = await make_deferred_yieldable(readBody(response))
507	506
508	507	if 200 <= response.code < 300:
509	508	return json.loads(body.decode("utf-8"))
510	509	else:
511		raise HttpResponseException(response.code, response.phrase, body)
512
513		@defer.inlineCallbacks
514		def get_raw(self, uri, args={}, headers=None):
	510	raise HttpResponseException(
	511	response.code, response.phrase.decode("ascii", errors="replace"), body
	512	)
	513
	514	async def get_raw(self, uri, args={}, headers=None):
515	515	""" Gets raw text from the given URI.
516	516
517	517	Args:

523	523	headers (dict[str\|bytes, List[str\|bytes]]\|None): If not None, a map from
524	524	header name to a list of values for that header
525	525	Returns:
526		Deferred: Succeeds when we get any 2xx HTTP response, with the
	526	Succeeds when we get any 2xx HTTP response, with the
527	527	HTTP body as bytes.
528	528	Raises:
529	529	HttpResponseException on a non-2xx HTTP response.

536	536	if headers:
537	537	actual_headers.update(headers)
538	538
539		response = yield self.request("GET", uri, headers=Headers(actual_headers))
540
541		body = yield make_deferred_yieldable(readBody(response))
	539	response = await self.request("GET", uri, headers=Headers(actual_headers))
	540
	541	body = await make_deferred_yieldable(readBody(response))
542	542
543	543	if 200 <= response.code < 300:
544	544	return body
545	545	else:
546		raise HttpResponseException(response.code, response.phrase, body)
	546	raise HttpResponseException(
	547	response.code, response.phrase.decode("ascii", errors="replace"), body
	548	)
547	549
548	550	# XXX: FIXME: This is horribly copy-pasted from matrixfederationclient.
549	551	# The two should be factored out.
550	552
551		@defer.inlineCallbacks
552		def get_file(self, url, output_stream, max_size=None, headers=None):
	553	async def get_file(self, url, output_stream, max_size=None, headers=None):
553	554	"""GETs a file from a given URL
554	555	Args:
555	556	url (str): The URL to GET

565	566	if headers:
566	567	actual_headers.update(headers)
567	568
568		response = yield self.request("GET", url, headers=Headers(actual_headers))
	569	response = await self.request("GET", url, headers=Headers(actual_headers))
569	570
570	571	resp_headers = dict(response.headers.getAllRawHeaders())
571	572

589	590	# straight back in again
590	591
591	592	try:
592		length = yield make_deferred_yieldable(
	593	length = await make_deferred_yieldable(
593	594	_readBodyToFile(response, output_stream, max_size)
594	595	)
595	596	except SynapseError:

+1

-1

synapse/http/federation/matrix_federation_agent.py less more

246	246	port = server.port
247	247
248	248	try:
249		logger.info("Connecting to %s:%i", host.decode("ascii"), port)
	249	logger.debug("Connecting to %s:%i", host.decode("ascii"), port)
250	250	endpoint = HostnameEndpoint(self._reactor, host, port)
251	251	if self._tls_options:
252	252	endpoint = wrapClientTLS(self._tls_options, endpoint)

+107

-66

synapse/http/matrixfederationclient.py less more

28	28
29	29	from twisted.internet import defer, protocol
30	30	from twisted.internet.error import DNSLookupError
31		from twisted.internet.interfaces import IReactorPluggableNameResolver
	31	from twisted.internet.interfaces import IReactorPluggableNameResolver, IReactorTime
32	32	from twisted.internet.task import _EPSILON, Cooperator
33	33	from twisted.web._newclient import ResponseDone
34	34	from twisted.web.http_headers import Headers
	35	from twisted.web.iweb import IResponse
35	36
36	37	import synapse.metrics
37	38	import synapse.util.retryutils

73	74	_next_id = 1
74	75
75	76
76		@attr.s
	77	@attr.s(frozen=True)
77	78	class MatrixFederationRequest(object):
78	79	method = attr.ib()
79	80	"""HTTP method

109	110	:type: str\|None
110	111	"""
111	112
	113	uri = attr.ib(init=False, type=bytes)
	114	"""The URI of this request
	115	"""
	116
112	117	def __attrs_post_init__(self):
113	118	global _next_id
114		self.txn_id = "%s-O-%s" % (self.method, _next_id)
	119	txn_id = "%s-O-%s" % (self.method, _next_id)
115	120	_next_id = (_next_id + 1) % (MAXINT - 1)
	121
	122	object.__setattr__(self, "txn_id", txn_id)
	123
	124	destination_bytes = self.destination.encode("ascii")
	125	path_bytes = self.path.encode("ascii")
	126	if self.query:
	127	query_bytes = encode_query_args(self.query)
	128	else:
	129	query_bytes = b""
	130
	131	# The object is frozen so we can pre-compute this.
	132	uri = urllib.parse.urlunparse(
	133	(b"matrix", destination_bytes, path_bytes, None, query_bytes, b"")
	134	)
	135	object.__setattr__(self, "uri", uri)
116	136
117	137	def get_json(self):
118	138	if self.json_callback:

120	140	return self.json
121	141
122	142
123		@defer.inlineCallbacks
124		def _handle_json_response(reactor, timeout_sec, request, response):
	143	async def _handle_json_response(
	144	reactor: IReactorTime,
	145	timeout_sec: float,
	146	request: MatrixFederationRequest,
	147	response: IResponse,
	148	start_ms: int,
	149	):
125	150	"""
126	151	Reads the JSON body of a response, with a timeout
127	152
128	153	Args:
129		reactor (IReactor): twisted reactor, for the timeout
130		timeout_sec (float): number of seconds to wait for response to complete
131		request (MatrixFederationRequest): the request that triggered the response
132		response (IResponse): response to the request
	154	reactor: twisted reactor, for the timeout
	155	timeout_sec: number of seconds to wait for response to complete
	156	request: the request that triggered the response
	157	response: response to the request
	158	start_ms: Timestamp when request was made
133	159
134	160	Returns:
135	161	dict: parsed JSON response

140	166	d = treq.json_content(response)
141	167	d = timeout_deferred(d, timeout=timeout_sec, reactor=reactor)
142	168
143		body = yield make_deferred_yieldable(d)
	169	body = await make_deferred_yieldable(d)
144	170	except TimeoutError as e:
145	171	logger.warning(
146		"{%s} [%s] Timed out reading response", request.txn_id, request.destination,
	172	"{%s} [%s] Timed out reading response - %s %s",
	173	request.txn_id,
	174	request.destination,
	175	request.method,
	176	request.uri.decode("ascii"),
147	177	)
148	178	raise RequestSendFailed(e, can_retry=True) from e
149	179	except Exception as e:
150	180	logger.warning(
151		"{%s} [%s] Error reading response: %s",
	181	"{%s} [%s] Error reading response %s %s: %s",
152	182	request.txn_id,
153	183	request.destination,
	184	request.method,
	185	request.uri.decode("ascii"),
154	186	e,
155	187	)
156	188	raise
	189
	190	time_taken_secs = reactor.seconds() - start_ms / 1000
	191
157	192	logger.info(
158		"{%s} [%s] Completed: %d %s",
	193	"{%s} [%s] Completed request: %d %s in %.2f secs - %s %s",
159	194	request.txn_id,
160	195	request.destination,
161	196	response.code,
162	197	response.phrase.decode("ascii", errors="replace"),
	198	time_taken_secs,
	199	request.method,
	200	request.uri.decode("ascii"),
163	201	)
164	202	return body
165	203

223	261
224	262	self._cooperator = Cooperator(scheduler=schedule)
225	263
226		@defer.inlineCallbacks
227		def _send_request_with_optional_trailing_slash(
	264	async def _send_request_with_optional_trailing_slash(
228	265	self, request, try_trailing_slash_on_400=False, **send_request_args
229	266	):
230	267	"""Wrapper for _send_request which can optionally retry the request

245	282	(except 429).
246	283
247	284	Returns:
248		Deferred[Dict]: Parsed JSON response body.
	285	Dict: Parsed JSON response body.
249	286	"""
250	287	try:
251		response = yield self._send_request(request, **send_request_args)
	288	response = await self._send_request(request, **send_request_args)
252	289	except HttpResponseException as e:
253	290	# Received an HTTP error > 300. Check if it meets the requirements
254	291	# to retry with a trailing slash

262	299	# 'M_UNRECOGNIZED' which some endpoints can return when omitting a
263	300	# trailing slash on Synapse <= v0.99.3.
264	301	logger.info("Retrying request with trailing slash")
265		request.path += "/"
266
267		response = yield self._send_request(request, **send_request_args)
	302
	303	# Request is frozen so we create a new instance
	304	request = attr.evolve(request, path=request.path + "/")
	305
	306	response = await self._send_request(request, **send_request_args)
268	307
269	308	return response
270	309
271		@defer.inlineCallbacks
272		def _send_request(
	310	async def _send_request(
273	311	self,
274	312	request,
275	313	retry_on_dns_fail=True,

310	348	backoff_on_404 (bool): Back off if we get a 404
311	349
312	350	Returns:
313		Deferred[twisted.web.client.Response]: resolves with the HTTP
	351	twisted.web.client.Response: resolves with the HTTP
314	352	response object on success.
315	353
316	354	Raises:

334	372	):
335	373	raise FederationDeniedError(request.destination)
336	374
337		limiter = yield synapse.util.retryutils.get_retry_limiter(
	375	limiter = await synapse.util.retryutils.get_retry_limiter(
338	376	request.destination,
339	377	self.clock,
340	378	self._store,

375	413	else:
376	414	retries_left = MAX_SHORT_RETRIES
377	415
378		url_bytes = urllib.parse.urlunparse(
379		(b"matrix", destination_bytes, path_bytes, None, query_bytes, b"")
380		)
	416	url_bytes = request.uri
381	417	url_str = url_bytes.decode("ascii")
382	418
383	419	url_to_sign_bytes = urllib.parse.urlunparse(

404	440
405	441	headers_dict[b"Authorization"] = auth_headers
406	442
407		logger.info(
	443	logger.debug(
408	444	"{%s} [%s] Sending request: %s %s; timeout %fs",
409	445	request.txn_id,
410	446	request.destination,

432	468	reactor=self.reactor,
433	469	)
434	470
435		response = yield request_deferred
	471	response = await request_deferred
436	472	except TimeoutError as e:
437	473	raise RequestSendFailed(e, can_retry=True) from e
438	474	except DNSLookupError as e:
439	475	raise RequestSendFailed(e, can_retry=retry_on_dns_fail) from e
440	476	except Exception as e:
441		logger.info("Failed to send request: %s", e)
442	477	raise RequestSendFailed(e, can_retry=True) from e
443	478
444	479	incoming_responses_counter.labels(

446	481	).inc()
447	482
448	483	set_tag(tags.HTTP_STATUS_CODE, response.code)
	484	response_phrase = response.phrase.decode("ascii", errors="replace")
449	485
450	486	if 200 <= response.code < 300:
451	487	logger.debug(

453	489	request.txn_id,
454	490	request.destination,
455	491	response.code,
456		response.phrase.decode("ascii", errors="replace"),
	492	response_phrase,
457	493	)
458	494	pass
459	495	else:

462	498	request.txn_id,
463	499	request.destination,
464	500	response.code,
465		response.phrase.decode("ascii", errors="replace"),
	501	response_phrase,
466	502	)
467	503	# :'(
468	504	# Update transactions table?

472	508	)
473	509
474	510	try:
475		body = yield make_deferred_yieldable(d)
	511	body = await make_deferred_yieldable(d)
476	512	except Exception as e:
477	513	# Eh, we're already going to raise an exception so lets
478	514	# ignore if this fails.

486	522	)
487	523	body = None
488	524
489		e = HttpResponseException(response.code, response.phrase, body)
	525	e = HttpResponseException(response.code, response_phrase, body)
490	526
491	527	# Retry if the error is a 429 (Too Many Requests),
492	528	# otherwise just raise a standard HttpResponseException

497	533
498	534	break
499	535	except RequestSendFailed as e:
500		logger.warning(
	536	logger.info(
501	537	"{%s} [%s] Request failed: %s %s: %s",
502	538	request.txn_id,
503	539	request.destination,

526	562	delay,
527	563	)
528	564
529		yield self.clock.sleep(delay)
	565	await self.clock.sleep(delay)
530	566	retries_left -= 1
531	567	else:
532	568	raise

589	625	)
590	626	return auth_headers
591	627
592		@defer.inlineCallbacks
593		def put_json(
	628	async def put_json(
594	629	self,
595	630	destination,
596	631	path,

634	669	enabled.
635	670
636	671	Returns:
637		Deferred[dict\|list]: Succeeds when we get a 2xx HTTP response. The
	672	dict\|list: Succeeds when we get a 2xx HTTP response. The
638	673	result will be the decoded JSON body.
639	674
640	675	Raises:

656	691	json=data,
657	692	)
658	693
659		response = yield self._send_request_with_optional_trailing_slash(
	694	start_ms = self.clock.time_msec()
	695
	696	response = await self._send_request_with_optional_trailing_slash(
660	697	request,
661	698	try_trailing_slash_on_400,
662	699	backoff_on_404=backoff_on_404,

665	702	timeout=timeout,
666	703	)
667	704
668		body = yield _handle_json_response(
669		self.reactor, self.default_timeout, request, response
	705	body = await _handle_json_response(
	706	self.reactor, self.default_timeout, request, response, start_ms
670	707	)
671	708
672	709	return body
673	710
674		@defer.inlineCallbacks
675		def post_json(
	711	async def post_json(
676	712	self,
677	713	destination,
678	714	path,

705	741
706	742	args (dict): query params
707	743	Returns:
708		Deferred[dict\|list]: Succeeds when we get a 2xx HTTP response. The
	744	dict\|list: Succeeds when we get a 2xx HTTP response. The
709	745	result will be the decoded JSON body.
710	746
711	747	Raises:

723	759	method="POST", destination=destination, path=path, query=args, json=data
724	760	)
725	761
726		response = yield self._send_request(
	762	start_ms = self.clock.time_msec()
	763
	764	response = await self._send_request(
727	765	request,
728	766	long_retries=long_retries,
729	767	timeout=timeout,

735	773	else:
736	774	_sec_timeout = self.default_timeout
737	775
738		body = yield _handle_json_response(
739		self.reactor, _sec_timeout, request, response
	776	body = await _handle_json_response(
	777	self.reactor, _sec_timeout, request, response, start_ms,
740	778	)
741	779	return body
742	780
743		@defer.inlineCallbacks
744		def get_json(
	781	async def get_json(
745	782	self,
746	783	destination,
747	784	path,

773	810	response we should try appending a trailing slash to the end of
774	811	the request. Workaround for #3622 in Synapse <= v0.99.3.
775	812	Returns:
776		Deferred[dict\|list]: Succeeds when we get a 2xx HTTP response. The
	813	dict\|list: Succeeds when we get a 2xx HTTP response. The
777	814	result will be the decoded JSON body.
778	815
779	816	Raises:

790	827	method="GET", destination=destination, path=path, query=args
791	828	)
792	829
793		response = yield self._send_request_with_optional_trailing_slash(
	830	start_ms = self.clock.time_msec()
	831
	832	response = await self._send_request_with_optional_trailing_slash(
794	833	request,
795	834	try_trailing_slash_on_400,
796	835	backoff_on_404=False,

799	838	timeout=timeout,
800	839	)
801	840
802		body = yield _handle_json_response(
803		self.reactor, self.default_timeout, request, response
	841	body = await _handle_json_response(
	842	self.reactor, self.default_timeout, request, response, start_ms
804	843	)
805	844
806	845	return body
807	846
808		@defer.inlineCallbacks
809		def delete_json(
	847	async def delete_json(
810	848	self,
811	849	destination,
812	850	path,

834	872
835	873	args (dict): query params
836	874	Returns:
837		Deferred[dict\|list]: Succeeds when we get a 2xx HTTP response. The
	875	dict\|list: Succeeds when we get a 2xx HTTP response. The
838	876	result will be the decoded JSON body.
839	877
840	878	Raises:

851	889	method="DELETE", destination=destination, path=path, query=args
852	890	)
853	891
854		response = yield self._send_request(
	892	start_ms = self.clock.time_msec()
	893
	894	response = await self._send_request(
855	895	request,
856	896	long_retries=long_retries,
857	897	timeout=timeout,
858	898	ignore_backoff=ignore_backoff,
859	899	)
860	900
861		body = yield _handle_json_response(
862		self.reactor, self.default_timeout, request, response
	901	body = await _handle_json_response(
	902	self.reactor, self.default_timeout, request, response, start_ms
863	903	)
864	904	return body
865	905
866		@defer.inlineCallbacks
867		def get_file(
	906	async def get_file(
868	907	self,
869	908	destination,
870	909	path,

884	923	and try the request anyway.
885	924
886	925	Returns:
887		Deferred[tuple[int, dict]]: Resolves with an (int,dict) tuple of
	926	tuple[int, dict]: Resolves with an (int,dict) tuple of
888	927	the file length and a dict of the response headers.
889	928
890	929	Raises:

901	940	method="GET", destination=destination, path=path, query=args
902	941	)
903	942
904		response = yield self._send_request(
	943	response = await self._send_request(
905	944	request, retry_on_dns_fail=retry_on_dns_fail, ignore_backoff=ignore_backoff
906	945	)
907	946

910	949	try:
911	950	d = _readBodyToFile(response, output_stream, max_size)
912	951	d.addTimeout(self.default_timeout, self.reactor)
913		length = yield make_deferred_yieldable(d)
	952	length = await make_deferred_yieldable(d)
914	953	except Exception as e:
915	954	logger.warning(
916	955	"{%s} [%s] Error reading response: %s",

920	959	)
921	960	raise
922	961	logger.info(
923		"{%s} [%s] Completed: %d %s [%d bytes]",
	962	"{%s} [%s] Completed: %d %s [%d bytes] %s %s",
924	963	request.txn_id,
925	964	request.destination,
926	965	response.code,
927	966	response.phrase.decode("ascii", errors="replace"),
928	967	length,
	968	request.method,
	969	request.uri.decode("ascii"),
929	970	)
930	971	return (length, headers)
931	972

+15

-8

synapse/http/server.py less more

24	24	from typing import Any, Callable, Dict, Tuple, Union
25	25
26	26	import jinja2
27		from canonicaljson import encode_canonical_json, encode_pretty_printed_json, json
	27	from canonicaljson import encode_canonical_json, encode_pretty_printed_json
28	28
29	29	from twisted.internet import defer
30	30	from twisted.python import failure

45	45	from synapse.http.site import SynapseRequest
46	46	from synapse.logging.context import preserve_fn
47	47	from synapse.logging.opentracing import trace_servlet
	48	from synapse.util import json_encoder
48	49	from synapse.util.caches import intern_dict
49	50
50	51	logger = logging.getLogger(__name__)

241	242	no appropriate method exists. Can be overriden in sub classes for
242	243	different routing.
243	244	"""
244
245		method_handler = getattr(
246		self, "_async_render_%s" % (request.method.decode("ascii"),), None
247		)
	245	# Treat HEAD requests as GET requests.
	246	request_method = request.method.decode("ascii")
	247	if request_method == "HEAD":
	248	request_method = "GET"
	249
	250	method_handler = getattr(self, "_async_render_%s" % (request_method,), None)
248	251	if method_handler:
249	252	raw_callback_return = method_handler(request)
250	253

361	364	A tuple of the callback to use, the name of the servlet, and the
362	365	key word arguments to pass to the callback
363	366	"""
	367	# Treat HEAD requests as GET requests.
364	368	request_path = request.path.decode("ascii")
	369	request_method = request.method
	370	if request_method == b"HEAD":
	371	request_method = b"GET"
365	372
366	373	# Loop through all the registered callbacks to check if the method
367	374	# and path regex match
368		for path_entry in self.path_regexs.get(request.method, []):
	375	for path_entry in self.path_regexs.get(request_method, []):
369	376	m = path_entry.pattern.match(request_path)
370	377	if m:
371	378	# We found a match!

531	538	# canonicaljson already encodes to bytes
532	539	json_bytes = encode_canonical_json(json_object)
533	540	else:
534		json_bytes = json.dumps(json_object).encode("utf-8")
	541	json_bytes = json_encoder.encode(json_object).encode("utf-8")
535	542
536	543	return respond_with_json_bytes(request, code, json_bytes, send_cors=send_cors)
537	544

578	585	"""
579	586	request.setHeader(b"Access-Control-Allow-Origin", b"*")
580	587	request.setHeader(
581		b"Access-Control-Allow-Methods", b"GET, POST, PUT, DELETE, OPTIONS"
	588	b"Access-Control-Allow-Methods", b"GET, HEAD, POST, PUT, DELETE, OPTIONS"
582	589	)
583	590	request.setHeader(
584	591	b"Access-Control-Allow-Headers",

+16

-4

synapse/http/site.py less more

145	145
146	146	Returns a context manager; the correct way to use this is:
147	147
148		@defer.inlineCallbacks
149		def handle_request(request):
	148	async def handle_request(request):
150	149	with request.processing("FooServlet"):
151		yield really_handle_the_request()
	150	await really_handle_the_request()
152	151
153	152	Once the context manager is closed, the completion of the request will be logged,
154	153	and the various metrics will be updated.

286	285	# the connection dropped)
287	286	code += "!"
288	287
289		self.site.access_logger.info(
	288	log_level = logging.INFO if self._should_log_request() else logging.DEBUG
	289	self.site.access_logger.log(
	290	log_level,
290	291	"%s - %s - {%s}"
291	292	" Processed request: %.3fsec/%.3fsec (%.3fsec, %.3fsec) (%.3fsec/%.3fsec/%d)"
292	293	' %sB %s "%s %s %s" "%s" [%d dbevts]',

314	315	except Exception as e:
315	316	logger.warning("Failed to stop metrics: %r", e)
316	317
	318	def _should_log_request(self) -> bool:
	319	"""Whether we should log at INFO that we processed the request.
	320	"""
	321	if self.path == b"/health":
	322	return False
	323
	324	if self.method == b"OPTIONS":
	325	return False
	326
	327	return True
	328
317	329
318	330	class XForwardedForRequest(SynapseRequest):
319	331	def __init__(self, args, *kw):

+13

-23

synapse/metrics/background_process_metrics.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
	15	import inspect
15	16	import logging
16	17	import threading
17		from asyncio import iscoroutine
18	18	from functools import wraps
19	19	from typing import TYPE_CHECKING, Dict, Optional, Set
20	20
21	21	from prometheus_client.core import REGISTRY, Counter, Gauge
22	22
23	23	from twisted.internet import defer
24		from twisted.python.failure import Failure
25	24
26	25	from synapse.logging.context import LoggingContext, PreserveLoggingContext
27	26

166	165	)
167	166
168	167
169		def run_as_background_process(desc, func, args, *kwargs):
	168	def run_as_background_process(desc: str, func, args, *kwargs):
170	169	"""Run the given function in its own logcontext, with resource metrics
171	170
172	171	This should be used to wrap processes which are fired off to run in the

178	177	normal synapse inlineCallbacks function).
179	178
180	179	Args:
181		desc (str): a description for this background process type
	180	desc: a description for this background process type
182	181	func: a function, which may return a Deferred or a coroutine
183	182	args: positional args for func
184	183	kwargs: keyword args for func

187	186	follow the synapse logcontext rules.
188	187	"""
189	188
190		@defer.inlineCallbacks
191		def run():
	189	async def run():
192	190	with _bg_metrics_lock:
193	191	count = _background_process_counts.get(desc, 0)
194	192	_background_process_counts[desc] = count + 1

202	200	try:
203	201	result = func(args, *kwargs)
204	202
205		# We probably don't have an ensureDeferred in our call stack to handle
206		# coroutine results, so we need to ensureDeferred here.
207		#
208		# But we need this check because ensureDeferred doesn't like being
209		# called on immediate values (as opposed to Deferreds or coroutines).
210		if iscoroutine(result):
211		result = defer.ensureDeferred(result)
212
213		return (yield result)
	203	if inspect.isawaitable(result):
	204	result = await result
	205
	206	return result
214	207	except Exception:
215		# failure.Failure() fishes the original Failure out of our stack, and
216		# thus gives us a sensible stack trace.
217		f = Failure()
218		logger.error(
219		"Background process '%s' threw an exception",
220		desc,
221		exc_info=(f.type, f.value, f.getTracebackObject()),
	208	logger.exception(
	209	"Background process '%s' threw an exception", desc,
222	210	)
223	211	finally:
224	212	_background_process_in_flight_count.labels(desc).dec()
225	213
226	214	with PreserveLoggingContext():
227		return run()
	215	# Note that we return a Deferred here so that it can be used in a
	216	# looping_call and other places that expect a Deferred.
	217	return defer.ensureDeferred(run())
228	218
229	219
230	220	def wrap_as_background_process(desc):

+7

-3

synapse/module_api/__init__.py less more

193	193	synapse.api.errors.AuthError: the access token is invalid
194	194	"""
195	195	# see if the access token corresponds to a device
196		user_info = yield self._auth.get_user_by_access_token(access_token)
	196	user_info = yield defer.ensureDeferred(
	197	self._auth.get_user_by_access_token(access_token)
	198	)
197	199	device_id = user_info.get("device_id")
198	200	user_id = user_info["user"].to_string()
199	201	if device_id:
200	202	# delete the device, which will also delete its access tokens
201		yield self._hs.get_device_handler().delete_device(user_id, device_id)
	203	yield defer.ensureDeferred(
	204	self._hs.get_device_handler().delete_device(user_id, device_id)
	205	)
202	206	else:
203	207	# no associated device. Just delete the access token.
204	208	yield defer.ensureDeferred(

218	222	Returns:
219	223	Deferred[object]: result of func
220	224	"""
221		return self._store.db.runInteraction(desc, func, args, *kwargs)
	225	return self._store.db_pool.runInteraction(desc, func, args, *kwargs)
222	226
223	227	def complete_sso_login(
224	228	self, registered_user_id: str, request: SynapseRequest, client_redirect_url: str

+90

-51

synapse/notifier.py less more

14	14
15	15	import logging
16	16	from collections import namedtuple
17		from typing import Callable, Iterable, List, TypeVar
	17	from typing import (
	18	Awaitable,
	19	Callable,
	20	Dict,
	21	Iterable,
	22	List,
	23	Optional,
	24	Set,
	25	Tuple,
	26	TypeVar,
	27	Union,
	28	)
18	29
19	30	from prometheus_client import Counter
20	31

23	34	import synapse.server
24	35	from synapse.api.constants import EventTypes, Membership
25	36	from synapse.api.errors import AuthError
	37	from synapse.events import EventBase
26	38	from synapse.handlers.presence import format_user_presence_state
27	39	from synapse.logging.context import PreserveLoggingContext
28	40	from synapse.logging.utils import log_function
29	41	from synapse.metrics import LaterGauge
30	42	from synapse.metrics.background_process_metrics import run_as_background_process
31		from synapse.types import StreamToken
	43	from synapse.streams.config import PaginationConfig
	44	from synapse.types import Collection, StreamToken, UserID
32	45	from synapse.util.async_helpers import ObservableDeferred, timeout_deferred
33	46	from synapse.util.metrics import Measure
34	47	from synapse.visibility import filter_events_for_client

76	89	so that it can remove itself from the indexes in the Notifier class.
77	90	"""
78	91
79		def __init__(self, user_id, rooms, current_token, time_now_ms):
	92	def __init__(
	93	self,
	94	user_id: str,
	95	rooms: Collection[str],
	96	current_token: StreamToken,
	97	time_now_ms: int,
	98	):
80	99	self.user_id = user_id
81	100	self.rooms = set(rooms)
82	101	self.current_token = current_token

92	111	with PreserveLoggingContext():
93	112	self.notify_deferred = ObservableDeferred(defer.Deferred())
94	113
95		def notify(self, stream_key, stream_id, time_now_ms):
	114	def notify(self, stream_key: str, stream_id: int, time_now_ms: int):
96	115	"""Notify any listeners for this user of a new event from an
97	116	event source.
98	117	Args:
99		stream_key(str): The stream the event came from.
100		stream_id(str): The new id for the stream the event came from.
101		time_now_ms(int): The current time in milliseconds.
	118	stream_key: The stream the event came from.
	119	stream_id: The new id for the stream the event came from.
	120	time_now_ms: The current time in milliseconds.
102	121	"""
103	122	self.current_token = self.current_token.copy_and_advance(stream_key, stream_id)
104	123	self.last_notified_token = self.current_token

111	130	self.notify_deferred = ObservableDeferred(defer.Deferred())
112	131	noify_deferred.callback(self.current_token)
113	132
114		def remove(self, notifier):
	133	def remove(self, notifier: "Notifier"):
115	134	""" Remove this listener from all the indexes in the Notifier
116	135	it knows about.
117	136	"""

122	141
123	142	notifier.user_to_user_stream.pop(self.user_id)
124	143
125		def count_listeners(self):
	144	def count_listeners(self) -> int:
126	145	return len(self.notify_deferred.observers())
127	146
128		def new_listener(self, token):
	147	def new_listener(self, token: StreamToken) -> _NotificationListener:
129	148	"""Returns a deferred that is resolved when there is a new token
130	149	greater than the given token.
131	150

158	177	UNUSED_STREAM_EXPIRY_MS = 10 * 60 * 1000
159	178
160	179	def __init__(self, hs: "synapse.server.HomeServer"):
161		self.user_to_user_stream = {}
162		self.room_to_user_streams = {}
	180	self.user_to_user_stream = {} # type: Dict[str, _NotifierUserStream]
	181	self.room_to_user_streams = {} # type: Dict[str, Set[_NotifierUserStream]]
163	182
164	183	self.hs = hs
165	184	self.storage = hs.get_storage()
166	185	self.event_sources = hs.get_event_sources()
167	186	self.store = hs.get_datastore()
168		self.pending_new_room_events = []
	187	self.pending_new_room_events = (
	188	[]
	189	) # type: List[Tuple[int, EventBase, Collection[Union[str, UserID]]]]
169	190
170	191	# Called when there are new things to stream over replication
171	192	self.replication_callbacks = [] # type: List[Callable[[], None]]

177	198	self.clock = hs.get_clock()
178	199	self.appservice_handler = hs.get_application_service_handler()
179	200
	201	self.federation_sender = None
180	202	if hs.should_send_federation():
181	203	self.federation_sender = hs.get_federation_sender()
182		else:
183		self.federation_sender = None
184	204
185	205	self.state_handler = hs.get_state_handler()
186	206

192	212	# when rendering the metrics page, which is likely once per minute at
193	213	# most when scraping it.
194	214	def count_listeners():
195		all_user_streams = set()
196
197		for x in list(self.room_to_user_streams.values()):
198		all_user_streams \|= x
199		for x in list(self.user_to_user_stream.values()):
200		all_user_streams.add(x)
	215	all_user_streams = set() # type: Set[_NotifierUserStream]
	216
	217	for streams in list(self.room_to_user_streams.values()):
	218	all_user_streams \|= streams
	219	for stream in list(self.user_to_user_stream.values()):
	220	all_user_streams.add(stream)
201	221
202	222	return sum(stream.count_listeners() for stream in all_user_streams)
203	223

222	242	self.replication_callbacks.append(cb)
223	243
224	244	def on_new_room_event(
225		self, event, room_stream_id, max_room_stream_id, extra_users=[]
	245	self,
	246	event: EventBase,
	247	room_stream_id: int,
	248	max_room_stream_id: int,
	249	extra_users: Collection[Union[str, UserID]] = [],
226	250	):
227	251	""" Used by handlers to inform the notifier something has happened
228	252	in the room, room event wise.

240	264
241	265	self.notify_replication()
242	266
243		def _notify_pending_new_room_events(self, max_room_stream_id):
	267	def _notify_pending_new_room_events(self, max_room_stream_id: int):
244	268	"""Notify for the room events that were queued waiting for a previous
245	269	event to be persisted.
246	270	Args:
247		max_room_stream_id(int): The highest stream_id below which all
	271	max_room_stream_id: The highest stream_id below which all
248	272	events have been persisted.
249	273	"""
250	274	pending = self.pending_new_room_events

257	281	else:
258	282	self._on_new_room_event(event, room_stream_id, extra_users)
259	283
260		def _on_new_room_event(self, event, room_stream_id, extra_users=[]):
	284	def _on_new_room_event(
	285	self,
	286	event: EventBase,
	287	room_stream_id: int,
	288	extra_users: Collection[Union[str, UserID]] = [],
	289	):
261	290	"""Notify any user streams that are interested in this room event"""
262	291	# poke any interested application service.
263	292	run_as_background_process(

274	303	"room_key", room_stream_id, users=extra_users, rooms=[event.room_id]
275	304	)
276	305
277		async def _notify_app_services(self, room_stream_id):
	306	async def _notify_app_services(self, room_stream_id: int):
278	307	try:
279	308	await self.appservice_handler.notify_interested_services(room_stream_id)
280	309	except Exception:
281	310	logger.exception("Error notifying application services of event")
282	311
283		def on_new_event(self, stream_key, new_token, users=[], rooms=[]):
	312	def on_new_event(
	313	self,
	314	stream_key: str,
	315	new_token: int,
	316	users: Collection[Union[str, UserID]] = [],
	317	rooms: Collection[str] = [],
	318	):
284	319	""" Used to inform listeners that something has happened event wise.
285	320
286	321	Will wake up all listeners for the given users and rooms.

306	341
307	342	self.notify_replication()
308	343
309		def on_new_replication_data(self):
	344	def on_new_replication_data(self) -> None:
310	345	"""Used to inform replication listeners that something has happend
311	346	without waking up any of the normal user event streams"""
312	347	self.notify_replication()
313	348
314	349	async def wait_for_events(
315		self, user_id, timeout, callback, room_ids=None, from_token=StreamToken.START
316		):
	350	self,
	351	user_id: str,
	352	timeout: int,
	353	callback: Callable[[StreamToken, StreamToken], Awaitable[T]],
	354	room_ids=None,
	355	from_token=StreamToken.START,
	356	) -> T:
317	357	"""Wait until the callback returns a non empty response or the
318	358	timeout fires.
319	359	"""
320	360	user_stream = self.user_to_user_stream.get(user_id)
321	361	if user_stream is None:
322		current_token = await self.event_sources.get_current_token()
	362	current_token = self.event_sources.get_current_token()
323	363	if room_ids is None:
324	364	room_ids = await self.store.get_rooms_for_user(user_id)
325	365	user_stream = _NotifierUserStream(

376	416
377	417	async def get_events_for(
378	418	self,
379		user,
380		pagination_config,
381		timeout,
382		only_keys=None,
383		is_guest=False,
384		explicit_room_id=None,
385		):
	419	user: UserID,
	420	pagination_config: PaginationConfig,
	421	timeout: int,
	422	is_guest: bool = False,
	423	explicit_room_id: str = None,
	424	) -> EventStreamResult:
386	425	""" For the given user and rooms, return any new events for them. If
387	426	there are no new events wait for up to `timeout` milliseconds for any
388	427	new events to happen before returning.
389
390		If `only_keys` is not None, events from keys will be sent down.
391	428
392	429	If explicit_room_id is not set, the user's joined rooms will be polled
393	430	for events.

396	433	"""
397	434	from_token = pagination_config.from_token
398	435	if not from_token:
399		from_token = await self.event_sources.get_current_token()
	436	from_token = self.event_sources.get_current_token()
400	437
401	438	limit = pagination_config.limit
402	439
403	440	room_ids, is_joined = await self._get_room_ids(user, explicit_room_id)
404	441	is_peeking = not is_joined
405	442
406		async def check_for_updates(before_token, after_token):
	443	async def check_for_updates(
	444	before_token: StreamToken, after_token: StreamToken
	445	) -> EventStreamResult:
407	446	if not after_token.is_after(before_token):
408	447	return EventStreamResult([], (from_token, from_token))
409	448
410		events = []
	449	events = [] # type: List[EventBase]
411	450	end_token = from_token
412	451
413	452	for name, source in self.event_sources.sources.items():

415	454	before_id = getattr(before_token, keyname)
416	455	after_id = getattr(after_token, keyname)
417	456	if before_id == after_id:
418		continue
419		if only_keys and name not in only_keys:
420	457	continue
421	458
422	459	new_events, new_key = await source.get_new_events(

475	512
476	513	return result
477	514
478		async def _get_room_ids(self, user, explicit_room_id):
	515	async def _get_room_ids(
	516	self, user: UserID, explicit_room_id: Optional[str]
	517	) -> Tuple[Collection[str], bool]:
479	518	joined_room_ids = await self.store.get_rooms_for_user(user.to_string())
480	519	if explicit_room_id:
481	520	if explicit_room_id in joined_room_ids:

485	524	raise AuthError(403, "Non-joined access not allowed")
486	525	return joined_room_ids, True
487	526
488		async def _is_world_readable(self, room_id):
	527	async def _is_world_readable(self, room_id: str) -> bool:
489	528	state = await self.state_handler.get_current_state(
490	529	room_id, EventTypes.RoomHistoryVisibility, ""
491	530	)

495	534	return False
496	535
497	536	@log_function
498		def remove_expired_streams(self):
	537	def remove_expired_streams(self) -> None:
499	538	time_now_ms = self.clock.time_msec()
500	539	expired_streams = []
501	540	expire_before_ts = time_now_ms - self.UNUSED_STREAM_EXPIRY_MS

509	548	expired_stream.remove(self)
510	549
511	550	@log_function
512		def _register_with_keys(self, user_stream):
	551	def _register_with_keys(self, user_stream: _NotifierUserStream):
513	552	self.user_to_user_stream[user_stream.user_id] = user_stream
514	553
515	554	for room in user_stream.rooms:
516	555	s = self.room_to_user_streams.setdefault(room, set())
517	556	s.add(user_stream)
518	557
519		def _user_joined_room(self, user_id, room_id):
	558	def _user_joined_room(self, user_id: str, room_id: str):
520	559	new_user_stream = self.user_to_user_stream.get(user_id)
521	560	if new_user_stream is not None:
522	561	room_streams = self.room_to_user_streams.setdefault(room_id, set())
523	562	room_streams.add(new_user_stream)
524	563	new_user_stream.rooms.add(room_id)
525	564
526		def notify_replication(self):
	565	def notify_replication(self) -> None:
527	566	"""Notify the any replication listeners that there's a new event"""
528	567	for cb in self.replication_callbacks:
529	568	cb()

+2

-5

synapse/push/action_generator.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
16
17		from twisted.internet import defer
18	16
19	17	from synapse.util.metrics import Measure
20	18

36	34	# event stream, so we just run the rules for a client with no profile
37	35	# tag (ie. we just need all the users).
38	36
39		@defer.inlineCallbacks
40		def handle_push_actions_for_event(self, event, context):
	37	async def handle_push_actions_for_event(self, event, context):
41	38	with Measure(self.clock, "action_for_event_by_user"):
42		yield self.bulk_evaluator.action_for_event_by_user(event, context)
	39	await self.bulk_evaluator.action_for_event_by_user(event, context)

+211

-11

synapse/push/baserules.py less more

18	18	from synapse.push.rulekinds import PRIORITY_CLASS_INVERSE_MAP, PRIORITY_CLASS_MAP
19	19
20	20
21		def list_with_base_rules(rawrules):
	21	def list_with_base_rules(rawrules, use_new_defaults=False):
22	22	"""Combine the list of rules set by the user with the default push rules
23	23
24	24	Args:
25	25	rawrules(list): The rules the user has modified or set.
	26	use_new_defaults(bool): Whether to use the new experimental default rules when
	27	appending or prepending default rules.
26	28
27	29	Returns:
28	30	A new list with the rules set by the user combined with the defaults.

42	44
43	45	ruleslist.extend(
44	46	make_base_prepend_rules(
45		PRIORITY_CLASS_INVERSE_MAP[current_prio_class], modified_base_rules
	47	PRIORITY_CLASS_INVERSE_MAP[current_prio_class],
	48	modified_base_rules,
	49	use_new_defaults,
46	50	)
47	51	)
48	52

53	57	make_base_append_rules(
54	58	PRIORITY_CLASS_INVERSE_MAP[current_prio_class],
55	59	modified_base_rules,
	60	use_new_defaults,
56	61	)
57	62	)
58	63	current_prio_class -= 1

61	66	make_base_prepend_rules(
62	67	PRIORITY_CLASS_INVERSE_MAP[current_prio_class],
63	68	modified_base_rules,
	69	use_new_defaults,
64	70	)
65	71	)
66	72

69	75	while current_prio_class > 0:
70	76	ruleslist.extend(
71	77	make_base_append_rules(
72		PRIORITY_CLASS_INVERSE_MAP[current_prio_class], modified_base_rules
	78	PRIORITY_CLASS_INVERSE_MAP[current_prio_class],
	79	modified_base_rules,
	80	use_new_defaults,
73	81	)
74	82	)
75	83	current_prio_class -= 1
76	84	if current_prio_class > 0:
77	85	ruleslist.extend(
78	86	make_base_prepend_rules(
79		PRIORITY_CLASS_INVERSE_MAP[current_prio_class], modified_base_rules
	87	PRIORITY_CLASS_INVERSE_MAP[current_prio_class],
	88	modified_base_rules,
	89	use_new_defaults,
80	90	)
81	91	)
82	92
83	93	return ruleslist
84	94
85	95
86		def make_base_append_rules(kind, modified_base_rules):
	96	def make_base_append_rules(kind, modified_base_rules, use_new_defaults=False):
87	97	rules = []
88	98
89	99	if kind == "override":
90		rules = BASE_APPEND_OVERRIDE_RULES
	100	rules = (
	101	NEW_APPEND_OVERRIDE_RULES
	102	if use_new_defaults
	103	else BASE_APPEND_OVERRIDE_RULES
	104	)
91	105	elif kind == "underride":
92		rules = BASE_APPEND_UNDERRIDE_RULES
	106	rules = (
	107	NEW_APPEND_UNDERRIDE_RULES
	108	if use_new_defaults
	109	else BASE_APPEND_UNDERRIDE_RULES
	110	)
93	111	elif kind == "content":
94	112	rules = BASE_APPEND_CONTENT_RULES
95	113

104	122	return rules
105	123
106	124
107		def make_base_prepend_rules(kind, modified_base_rules):
	125	def make_base_prepend_rules(kind, modified_base_rules, use_new_defaults=False):
108	126	rules = []
109	127
110	128	if kind == "override":

269	287	]
270	288
271	289
272		BASE_APPEND_UNDERRIDE_RULES = [
273		{
274		"rule_id": "global/underride/.m.rule.call",
	290	NEW_APPEND_OVERRIDE_RULES = [
	291	{
	292	"rule_id": "global/override/.m.rule.encrypted",
	293	"conditions": [
	294	{
	295	"kind": "event_match",
	296	"key": "type",
	297	"pattern": "m.room.encrypted",
	298	"_id": "_encrypted",
	299	}
	300	],
	301	"actions": ["notify"],
	302	},
	303	{
	304	"rule_id": "global/override/.m.rule.suppress_notices",
	305	"conditions": [
	306	{
	307	"kind": "event_match",
	308	"key": "type",
	309	"pattern": "m.room.message",
	310	"_id": "_suppress_notices_type",
	311	},
	312	{
	313	"kind": "event_match",
	314	"key": "content.msgtype",
	315	"pattern": "m.notice",
	316	"_id": "_suppress_notices",
	317	},
	318	],
	319	"actions": [],
	320	},
	321	{
	322	"rule_id": "global/underride/.m.rule.suppress_edits",
	323	"conditions": [
	324	{
	325	"kind": "event_match",
	326	"key": "m.relates_to.m.rel_type",
	327	"pattern": "m.replace",
	328	"_id": "_suppress_edits",
	329	}
	330	],
	331	"actions": [],
	332	},
	333	{
	334	"rule_id": "global/override/.m.rule.invite_for_me",
	335	"conditions": [
	336	{
	337	"kind": "event_match",
	338	"key": "type",
	339	"pattern": "m.room.member",
	340	"_id": "_member",
	341	},
	342	{
	343	"kind": "event_match",
	344	"key": "content.membership",
	345	"pattern": "invite",
	346	"_id": "_invite_member",
	347	},
	348	{"kind": "event_match", "key": "state_key", "pattern_type": "user_id"},
	349	],
	350	"actions": ["notify", {"set_tweak": "sound", "value": "default"}],
	351	},
	352	{
	353	"rule_id": "global/override/.m.rule.contains_display_name",
	354	"conditions": [{"kind": "contains_display_name"}],
	355	"actions": [
	356	"notify",
	357	{"set_tweak": "sound", "value": "default"},
	358	{"set_tweak": "highlight"},
	359	],
	360	},
	361	{
	362	"rule_id": "global/override/.m.rule.tombstone",
	363	"conditions": [
	364	{
	365	"kind": "event_match",
	366	"key": "type",
	367	"pattern": "m.room.tombstone",
	368	"_id": "_tombstone",
	369	},
	370	{
	371	"kind": "event_match",
	372	"key": "state_key",
	373	"pattern": "",
	374	"_id": "_tombstone_statekey",
	375	},
	376	],
	377	"actions": [
	378	"notify",
	379	{"set_tweak": "sound", "value": "default"},
	380	{"set_tweak": "highlight"},
	381	],
	382	},
	383	{
	384	"rule_id": "global/override/.m.rule.roomnotif",
	385	"conditions": [
	386	{
	387	"kind": "event_match",
	388	"key": "content.body",
	389	"pattern": "@room",
	390	"_id": "_roomnotif_content",
	391	},
	392	{
	393	"kind": "sender_notification_permission",
	394	"key": "room",
	395	"_id": "_roomnotif_pl",
	396	},
	397	],
	398	"actions": [
	399	"notify",
	400	{"set_tweak": "highlight"},
	401	{"set_tweak": "sound", "value": "default"},
	402	],
	403	},
	404	{
	405	"rule_id": "global/override/.m.rule.call",
275	406	"conditions": [
276	407	{
277	408	"kind": "event_match",

280	411	"_id": "_call",
281	412	}
282	413	],
	414	"actions": ["notify", {"set_tweak": "sound", "value": "ring"}],
	415	},
	416	]
	417
	418
	419	BASE_APPEND_UNDERRIDE_RULES = [
	420	{
	421	"rule_id": "global/underride/.m.rule.call",
	422	"conditions": [
	423	{
	424	"kind": "event_match",
	425	"key": "type",
	426	"pattern": "m.call.invite",
	427	"_id": "_call",
	428	}
	429	],
283	430	"actions": [
284	431	"notify",
285	432	{"set_tweak": "sound", "value": "ring"},

353	500	]
354	501
355	502
	503	NEW_APPEND_UNDERRIDE_RULES = [
	504	{
	505	"rule_id": "global/underride/.m.rule.room_one_to_one",
	506	"conditions": [
	507	{"kind": "room_member_count", "is": "2", "_id": "member_count"},
	508	{
	509	"kind": "event_match",
	510	"key": "content.body",
	511	"pattern": "*",
	512	"_id": "body",
	513	},
	514	],
	515	"actions": ["notify", {"set_tweak": "sound", "value": "default"}],
	516	},
	517	{
	518	"rule_id": "global/underride/.m.rule.message",
	519	"conditions": [
	520	{
	521	"kind": "event_match",
	522	"key": "content.body",
	523	"pattern": "*",
	524	"_id": "body",
	525	},
	526	],
	527	"actions": ["notify"],
	528	"enabled": False,
	529	},
	530	]
	531
	532
356	533	BASE_RULE_IDS = set()
357	534
358	535	for r in BASE_APPEND_CONTENT_RULES:

374	551	r["priority_class"] = PRIORITY_CLASS_MAP["underride"]
375	552	r["default"] = True
376	553	BASE_RULE_IDS.add(r["rule_id"])
	554
	555
	556	NEW_RULE_IDS = set()
	557
	558	for r in BASE_APPEND_CONTENT_RULES:
	559	r["priority_class"] = PRIORITY_CLASS_MAP["content"]
	560	r["default"] = True
	561	NEW_RULE_IDS.add(r["rule_id"])
	562
	563	for r in BASE_PREPEND_OVERRIDE_RULES:
	564	r["priority_class"] = PRIORITY_CLASS_MAP["override"]
	565	r["default"] = True
	566	NEW_RULE_IDS.add(r["rule_id"])
	567
	568	for r in NEW_APPEND_OVERRIDE_RULES:
	569	r["priority_class"] = PRIORITY_CLASS_MAP["override"]
	570	r["default"] = True
	571	NEW_RULE_IDS.add(r["rule_id"])
	572
	573	for r in NEW_APPEND_UNDERRIDE_RULES:
	574	r["priority_class"] = PRIORITY_CLASS_MAP["underride"]
	575	r["default"] = True
	576	NEW_RULE_IDS.add(r["rule_id"])

+27

-39

synapse/push/bulk_push_rule_evaluator.py less more

18	18
19	19	from prometheus_client import Counter
20	20
21		from twisted.internet import defer
22
23	21	from synapse.api.constants import EventTypes, Membership
24	22	from synapse.event_auth import get_user_power_level
25	23	from synapse.state import POWER_KEY

69	67	resizable=False,
70	68	)
71	69
72		@defer.inlineCallbacks
73		def _get_rules_for_event(self, event, context):
	70	async def _get_rules_for_event(self, event, context):
74	71	"""This gets the rules for all users in the room at the time of the event,
75	72	as well as the push rules for the invitee if the event is an invite.
76	73

78	75	dict of user_id -> push_rules
79	76	"""
80	77	room_id = event.room_id
81		rules_for_room = yield self._get_rules_for_room(room_id)
82
83		rules_by_user = yield rules_for_room.get_rules(event, context)
	78	rules_for_room = await self._get_rules_for_room(room_id)
	79
	80	rules_by_user = await rules_for_room.get_rules(event, context)
84	81
85	82	# if this event is an invite event, we may need to run rules for the user
86	83	# who's been invited, otherwise they won't get told they've been invited
87	84	if event.type == "m.room.member" and event.content["membership"] == "invite":
88	85	invited = event.state_key
89	86	if invited and self.hs.is_mine_id(invited):
90		has_pusher = yield self.store.user_has_pusher(invited)
	87	has_pusher = await self.store.user_has_pusher(invited)
91	88	if has_pusher:
92	89	rules_by_user = dict(rules_by_user)
93		rules_by_user[invited] = yield self.store.get_push_rules_for_user(
	90	rules_by_user[invited] = await self.store.get_push_rules_for_user(
94	91	invited
95	92	)
96	93

113	110	self.room_push_rule_cache_metrics,
114	111	)
115	112
116		@defer.inlineCallbacks
117		def _get_power_levels_and_sender_level(self, event, context):
118		prev_state_ids = yield context.get_prev_state_ids()
	113	async def _get_power_levels_and_sender_level(self, event, context):
	114	prev_state_ids = await context.get_prev_state_ids()
119	115	pl_event_id = prev_state_ids.get(POWER_KEY)
120	116	if pl_event_id:
121	117	# fastpath: if there's a power level event, that's all we need, and
122	118	# not having a power level event is an extreme edge case
123		pl_event = yield self.store.get_event(pl_event_id)
	119	pl_event = await self.store.get_event(pl_event_id)
124	120	auth_events = {POWER_KEY: pl_event}
125	121	else:
126		auth_events_ids = yield self.auth.compute_auth_events(
	122	auth_events_ids = self.auth.compute_auth_events(
127	123	event, prev_state_ids, for_verification=False
128	124	)
129		auth_events = yield self.store.get_events(auth_events_ids)
	125	auth_events = await self.store.get_events(auth_events_ids)
130	126	auth_events = {(e.type, e.state_key): e for e in auth_events.values()}
131	127
132	128	sender_level = get_user_power_level(event.sender, auth_events)

135	131
136	132	return pl_event.content if pl_event else {}, sender_level
137	133
138		@defer.inlineCallbacks
139		def action_for_event_by_user(self, event, context):
	134	async def action_for_event_by_user(self, event, context) -> None:
140	135	"""Given an event and context, evaluate the push rules and insert the
141	136	results into the event_push_actions_staging table.
142
143		Returns:
144		Deferred
145		"""
146		rules_by_user = yield self._get_rules_for_event(event, context)
	137	"""
	138	rules_by_user = await self._get_rules_for_event(event, context)
147	139	actions_by_user = {}
148	140
149		room_members = yield self.store.get_joined_users_from_context(event, context)
	141	room_members = await self.store.get_joined_users_from_context(event, context)
150	142
151	143	(
152	144	power_levels,
153	145	sender_power_level,
154		) = yield self._get_power_levels_and_sender_level(event, context)
	146	) = await self._get_power_levels_and_sender_level(event, context)
155	147
156	148	evaluator = PushRuleEvaluatorForEvent(
157	149	event, len(room_members), sender_power_level, power_levels

164	156	continue
165	157
166	158	if not event.is_state():
167		is_ignored = yield self.store.is_ignored_by(event.sender, uid)
	159	is_ignored = await self.store.is_ignored_by(event.sender, uid)
168	160	if is_ignored:
169	161	continue
170	162

196	188	# Mark in the DB staging area the push actions for users who should be
197	189	# notified for this event. (This will then get handled when we persist
198	190	# the event)
199		yield self.store.add_push_actions_to_staging(event.event_id, actions_by_user)
	191	await self.store.add_push_actions_to_staging(event.event_id, actions_by_user)
200	192
201	193
202	194	def _condition_checker(evaluator, conditions, uid, display_name, cache):

273	265	# to self around in the callback.
274	266	self.invalidate_all_cb = _Invalidation(rules_for_room_cache, room_id)
275	267
276		@defer.inlineCallbacks
277		def get_rules(self, event, context):
	268	async def get_rules(self, event, context):
278	269	"""Given an event context return the rules for all users who are
279	270	currently in the room.
280	271	"""

285	276	self.room_push_rule_cache_metrics.inc_hits()
286	277	return self.rules_by_user
287	278
288		with (yield self.linearizer.queue(())):
	279	with (await self.linearizer.queue(())):
289	280	if state_group and self.state_group == state_group:
290	281	logger.debug("Using cached rules for %r", self.room_id)
291	282	self.room_push_rule_cache_metrics.inc_hits()

303	294
304	295	push_rules_delta_state_cache_metric.inc_hits()
305	296	else:
306		current_state_ids = yield defer.ensureDeferred(
307		context.get_current_state_ids()
308		)
	297	current_state_ids = await context.get_current_state_ids()
309	298	push_rules_delta_state_cache_metric.inc_misses()
310	299
311	300	push_rules_state_size_counter.inc(len(current_state_ids))

352	341	# If we have some memebr events we haven't seen, look them up
353	342	# and fetch push rules for them if appropriate.
354	343	logger.debug("Found new member events %r", missing_member_event_ids)
355		yield self._update_rules_with_member_event_ids(
	344	await self._update_rules_with_member_event_ids(
356	345	ret_rules_by_user, missing_member_event_ids, state_group, event
357	346	)
358	347	else:

370	359	)
371	360	return ret_rules_by_user
372	361
373		@defer.inlineCallbacks
374		def _update_rules_with_member_event_ids(
	362	async def _update_rules_with_member_event_ids(
375	363	self, ret_rules_by_user, member_event_ids, state_group, event
376	364	):
377	365	"""Update the partially filled rules_by_user dict by fetching rules for

387	375	"""
388	376	sequence = self.sequence
389	377
390		rows = yield self.store.get_membership_from_event_ids(member_event_ids.values())
	378	rows = await self.store.get_membership_from_event_ids(member_event_ids.values())
391	379
392	380	members = {row["event_id"]: (row["user_id"], row["membership"]) for row in rows}
393	381

409	397
410	398	logger.debug("Joined: %r", interested_in_user_ids)
411	399
412		if_users_with_pushers = yield self.store.get_if_users_have_pushers(
	400	if_users_with_pushers = await self.store.get_if_users_have_pushers(
413	401	interested_in_user_ids, on_invalidate=self.invalidate_all_cb
414	402	)
415	403

419	407
420	408	logger.debug("With pushers: %r", user_ids)
421	409
422		users_with_receipts = yield self.store.get_users_with_read_receipts_in_room(
	410	users_with_receipts = await self.store.get_users_with_read_receipts_in_room(
423	411	self.room_id, on_invalidate=self.invalidate_all_cb
424	412	)
425	413

430	418	if uid in interested_in_user_ids:
431	419	user_ids.add(uid)
432	420
433		rules_by_user = yield self.store.bulk_get_push_rules(
	421	rules_by_user = await self.store.bulk_get_push_rules(
434	422	user_ids, on_invalidate=self.invalidate_all_cb
435	423	)
436	424

+26

-34

synapse/push/httppusher.py less more

16	16
17	17	from prometheus_client import Counter
18	18
19		from twisted.internet import defer
20	19	from twisted.internet.error import AlreadyCalled, AlreadyCancelled
21	20
22	21	from synapse.api.constants import EventTypes

127	126	# but currently that's the only type of receipt anyway...
128	127	run_as_background_process("http_pusher.on_new_receipts", self._update_badge)
129	128
130		@defer.inlineCallbacks
131		def _update_badge(self):
	129	async def _update_badge(self):
132	130	# XXX as per https://github.com/matrix-org/matrix-doc/issues/2627, this seems
133	131	# to be largely redundant. perhaps we can remove it.
134		badge = yield push_tools.get_badge_count(self.hs.get_datastore(), self.user_id)
135		yield self._send_badge(badge)
	132	badge = await push_tools.get_badge_count(self.hs.get_datastore(), self.user_id)
	133	await self._send_badge(badge)
136	134
137	135	def on_timer(self):
138	136	self._start_processing()

151	149
152	150	run_as_background_process("httppush.process", self._process)
153	151
154		@defer.inlineCallbacks
155		def _process(self):
	152	async def _process(self):
156	153	# we should never get here if we are already processing
157	154	assert not self._is_processing
158	155

163	160	while True:
164	161	starting_max_ordering = self.max_stream_ordering
165	162	try:
166		yield self._unsafe_process()
	163	await self._unsafe_process()
167	164	except Exception:
168	165	logger.exception("Exception processing notifs")
169	166	if self.max_stream_ordering == starting_max_ordering:

171	168	finally:
172	169	self._is_processing = False
173	170
174		@defer.inlineCallbacks
175		def _unsafe_process(self):
	171	async def _unsafe_process(self):
176	172	"""
177	173	Looks for unset notifications and dispatch them, in order
178	174	Never call this directly: use _process which will only allow this to

180	176	"""
181	177
182	178	fn = self.store.get_unread_push_actions_for_user_in_range_for_http
183		unprocessed = yield fn(
	179	unprocessed = await fn(
184	180	self.user_id, self.last_stream_ordering, self.max_stream_ordering
185	181	)
186	182

202	198	"app_display_name": self.app_display_name,
203	199	},
204	200	):
205		processed = yield self._process_one(push_action)
	201	processed = await self._process_one(push_action)
206	202
207	203	if processed:
208	204	http_push_processed_counter.inc()
209	205	self.backoff_delay = HttpPusher.INITIAL_BACKOFF_SEC
210	206	self.last_stream_ordering = push_action["stream_ordering"]
211		pusher_still_exists = yield self.store.update_pusher_last_stream_ordering_and_success(
	207	pusher_still_exists = await self.store.update_pusher_last_stream_ordering_and_success(
212	208	self.app_id,
213	209	self.pushkey,
214	210	self.user_id,

223	219
224	220	if self.failing_since:
225	221	self.failing_since = None
226		yield self.store.update_pusher_failing_since(
	222	await self.store.update_pusher_failing_since(
227	223	self.app_id, self.pushkey, self.user_id, self.failing_since
228	224	)
229	225	else:
230	226	http_push_failed_counter.inc()
231	227	if not self.failing_since:
232	228	self.failing_since = self.clock.time_msec()
233		yield self.store.update_pusher_failing_since(
	229	await self.store.update_pusher_failing_since(
234	230	self.app_id, self.pushkey, self.user_id, self.failing_since
235	231	)
236	232

249	245	)
250	246	self.backoff_delay = HttpPusher.INITIAL_BACKOFF_SEC
251	247	self.last_stream_ordering = push_action["stream_ordering"]
252		pusher_still_exists = yield self.store.update_pusher_last_stream_ordering(
	248	pusher_still_exists = await self.store.update_pusher_last_stream_ordering(
253	249	self.app_id,
254	250	self.pushkey,
255	251	self.user_id,

262	258	return
263	259
264	260	self.failing_since = None
265		yield self.store.update_pusher_failing_since(
	261	await self.store.update_pusher_failing_since(
266	262	self.app_id, self.pushkey, self.user_id, self.failing_since
267	263	)
268	264	else:

275	271	)
276	272	break
277	273
278		@defer.inlineCallbacks
279		def _process_one(self, push_action):
	274	async def _process_one(self, push_action):
280	275	if "notify" not in push_action["actions"]:
281	276	return True
282	277
283	278	tweaks = push_rule_evaluator.tweaks_for_actions(push_action["actions"])
284		badge = yield push_tools.get_badge_count(self.hs.get_datastore(), self.user_id)
285
286		event = yield self.store.get_event(push_action["event_id"], allow_none=True)
	279	badge = await push_tools.get_badge_count(self.hs.get_datastore(), self.user_id)
	280
	281	event = await self.store.get_event(push_action["event_id"], allow_none=True)
287	282	if event is None:
288	283	return True # It's been redacted
289		rejected = yield self.dispatch_push(event, tweaks, badge)
	284	rejected = await self.dispatch_push(event, tweaks, badge)
290	285	if rejected is False:
291	286	return False
292	287

300	295	)
301	296	else:
302	297	logger.info("Pushkey %s was rejected: removing", pk)
303		yield self.hs.remove_pusher(self.app_id, pk, self.user_id)
	298	await self.hs.remove_pusher(self.app_id, pk, self.user_id)
304	299	return True
305	300
306		@defer.inlineCallbacks
307		def _build_notification_dict(self, event, tweaks, badge):
	301	async def _build_notification_dict(self, event, tweaks, badge):
308	302	priority = "low"
309	303	if (
310	304	event.type == EventTypes.Encrypted

334	328	}
335	329	return d
336	330
337		ctx = yield push_tools.get_context_for_event(
	331	ctx = await push_tools.get_context_for_event(
338	332	self.storage, self.state_handler, event, self.user_id
339	333	)
340	334

376	370
377	371	return d
378	372
379		@defer.inlineCallbacks
380		def dispatch_push(self, event, tweaks, badge):
381		notification_dict = yield self._build_notification_dict(event, tweaks, badge)
	373	async def dispatch_push(self, event, tweaks, badge):
	374	notification_dict = await self._build_notification_dict(event, tweaks, badge)
382	375	if not notification_dict:
383	376	return []
384	377	try:
385		resp = yield self.http_client.post_json_get_json(
	378	resp = await self.http_client.post_json_get_json(
386	379	self.url, notification_dict
387	380	)
388	381	except Exception as e:

399	392	rejected = resp["rejected"]
400	393	return rejected
401	394
402		@defer.inlineCallbacks
403		def _send_badge(self, badge):
	395	async def _send_badge(self, badge):
404	396	"""
405	397	Args:
406	398	badge (int): number of unread messages

423	415	}
424	416	}
425	417	try:
426		yield self.http_client.post_json_get_json(self.url, d)
	418	await self.http_client.post_json_get_json(self.url, d)
427	419	http_badges_processed_counter.inc()
428	420	except Exception as e:
429	421	logger.warning(

+6

-9

synapse/push/presentable_names.py less more

15	15	import logging
16	16	import re
17	17
18		from twisted.internet import defer
19
20	18	from synapse.api.constants import EventTypes
21	19
22	20	logger = logging.getLogger(__name__)

28	26	ALL_ALONE = "Empty Room"
29	27
30	28
31		@defer.inlineCallbacks
32		def calculate_room_name(
	29	async def calculate_room_name(
33	30	store,
34	31	room_state_ids,
35	32	user_id,

52	49	"""
53	50	# does it have a name?
54	51	if (EventTypes.Name, "") in room_state_ids:
55		m_room_name = yield store.get_event(
	52	m_room_name = await store.get_event(
56	53	room_state_ids[(EventTypes.Name, "")], allow_none=True
57	54	)
58	55	if m_room_name and m_room_name.content and m_room_name.content["name"]:

60	57
61	58	# does it have a canonical alias?
62	59	if (EventTypes.CanonicalAlias, "") in room_state_ids:
63		canon_alias = yield store.get_event(
	60	canon_alias = await store.get_event(
64	61	room_state_ids[(EventTypes.CanonicalAlias, "")], allow_none=True
65	62	)
66	63	if (

80	77
81	78	my_member_event = None
82	79	if (EventTypes.Member, user_id) in room_state_ids:
83		my_member_event = yield store.get_event(
	80	my_member_event = await store.get_event(
84	81	room_state_ids[(EventTypes.Member, user_id)], allow_none=True
85	82	)
86	83

89	86	and my_member_event.content["membership"] == "invite"
90	87	):
91	88	if (EventTypes.Member, my_member_event.sender) in room_state_ids:
92		inviter_member_event = yield store.get_event(
	89	inviter_member_event = await store.get_event(
93	90	room_state_ids[(EventTypes.Member, my_member_event.sender)],
94	91	allow_none=True,
95	92	)

106	103	# we're going to have to generate a name based on who's in the room,
107	104	# so find out who is in the room that isn't the user.
108	105	if EventTypes.Member in room_state_bytype_ids:
109		member_events = yield store.get_events(
	106	member_events = await store.get_events(
110	107	list(room_state_bytype_ids[EventTypes.Member].values())
111	108	)
112	109	all_members = [

+9

-13

synapse/push/push_tools.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		from twisted.internet import defer
16
17	15	from synapse.push.presentable_names import calculate_room_name, name_from_member_event
18	16	from synapse.storage import Storage
19	17
20	18
21		@defer.inlineCallbacks
22		def get_badge_count(store, user_id):
23		invites = yield store.get_invited_rooms_for_local_user(user_id)
24		joins = yield store.get_rooms_for_user(user_id)
	19	async def get_badge_count(store, user_id):
	20	invites = await store.get_invited_rooms_for_local_user(user_id)
	21	joins = await store.get_rooms_for_user(user_id)
25	22
26		my_receipts_by_room = yield store.get_receipts_for_user(user_id, "m.read")
	23	my_receipts_by_room = await store.get_receipts_for_user(user_id, "m.read")
27	24
28	25	badge = len(invites)
29	26

31	28	if room_id in my_receipts_by_room:
32	29	last_unread_event_id = my_receipts_by_room[room_id]
33	30
34		notifs = yield (
	31	notifs = await (
35	32	store.get_unread_event_push_actions_by_room_for_user(
36	33	room_id, user_id, last_unread_event_id
37	34	)

42	39	return badge
43	40
44	41
45		@defer.inlineCallbacks
46		def get_context_for_event(storage: Storage, state_handler, ev, user_id):
	42	async def get_context_for_event(storage: Storage, state_handler, ev, user_id):
47	43	ctx = {}
48	44
49		room_state_ids = yield storage.state.get_state_ids_for_event(ev.event_id)
	45	room_state_ids = await storage.state.get_state_ids_for_event(ev.event_id)
50	46
51	47	# we no longer bother setting room_alias, and make room_name the
52	48	# human-readable name instead, be that m.room.name, an alias or
53	49	# a list of people in the room
54		name = yield calculate_room_name(
	50	name = await calculate_room_name(
55	51	storage.main, room_state_ids, user_id, fallback_to_single_member=False
56	52	)
57	53	if name:
58	54	ctx["name"] = name
59	55
60	56	sender_state_event_id = room_state_ids[("m.room.member", ev.sender)]
61		sender_state_event = yield storage.main.get_event(sender_state_event_id)
	57	sender_state_event = await storage.main.get_event(sender_state_event_id)
62	58	ctx["sender_display_name"] = name_from_member_event(sender_state_event)
63	59
64	60	return ctx

+32

-46

synapse/push/pusherpool.py less more

18	18
19	19	from prometheus_client import Gauge
20	20
21		from twisted.internet import defer
22
23	21	from synapse.metrics.background_process_metrics import run_as_background_process
24	22	from synapse.push import PusherConfigException
25	23	from synapse.push.emailpusher import EmailPusher

51	49	Note that it is expected that each pusher will have its own 'processing' loop which
52	50	will send out the notifications in the background, rather than blocking until the
53	51	notifications are sent; accordingly Pusher.on_started, Pusher.on_new_notifications and
54		Pusher.on_new_receipts are not expected to return deferreds.
	52	Pusher.on_new_receipts are not expected to return awaitables.
55	53	"""
56	54
57	55	def __init__(self, hs: "HomeServer"):

76	74	return
77	75	run_as_background_process("start_pushers", self._start_pushers)
78	76
79		@defer.inlineCallbacks
80		def add_pusher(
	77	async def add_pusher(
81	78	self,
82	79	user_id,
83	80	access_token,

93	90	"""Creates a new pusher and adds it to the pool
94	91
95	92	Returns:
96		Deferred[EmailPusher\|HttpPusher]
	93	EmailPusher\|HttpPusher
97	94	"""
98	95
99	96	time_now_msec = self.clock.time_msec()

123	120	# create the pusher setting last_stream_ordering to the current maximum
124	121	# stream ordering in event_push_actions, so it will process
125	122	# pushes from this point onwards.
126		last_stream_ordering = yield self.store.get_latest_push_action_stream_ordering()
127
128		yield self.store.add_pusher(
	123	last_stream_ordering = await self.store.get_latest_push_action_stream_ordering()
	124
	125	await self.store.add_pusher(
129	126	user_id=user_id,
130	127	access_token=access_token,
131	128	kind=kind,

139	136	last_stream_ordering=last_stream_ordering,
140	137	profile_tag=profile_tag,
141	138	)
142		pusher = yield self.start_pusher_by_id(app_id, pushkey, user_id)
	139	pusher = await self.start_pusher_by_id(app_id, pushkey, user_id)
143	140
144	141	return pusher
145	142
146		@defer.inlineCallbacks
147		def remove_pushers_by_app_id_and_pushkey_not_user(
	143	async def remove_pushers_by_app_id_and_pushkey_not_user(
148	144	self, app_id, pushkey, not_user_id
149	145	):
150		to_remove = yield self.store.get_pushers_by_app_id_and_pushkey(app_id, pushkey)
	146	to_remove = await self.store.get_pushers_by_app_id_and_pushkey(app_id, pushkey)
151	147	for p in to_remove:
152	148	if p["user_name"] != not_user_id:
153	149	logger.info(

156	152	pushkey,
157	153	p["user_name"],
158	154	)
159		yield self.remove_pusher(p["app_id"], p["pushkey"], p["user_name"])
160
161		@defer.inlineCallbacks
162		def remove_pushers_by_access_token(self, user_id, access_tokens):
	155	await self.remove_pusher(p["app_id"], p["pushkey"], p["user_name"])
	156
	157	async def remove_pushers_by_access_token(self, user_id, access_tokens):
163	158	"""Remove the pushers for a given user corresponding to a set of
164	159	access_tokens.
165	160

172	167	return
173	168
174	169	tokens = set(access_tokens)
175		for p in (yield self.store.get_pushers_by_user_id(user_id)):
	170	for p in await self.store.get_pushers_by_user_id(user_id):
176	171	if p["access_token"] in tokens:
177	172	logger.info(
178	173	"Removing pusher for app id %s, pushkey %s, user %s",

180	175	p["pushkey"],
181	176	p["user_name"],
182	177	)
183		yield self.remove_pusher(p["app_id"], p["pushkey"], p["user_name"])
184
185		@defer.inlineCallbacks
186		def on_new_notifications(self, min_stream_id, max_stream_id):
	178	await self.remove_pusher(p["app_id"], p["pushkey"], p["user_name"])
	179
	180	async def on_new_notifications(self, min_stream_id, max_stream_id):
187	181	if not self.pushers:
188	182	# nothing to do here.
189	183	return
190	184
191	185	try:
192		users_affected = yield self.store.get_push_action_users_in_range(
	186	users_affected = await self.store.get_push_action_users_in_range(
193	187	min_stream_id, max_stream_id
194	188	)
195	189

201	195	except Exception:
202	196	logger.exception("Exception in pusher on_new_notifications")
203	197
204		@defer.inlineCallbacks
205		def on_new_receipts(self, min_stream_id, max_stream_id, affected_room_ids):
	198	async def on_new_receipts(self, min_stream_id, max_stream_id, affected_room_ids):
206	199	if not self.pushers:
207	200	# nothing to do here.
208	201	return

210	203	try:
211	204	# Need to subtract 1 from the minimum because the lower bound here
212	205	# is not inclusive
213		users_affected = yield self.store.get_users_sent_receipts_between(
	206	users_affected = await self.store.get_users_sent_receipts_between(
214	207	min_stream_id - 1, max_stream_id
215	208	)
216	209

222	215	except Exception:
223	216	logger.exception("Exception in pusher on_new_receipts")
224	217
225		@defer.inlineCallbacks
226		def start_pusher_by_id(self, app_id, pushkey, user_id):
	218	async def start_pusher_by_id(self, app_id, pushkey, user_id):
227	219	"""Look up the details for the given pusher, and start it
228	220
229	221	Returns:
230		Deferred[EmailPusher\|HttpPusher\|None]: The pusher started, if any
	222	EmailPusher\|HttpPusher\|None: The pusher started, if any
231	223	"""
232	224	if not self._should_start_pushers:
233	225	return

235	227	if not self._pusher_shard_config.should_handle(self._instance_name, user_id):
236	228	return
237	229
238		resultlist = yield self.store.get_pushers_by_app_id_and_pushkey(app_id, pushkey)
	230	resultlist = await self.store.get_pushers_by_app_id_and_pushkey(app_id, pushkey)
239	231
240	232	pusher_dict = None
241	233	for r in resultlist:

244	236
245	237	pusher = None
246	238	if pusher_dict:
247		pusher = yield self._start_pusher(pusher_dict)
	239	pusher = await self._start_pusher(pusher_dict)
248	240
249	241	return pusher
250	242
251		@defer.inlineCallbacks
252		def _start_pushers(self):
	243	async def _start_pushers(self) -> None:
253	244	"""Start all the pushers
254
255		Returns:
256		Deferred
257		"""
258		pushers = yield self.store.get_all_pushers()
	245	"""
	246	pushers = await self.store.get_all_pushers()
259	247
260	248	# Stagger starting up the pushers so we don't completely drown the
261	249	# process on start up.
262		yield concurrently_execute(self._start_pusher, pushers, 10)
	250	await concurrently_execute(self._start_pusher, pushers, 10)
263	251
264	252	logger.info("Started pushers")
265	253
266		@defer.inlineCallbacks
267		def _start_pusher(self, pusherdict):
	254	async def _start_pusher(self, pusherdict):
268	255	"""Start the given pusher
269	256
270	257	Args:
271	258	pusherdict (dict): dict with the values pulled from the db table
272	259
273	260	Returns:
274		Deferred[EmailPusher\|HttpPusher]
	261	EmailPusher\|HttpPusher
275	262	"""
276	263	if not self._pusher_shard_config.should_handle(
277	264	self._instance_name, pusherdict["user_name"]

314	301	user_id = pusherdict["user_name"]
315	302	last_stream_ordering = pusherdict["last_stream_ordering"]
316	303	if last_stream_ordering:
317		have_notifs = yield self.store.get_if_maybe_push_in_range_for_user(
	304	have_notifs = await self.store.get_if_maybe_push_in_range_for_user(
318	305	user_id, last_stream_ordering
319	306	)
320	307	else:

326	313
327	314	return p
328	315
329		@defer.inlineCallbacks
330		def remove_pusher(self, app_id, pushkey, user_id):
	316	async def remove_pusher(self, app_id, pushkey, user_id):
331	317	appid_pushkey = "%s:%s" % (app_id, pushkey)
332	318
333	319	byuser = self.pushers.get(user_id, {})

339	325
340	326	synapse_pushers.labels(type(pusher).__name__, pusher.app_id).dec()
341	327
342		yield self.store.delete_pusher_by_app_id_pushkey_user_id(
	328	await self.store.delete_pusher_by_app_id_pushkey_user_id(
343	329	app_id, pushkey, user_id
344	330	)

+1

-2

synapse/python_dependencies.py less more

42	42	"jsonschema>=2.5.1",
43	43	"frozendict>=1",
44	44	"unpaddedbase64>=1.1.0",
45		"canonicaljson>=1.1.3",
	45	"canonicaljson>=1.2.0",
46	46	# we use the type definitions added in signedjson 1.1.
47	47	"signedjson>=1.1.0",
48	48	"pynacl>=1.2.1",

58	58	"pyyaml>=3.11",
59	59	"pyasn1>=0.1.9",
60	60	"pyasn1-modules>=0.0.7",
61		"daemonize>=2.3.1",
62	61	"bcrypt>=3.1.0",
63	62	"pillow>=4.3.0",
64	63	"sortedcontainers>=1.4.4",

+7

-11

synapse/replication/http/_base.py less more

18	18	import urllib
19	19	from inspect import signature
20	20	from typing import Dict, List, Tuple
21
22		from twisted.internet import defer
23	21
24	22	from synapse.api.errors import (
25	23	CodeMessageException,

100	98	assert self.METHOD in ("PUT", "POST", "GET")
101	99
102	100	@abc.abstractmethod
103		def _serialize_payload(**kwargs):
	101	async def _serialize_payload(**kwargs):
104	102	"""Static method that is called when creating a request.
105	103
106	104	Concrete implementations should have explicit parameters (rather than

109	107	argument list.
110	108
111	109	Returns:
112		Deferred[dict]\|dict: If POST/PUT request then dictionary must be
113		JSON serialisable, otherwise must be appropriate for adding as
114		query args.
	110	dict: If POST/PUT request then dictionary must be JSON serialisable,
	111	otherwise must be appropriate for adding as query args.
115	112	"""
116	113	return {}
117	114

143	140	instance_map = hs.config.worker.instance_map
144	141
145	142	@trace(opname="outgoing_replication_request")
146		@defer.inlineCallbacks
147		def send_request(instance_name="master", **kwargs):
	143	async def send_request(instance_name="master", **kwargs):
148	144	if instance_name == local_instance_name:
149	145	raise Exception("Trying to send HTTP request to self")
150	146	if instance_name == "master":

158	154	"Instance %r not in 'instance_map' config" % (instance_name,)
159	155	)
160	156
161		data = yield cls._serialize_payload(**kwargs)
	157	data = await cls._serialize_payload(**kwargs)
162	158
163	159	url_args = [
164	160	urllib.parse.quote(kwargs[name], safe="") for name in cls.PATH_ARGS

196	192	headers = {} # type: Dict[bytes, List[bytes]]
197	193	inject_active_span_byte_dict(headers, None, check_destination=False)
198	194	try:
199		result = yield request_func(uri, data, headers=headers)
	195	result = await request_func(uri, data, headers=headers)
200	196	break
201	197	except CodeMessageException as e:
202	198	if e.code != 504 or not cls.RETRY_ON_TIMEOUT:

206	202
207	203	# If we timed out we probably don't need to worry about backing
208	204	# off too much, but lets just wait a little anyway.
209		yield clock.sleep(1)
	205	await clock.sleep(1)
210	206	except HttpResponseException as e:
211	207	# We convert to SynapseError as we know that it was a SynapseError
212	208	# on the master process that we should send to the client. (And

+1

-1

synapse/replication/http/devices.py less more

59	59	self.clock = hs.get_clock()
60	60
61	61	@staticmethod
62		def _serialize_payload(user_id):
	62	async def _serialize_payload(user_id):
63	63	return {}
64	64
65	65	async def _handle_request(self, request, user_id):

+6

-9

synapse/replication/http/federation.py less more

14	14
15	15	import logging
16	16
17		from twisted.internet import defer
18
19	17	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
20	18	from synapse.events import make_event_from_dict
21	19	from synapse.events.snapshot import EventContext

66	64	self.federation_handler = hs.get_handlers().federation_handler
67	65
68	66	@staticmethod
69		@defer.inlineCallbacks
70		def _serialize_payload(store, event_and_contexts, backfilled):
	67	async def _serialize_payload(store, event_and_contexts, backfilled):
71	68	"""
72	69	Args:
73	70	store

77	74	"""
78	75	event_payloads = []
79	76	for event, context in event_and_contexts:
80		serialized_context = yield context.serialize(event, store)
	77	serialized_context = await context.serialize(event, store)
81	78
82	79	event_payloads.append(
83	80	{

153	150	self.registry = hs.get_federation_registry()
154	151
155	152	@staticmethod
156		def _serialize_payload(edu_type, origin, content):
	153	async def _serialize_payload(edu_type, origin, content):
157	154	return {"origin": origin, "content": content}
158	155
159	156	async def _handle_request(self, request, edu_type):

196	193	self.registry = hs.get_federation_registry()
197	194
198	195	@staticmethod
199		def _serialize_payload(query_type, args):
	196	async def _serialize_payload(query_type, args):
200	197	"""
201	198	Args:
202	199	query_type (str)

237	234	self.store = hs.get_datastore()
238	235
239	236	@staticmethod
240		def _serialize_payload(room_id, args):
	237	async def _serialize_payload(room_id, args):
241	238	"""
242	239	Args:
243	240	room_id (str)

272	269	self.store = hs.get_datastore()
273	270
274	271	@staticmethod
275		def _serialize_payload(room_id, room_version):
	272	async def _serialize_payload(room_id, room_version):
276	273	return {"room_version": room_version.identifier}
277	274
278	275	async def _handle_request(self, request, room_id):

+1

-1

synapse/replication/http/login.py less more

35	35	self.registration_handler = hs.get_registration_handler()
36	36
37	37	@staticmethod
38		def _serialize_payload(user_id, device_id, initial_display_name, is_guest):
	38	async def _serialize_payload(user_id, device_id, initial_display_name, is_guest):
39	39	"""
40	40	Args:
41	41	device_id (str\|None): Device ID to use, if None a new one is

+5

-3

synapse/replication/http/membership.py less more

51	51	self.clock = hs.get_clock()
52	52
53	53	@staticmethod
54		def _serialize_payload(requester, room_id, user_id, remote_room_hosts, content):
	54	async def _serialize_payload(
	55	requester, room_id, user_id, remote_room_hosts, content
	56	):
55	57	"""
56	58	Args:
57	59	requester(Requester)

111	113	self.member_handler = hs.get_room_member_handler()
112	114
113	115	@staticmethod
114		def _serialize_payload( # type: ignore
	116	async def _serialize_payload( # type: ignore
115	117	invite_event_id: str,
116	118	txn_id: Optional[str],
117	119	requester: Requester,

173	175	self.distributor = hs.get_distributor()
174	176
175	177	@staticmethod
176		def _serialize_payload(room_id, user_id, change):
	178	async def _serialize_payload(room_id, user_id, change):
177	179	"""
178	180	Args:
179	181	room_id (str)

+2

-2

synapse/replication/http/presence.py less more

49	49	self._presence_handler = hs.get_presence_handler()
50	50
51	51	@staticmethod
52		def _serialize_payload(user_id):
	52	async def _serialize_payload(user_id):
53	53	return {}
54	54
55	55	async def _handle_request(self, request, user_id):

91	91	self._presence_handler = hs.get_presence_handler()
92	92
93	93	@staticmethod
94		def _serialize_payload(user_id, state, ignore_status_msg=False):
	94	async def _serialize_payload(user_id, state, ignore_status_msg=False):
95	95	return {
96	96	"state": state,
97	97	"ignore_status_msg": ignore_status_msg,

+2

-2

synapse/replication/http/register.py less more

33	33	self.registration_handler = hs.get_registration_handler()
34	34
35	35	@staticmethod
36		def _serialize_payload(
	36	async def _serialize_payload(
37	37	user_id,
38	38	password_hash,
39	39	was_guest,

104	104	self.registration_handler = hs.get_registration_handler()
105	105
106	106	@staticmethod
107		def _serialize_payload(user_id, auth_result, access_token):
	107	async def _serialize_payload(user_id, auth_result, access_token):
108	108	"""
109	109	Args:
110	110	user_id (str): The user ID that consented

+2

-5

synapse/replication/http/send_event.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
16
17		from twisted.internet import defer
18	16
19	17	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
20	18	from synapse.events import make_event_from_dict

61	59	self.clock = hs.get_clock()
62	60
63	61	@staticmethod
64		@defer.inlineCallbacks
65		def _serialize_payload(
	62	async def _serialize_payload(
66	63	event_id, store, event, context, requester, ratelimit, extra_users
67	64	):
68	65	"""

76	73	extra_users (list(UserID)): Any extra users to notify about event
77	74	"""
78	75
79		serialized_context = yield context.serialize(event, store)
	76	serialized_context = await context.serialize(event, store)
80	77
81	78	payload = {
82	79	"event": event.get_pdu_json(),

+1

-1

synapse/replication/http/streams.py less more

53	53	self.streams = hs.get_replication_streams()
54	54
55	55	@staticmethod
56		def _serialize_payload(stream_name, from_token, upto_token):
	56	async def _serialize_payload(stream_name, from_token, upto_token):
57	57	return {"from_token": from_token, "upto_token": upto_token}
58	58
59	59	async def _handle_request(self, request, stream_name):

+3

-3

synapse/replication/slave/storage/_base.py less more

15	15	import logging
16	16	from typing import Optional
17	17
18		from synapse.storage.data_stores.main.cache import CacheInvalidationWorkerStore
19		from synapse.storage.database import Database
	18	from synapse.storage.database import DatabasePool
	19	from synapse.storage.databases.main.cache import CacheInvalidationWorkerStore
20	20	from synapse.storage.engines import PostgresEngine
21	21	from synapse.storage.util.id_generators import MultiWriterIdGenerator
22	22

24	24
25	25
26	26	class BaseSlavedStore(CacheInvalidationWorkerStore):
27		def __init__(self, database: Database, db_conn, hs):
	27	def __init__(self, database: DatabasePool, db_conn, hs):
28	28	super(BaseSlavedStore, self).__init__(database, db_conn, hs)
29	29	if isinstance(self.database_engine, PostgresEngine):
30	30	self._cache_id_gen = MultiWriterIdGenerator(

+4

-4

synapse/replication/slave/storage/account_data.py less more

16	16	from synapse.replication.slave.storage._base import BaseSlavedStore
17	17	from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
18	18	from synapse.replication.tcp.streams import AccountDataStream, TagAccountDataStream
19		from synapse.storage.data_stores.main.account_data import AccountDataWorkerStore
20		from synapse.storage.data_stores.main.tags import TagsWorkerStore
21		from synapse.storage.database import Database
	19	from synapse.storage.database import DatabasePool
	20	from synapse.storage.databases.main.account_data import AccountDataWorkerStore
	21	from synapse.storage.databases.main.tags import TagsWorkerStore
22	22
23	23
24	24	class SlavedAccountDataStore(TagsWorkerStore, AccountDataWorkerStore, BaseSlavedStore):
25		def __init__(self, database: Database, db_conn, hs):
	25	def __init__(self, database: DatabasePool, db_conn, hs):
26	26	self._account_data_id_gen = SlavedIdTracker(
27	27	db_conn,
28	28	"account_data",

+1

-1

synapse/replication/slave/storage/appservice.py less more

13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
15	15
16		from synapse.storage.data_stores.main.appservice import (
	16	from synapse.storage.databases.main.appservice import (
17	17	ApplicationServiceTransactionWorkerStore,
18	18	ApplicationServiceWorkerStore,
19	19	)

+4

-4

synapse/replication/slave/storage/client_ips.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		from synapse.storage.data_stores.main.client_ips import LAST_SEEN_GRANULARITY
16		from synapse.storage.database import Database
	15	from synapse.storage.database import DatabasePool
	16	from synapse.storage.databases.main.client_ips import LAST_SEEN_GRANULARITY
17	17	from synapse.util.caches.descriptors import Cache
18	18
19	19	from ._base import BaseSlavedStore
20	20
21	21
22	22	class SlavedClientIpStore(BaseSlavedStore):
23		def __init__(self, database: Database, db_conn, hs):
	23	def __init__(self, database: DatabasePool, db_conn, hs):
24	24	super(SlavedClientIpStore, self).__init__(database, db_conn, hs)
25	25
26	26	self.client_ip_last_seen = Cache(
27	27	name="client_ip_last_seen", keylen=4, max_entries=50000
28	28	)
29	29
30		def insert_client_ip(self, user_id, access_token, ip, user_agent, device_id):
	30	async def insert_client_ip(self, user_id, access_token, ip, user_agent, device_id):
31	31	now = int(self._clock.time_msec())
32	32	key = (user_id, access_token, ip)
33	33

+3

-3

synapse/replication/slave/storage/deviceinbox.py less more

15	15	from synapse.replication.slave.storage._base import BaseSlavedStore
16	16	from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
17	17	from synapse.replication.tcp.streams import ToDeviceStream
18		from synapse.storage.data_stores.main.deviceinbox import DeviceInboxWorkerStore
19		from synapse.storage.database import Database
	18	from synapse.storage.database import DatabasePool
	19	from synapse.storage.databases.main.deviceinbox import DeviceInboxWorkerStore
20	20	from synapse.util.caches.expiringcache import ExpiringCache
21	21	from synapse.util.caches.stream_change_cache import StreamChangeCache
22	22
23	23
24	24	class SlavedDeviceInboxStore(DeviceInboxWorkerStore, BaseSlavedStore):
25		def __init__(self, database: Database, db_conn, hs):
	25	def __init__(self, database: DatabasePool, db_conn, hs):
26	26	super(SlavedDeviceInboxStore, self).__init__(database, db_conn, hs)
27	27	self._device_inbox_id_gen = SlavedIdTracker(
28	28	db_conn, "device_inbox", "stream_id"

+4

-4

synapse/replication/slave/storage/devices.py less more

15	15	from synapse.replication.slave.storage._base import BaseSlavedStore
16	16	from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
17	17	from synapse.replication.tcp.streams._base import DeviceListsStream, UserSignatureStream
18		from synapse.storage.data_stores.main.devices import DeviceWorkerStore
19		from synapse.storage.data_stores.main.end_to_end_keys import EndToEndKeyWorkerStore
20		from synapse.storage.database import Database
	18	from synapse.storage.database import DatabasePool
	19	from synapse.storage.databases.main.devices import DeviceWorkerStore
	20	from synapse.storage.databases.main.end_to_end_keys import EndToEndKeyWorkerStore
21	21	from synapse.util.caches.stream_change_cache import StreamChangeCache
22	22
23	23
24	24	class SlavedDeviceStore(EndToEndKeyWorkerStore, DeviceWorkerStore, BaseSlavedStore):
25		def __init__(self, database: Database, db_conn, hs):
	25	def __init__(self, database: DatabasePool, db_conn, hs):
26	26	super(SlavedDeviceStore, self).__init__(database, db_conn, hs)
27	27
28	28	self.hs = hs

+1

-1

synapse/replication/slave/storage/directory.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		from synapse.storage.data_stores.main.directory import DirectoryWorkerStore
	15	from synapse.storage.databases.main.directory import DirectoryWorkerStore
16	16
17	17	from ._base import BaseSlavedStore
18	18

+12

-12

synapse/replication/slave/storage/events.py less more

14	14	# limitations under the License.
15	15	import logging
16	16
17		from synapse.storage.data_stores.main.event_federation import EventFederationWorkerStore
18		from synapse.storage.data_stores.main.event_push_actions import (
	17	from synapse.storage.database import DatabasePool
	18	from synapse.storage.databases.main.event_federation import EventFederationWorkerStore
	19	from synapse.storage.databases.main.event_push_actions import (
19	20	EventPushActionsWorkerStore,
20	21	)
21		from synapse.storage.data_stores.main.events_worker import EventsWorkerStore
22		from synapse.storage.data_stores.main.relations import RelationsWorkerStore
23		from synapse.storage.data_stores.main.roommember import RoomMemberWorkerStore
24		from synapse.storage.data_stores.main.signatures import SignatureWorkerStore
25		from synapse.storage.data_stores.main.state import StateGroupWorkerStore
26		from synapse.storage.data_stores.main.stream import StreamWorkerStore
27		from synapse.storage.data_stores.main.user_erasure_store import UserErasureWorkerStore
28		from synapse.storage.database import Database
	22	from synapse.storage.databases.main.events_worker import EventsWorkerStore
	23	from synapse.storage.databases.main.relations import RelationsWorkerStore
	24	from synapse.storage.databases.main.roommember import RoomMemberWorkerStore
	25	from synapse.storage.databases.main.signatures import SignatureWorkerStore
	26	from synapse.storage.databases.main.state import StateGroupWorkerStore
	27	from synapse.storage.databases.main.stream import StreamWorkerStore
	28	from synapse.storage.databases.main.user_erasure_store import UserErasureWorkerStore
29	29	from synapse.util.caches.stream_change_cache import StreamChangeCache
30	30
31	31	from ._base import BaseSlavedStore

54	54	RelationsWorkerStore,
55	55	BaseSlavedStore,
56	56	):
57		def __init__(self, database: Database, db_conn, hs):
	57	def __init__(self, database: DatabasePool, db_conn, hs):
58	58	super(SlavedEventStore, self).__init__(database, db_conn, hs)
59	59
60	60	events_max = self._stream_id_gen.get_current_token()
61		curr_state_delta_prefill, min_curr_state_delta_id = self.db.get_cache_dict(
	61	curr_state_delta_prefill, min_curr_state_delta_id = self.db_pool.get_cache_dict(
62	62	db_conn,
63	63	"current_state_delta_stream",
64	64	entity_column="room_id",

+3

-3

synapse/replication/slave/storage/filtering.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		from synapse.storage.data_stores.main.filtering import FilteringStore
16		from synapse.storage.database import Database
	15	from synapse.storage.database import DatabasePool
	16	from synapse.storage.databases.main.filtering import FilteringStore
17	17
18	18	from ._base import BaseSlavedStore
19	19
20	20
21	21	class SlavedFilteringStore(BaseSlavedStore):
22		def __init__(self, database: Database, db_conn, hs):
	22	def __init__(self, database: DatabasePool, db_conn, hs):
23	23	super(SlavedFilteringStore, self).__init__(database, db_conn, hs)
24	24
25	25	# Filters are immutable so this cache doesn't need to be expired

+3

-3

synapse/replication/slave/storage/groups.py less more

15	15	from synapse.replication.slave.storage._base import BaseSlavedStore
16	16	from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
17	17	from synapse.replication.tcp.streams import GroupServerStream
18		from synapse.storage.data_stores.main.group_server import GroupServerWorkerStore
19		from synapse.storage.database import Database
	18	from synapse.storage.database import DatabasePool
	19	from synapse.storage.databases.main.group_server import GroupServerWorkerStore
20	20	from synapse.util.caches.stream_change_cache import StreamChangeCache
21	21
22	22
23	23	class SlavedGroupServerStore(GroupServerWorkerStore, BaseSlavedStore):
24		def __init__(self, database: Database, db_conn, hs):
	24	def __init__(self, database: DatabasePool, db_conn, hs):
25	25	super(SlavedGroupServerStore, self).__init__(database, db_conn, hs)
26	26
27	27	self.hs = hs

+1

-1

synapse/replication/slave/storage/keys.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		from synapse.storage.data_stores.main.keys import KeyStore
	15	from synapse.storage.databases.main.keys import KeyStore
16	16
17	17	# KeyStore isn't really safe to use from a worker, but for now we do so and hope that
18	18	# the races it creates aren't too bad.

+3

-3

synapse/replication/slave/storage/presence.py less more

14	14
15	15	from synapse.replication.tcp.streams import PresenceStream
16	16	from synapse.storage import DataStore
17		from synapse.storage.data_stores.main.presence import PresenceStore
18		from synapse.storage.database import Database
	17	from synapse.storage.database import DatabasePool
	18	from synapse.storage.databases.main.presence import PresenceStore
19	19	from synapse.util.caches.stream_change_cache import StreamChangeCache
20	20
21	21	from ._base import BaseSlavedStore

23	23
24	24
25	25	class SlavedPresenceStore(BaseSlavedStore):
26		def __init__(self, database: Database, db_conn, hs):
	26	def __init__(self, database: DatabasePool, db_conn, hs):
27	27	super(SlavedPresenceStore, self).__init__(database, db_conn, hs)
28	28	self._presence_id_gen = SlavedIdTracker(db_conn, "presence_stream", "stream_id")
29	29

+1

-1

synapse/replication/slave/storage/profile.py less more

13	13	# limitations under the License.
14	14
15	15	from synapse.replication.slave.storage._base import BaseSlavedStore
16		from synapse.storage.data_stores.main.profile import ProfileWorkerStore
	16	from synapse.storage.databases.main.profile import ProfileWorkerStore
17	17
18	18
19	19	class SlavedProfileStore(ProfileWorkerStore, BaseSlavedStore):

+1

-1

synapse/replication/slave/storage/push_rule.py less more

14	14	# limitations under the License.
15	15
16	16	from synapse.replication.tcp.streams import PushRulesStream
17		from synapse.storage.data_stores.main.push_rule import PushRulesWorkerStore
	17	from synapse.storage.databases.main.push_rule import PushRulesWorkerStore
18	18
19	19	from .events import SlavedEventStore
20	20

+3

-3

synapse/replication/slave/storage/pushers.py less more

14	14	# limitations under the License.
15	15
16	16	from synapse.replication.tcp.streams import PushersStream
17		from synapse.storage.data_stores.main.pusher import PusherWorkerStore
18		from synapse.storage.database import Database
	17	from synapse.storage.database import DatabasePool
	18	from synapse.storage.databases.main.pusher import PusherWorkerStore
19	19
20	20	from ._base import BaseSlavedStore
21	21	from ._slaved_id_tracker import SlavedIdTracker
22	22
23	23
24	24	class SlavedPusherStore(PusherWorkerStore, BaseSlavedStore):
25		def __init__(self, database: Database, db_conn, hs):
	25	def __init__(self, database: DatabasePool, db_conn, hs):
26	26	super(SlavedPusherStore, self).__init__(database, db_conn, hs)
27	27	self._pushers_id_gen = SlavedIdTracker(
28	28	db_conn, "pushers", "id", extra_tables=[("deleted_pushers", "stream_id")]

+3

-3

synapse/replication/slave/storage/receipts.py less more

14	14	# limitations under the License.
15	15
16	16	from synapse.replication.tcp.streams import ReceiptsStream
17		from synapse.storage.data_stores.main.receipts import ReceiptsWorkerStore
18		from synapse.storage.database import Database
	17	from synapse.storage.database import DatabasePool
	18	from synapse.storage.databases.main.receipts import ReceiptsWorkerStore
19	19
20	20	from ._base import BaseSlavedStore
21	21	from ._slaved_id_tracker import SlavedIdTracker
22	22
23	23
24	24	class SlavedReceiptsStore(ReceiptsWorkerStore, BaseSlavedStore):
25		def __init__(self, database: Database, db_conn, hs):
	25	def __init__(self, database: DatabasePool, db_conn, hs):
26	26	# We instantiate this first as the ReceiptsWorkerStore constructor
27	27	# needs to be able to call get_max_receipt_stream_id
28	28	self._receipts_id_gen = SlavedIdTracker(

+1

-1

synapse/replication/slave/storage/registration.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		from synapse.storage.data_stores.main.registration import RegistrationWorkerStore
	15	from synapse.storage.databases.main.registration import RegistrationWorkerStore
16	16
17	17	from ._base import BaseSlavedStore
18	18

+3

-3

synapse/replication/slave/storage/room.py less more

13	13	# limitations under the License.
14	14
15	15	from synapse.replication.tcp.streams import PublicRoomsStream
16		from synapse.storage.data_stores.main.room import RoomWorkerStore
17		from synapse.storage.database import Database
	16	from synapse.storage.database import DatabasePool
	17	from synapse.storage.databases.main.room import RoomWorkerStore
18	18
19	19	from ._base import BaseSlavedStore
20	20	from ._slaved_id_tracker import SlavedIdTracker
21	21
22	22
23	23	class RoomStore(RoomWorkerStore, BaseSlavedStore):
24		def __init__(self, database: Database, db_conn, hs):
	24	def __init__(self, database: DatabasePool, db_conn, hs):
25	25	super(RoomStore, self).__init__(database, db_conn, hs)
26	26	self._public_room_id_gen = SlavedIdTracker(
27	27	db_conn, "public_room_list_stream", "stream_id"

+1

-1

synapse/replication/slave/storage/transactions.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		from synapse.storage.data_stores.main.transactions import TransactionStore
	15	from synapse.storage.databases.main.transactions import TransactionStore
16	16
17	17	from ._base import BaseSlavedStore
18	18

+3

-2

synapse/replication/tcp/commands.py less more

17	17	allowed to be sent by which side.
18	18	"""
19	19	import abc
20		import json
21	20	import logging
22	21	from typing import Tuple, Type
23	22
24		_json_encoder = json.JSONEncoder()
	23	from canonicaljson import json
	24
	25	from synapse.util import json_encoder as _json_encoder
25	26
26	27	logger = logging.getLogger(__name__)
27	28

+12

-5

synapse/res/templates/saml_error.html less more

1	1	<html lang="en">
2	2	<head>
3	3	<meta charset="UTF-8">
4		<title>SSO error</title>
	4	<title>SSO login error</title>
5	5	</head>
6	6	<body>
7		<p>Oops! Something went wrong during authentication<span id="errormsg"></span>.</p>
	7	{# a 403 means we have actively rejected their login #}
	8	{% if code == 403 %}
	9	<p>You are not allowed to log in here.</p>
	10	{% else %}
	11	<p>
	12	There was an error during authentication:
	13	</p>
	14	<div id="errormsg" style="margin:20px 80px">{{ msg }}</div>
8	15	<p>
9	16	If you are seeing this page after clicking a link sent to you via email, make
10	17	sure you only click the confirmation link once, and that you open the

36	43	// to print one.
37	44	let errorDesc = new URLSearchParams(searchStr).get("error_description")
38	45	if (errorDesc) {
39
40		document.getElementById("errormsg").innerText = ` ("${errorDesc}")`;
	46	document.getElementById("errormsg").innerText = errorDesc;
41	47	}
42	48	</script>
	49	{% endif %}
43	50	</body>
44		</html>⏎
	51	</html>

+11

-2

synapse/rest/admin/rooms.py less more

30	30	assert_user_is_admin,
31	31	historical_admin_path_patterns,
32	32	)
33		from synapse.storage.data_stores.main.room import RoomSortOrder
	33	from synapse.storage.databases.main.room import RoomSortOrder
34	34	from synapse.types import RoomAlias, RoomID, UserID, create_requester
35	35
36	36	logger = logging.getLogger(__name__)

102	102	Codes.BAD_JSON,
103	103	)
104	104
	105	purge = content.get("purge", True)
	106	if not isinstance(purge, bool):
	107	raise SynapseError(
	108	HTTPStatus.BAD_REQUEST,
	109	"Param 'purge' must be a boolean, if given",
	110	Codes.BAD_JSON,
	111	)
	112
105	113	ret = await self.room_shutdown_handler.shutdown_room(
106	114	room_id=room_id,
107	115	new_room_user_id=content.get("new_room_user_id"),

112	120	)
113	121
114	122	# Purge room
115		await self.pagination_handler.purge_room(room_id)
	123	if purge:
	124	await self.pagination_handler.purge_room(room_id)
116	125
117	126	return (200, ret)
118	127

+1

-1

synapse/rest/client/v1/directory.py less more

88	88	dir_handler = self.handlers.directory_handler
89	89
90	90	try:
91		service = await self.auth.get_appservice_by_req(request)
	91	service = self.auth.get_appservice_by_req(request)
92	92	room_alias = RoomAlias.from_string(room_alias)
93	93	await dir_handler.delete_appservice_association(service, room_alias)
94	94	logger.info(

+9

-2

synapse/rest/client/v1/push_rule.py less more

24	24	parse_json_value_from_request,
25	25	parse_string,
26	26	)
27		from synapse.push.baserules import BASE_RULE_IDS
	27	from synapse.push.baserules import BASE_RULE_IDS, NEW_RULE_IDS
28	28	from synapse.push.clientformat import format_push_rules_for_user
29	29	from synapse.push.rulekinds import PRIORITY_CLASS_MAP
30	30	from synapse.rest.client.v2_alpha._base import client_patterns

43	43	self.store = hs.get_datastore()
44	44	self.notifier = hs.get_notifier()
45	45	self._is_worker = hs.config.worker_app is not None
	46
	47	self._users_new_default_push_rules = hs.config.users_new_default_push_rules
46	48
47	49	async def on_PUT(self, request, path):
48	50	if self._is_worker:

178	180	rule_id = spec["rule_id"]
179	181	is_default_rule = rule_id.startswith(".")
180	182	if is_default_rule:
181		if namespaced_rule_id not in BASE_RULE_IDS:
	183	if user_id in self._users_new_default_push_rules:
	184	rule_ids = NEW_RULE_IDS
	185	else:
	186	rule_ids = BASE_RULE_IDS
	187
	188	if namespaced_rule_id not in rule_ids:
182	189	raise SynapseError(404, "Unknown rule %r" % (namespaced_rule_id,))
183	190	return self.store.set_push_rule_actions(
184	191	user_id, namespaced_rule_id, actions, is_default_rule

+1

-1

synapse/rest/client/v1/room.py less more

443	443
444	444	async def on_GET(self, request, room_id):
445	445	# TODO support Pagination stream API (limit/tokens)
446		requester = await self.auth.get_user_by_req(request)
	446	requester = await self.auth.get_user_by_req(request, allow_guest=True)
447	447	handler = self.message_handler
448	448
449	449	# request the state as of a given event, as identified by a stream token,

+60

-26

synapse/rest/client/v2_alpha/account.py less more

17	17	from http import HTTPStatus
18	18
19	19	from synapse.api.constants import LoginType
20		from synapse.api.errors import Codes, SynapseError, ThreepidValidationError
	20	from synapse.api.errors import (
	21	Codes,
	22	InteractiveAuthIncompleteError,
	23	SynapseError,
	24	ThreepidValidationError,
	25	)
21	26	from synapse.config.emailconfig import ThreepidBehaviour
22	27	from synapse.http.server import finish_request, respond_with_html
23	28	from synapse.http.servlet import (

238	243
239	244	# we do basic sanity checks here because the auth layer will store these
240	245	# in sessions. Pull out the new password provided to us.
241		if "new_password" in body:
242		new_password = body.pop("new_password")
	246	new_password = body.pop("new_password", None)
	247	if new_password is not None:
243	248	if not isinstance(new_password, str) or len(new_password) > 512:
244	249	raise SynapseError(400, "Invalid password")
245	250	self.password_policy_handler.validate_password(new_password)
246
247		# If the password is valid, hash it and store it back on the body.
248		# This ensures that only the hashed password is handled everywhere.
249		if "new_password_hash" in body:
250		raise SynapseError(400, "Unexpected property: new_password_hash")
251		body["new_password_hash"] = await self.auth_handler.hash(new_password)
252	251
253	252	# there are two possibilities here. Either the user does not have an
254	253	# access token, and needs to do a password reset; or they have one and

262	261
263	262	if self.auth.has_access_token(request):
264	263	requester = await self.auth.get_user_by_req(request)
265		params = await self.auth_handler.validate_user_via_ui_auth(
266		requester,
267		request,
268		body,
269		self.hs.get_ip_from_request(request),
270		"modify your account password",
271		)
	264	try:
	265	params, session_id = await self.auth_handler.validate_user_via_ui_auth(
	266	requester,
	267	request,
	268	body,
	269	self.hs.get_ip_from_request(request),
	270	"modify your account password",
	271	)
	272	except InteractiveAuthIncompleteError as e:
	273	# The user needs to provide more steps to complete auth, but
	274	# they're not required to provide the password again.
	275	#
	276	# If a password is available now, hash the provided password and
	277	# store it for later.
	278	if new_password:
	279	password_hash = await self.auth_handler.hash(new_password)
	280	await self.auth_handler.set_session_data(
	281	e.session_id, "password_hash", password_hash
	282	)
	283	raise
272	284	user_id = requester.user.to_string()
273	285	else:
274	286	requester = None
275		result, params, _ = await self.auth_handler.check_auth(
276		[[LoginType.EMAIL_IDENTITY]],
277		request,
278		body,
279		self.hs.get_ip_from_request(request),
280		"modify your account password",
281		)
	287	try:
	288	result, params, session_id = await self.auth_handler.check_ui_auth(
	289	[[LoginType.EMAIL_IDENTITY]],
	290	request,
	291	body,
	292	self.hs.get_ip_from_request(request),
	293	"modify your account password",
	294	)
	295	except InteractiveAuthIncompleteError as e:
	296	# The user needs to provide more steps to complete auth, but
	297	# they're not required to provide the password again.
	298	#
	299	# If a password is available now, hash the provided password and
	300	# store it for later.
	301	if new_password:
	302	password_hash = await self.auth_handler.hash(new_password)
	303	await self.auth_handler.set_session_data(
	304	e.session_id, "password_hash", password_hash
	305	)
	306	raise
282	307
283	308	if LoginType.EMAIL_IDENTITY in result:
284	309	threepid = result[LoginType.EMAIL_IDENTITY]

303	328	logger.error("Auth succeeded but no known type! %r", result.keys())
304	329	raise SynapseError(500, "", Codes.UNKNOWN)
305	330
306		assert_params_in_dict(params, ["new_password_hash"])
307		new_password_hash = params["new_password_hash"]
	331	# If we have a password in this request, prefer it. Otherwise, there
	332	# must be a password hash from an earlier request.
	333	if new_password:
	334	password_hash = await self.auth_handler.hash(new_password)
	335	else:
	336	password_hash = await self.auth_handler.get_session_data(
	337	session_id, "password_hash", None
	338	)
	339	if not password_hash:
	340	raise SynapseError(400, "Missing params: password", Codes.MISSING_PARAM)
	341
308	342	logout_devices = params.get("logout_devices", True)
309	343
310	344	await self._set_password_handler.set_password(
311		user_id, new_password_hash, logout_devices, requester
	345	user_id, password_hash, logout_devices, requester
312	346	)
313	347
314	348	return 200, {}

+72

-42

synapse/rest/client/v2_alpha/register.py less more

23	23	from synapse.api.constants import LoginType
24	24	from synapse.api.errors import (
25	25	Codes,
	26	InteractiveAuthIncompleteError,
26	27	SynapseError,
27	28	ThreepidValidationError,
28	29	UnrecognizedRequestError,

386	387	self.ratelimiter = hs.get_registration_ratelimiter()
387	388	self.password_policy_handler = hs.get_password_policy_handler()
388	389	self.clock = hs.get_clock()
	390	self._registration_enabled = self.hs.config.enable_registration
389	391
390	392	self._registration_flows = _calculate_registration_flows(
391	393	hs.config, self.auth_handler

411	413	"Do not understand membership kind: %s" % (kind.decode("utf8"),)
412	414	)
413	415
414		# we do basic sanity checks here because the auth layer will store these
415		# in sessions. Pull out the username/password provided to us.
416		if "password" in body:
417		password = body.pop("password")
418		if not isinstance(password, str) or len(password) > 512:
419		raise SynapseError(400, "Invalid password")
420		self.password_policy_handler.validate_password(password)
421
422		# If the password is valid, hash it and store it back on the body.
423		# This ensures that only the hashed password is handled everywhere.
424		if "password_hash" in body:
425		raise SynapseError(400, "Unexpected property: password_hash")
426		body["password_hash"] = await self.auth_handler.hash(password)
427
	416	# Pull out the provided username and do basic sanity checks early since
	417	# the auth layer will store these in sessions.
428	418	desired_username = None
429	419	if "username" in body:
430	420	if not isinstance(body["username"], str) or len(body["username"]) > 512:

433	423
434	424	appservice = None
435	425	if self.auth.has_access_token(request):
436		appservice = await self.auth.get_appservice_by_req(request)
	426	appservice = self.auth.get_appservice_by_req(request)
437	427
438	428	# fork off as soon as possible for ASes which have completely
439	429	# different registration flows to normal users

458	448	)
459	449	return 200, result # we throw for non 200 responses
460	450
461		# for regular registration, downcase the provided username before
462		# attempting to register it. This should mean
463		# that people who try to register with upper-case in their usernames
464		# don't get a nasty surprise. (Note that we treat username
465		# case-insenstively in login, so they are free to carry on imagining
466		# that their username is CrAzYh4cKeR if that keeps them happy)
	451	# == Normal User Registration == (everyone else)
	452	if not self._registration_enabled:
	453	raise SynapseError(403, "Registration has been disabled")
	454
	455	# For regular registration, convert the provided username to lowercase
	456	# before attempting to register it. This should mean that people who try
	457	# to register with upper-case in their usernames don't get a nasty surprise.
	458	#
	459	# Note that we treat usernames case-insensitively in login, so they are
	460	# free to carry on imagining that their username is CrAzYh4cKeR if that
	461	# keeps them happy.
467	462	if desired_username is not None:
468	463	desired_username = desired_username.lower()
469	464
470		# == Normal User Registration == (everyone else)
471		if not self.hs.config.enable_registration:
472		raise SynapseError(403, "Registration has been disabled")
473
	465	# Check if this account is upgrading from a guest account.
474	466	guest_access_token = body.get("guest_access_token", None)
475	467
476		if "initial_device_display_name" in body and "password_hash" not in body:
	468	# Pull out the provided password and do basic sanity checks early.
	469	#
	470	# Note that we remove the password from the body since the auth layer
	471	# will store the body in the session and we don't want a plaintext
	472	# password store there.
	473	password = body.pop("password", None)
	474	if password is not None:
	475	if not isinstance(password, str) or len(password) > 512:
	476	raise SynapseError(400, "Invalid password")
	477	self.password_policy_handler.validate_password(password)
	478
	479	if "initial_device_display_name" in body and password is None:
477	480	# ignore 'initial_device_display_name' if sent without
478	481	# a password to work around a client bug where it sent
479	482	# the 'initial_device_display_name' param alone, wiping out

483	486
484	487	session_id = self.auth_handler.get_session_id(body)
485	488	registered_user_id = None
	489	password_hash = None
486	490	if session_id:
487	491	# if we get a registered user id out of here, it means we previously
488	492	# registered a user for this session, so we could just return the

491	495	registered_user_id = await self.auth_handler.get_session_data(
492	496	session_id, "registered_user_id", None
493	497	)
494
	498	# Extract the previously-hashed password from the session.
	499	password_hash = await self.auth_handler.get_session_data(
	500	session_id, "password_hash", None
	501	)
	502
	503	# Ensure that the username is valid.
495	504	if desired_username is not None:
496	505	await self.registration_handler.check_username(
497	506	desired_username,

499	508	assigned_user_id=registered_user_id,
500	509	)
501	510
502		auth_result, params, session_id = await self.auth_handler.check_auth(
503		self._registration_flows,
504		request,
505		body,
506		self.hs.get_ip_from_request(request),
507		"register a new account",
508		)
	511	# Check if the user-interactive authentication flows are complete, if
	512	# not this will raise a user-interactive auth error.
	513	try:
	514	auth_result, params, session_id = await self.auth_handler.check_ui_auth(
	515	self._registration_flows,
	516	request,
	517	body,
	518	self.hs.get_ip_from_request(request),
	519	"register a new account",
	520	)
	521	except InteractiveAuthIncompleteError as e:
	522	# The user needs to provide more steps to complete auth.
	523	#
	524	# Hash the password and store it with the session since the client
	525	# is not required to provide the password again.
	526	#
	527	# If a password hash was previously stored we will not attempt to
	528	# re-hash and store it for efficiency. This assumes the password
	529	# does not change throughout the authentication flow, but this
	530	# should be fine since the data is meant to be consistent.
	531	if not password_hash and password:
	532	password_hash = await self.auth_handler.hash(password)
	533	await self.auth_handler.set_session_data(
	534	e.session_id, "password_hash", password_hash
	535	)
	536	raise
509	537
510	538	# Check that we're not trying to register a denied 3pid.
511	539	#
512	540	# the user-facing checks will probably already have happened in
513	541	# /register/email/requestToken when we requested a 3pid, but that's not
514	542	# guaranteed.
515
516	543	if auth_result:
517	544	for login_type in [LoginType.EMAIL_IDENTITY, LoginType.MSISDN]:
518	545	if login_type in auth_result:

534	561	# don't re-register the threepids
535	562	registered = False
536	563	else:
537		# NB: This may be from the auth handler and NOT from the POST
538		assert_params_in_dict(params, ["password_hash"])
	564	# If we have a password in this request, prefer it. Otherwise, there
	565	# might be a password hash from an earlier request.
	566	if password:
	567	password_hash = await self.auth_handler.hash(password)
	568	if not password_hash:
	569	raise SynapseError(400, "Missing params: password", Codes.MISSING_PARAM)
539	570
540	571	desired_username = params.get("username", None)
541	572	guest_access_token = params.get("guest_access_token", None)
542		new_password_hash = params.get("password_hash", None)
543	573
544	574	if desired_username is not None:
545	575	desired_username = desired_username.lower()

581	611
582	612	registered_user_id = await self.registration_handler.register_user(
583	613	localpart=desired_username,
584		password_hash=new_password_hash,
	614	password_hash=password_hash,
585	615	guest_access_token=guest_access_token,
586	616	threepid=threepid,
587	617	address=client_addr,

594	624	):
595	625	await self.store.upsert_monthly_active_user(registered_user_id)
596	626
597		# remember that we've now registered that user account, and with
598		# what user ID (since the user may not have specified)
	627	# Remember that the user account has been registered (and the user
	628	# ID it was registered with, since it might not have been specified).
599	629	await self.auth_handler.set_session_data(
600	630	session_id, "registered_user_id", registered_user_id
601	631	)

634	664	(object) params: registration parameters, from which we pull
635	665	device_id, initial_device_name and inhibit_login
636	666	Returns:
637		defer.Deferred: (object) dictionary for response from /register
	667	(object) dictionary for response from /register
638	668	"""
639	669	result = {"user_id": user_id, "home_server": self.hs.hostname}
640	670	if not params.get("inhibit_login", False):

+1

-3

synapse/rest/consent/consent_resource.py less more

21	21	import jinja2
22	22	from jinja2 import TemplateNotFound
23	23
24		from twisted.internet import defer
25
26	24	from synapse.api.errors import NotFoundError, StoreError, SynapseError
27	25	from synapse.config import ConfigError
28	26	from synapse.http.server import DirectServeHtmlResource, respond_with_html

134	132	else:
135	133	qualified_user_id = UserID(username, self.hs.hostname).to_string()
136	134
137		u = await defer.maybeDeferred(self.store.get_user_by_id, qualified_user_id)
	135	u = await self.store.get_user_by_id(qualified_user_id)
138	136	if u is None:
139	137	raise NotFoundError("Unknown user")
140	138

+31

-0

synapse/rest/health.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from twisted.web.resource import Resource
	16
	17
	18	class HealthResource(Resource):
	19	"""A resource that does nothing except return a 200 with a body of `OK`,
	20	which can be used as a health check.
	21
	22	Note: `SynapseRequest._should_log_request` ensures that requests to
	23	`/health` do not get logged at INFO.
	24	"""
	25
	26	isLeaf = 1
	27
	28	def render_GET(self, request):
	29	request.setHeader(b"Content-Type", b"text/plain")
	30	return b"OK"

+6

-4

synapse/rest/media/v1/_base.py less more

16	16	import logging
17	17	import os
18	18	import urllib
19
	19	from typing import Awaitable
	20
	21	from twisted.internet.interfaces import IConsumer
20	22	from twisted.protocols.basic import FileSender
21	23
22	24	from synapse.api.errors import Codes, SynapseError, cs_error

239	241	held can be cleaned up.
240	242	"""
241	243
242		def write_to_consumer(self, consumer):
	244	def write_to_consumer(self, consumer: IConsumer) -> Awaitable:
243	245	"""Stream response into consumer
244	246
245	247	Args:
246		consumer (IConsumer)
	248	consumer: The consumer to stream into.
247	249
248	250	Returns:
249		Deferred: Resolves once the response has finished being written
	251	Resolves once the response has finished being written
250	252	"""
251	253	pass
252	254

+61

-44

synapse/rest/media/v1/media_repository.py less more

17	17	import logging
18	18	import os
19	19	import shutil
20		from typing import Dict, Tuple
	20	from typing import IO, Dict, Optional, Tuple
21	21
22	22	import twisted.internet.error
23	23	import twisted.web.http
	24	from twisted.web.http import Request
24	25	from twisted.web.resource import Resource
25	26
26	27	from synapse.api.errors import (

39	40
40	41	from ._base import (
41	42	FileInfo,
	43	Responder,
42	44	get_filename_from_headers,
43	45	respond_404,
44	46	respond_with_responder,

134	136	self.recently_accessed_locals.add(media_id)
135	137
136	138	async def create_content(
137		self, media_type, upload_name, content, content_length, auth_user
138		):
	139	self,
	140	media_type: str,
	141	upload_name: str,
	142	content: IO,
	143	content_length: int,
	144	auth_user: str,
	145	) -> str:
139	146	"""Store uploaded content for a local user and return the mxc URL
140	147
141	148	Args:
142		media_type(str): The content type of the file
143		upload_name(str): The name of the file
	149	media_type: The content type of the file
	150	upload_name: The name of the file
144	151	content: A file like object that is the content to store
145		content_length(int): The length of the content
146		auth_user(str): The user_id of the uploader
	152	content_length: The length of the content
	153	auth_user: The user_id of the uploader
147	154
148	155	Returns:
149		Deferred[str]: The mxc url of the stored content
	156	The mxc url of the stored content
150	157	"""
151	158	media_id = random_string(24)
152	159

169	176
170	177	return "mxc://%s/%s" % (self.server_name, media_id)
171	178
172		async def get_local_media(self, request, media_id, name):
	179	async def get_local_media(
	180	self, request: Request, media_id: str, name: Optional[str]
	181	) -> None:
173	182	"""Responds to reqests for local media, if exists, or returns 404.
174	183
175	184	Args:
176		request(twisted.web.http.Request)
177		media_id (str): The media ID of the content. (This is the same as
	185	request: The incoming request.
	186	media_id: The media ID of the content. (This is the same as
178	187	the file_id for local content.)
179		name (str\|None): Optional name that, if specified, will be used as
	188	name: Optional name that, if specified, will be used as
180	189	the filename in the Content-Disposition header of the response.
181	190
182	191	Returns:
183		Deferred: Resolves once a response has successfully been written
184		to request
	192	Resolves once a response has successfully been written to request
185	193	"""
186	194	media_info = await self.store.get_local_media(media_id)
187	195	if not media_info or media_info["quarantined_by"]:

202	210	request, responder, media_type, media_length, upload_name
203	211	)
204	212
205		async def get_remote_media(self, request, server_name, media_id, name):
	213	async def get_remote_media(
	214	self, request: Request, server_name: str, media_id: str, name: Optional[str]
	215	) -> None:
206	216	"""Respond to requests for remote media.
207	217
208	218	Args:
209		request(twisted.web.http.Request)
210		server_name (str): Remote server_name where the media originated.
211		media_id (str): The media ID of the content (as defined by the
212		remote server).
213		name (str\|None): Optional name that, if specified, will be used as
	219	request: The incoming request.
	220	server_name: Remote server_name where the media originated.
	221	media_id: The media ID of the content (as defined by the remote server).
	222	name: Optional name that, if specified, will be used as
214	223	the filename in the Content-Disposition header of the response.
215	224
216	225	Returns:
217		Deferred: Resolves once a response has successfully been written
218		to request
	226	Resolves once a response has successfully been written to request
219	227	"""
220	228	if (
221	229	self.federation_domain_whitelist is not None

244	252	else:
245	253	respond_404(request)
246	254
247		async def get_remote_media_info(self, server_name, media_id):
	255	async def get_remote_media_info(self, server_name: str, media_id: str) -> dict:
248	256	"""Gets the media info associated with the remote file, downloading
249	257	if necessary.
250	258
251	259	Args:
252		server_name (str): Remote server_name where the media originated.
253		media_id (str): The media ID of the content (as defined by the
254		remote server).
	260	server_name: Remote server_name where the media originated.
	261	media_id: The media ID of the content (as defined by the remote server).
255	262
256	263	Returns:
257		Deferred[dict]: The media_info of the file
	264	The media info of the file
258	265	"""
259	266	if (
260	267	self.federation_domain_whitelist is not None

277	284
278	285	return media_info
279	286
280		async def _get_remote_media_impl(self, server_name, media_id):
	287	async def _get_remote_media_impl(
	288	self, server_name: str, media_id: str
	289	) -> Tuple[Optional[Responder], dict]:
281	290	"""Looks for media in local cache, if not there then attempt to
282	291	download from remote server.
283	292

287	296	remote server).
288	297
289	298	Returns:
290		Deferred[(Responder, media_info)]
	299	A tuple of responder and the media info of the file.
291	300	"""
292	301	media_info = await self.store.get_cached_remote_media(server_name, media_id)
293	302

318	327	responder = await self.media_storage.fetch_media(file_info)
319	328	return responder, media_info
320	329
321		async def _download_remote_file(self, server_name, media_id, file_id):
	330	async def _download_remote_file(
	331	self, server_name: str, media_id: str, file_id: str
	332	) -> dict:
322	333	"""Attempt to download the remote file from the given server name,
323	334	using the given file_id as the local id.
324	335
325	336	Args:
326		server_name (str): Originating server
327		media_id (str): The media ID of the content (as defined by the
	337	server_name: Originating server
	338	media_id: The media ID of the content (as defined by the
328	339	remote server). This is different than the file_id, which is
329	340	locally generated.
330		file_id (str): Local file ID
	341	file_id: Local file ID
331	342
332	343	Returns:
333		Deferred[MediaInfo]
	344	The media info of the file.
334	345	"""
335	346
336	347	file_info = FileInfo(server_name=server_name, file_id=file_id)

548	559	return output_path
549	560
550	561	async def _generate_thumbnails(
551		self, server_name, media_id, file_id, media_type, url_cache=False
552		):
	562	self,
	563	server_name: Optional[str],
	564	media_id: str,
	565	file_id: str,
	566	media_type: str,
	567	url_cache: bool = False,
	568	) -> Optional[dict]:
553	569	"""Generate and store thumbnails for an image.
554	570
555	571	Args:
556		server_name (str\|None): The server name if remote media, else None if local
557		media_id (str): The media ID of the content. (This is the same as
	572	server_name: The server name if remote media, else None if local
	573	media_id: The media ID of the content. (This is the same as
558	574	the file_id for local content)
559		file_id (str): Local file ID
560		media_type (str): The content type of the file
561		url_cache (bool): If we are thumbnailing images downloaded for the URL cache,
	575	file_id: Local file ID
	576	media_type: The content type of the file
	577	url_cache: If we are thumbnailing images downloaded for the URL cache,
562	578	used exclusively by the url previewer
563	579
564	580	Returns:
565		Deferred[dict]: Dict with "width" and "height" keys of original image
	581	Dict with "width" and "height" keys of original image or None if the
	582	media cannot be thumbnailed.
566	583	"""
567	584	requirements = self._get_thumbnail_requirements(media_type)
568	585	if not requirements:
569		return
	586	return None
570	587
571	588	input_path = await self.media_storage.ensure_media_is_in_local_cache(
572	589	FileInfo(server_name, file_id, url_cache=url_cache)

583	600	m_height,
584	601	self.max_image_pixels,
585	602	)
586		return
	603	return None
587	604
588	605	if thumbnailer.transpose_method is not None:
589	606	m_width, m_height = await defer_to_thread(

+25

-34

synapse/rest/media/v1/media_storage.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14
15	14	import contextlib
16		import inspect
17	15	import logging
18	16	import os
19	17	import shutil
20		from typing import Optional
	18	from typing import IO, TYPE_CHECKING, Any, Optional, Sequence
21	19
22	20	from twisted.protocols.basic import FileSender
23	21

25	23	from synapse.util.file_consumer import BackgroundFileConsumer
26	24
27	25	from ._base import FileInfo, Responder
	26	from .filepath import MediaFilePaths
	27
	28	if TYPE_CHECKING:
	29	from synapse.server import HomeServer
	30
	31	from .storage_provider import StorageProviderWrapper
28	32
29	33	logger = logging.getLogger(__name__)
30	34

33	37	"""Responsible for storing/fetching files from local sources.
34	38
35	39	Args:
36		hs (synapse.server.Homeserver)
37		local_media_directory (str): Base path where we store media on disk
38		filepaths (MediaFilePaths)
39		storage_providers ([StorageProvider]): List of StorageProvider that are
40		used to fetch and store files.
	40	hs
	41	local_media_directory: Base path where we store media on disk
	42	filepaths
	43	storage_providers: List of StorageProvider that are used to fetch and store files.
41	44	"""
42	45
43		def __init__(self, hs, local_media_directory, filepaths, storage_providers):
	46	def __init__(
	47	self,
	48	hs: "HomeServer",
	49	local_media_directory: str,
	50	filepaths: MediaFilePaths,
	51	storage_providers: Sequence["StorageProviderWrapper"],
	52	):
44	53	self.hs = hs
45	54	self.local_media_directory = local_media_directory
46	55	self.filepaths = filepaths
47	56	self.storage_providers = storage_providers
48	57
49		async def store_file(self, source, file_info: FileInfo) -> str:
	58	async def store_file(self, source: IO, file_info: FileInfo) -> str:
50	59	"""Write `source` to the on disk media store, and also any other
51	60	configured storage providers
52	61

68	77	return fname
69	78
70	79	@contextlib.contextmanager
71		def store_into_file(self, file_info):
	80	def store_into_file(self, file_info: FileInfo):
72	81	"""Context manager used to get a file like object to write into, as
73	82	described by file_info.
74	83

84	93	error.
85	94
86	95	Args:
87		file_info (FileInfo): Info about the file to store
	96	file_info: Info about the file to store
88	97
89	98	Example:
90	99

104	113
105	114	async def finish():
106	115	for provider in self.storage_providers:
107		# store_file is supposed to return an Awaitable, but guard
108		# against improper implementations.
109		result = provider.store_file(path, file_info)
110		if inspect.isawaitable(result):
111		await result
	116	await provider.store_file(path, file_info)
112	117
113	118	finished_called[0] = True
114	119

142	147	return FileResponder(open(local_path, "rb"))
143	148
144	149	for provider in self.storage_providers:
145		res = provider.fetch(path, file_info)
146		# Fetch is supposed to return an Awaitable, but guard against
147		# improper implementations.
148		if inspect.isawaitable(res):
149		res = await res
	150	res = await provider.fetch(path, file_info) # type: Any
150	151	if res:
151	152	logger.debug("Streaming %s from %s", path, provider)
152	153	return res

173	174	os.makedirs(dirname)
174	175
175	176	for provider in self.storage_providers:
176		res = provider.fetch(path, file_info)
177		# Fetch is supposed to return an Awaitable, but guard against
178		# improper implementations.
179		if inspect.isawaitable(res):
180		res = await res
	177	res = await provider.fetch(path, file_info) # type: Any
181	178	if res:
182	179	with res:
183	180	consumer = BackgroundFileConsumer(

189	186
190	187	raise Exception("file could not be found")
191	188
192		def _file_info_to_path(self, file_info):
	189	def _file_info_to_path(self, file_info: FileInfo) -> str:
193	190	"""Converts file_info into a relative path.
194	191
195	192	The path is suitable for storing files under a directory, e.g. used to
196	193	store files on local FS under the base media repository directory.
197
198		Args:
199		file_info (FileInfo)
200
201		Returns:
202		str
203	194	"""
204	195	if file_info.url_cache:
205	196	if file_info.thumbnail:

+10

-11

synapse/rest/media/v1/preview_url_resource.py less more

26	26	from urllib import parse as urlparse
27	27
28	28	import attr
29		from canonicaljson import json
30
31		from twisted.internet import defer
	29
32	30	from twisted.internet.error import DNSLookupError
33	31
34	32	from synapse.api.errors import Codes, SynapseError

42	40	from synapse.logging.context import make_deferred_yieldable, run_in_background
43	41	from synapse.metrics.background_process_metrics import run_as_background_process
44	42	from synapse.rest.media.v1._base import get_filename_from_headers
	43	from synapse.util import json_encoder
45	44	from synapse.util.async_helpers import ObservableDeferred
46	45	from synapse.util.caches.expiringcache import ExpiringCache
47	46	from synapse.util.stringutils import random_string

227	226	else:
228	227	logger.info("Returning cached response")
229	228
230		og = await make_deferred_yieldable(defer.maybeDeferred(observable.observe))
	229	og = await make_deferred_yieldable(observable.observe())
231	230	respond_with_json_bytes(request, 200, og, send_cors=True)
232	231
233		async def _do_preview(self, url, user, ts):
	232	async def _do_preview(self, url: str, user: str, ts: int) -> bytes:
234	233	"""Check the db, and download the URL and build a preview
235	234
236	235	Args:
237		url (str):
238		user (str):
239		ts (int):
	236	url: The URL to preview.
	237	user: The user requesting the preview.
	238	ts: The timestamp requested for the preview.
240	239
241	240	Returns:
242		Deferred[bytes]: json-encoded og data
	241	json-encoded og data
243	242	"""
244	243	# check the URL cache in the DB (which will also provide us with
245	244	# historical previews, if we have any)

354	353
355	354	logger.debug("Calculated OG for %s as %s", url, og)
356	355
357		jsonog = json.dumps(og)
	356	jsonog = json_encoder.encode(og)
358	357
359	358	# store OG in history-aware DB cache
360	359	await self.store.store_url_cache(

585	584
586	585	logger.debug("Running url preview cache expiry")
587	586
588		if not (await self.store.db.updates.has_completed_background_updates()):
	587	if not (await self.store.db_pool.updates.has_completed_background_updates()):
589	588	logger.info("Still running DB updates; skipping expiry")
590	589	return
591	590

+45

-34

synapse/rest/media/v1/storage_provider.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
	15	import inspect
15	16	import logging
16	17	import os
17	18	import shutil
18
19		from twisted.internet import defer
	19	from typing import Optional
20	20
21	21	from synapse.config._base import Config
22	22	from synapse.logging.context import defer_to_thread, run_in_background
23	23
	24	from ._base import FileInfo, Responder
24	25	from .media_storage import FileResponder
25	26
26	27	logger = logging.getLogger(__name__)
27	28
28	29
29		class StorageProvider(object):
	30	class StorageProvider:
30	31	"""A storage provider is a service that can store uploaded media and
31	32	retrieve them.
32	33	"""
33	34
34		def store_file(self, path, file_info):
	35	async def store_file(self, path: str, file_info: FileInfo):
35	36	"""Store the file described by file_info. The actual contents can be
36	37	retrieved by reading the file in file_info.upload_path.
37	38
38	39	Args:
39		path (str): Relative path of file in local cache
40		file_info (FileInfo)
	40	path: Relative path of file in local cache
	41	file_info: The metadata of the file.
	42	"""
41	43
42		Returns:
43		Deferred
44		"""
45		pass
46
47		def fetch(self, path, file_info):
	44	async def fetch(self, path: str, file_info: FileInfo) -> Optional[Responder]:
48	45	"""Attempt to fetch the file described by file_info and stream it
49	46	into writer.
50	47
51	48	Args:
52		path (str): Relative path of file in local cache
53		file_info (FileInfo)
	49	path: Relative path of file in local cache
	50	file_info: The metadata of the file.
54	51
55	52	Returns:
56		Deferred(Responder): Returns a Responder if the provider has the file,
57		otherwise returns None.
	53	Returns a Responder if the provider has the file, otherwise returns None.
58	54	"""
59		pass
60	55
61	56
62	57	class StorageProviderWrapper(StorageProvider):
63	58	"""Wraps a storage provider and provides various config options
64	59
65	60	Args:
66		backend (StorageProvider)
67		store_local (bool): Whether to store new local files or not.
68		store_synchronous (bool): Whether to wait for file to be successfully
	61	backend: The storage provider to wrap.
	62	store_local: Whether to store new local files or not.
	63	store_synchronous: Whether to wait for file to be successfully
69	64	uploaded, or todo the upload in the background.
70		store_remote (bool): Whether remote media should be uploaded
	65	store_remote: Whether remote media should be uploaded
71	66	"""
72	67
73		def __init__(self, backend, store_local, store_synchronous, store_remote):
	68	def __init__(
	69	self,
	70	backend: StorageProvider,
	71	store_local: bool,
	72	store_synchronous: bool,
	73	store_remote: bool,
	74	):
74	75	self.backend = backend
75	76	self.store_local = store_local
76	77	self.store_synchronous = store_synchronous

79	80	def __str__(self):
80	81	return "StorageProviderWrapper[%s]" % (self.backend,)
81	82
82		def store_file(self, path, file_info):
	83	async def store_file(self, path, file_info):
83	84	if not file_info.server_name and not self.store_local:
84		return defer.succeed(None)
	85	return None
85	86
86	87	if file_info.server_name and not self.store_remote:
87		return defer.succeed(None)
	88	return None
88	89
89	90	if self.store_synchronous:
90		return self.backend.store_file(path, file_info)
	91	# store_file is supposed to return an Awaitable, but guard
	92	# against improper implementations.
	93	result = self.backend.store_file(path, file_info)
	94	if inspect.isawaitable(result):
	95	return await result
91	96	else:
92	97	# TODO: Handle errors.
93		def store():
	98	async def store():
94	99	try:
95		return self.backend.store_file(path, file_info)
	100	result = self.backend.store_file(path, file_info)
	101	if inspect.isawaitable(result):
	102	return await result
96	103	except Exception:
97	104	logger.exception("Error storing file")
98	105
99	106	run_in_background(store)
100		return defer.succeed(None)
	107	return None
101	108
102		def fetch(self, path, file_info):
103		return self.backend.fetch(path, file_info)
	109	async def fetch(self, path, file_info):
	110	# store_file is supposed to return an Awaitable, but guard
	111	# against improper implementations.
	112	result = self.backend.fetch(path, file_info)
	113	if inspect.isawaitable(result):
	114	return await result
104	115
105	116
106	117	class FileStorageProviderBackend(StorageProvider):

119	130	def __str__(self):
120	131	return "FileStorageProviderBackend[%s]" % (self.base_directory,)
121	132
122		def store_file(self, path, file_info):
	133	async def store_file(self, path, file_info):
123	134	"""See StorageProvider.store_file"""
124	135
125	136	primary_fname = os.path.join(self.cache_directory, path)

129	140	if not os.path.exists(dirname):
130	141	os.makedirs(dirname)
131	142
132		return defer_to_thread(
	143	return await defer_to_thread(
133	144	self.hs.get_reactor(), shutil.copyfile, primary_fname, backup_fname
134	145	)
135	146
136		def fetch(self, path, file_info):
	147	async def fetch(self, path, file_info):
137	148	"""See StorageProvider.fetch"""
138	149
139	150	backup_fname = os.path.join(self.base_directory, path)

+6

-2

synapse/secrets.py less more

24	24	if sys.version_info[0:2] >= (3, 6):
25	25	import secrets
26	26
27		def Secrets():
28		return secrets
	27	class Secrets:
	28	def token_bytes(self, nbytes=32):
	29	return secrets.token_bytes(nbytes)
	30
	31	def token_hex(self, nbytes=32):
	32	return secrets.token_hex(nbytes)
29	33
30	34
31	35	else:

+241

-228

synapse/server.py less more

21	21
22	22	# Imports required for the default HomeServer() implementation
23	23	import abc
	24	import functools
24	25	import logging
25	26	import os
26
	27	from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, TypeVar, cast
	28
	29	import twisted
27	30	from twisted.mail.smtp import sendmail
	31	from twisted.web.iweb import IPolicyForHTTPS
28	32
29	33	from synapse.api.auth import Auth
30	34	from synapse.api.filtering import Filtering

92	96	from synapse.replication.tcp.client import ReplicationDataHandler
93	97	from synapse.replication.tcp.handler import ReplicationCommandHandler
94	98	from synapse.replication.tcp.resource import ReplicationStreamer
95		from synapse.replication.tcp.streams import STREAMS_MAP
	99	from synapse.replication.tcp.streams import STREAMS_MAP, Stream
96	100	from synapse.rest.media.v1.media_repository import (
97	101	MediaRepository,
98	102	MediaRepositoryResource,

104	108	WorkerServerNoticesSender,
105	109	)
106	110	from synapse.state import StateHandler, StateResolutionHandler
107		from synapse.storage import DataStore, DataStores, Storage
	111	from synapse.storage import Databases, DataStore, Storage
108	112	from synapse.streams.events import EventSources
	113	from synapse.types import DomainSpecificString
109	114	from synapse.util import Clock
110	115	from synapse.util.distributor import Distributor
111	116	from synapse.util.stringutils import random_string
112	117
113	118	logger = logging.getLogger(__name__)
114	119
115
116		class HomeServer(object):
	120	if TYPE_CHECKING:
	121	from synapse.handlers.oidc_handler import OidcHandler
	122	from synapse.handlers.saml_handler import SamlHandler
	123
	124
	125	T = TypeVar("T", bound=Callable[..., Any])
	126
	127
	128	def cache_in_self(builder: T) -> T:
	129	"""Wraps a function called e.g. `get_foo`, checking if `self.foo` exists and
	130	returning if so. If not, calls the given function and sets `self.foo` to it.
	131
	132	Also ensures that dependency cycles throw an exception correctly, rather
	133	than overflowing the stack.
	134	"""
	135
	136	if not builder.__name__.startswith("get_"):
	137	raise Exception(
	138	"@cache_in_self can only be used on functions starting with `get_`"
	139	)
	140
	141	depname = builder.__name__[len("get_") :]
	142
	143	building = [False]
	144
	145	@functools.wraps(builder)
	146	def _get(self):
	147	try:
	148	return getattr(self, depname)
	149	except AttributeError:
	150	pass
	151
	152	# Prevent cyclic dependencies from deadlocking
	153	if building[0]:
	154	raise ValueError("Cyclic dependency while building %s" % (depname,))
	155
	156	building[0] = True
	157	try:
	158	dep = builder(self)
	159	setattr(self, depname, dep)
	160	finally:
	161	building[0] = False
	162
	163	return dep
	164
	165	# We cast here as we need to tell mypy that `_get` has the same signature as
	166	# `builder`.
	167	return cast(T, _get)
	168
	169
	170	class HomeServer(metaclass=abc.ABCMeta):
117	171	"""A basic homeserver object without lazy component builders.
118	172
119	173	This will need all of the components it requires to either be passed as
120	174	constructor arguments, or the relevant methods overriding to create them.
121	175	Typically this would only be used for unit tests.
122	176
123		For every dependency in the DEPENDENCIES list below, this class creates one
124		method,
125		def get_DEPENDENCY(self)
126		which returns the value of that dependency. If no value has yet been set
127		nor was provided to the constructor, it will attempt to call a lazy builder
128		method called
129		def build_DEPENDENCY(self)
130		which must be implemented by the subclass. This code may call any of the
131		required "get" methods on the instance to obtain the sub-dependencies that
132		one requires.
	177	Dependencies should be added by creating a `def get_<depname>(self)`
	178	function, wrapping it in `@cache_in_self`.
133	179
134	180	Attributes:
135	181	config (synapse.config.homeserver.HomeserverConfig):

137	183	we are listening on to provide HTTP services.
138	184	"""
139	185
140		__metaclass__ = abc.ABCMeta
141
142		DEPENDENCIES = [
143		"http_client",
144		"federation_client",
145		"federation_server",
146		"handlers",
147		"auth",
148		"room_creation_handler",
149		"room_shutdown_handler",
150		"state_handler",
151		"state_resolution_handler",
152		"presence_handler",
153		"sync_handler",
154		"typing_handler",
155		"room_list_handler",
156		"acme_handler",
157		"auth_handler",
158		"device_handler",
159		"stats_handler",
160		"e2e_keys_handler",
161		"e2e_room_keys_handler",
162		"event_handler",
163		"event_stream_handler",
164		"initial_sync_handler",
165		"application_service_api",
166		"application_service_scheduler",
167		"application_service_handler",
168		"device_message_handler",
169		"profile_handler",
170		"event_creation_handler",
171		"deactivate_account_handler",
172		"set_password_handler",
173		"notifier",
174		"event_sources",
175		"keyring",
176		"pusherpool",
177		"event_builder_factory",
178		"filtering",
179		"http_client_context_factory",
180		"simple_http_client",
181		"proxied_http_client",
182		"media_repository",
183		"media_repository_resource",
184		"federation_transport_client",
185		"federation_sender",
186		"receipts_handler",
187		"macaroon_generator",
188		"tcp_replication",
189		"read_marker_handler",
190		"action_generator",
191		"user_directory_handler",
192		"groups_local_handler",
193		"groups_server_handler",
194		"groups_attestation_signing",
195		"groups_attestation_renewer",
196		"secrets",
197		"spam_checker",
198		"third_party_event_rules",
199		"room_member_handler",
200		"federation_registry",
201		"server_notices_manager",
202		"server_notices_sender",
203		"message_handler",
204		"pagination_handler",
205		"room_context_handler",
206		"sendmail",
207		"registration_handler",
208		"account_validity_handler",
209		"cas_handler",
210		"saml_handler",
211		"oidc_handler",
212		"event_client_serializer",
213		"password_policy_handler",
214		"storage",
215		"replication_streamer",
216		"replication_data_handler",
217		"replication_streams",
218		]
219
220	186	REQUIRED_ON_MASTER_STARTUP = ["user_directory_handler", "stats_handler"]
221	187
222	188	# This is overridden in derived application classes

231	197	config: The full config for the homeserver.
232	198	"""
233	199	if not reactor:
234		from twisted.internet import reactor
	200	from twisted.internet import reactor as _reactor
	201
	202	reactor = _reactor
235	203
236	204	self._reactor = reactor
237	205	self.hostname = hostname
238	206	# the key we use to sign events and requests
239	207	self.signing_key = config.key.signing_key[0]
240	208	self.config = config
241		self._building = {}
242		self._listening_services = []
243		self.start_time = None
	209	self._listening_services = [] # type: List[twisted.internet.tcp.Port]
	210	self.start_time = None # type: Optional[int]
244	211
245	212	self._instance_id = random_string(5)
246	213	self._instance_name = config.worker_name or "master"

254	221	burst_count=config.rc_registration.burst_count,
255	222	)
256	223
257		self.datastores = None
	224	self.datastores = None # type: Optional[Databases]
258	225
259	226	# Other kwargs are explicit dependencies
260	227	for depname in kwargs:
261	228	setattr(self, depname, kwargs[depname])
262	229
263		def get_instance_id(self):
	230	def get_instance_id(self) -> str:
264	231	"""A unique ID for this synapse process instance.
265	232
266	233	This is used to distinguish running instances in worker-based

276	243	"""
277	244	return self._instance_name
278	245
279		def setup(self):
	246	def setup(self) -> None:
280	247	logger.info("Setting up.")
281	248	self.start_time = int(self.get_clock().time())
282		self.datastores = DataStores(self.DATASTORE_CLASS, self)
	249	self.datastores = Databases(self.DATASTORE_CLASS, self)
283	250	logger.info("Finished setting up.")
284	251
285		def setup_master(self):
	252	def setup_master(self) -> None:
286	253	"""
287	254	Some handlers have side effects on instantiation (like registering
288	255	background updates). This function causes them to be fetched, and

291	258	for i in self.REQUIRED_ON_MASTER_STARTUP:
292	259	getattr(self, "get_" + i)()
293	260
294		def get_reactor(self):
	261	def get_reactor(self) -> twisted.internet.base.ReactorBase:
295	262	"""
296	263	Fetch the Twisted reactor in use by this HomeServer.
297	264	"""
298	265	return self._reactor
299	266
300		def get_ip_from_request(self, request):
	267	def get_ip_from_request(self, request) -> str:
301	268	# X-Forwarded-For is handled by our custom request type.
302	269	return request.getClientIP()
303	270
304		def is_mine(self, domain_specific_string):
	271	def is_mine(self, domain_specific_string: DomainSpecificString) -> bool:
305	272	return domain_specific_string.domain == self.hostname
306	273
307		def is_mine_id(self, string):
	274	def is_mine_id(self, string: str) -> bool:
308	275	return string.split(":", 1)[1] == self.hostname
309	276
310		def get_clock(self):
	277	def get_clock(self) -> Clock:
311	278	return self.clock
312	279
313	280	def get_datastore(self) -> DataStore:
	281	if not self.datastores:
	282	raise Exception("HomeServer.setup must be called before getting datastores")
	283
314	284	return self.datastores.main
315	285
316		def get_datastores(self):
	286	def get_datastores(self) -> Databases:
	287	if not self.datastores:
	288	raise Exception("HomeServer.setup must be called before getting datastores")
	289
317	290	return self.datastores
318	291
319		def get_config(self):
	292	def get_config(self) -> HomeServerConfig:
320	293	return self.config
321	294
322		def get_distributor(self):
	295	def get_distributor(self) -> Distributor:
323	296	return self.distributor
324	297
325	298	def get_registration_ratelimiter(self) -> Ratelimiter:
326	299	return self.registration_ratelimiter
327	300
328		def build_federation_client(self):
	301	@cache_in_self
	302	def get_federation_client(self) -> FederationClient:
329	303	return FederationClient(self)
330	304
331		def build_federation_server(self):
	305	@cache_in_self
	306	def get_federation_server(self) -> FederationServer:
332	307	return FederationServer(self)
333	308
334		def build_handlers(self):
	309	@cache_in_self
	310	def get_handlers(self) -> Handlers:
335	311	return Handlers(self)
336	312
337		def build_notifier(self):
	313	@cache_in_self
	314	def get_notifier(self) -> Notifier:
338	315	return Notifier(self)
339	316
340		def build_auth(self):
	317	@cache_in_self
	318	def get_auth(self) -> Auth:
341	319	return Auth(self)
342	320
343		def build_http_client_context_factory(self):
	321	@cache_in_self
	322	def get_http_client_context_factory(self) -> IPolicyForHTTPS:
344	323	return (
345	324	InsecureInterceptableContextFactory()
346	325	if self.config.use_insecure_ssl_client_just_for_testing_do_not_use
347	326	else RegularPolicyForHTTPS()
348	327	)
349	328
350		def build_simple_http_client(self):
	329	@cache_in_self
	330	def get_simple_http_client(self) -> SimpleHttpClient:
351	331	return SimpleHttpClient(self)
352	332
353		def build_proxied_http_client(self):
	333	@cache_in_self
	334	def get_proxied_http_client(self) -> SimpleHttpClient:
354	335	return SimpleHttpClient(
355	336	self,
356	337	http_proxy=os.getenvb(b"http_proxy"),
357	338	https_proxy=os.getenvb(b"HTTPS_PROXY"),
358	339	)
359	340
360		def build_room_creation_handler(self):
	341	@cache_in_self
	342	def get_room_creation_handler(self) -> RoomCreationHandler:
361	343	return RoomCreationHandler(self)
362	344
363		def build_room_shutdown_handler(self):
	345	@cache_in_self
	346	def get_room_shutdown_handler(self) -> RoomShutdownHandler:
364	347	return RoomShutdownHandler(self)
365	348
366		def build_sendmail(self):
	349	@cache_in_self
	350	def get_sendmail(self) -> sendmail:
367	351	return sendmail
368	352
369		def build_state_handler(self):
	353	@cache_in_self
	354	def get_state_handler(self) -> StateHandler:
370	355	return StateHandler(self)
371	356
372		def build_state_resolution_handler(self):
	357	@cache_in_self
	358	def get_state_resolution_handler(self) -> StateResolutionHandler:
373	359	return StateResolutionHandler(self)
374	360
375		def build_presence_handler(self):
	361	@cache_in_self
	362	def get_presence_handler(self) -> PresenceHandler:
376	363	return PresenceHandler(self)
377	364
378		def build_typing_handler(self):
	365	@cache_in_self
	366	def get_typing_handler(self):
379	367	if self.config.worker.writers.typing == self.get_instance_name():
380	368	return TypingWriterHandler(self)
381	369	else:
382	370	return FollowerTypingHandler(self)
383	371
384		def build_sync_handler(self):
	372	@cache_in_self
	373	def get_sync_handler(self) -> SyncHandler:
385	374	return SyncHandler(self)
386	375
387		def build_room_list_handler(self):
	376	@cache_in_self
	377	def get_room_list_handler(self) -> RoomListHandler:
388	378	return RoomListHandler(self)
389	379
390		def build_auth_handler(self):
	380	@cache_in_self
	381	def get_auth_handler(self) -> AuthHandler:
391	382	return AuthHandler(self)
392	383
393		def build_macaroon_generator(self):
	384	@cache_in_self
	385	def get_macaroon_generator(self) -> MacaroonGenerator:
394	386	return MacaroonGenerator(self)
395	387
396		def build_device_handler(self):
	388	@cache_in_self
	389	def get_device_handler(self):
397	390	if self.config.worker_app:
398	391	return DeviceWorkerHandler(self)
399	392	else:
400	393	return DeviceHandler(self)
401	394
402		def build_device_message_handler(self):
	395	@cache_in_self
	396	def get_device_message_handler(self) -> DeviceMessageHandler:
403	397	return DeviceMessageHandler(self)
404	398
405		def build_e2e_keys_handler(self):
	399	@cache_in_self
	400	def get_e2e_keys_handler(self) -> E2eKeysHandler:
406	401	return E2eKeysHandler(self)
407	402
408		def build_e2e_room_keys_handler(self):
	403	@cache_in_self
	404	def get_e2e_room_keys_handler(self) -> E2eRoomKeysHandler:
409	405	return E2eRoomKeysHandler(self)
410	406
411		def build_acme_handler(self):
	407	@cache_in_self
	408	def get_acme_handler(self) -> AcmeHandler:
412	409	return AcmeHandler(self)
413	410
414		def build_application_service_api(self):
	411	@cache_in_self
	412	def get_application_service_api(self) -> ApplicationServiceApi:
415	413	return ApplicationServiceApi(self)
416	414
417		def build_application_service_scheduler(self):
	415	@cache_in_self
	416	def get_application_service_scheduler(self) -> ApplicationServiceScheduler:
418	417	return ApplicationServiceScheduler(self)
419	418
420		def build_application_service_handler(self):
	419	@cache_in_self
	420	def get_application_service_handler(self) -> ApplicationServicesHandler:
421	421	return ApplicationServicesHandler(self)
422	422
423		def build_event_handler(self):
	423	@cache_in_self
	424	def get_event_handler(self) -> EventHandler:
424	425	return EventHandler(self)
425	426
426		def build_event_stream_handler(self):
	427	@cache_in_self
	428	def get_event_stream_handler(self) -> EventStreamHandler:
427	429	return EventStreamHandler(self)
428	430
429		def build_initial_sync_handler(self):
	431	@cache_in_self
	432	def get_initial_sync_handler(self) -> InitialSyncHandler:
430	433	return InitialSyncHandler(self)
431	434
432		def build_profile_handler(self):
	435	@cache_in_self
	436	def get_profile_handler(self):
433	437	if self.config.worker_app:
434	438	return BaseProfileHandler(self)
435	439	else:
436	440	return MasterProfileHandler(self)
437	441
438		def build_event_creation_handler(self):
	442	@cache_in_self
	443	def get_event_creation_handler(self) -> EventCreationHandler:
439	444	return EventCreationHandler(self)
440	445
441		def build_deactivate_account_handler(self):
	446	@cache_in_self
	447	def get_deactivate_account_handler(self) -> DeactivateAccountHandler:
442	448	return DeactivateAccountHandler(self)
443	449
444		def build_set_password_handler(self):
	450	@cache_in_self
	451	def get_set_password_handler(self) -> SetPasswordHandler:
445	452	return SetPasswordHandler(self)
446	453
447		def build_event_sources(self):
	454	@cache_in_self
	455	def get_event_sources(self) -> EventSources:
448	456	return EventSources(self)
449	457
450		def build_keyring(self):
	458	@cache_in_self
	459	def get_keyring(self) -> Keyring:
451	460	return Keyring(self)
452	461
453		def build_event_builder_factory(self):
	462	@cache_in_self
	463	def get_event_builder_factory(self) -> EventBuilderFactory:
454	464	return EventBuilderFactory(self)
455	465
456		def build_filtering(self):
	466	@cache_in_self
	467	def get_filtering(self) -> Filtering:
457	468	return Filtering(self)
458	469
459		def build_pusherpool(self):
	470	@cache_in_self
	471	def get_pusherpool(self) -> PusherPool:
460	472	return PusherPool(self)
461	473
462		def build_http_client(self):
	474	@cache_in_self
	475	def get_http_client(self) -> MatrixFederationHttpClient:
463	476	tls_client_options_factory = context_factory.FederationPolicyForHTTPS(
464	477	self.config
465	478	)
466	479	return MatrixFederationHttpClient(self, tls_client_options_factory)
467	480
468		def build_media_repository_resource(self):
	481	@cache_in_self
	482	def get_media_repository_resource(self) -> MediaRepositoryResource:
469	483	# build the media repo resource. This indirects through the HomeServer
470	484	# to ensure that we only have a single instance of
471	485	return MediaRepositoryResource(self)
472	486
473		def build_media_repository(self):
	487	@cache_in_self
	488	def get_media_repository(self) -> MediaRepository:
474	489	return MediaRepository(self)
475	490
476		def build_federation_transport_client(self):
	491	@cache_in_self
	492	def get_federation_transport_client(self) -> TransportLayerClient:
477	493	return TransportLayerClient(self)
478	494
479		def build_federation_sender(self):
	495	@cache_in_self
	496	def get_federation_sender(self):
480	497	if self.should_send_federation():
481	498	return FederationSender(self)
482	499	elif not self.config.worker_app:

484	501	else:
485	502	raise Exception("Workers cannot send federation traffic")
486	503
487		def build_receipts_handler(self):
	504	@cache_in_self
	505	def get_receipts_handler(self) -> ReceiptsHandler:
488	506	return ReceiptsHandler(self)
489	507
490		def build_read_marker_handler(self):
	508	@cache_in_self
	509	def get_read_marker_handler(self) -> ReadMarkerHandler:
491	510	return ReadMarkerHandler(self)
492	511
493		def build_tcp_replication(self):
	512	@cache_in_self
	513	def get_tcp_replication(self) -> ReplicationCommandHandler:
494	514	return ReplicationCommandHandler(self)
495	515
496		def build_action_generator(self):
	516	@cache_in_self
	517	def get_action_generator(self) -> ActionGenerator:
497	518	return ActionGenerator(self)
498	519
499		def build_user_directory_handler(self):
	520	@cache_in_self
	521	def get_user_directory_handler(self) -> UserDirectoryHandler:
500	522	return UserDirectoryHandler(self)
501	523
502		def build_groups_local_handler(self):
	524	@cache_in_self
	525	def get_groups_local_handler(self):
503	526	if self.config.worker_app:
504	527	return GroupsLocalWorkerHandler(self)
505	528	else:
506	529	return GroupsLocalHandler(self)
507	530
508		def build_groups_server_handler(self):
	531	@cache_in_self
	532	def get_groups_server_handler(self):
509	533	if self.config.worker_app:
510	534	return GroupsServerWorkerHandler(self)
511	535	else:
512	536	return GroupsServerHandler(self)
513	537
514		def build_groups_attestation_signing(self):
	538	@cache_in_self
	539	def get_groups_attestation_signing(self) -> GroupAttestationSigning:
515	540	return GroupAttestationSigning(self)
516	541
517		def build_groups_attestation_renewer(self):
	542	@cache_in_self
	543	def get_groups_attestation_renewer(self) -> GroupAttestionRenewer:
518	544	return GroupAttestionRenewer(self)
519	545
520		def build_secrets(self):
	546	@cache_in_self
	547	def get_secrets(self) -> Secrets:
521	548	return Secrets()
522	549
523		def build_stats_handler(self):
	550	@cache_in_self
	551	def get_stats_handler(self) -> StatsHandler:
524	552	return StatsHandler(self)
525	553
526		def build_spam_checker(self):
	554	@cache_in_self
	555	def get_spam_checker(self):
527	556	return SpamChecker(self)
528	557
529		def build_third_party_event_rules(self):
	558	@cache_in_self
	559	def get_third_party_event_rules(self) -> ThirdPartyEventRules:
530	560	return ThirdPartyEventRules(self)
531	561
532		def build_room_member_handler(self):
	562	@cache_in_self
	563	def get_room_member_handler(self):
533	564	if self.config.worker_app:
534	565	return RoomMemberWorkerHandler(self)
535	566	return RoomMemberMasterHandler(self)
536	567
537		def build_federation_registry(self):
	568	@cache_in_self
	569	def get_federation_registry(self) -> FederationHandlerRegistry:
538	570	return FederationHandlerRegistry(self)
539	571
540		def build_server_notices_manager(self):
	572	@cache_in_self
	573	def get_server_notices_manager(self):
541	574	if self.config.worker_app:
542	575	raise Exception("Workers cannot send server notices")
543	576	return ServerNoticesManager(self)
544	577
545		def build_server_notices_sender(self):
	578	@cache_in_self
	579	def get_server_notices_sender(self):
546	580	if self.config.worker_app:
547	581	return WorkerServerNoticesSender(self)
548	582	return ServerNoticesSender(self)
549	583
550		def build_message_handler(self):
	584	@cache_in_self
	585	def get_message_handler(self) -> MessageHandler:
551	586	return MessageHandler(self)
552	587
553		def build_pagination_handler(self):
	588	@cache_in_self
	589	def get_pagination_handler(self) -> PaginationHandler:
554	590	return PaginationHandler(self)
555	591
556		def build_room_context_handler(self):
	592	@cache_in_self
	593	def get_room_context_handler(self) -> RoomContextHandler:
557	594	return RoomContextHandler(self)
558	595
559		def build_registration_handler(self):
	596	@cache_in_self
	597	def get_registration_handler(self) -> RegistrationHandler:
560	598	return RegistrationHandler(self)
561	599
562		def build_account_validity_handler(self):
	600	@cache_in_self
	601	def get_account_validity_handler(self) -> AccountValidityHandler:
563	602	return AccountValidityHandler(self)
564	603
565		def build_cas_handler(self):
	604	@cache_in_self
	605	def get_cas_handler(self) -> CasHandler:
566	606	return CasHandler(self)
567	607
568		def build_saml_handler(self):
	608	@cache_in_self
	609	def get_saml_handler(self) -> "SamlHandler":
569	610	from synapse.handlers.saml_handler import SamlHandler
570	611
571	612	return SamlHandler(self)
572	613
573		def build_oidc_handler(self):
	614	@cache_in_self
	615	def get_oidc_handler(self) -> "OidcHandler":
574	616	from synapse.handlers.oidc_handler import OidcHandler
575	617
576	618	return OidcHandler(self)
577	619
578		def build_event_client_serializer(self):
	620	@cache_in_self
	621	def get_event_client_serializer(self) -> EventClientSerializer:
579	622	return EventClientSerializer(self)
580	623
581		def build_password_policy_handler(self):
	624	@cache_in_self
	625	def get_password_policy_handler(self) -> PasswordPolicyHandler:
582	626	return PasswordPolicyHandler(self)
583	627
584		def build_storage(self) -> Storage:
585		return Storage(self, self.datastores)
586
587		def build_replication_streamer(self) -> ReplicationStreamer:
	628	@cache_in_self
	629	def get_storage(self) -> Storage:
	630	return Storage(self, self.get_datastores())
	631
	632	@cache_in_self
	633	def get_replication_streamer(self) -> ReplicationStreamer:
588	634	return ReplicationStreamer(self)
589	635
590		def build_replication_data_handler(self):
	636	@cache_in_self
	637	def get_replication_data_handler(self) -> ReplicationDataHandler:
591	638	return ReplicationDataHandler(self)
592	639
593		def build_replication_streams(self):
	640	@cache_in_self
	641	def get_replication_streams(self) -> Dict[str, Stream]:
594	642	return {stream.NAME: stream(self) for stream in STREAMS_MAP.values()}
595	643
596		def remove_pusher(self, app_id, push_key, user_id):
597		return self.get_pusherpool().remove_pusher(app_id, push_key, user_id)
598
599		def should_send_federation(self):
	644	async def remove_pusher(self, app_id: str, push_key: str, user_id: str):
	645	return await self.get_pusherpool().remove_pusher(app_id, push_key, user_id)
	646
	647	def should_send_federation(self) -> bool:
600	648	"Should this server be sending federation traffic directly?"
601	649	return self.config.send_federation and (
602	650	not self.config.worker_app
603	651	or self.config.worker_app == "synapse.app.federation_sender"
604	652	)
605
606
607		def _make_dependency_method(depname):
608		def _get(hs):
609		try:
610		return getattr(hs, depname)
611		except AttributeError:
612		pass
613
614		try:
615		builder = getattr(hs, "build_%s" % (depname))
616		except AttributeError:
617		raise NotImplementedError(
618		"%s has no %s nor a builder for it" % (type(hs).__name__, depname)
619		)
620
621		# Prevent cyclic dependencies from deadlocking
622		if depname in hs._building:
623		raise ValueError("Cyclic dependency while building %s" % (depname,))
624
625		hs._building[depname] = 1
626		try:
627		dep = builder()
628		setattr(hs, depname, dep)
629		finally:
630		del hs._building[depname]
631
632		return dep
633
634		setattr(HomeServer, "get_%s" % (depname), _get)
635
636
637		# Build magic accessors for every dependency
638		for depname in HomeServer.DEPENDENCIES:
639		_make_dependency_method(depname)

+0

-155

~~synapse/server.pyi~~ less more

0		from typing import Dict
1
2		import twisted.internet
3
4		import synapse.api.auth
5		import synapse.config.homeserver
6		import synapse.crypto.keyring
7		import synapse.federation.federation_server
8		import synapse.federation.sender
9		import synapse.federation.transport.client
10		import synapse.handlers
11		import synapse.handlers.auth
12		import synapse.handlers.deactivate_account
13		import synapse.handlers.device
14		import synapse.handlers.e2e_keys
15		import synapse.handlers.message
16		import synapse.handlers.presence
17		import synapse.handlers.register
18		import synapse.handlers.room
19		import synapse.handlers.room_member
20		import synapse.handlers.set_password
21		import synapse.http.client
22		import synapse.http.matrixfederationclient
23		import synapse.notifier
24		import synapse.push.pusherpool
25		import synapse.replication.tcp.client
26		import synapse.replication.tcp.handler
27		import synapse.rest.media.v1.media_repository
28		import synapse.server_notices.server_notices_manager
29		import synapse.server_notices.server_notices_sender
30		import synapse.state
31		import synapse.storage
32		from synapse.events.builder import EventBuilderFactory
33		from synapse.handlers.typing import FollowerTypingHandler
34		from synapse.replication.tcp.streams import Stream
35
36		class HomeServer(object):
37		@property
38		def config(self) -> synapse.config.homeserver.HomeServerConfig:
39		pass
40		@property
41		def hostname(self) -> str:
42		pass
43		def get_auth(self) -> synapse.api.auth.Auth:
44		pass
45		def get_auth_handler(self) -> synapse.handlers.auth.AuthHandler:
46		pass
47		def get_datastore(self) -> synapse.storage.DataStore:
48		pass
49		def get_device_handler(self) -> synapse.handlers.device.DeviceHandler:
50		pass
51		def get_e2e_keys_handler(self) -> synapse.handlers.e2e_keys.E2eKeysHandler:
52		pass
53		def get_handlers(self) -> synapse.handlers.Handlers:
54		pass
55		def get_state_handler(self) -> synapse.state.StateHandler:
56		pass
57		def get_state_resolution_handler(self) -> synapse.state.StateResolutionHandler:
58		pass
59		def get_simple_http_client(self) -> synapse.http.client.SimpleHttpClient:
60		"""Fetch an HTTP client implementation which doesn't do any blacklisting
61		or support any HTTP_PROXY settings"""
62		pass
63		def get_proxied_http_client(self) -> synapse.http.client.SimpleHttpClient:
64		"""Fetch an HTTP client implementation which doesn't do any blacklisting
65		but does support HTTP_PROXY settings"""
66		pass
67		def get_deactivate_account_handler(
68		self,
69		) -> synapse.handlers.deactivate_account.DeactivateAccountHandler:
70		pass
71		def get_room_creation_handler(self) -> synapse.handlers.room.RoomCreationHandler:
72		pass
73		def get_room_member_handler(self) -> synapse.handlers.room_member.RoomMemberHandler:
74		pass
75		def get_room_shutdown_handler(self) -> synapse.handlers.room.RoomShutdownHandler:
76		pass
77		def get_event_creation_handler(
78		self,
79		) -> synapse.handlers.message.EventCreationHandler:
80		pass
81		def get_set_password_handler(
82		self,
83		) -> synapse.handlers.set_password.SetPasswordHandler:
84		pass
85		def get_federation_sender(self) -> synapse.federation.sender.FederationSender:
86		pass
87		def get_federation_transport_client(
88		self,
89		) -> synapse.federation.transport.client.TransportLayerClient:
90		pass
91		def get_media_repository_resource(
92		self,
93		) -> synapse.rest.media.v1.media_repository.MediaRepositoryResource:
94		pass
95		def get_media_repository(
96		self,
97		) -> synapse.rest.media.v1.media_repository.MediaRepository:
98		pass
99		def get_server_notices_manager(
100		self,
101		) -> synapse.server_notices.server_notices_manager.ServerNoticesManager:
102		pass
103		def get_server_notices_sender(
104		self,
105		) -> synapse.server_notices.server_notices_sender.ServerNoticesSender:
106		pass
107		def get_notifier(self) -> synapse.notifier.Notifier:
108		pass
109		def get_presence_handler(self) -> synapse.handlers.presence.BasePresenceHandler:
110		pass
111		def get_clock(self) -> synapse.util.Clock:
112		pass
113		def get_reactor(self) -> twisted.internet.base.ReactorBase:
114		pass
115		def get_keyring(self) -> synapse.crypto.keyring.Keyring:
116		pass
117		def get_tcp_replication(
118		self,
119		) -> synapse.replication.tcp.handler.ReplicationCommandHandler:
120		pass
121		def get_replication_data_handler(
122		self,
123		) -> synapse.replication.tcp.client.ReplicationDataHandler:
124		pass
125		def get_federation_registry(
126		self,
127		) -> synapse.federation.federation_server.FederationHandlerRegistry:
128		pass
129		def is_mine_id(self, domain_id: str) -> bool:
130		pass
131		def get_instance_id(self) -> str:
132		pass
133		def get_instance_name(self) -> str:
134		pass
135		def get_event_builder_factory(self) -> EventBuilderFactory:
136		pass
137		def get_storage(self) -> synapse.storage.Storage:
138		pass
139		def get_registration_handler(self) -> synapse.handlers.register.RegistrationHandler:
140		pass
141		def get_macaroon_generator(self) -> synapse.handlers.auth.MacaroonGenerator:
142		pass
143		def get_pusherpool(self) -> synapse.push.pusherpool.PusherPool:
144		pass
145		def get_replication_streams(self) -> Dict[str, Stream]:
146		pass
147		def get_http_client(
148		self,
149		) -> synapse.http.matrixfederationclient.MatrixFederationHttpClient:
150		pass
151		def should_send_federation(self) -> bool:
152		pass
153		def get_typing_handler(self) -> FollowerTypingHandler:
154		pass

+5

-7

synapse/server_notices/consent_server_notices.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
	15	from typing import Any
15	16
16	17	from synapse.api.errors import SynapseError
17	18	from synapse.api.urls import ConsentURIBuilder

54	55
55	56	self._consent_uri_builder = ConsentURIBuilder(hs.config)
56	57
57		async def maybe_send_server_notice_to_user(self, user_id):
	58	async def maybe_send_server_notice_to_user(self, user_id: str) -> None:
58	59	"""Check if we need to send a notice to this user, and does so if so
59	60
60	61	Args:
61		user_id (str): user to check
62
63		Returns:
64		Deferred
	62	user_id: user to check
65	63	"""
66	64	if self._server_notice_content is None:
67	65	# not enabled

104	102	self._users_in_progress.remove(user_id)
105	103
106	104
107		def copy_with_str_subst(x, substitutions):
	105	def copy_with_str_subst(x: Any, substitutions: Any) -> Any:
108	106	"""Deep-copy a structure, carrying out string substitions on any strings
109	107
110	108	Args:

120	118	if isinstance(x, dict):
121	119	return {k: copy_with_str_subst(v, substitutions) for (k, v) in x.items()}
122	120	if isinstance(x, (list, tuple)):
123		return [copy_with_str_subst(y) for y in x]
	121	return [copy_with_str_subst(y, substitutions) for y in x]
124	122
125	123	# assume it's uninterested and can be shallow-copied.
126	124	return x

+21

-22

synapse/server_notices/resource_limits_server_notices.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
	15	from typing import List, Tuple
15	16
16	17	from synapse.api.constants import (
17	18	EventTypes,

51	52	and not hs.config.hs_disabled
52	53	)
53	54
54		async def maybe_send_server_notice_to_user(self, user_id):
	55	async def maybe_send_server_notice_to_user(self, user_id: str) -> None:
55	56	"""Check if we need to send a notice to this user, this will be true in
56	57	two cases.
57	58	1. The server has reached its limit does not reflect this

59	60	actually the server is fine
60	61
61	62	Args:
62		user_id (str): user to check
63
64		Returns:
65		Deferred
	63	user_id: user to check
66	64	"""
67	65	if not self._enabled:
68	66	return

114	112	elif not currently_blocked and limit_msg:
115	113	# Room is not notifying of a block, when it ought to be.
116	114	await self._apply_limit_block_notification(
117		user_id, limit_msg, limit_type
	115	user_id, limit_msg, limit_type # type: ignore
118	116	)
119	117	except SynapseError as e:
120	118	logger.error("Error sending resource limits server notice: %s", e)
121	119
122		async def _remove_limit_block_notification(self, user_id, ref_events):
	120	async def _remove_limit_block_notification(
	121	self, user_id: str, ref_events: List[str]
	122	) -> None:
123	123	"""Utility method to remove limit block notifications from the server
124	124	notices room.
125	125
126	126	Args:
127		user_id (str): user to notify
128		ref_events (list[str]): The event_ids of pinned events that are unrelated to
129		limit blocking and need to be preserved.
	127	user_id: user to notify
	128	ref_events: The event_ids of pinned events that are unrelated to
	129	limit blocking and need to be preserved.
130	130	"""
131	131	content = {"pinned": ref_events}
132	132	await self._server_notices_manager.send_notice(

134	134	)
135	135
136	136	async def _apply_limit_block_notification(
137		self, user_id, event_body, event_limit_type
138		):
	137	self, user_id: str, event_body: str, event_limit_type: str
	138	) -> None:
139	139	"""Utility method to apply limit block notifications in the server
140	140	notices room.
141	141
142	142	Args:
143		user_id (str): user to notify
144		event_body(str): The human readable text that describes the block.
145		event_limit_type(str): Specifies the type of block e.g. monthly active user
146		limit has been exceeded.
	143	user_id: user to notify
	144	event_body: The human readable text that describes the block.
	145	event_limit_type: Specifies the type of block e.g. monthly active user
	146	limit has been exceeded.
147	147	"""
148	148	content = {
149	149	"body": event_body,

161	161	user_id, content, EventTypes.Pinned, ""
162	162	)
163	163
164		async def _check_and_set_tags(self, user_id, room_id):
	164	async def _check_and_set_tags(self, user_id: str, room_id: str) -> None:
165	165	"""
166	166	Since server notices rooms were originally not with tags,
167	167	important to check that tags have been set correctly

181	181	)
182	182	self._notifier.on_new_event("account_data_key", max_id, users=[user_id])
183	183
184		async def _is_room_currently_blocked(self, room_id):
	184	async def _is_room_currently_blocked(self, room_id: str) -> Tuple[bool, List[str]]:
185	185	"""
186	186	Determines if the room is currently blocked
187	187
188	188	Args:
189		room_id(str): The room id of the server notices room
	189	room_id: The room id of the server notices room
190	190
191	191	Returns:
192		Deferred[Tuple[bool, List]]:
193	192	bool: Is the room currently blocked
194		list: The list of pinned events that are unrelated to limit blocking
	193	list: The list of pinned event IDs that are unrelated to limit blocking
195	194	This list can be used as a convenience in the case where the block
196	195	is to be lifted and the remaining pinned event references need to be
197	196	preserved

206	205	# The user has yet to join the server notices room
207	206	pass
208	207
209		referenced_events = []
	208	referenced_events = [] # type: List[str]
210	209	if pinned_state_event is not None:
211	210	referenced_events = list(pinned_state_event.content.get("pinned", []))
212	211

+16

-13

synapse/server_notices/server_notices_manager.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
	15	from typing import Optional
15	16
16	17	from synapse.api.constants import EventTypes, Membership, RoomCreationPreset
	18	from synapse.events import EventBase
17	19	from synapse.types import UserID, create_requester
18	20	from synapse.util.caches.descriptors import cached
19	21

49	51	return self._config.server_notices_mxid is not None
50	52
51	53	async def send_notice(
52		self, user_id, event_content, type=EventTypes.Message, state_key=None
53		):
	54	self,
	55	user_id: str,
	56	event_content: dict,
	57	type: str = EventTypes.Message,
	58	state_key: Optional[bool] = None,
	59	) -> EventBase:
54	60	"""Send a notice to the given user
55	61
56	62	Creates the server notices room, if none exists.
57	63
58	64	Args:
59		user_id (str): mxid of user to send event to.
60		event_content (dict): content of event to send
61		type(EventTypes): type of event
62		is_state_event(bool): Is the event a state event
63
64		Returns:
65		Deferred[FrozenEvent]
	65	user_id: mxid of user to send event to.
	66	event_content: content of event to send
	67	type: type of event
	68	is_state_event: Is the event a state event
66	69	"""
67	70	room_id = await self.get_or_create_notice_room_for_user(user_id)
68	71	await self.maybe_invite_user_to_room(user_id, room_id)

88	91	return event
89	92
90	93	@cached()
91		async def get_or_create_notice_room_for_user(self, user_id):
	94	async def get_or_create_notice_room_for_user(self, user_id: str) -> str:
92	95	"""Get the room for notices for a given user
93	96
94	97	If we have not yet created a notice room for this user, create it, but don't
95	98	invite the user to it.
96	99
97	100	Args:
98		user_id (str): complete user id for the user we want a room for
	101	user_id: complete user id for the user we want a room for
99	102
100	103	Returns:
101		str: room id of notice room.
	104	room id of notice room.
102	105	"""
103	106	if not self.is_enabled():
104	107	raise Exception("Server notices not enabled")

162	165	logger.info("Created server notices room %s for %s", room_id, user_id)
163	166	return room_id
164	167
165		async def maybe_invite_user_to_room(self, user_id: str, room_id: str):
	168	async def maybe_invite_user_to_room(self, user_id: str, room_id: str) -> None:
166	169	"""Invite the given user to the given server room, unless the user has already
167	170	joined or been invited to it.
168	171

+7

-5

synapse/server_notices/server_notices_sender.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14	from typing import Iterable, Union
	15
14	16	from synapse.server_notices.consent_server_notices import ConsentServerNotices
15	17	from synapse.server_notices.resource_limits_server_notices import (
16	18	ResourceLimitsServerNotices,

31	33	self._server_notices = (
32	34	ConsentServerNotices(hs),
33	35	ResourceLimitsServerNotices(hs),
34		)
	36	) # type: Iterable[Union[ConsentServerNotices, ResourceLimitsServerNotices]]
35	37
36		async def on_user_syncing(self, user_id):
	38	async def on_user_syncing(self, user_id: str) -> None:
37	39	"""Called when the user performs a sync operation.
38	40
39	41	Args:
40		user_id (str): mxid of user who synced
	42	user_id: mxid of user who synced
41	43	"""
42	44	for sn in self._server_notices:
43	45	await sn.maybe_send_server_notice_to_user(user_id)
44	46
45		async def on_user_ip(self, user_id):
	47	async def on_user_ip(self, user_id: str) -> None:
46	48	"""Called on the master when a worker process saw a client request.
47	49
48	50	Args:
49		user_id (str): mxid
	51	user_id: mxid
50	52	"""
51	53	# The synchrotrons use a stubbed version of ServerNoticesSender, so
52	54	# we check for notices to send to the user in on_user_ip as well as

+6

-13

synapse/server_notices/worker_server_notices_sender.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14		from twisted.internet import defer
15	14
16	15
17	16	class WorkerServerNoticesSender(object):

23	22	hs (synapse.server.HomeServer):
24	23	"""
25	24
26		def on_user_syncing(self, user_id):
	25	async def on_user_syncing(self, user_id: str) -> None:
27	26	"""Called when the user performs a sync operation.
28	27
29	28	Args:
30		user_id (str): mxid of user who synced
	29	user_id: mxid of user who synced
	30	"""
	31	return None
31	32
32		Returns:
33		Deferred
34		"""
35		return defer.succeed(None)
36
37		def on_user_ip(self, user_id):
	33	async def on_user_ip(self, user_id: str) -> None:
38	34	"""Called on the master when a worker process saw a client request.
39	35
40	36	Args:
41		user_id (str): mxid
42
43		Returns:
44		Deferred
	37	user_id: mxid
45	38	"""
46	39	raise AssertionError("on_user_ip unexpectedly called on worker")

+1

-1

synapse/state/__init__.py less more

27	27	from synapse.events.snapshot import EventContext
28	28	from synapse.logging.utils import log_function
29	29	from synapse.state import v1, v2
30		from synapse.storage.data_stores.main.events_worker import EventRedactBehaviour
	30	from synapse.storage.databases.main.events_worker import EventRedactBehaviour
31	31	from synapse.storage.roommember import ProfileInfo
32	32	from synapse.types import StateMap
33	33	from synapse.util import Clock

+8

-7

synapse/storage/__init__.py less more

16	16	"""
17	17	The storage layer is split up into multiple parts to allow Synapse to run
18	18	against different configurations of databases (e.g. single or multiple
19		databases). The `Database` class represents a single physical database. The
20		`data_stores` are classes that talk directly to a `Database` instance and have
21		associated schemas, background updates, etc. On top of those there are classes
22		that provide high level interfaces that combine calls to multiple `data_stores`.
	19	databases). The `DatabasePool` class represents connections to a single physical
	20	database. The `databases` are classes that talk directly to a `DatabasePool`
	21	instance and have associated schemas, background updates, etc. On top of those
	22	there are classes that provide high level interfaces that combine calls to
	23	multiple `databases`.
23	24
24	25	There are also schemas that get applied to every database, regardless of the
25	26	data stores associated with them (e.g. the schema version tables), which are
26	27	stored in `synapse.storage.schema`.
27	28	"""
28	29
29		from synapse.storage.data_stores import DataStores
30		from synapse.storage.data_stores.main import DataStore
	30	from synapse.storage.databases import Databases
	31	from synapse.storage.databases.main import DataStore
31	32	from synapse.storage.persist_events import EventsPersistenceStorage
32	33	from synapse.storage.purge_events import PurgeEventsStorage
33	34	from synapse.storage.state import StateGroupStorage

39	40	"""The high level interfaces for talking to various storage layers.
40	41	"""
41	42
42		def __init__(self, hs, stores: DataStores):
	43	def __init__(self, hs, stores: Databases):
43	44	# We include the main data store here mainly so that we don't have to
44	45	# rewrite all the existing code to split it into high vs low level
45	46	# interfaces.

+3

-4

synapse/storage/_base.py less more

22	22
23	23	from synapse.storage.database import LoggingTransaction # noqa: F401
24	24	from synapse.storage.database import make_in_list_sql_clause # noqa: F401
25		from synapse.storage.database import Database
	25	from synapse.storage.database import DatabasePool
26	26	from synapse.types import Collection, get_domain_from_id
27	27
28	28	logger = logging.getLogger(__name__)

36	36	per data store (and not one per physical database).
37	37	"""
38	38
39		def __init__(self, database: Database, db_conn, hs):
	39	def __init__(self, database: DatabasePool, db_conn, hs):
40	40	self.hs = hs
41	41	self._clock = hs.get_clock()
42	42	self.database_engine = database.engine
43		self.db = database
	43	self.db_pool = database
44	44	self.rand = random.SystemRandom()
45	45
46	46	def process_replication_rows(self, stream_name, instance_name, token, rows):

57	57	"""
58	58	for host in {get_domain_from_id(u) for u in members_changed}:
59	59	self._attempt_to_invalidate_cache("is_host_joined", (room_id, host))
60		self._attempt_to_invalidate_cache("was_host_joined", (room_id, host))
61	60
62	61	self._attempt_to_invalidate_cache("get_users_in_room", (room_id,))
63	62	self._attempt_to_invalidate_cache("get_room_summary", (room_id,))

+11

-11

synapse/storage/background_updates.py less more

87	87
88	88	def __init__(self, hs, database):
89	89	self._clock = hs.get_clock()
90		self.db = database
	90	self.db_pool = database
91	91
92	92	# if a background update is currently running, its name.
93	93	self._current_background_update = None # type: Optional[str]

138	138	# otherwise, check if there are updates to be run. This is important,
139	139	# as we may be running on a worker which doesn't perform the bg updates
140	140	# itself, but still wants to wait for them to happen.
141		updates = await self.db.simple_select_onecol(
	141	updates = await self.db_pool.simple_select_onecol(
142	142	"background_updates",
143	143	keyvalues=None,
144	144	retcol="1",

159	159	if update_name == self._current_background_update:
160	160	return False
161	161
162		update_exists = await self.db.simple_select_one_onecol(
	162	update_exists = await self.db_pool.simple_select_one_onecol(
163	163	"background_updates",
164	164	keyvalues={"update_name": update_name},
165	165	retcol="1",

188	188	ORDER BY ordering, update_name
189	189	"""
190	190	)
191		return self.db.cursor_to_dict(txn)
	191	return self.db_pool.cursor_to_dict(txn)
192	192
193	193	if not self._current_background_update:
194		all_pending_updates = await self.db.runInteraction(
	194	all_pending_updates = await self.db_pool.runInteraction(
195	195	"background_updates", get_background_updates_txn,
196	196	)
197	197	if not all_pending_updates:

242	242	else:
243	243	batch_size = self.DEFAULT_BACKGROUND_BATCH_SIZE
244	244
245		progress_json = await self.db.simple_select_one_onecol(
	245	progress_json = await self.db_pool.simple_select_one_onecol(
246	246	"background_updates",
247	247	keyvalues={"update_name": update_name},
248	248	retcol="progress_json",

401	401	logger.debug("[SQL] %s", sql)
402	402	c.execute(sql)
403	403
404		if isinstance(self.db.engine, engines.PostgresEngine):
	404	if isinstance(self.db_pool.engine, engines.PostgresEngine):
405	405	runner = create_index_psql
406	406	elif psql_only:
407	407	runner = None

412	412	def updater(progress, batch_size):
413	413	if runner is not None:
414	414	logger.info("Adding index %s to %s", index_name, table)
415		yield self.db.runWithConnection(runner)
	415	yield self.db_pool.runWithConnection(runner)
416	416	yield self._end_background_update(update_name)
417	417	return 1
418	418

432	432	% update_name
433	433	)
434	434	self._current_background_update = None
435		return self.db.simple_delete_one(
	435	return self.db_pool.simple_delete_one(
436	436	"background_updates", keyvalues={"update_name": update_name}
437	437	)
438	438

444	444	progress: The progress of the update.
445	445	"""
446	446
447		return self.db.runInteraction(
	447	return self.db_pool.runInteraction(
448	448	"background_update_progress",
449	449	self._background_update_progress_txn,
450	450	update_name,

462	462
463	463	progress_json = json.dumps(progress)
464	464
465		self.db.simple_update_one_txn(
	465	self.db_pool.simple_update_one_txn(
466	466	txn,
467	467	"background_updates",
468	468	keyvalues={"update_name": update_name},

+0

-97

~~synapse/storage/data_stores/__init__.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2019 The Matrix.org Foundation C.I.C.
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16
17		from synapse.storage.data_stores.main.events import PersistEventsStore
18		from synapse.storage.data_stores.state import StateGroupDataStore
19		from synapse.storage.database import Database, make_conn
20		from synapse.storage.engines import create_engine
21		from synapse.storage.prepare_database import prepare_database
22
23		logger = logging.getLogger(__name__)
24
25
26		class DataStores(object):
27		"""The various data stores.
28
29		These are low level interfaces to physical databases.
30
31		Attributes:
32		main (DataStore)
33		"""
34
35		def __init__(self, main_store_class, hs):
36		# Note we pass in the main store class here as workers use a different main
37		# store.
38
39		self.databases = []
40		self.main = None
41		self.state = None
42		self.persist_events = None
43
44		for database_config in hs.config.database.databases:
45		db_name = database_config.name
46		engine = create_engine(database_config.config)
47
48		with make_conn(database_config, engine) as db_conn:
49		logger.info("Preparing database %r...", db_name)
50
51		engine.check_database(db_conn)
52		prepare_database(
53		db_conn, engine, hs.config, data_stores=database_config.data_stores,
54		)
55
56		database = Database(hs, database_config, engine)
57
58		if "main" in database_config.data_stores:
59		logger.info("Starting 'main' data store")
60
61		# Sanity check we don't try and configure the main store on
62		# multiple databases.
63		if self.main:
64		raise Exception("'main' data store already configured")
65
66		self.main = main_store_class(database, db_conn, hs)
67
68		# If we're on a process that can persist events also
69		# instantiate a `PersistEventsStore`
70		if hs.config.worker.writers.events == hs.get_instance_name():
71		self.persist_events = PersistEventsStore(
72		hs, database, self.main
73		)
74
75		if "state" in database_config.data_stores:
76		logger.info("Starting 'state' data store")
77
78		# Sanity check we don't try and configure the state store on
79		# multiple databases.
80		if self.state:
81		raise Exception("'state' data store already configured")
82
83		self.state = StateGroupDataStore(database, db_conn, hs)
84
85		db_conn.commit()
86
87		self.databases.append(database)
88
89		logger.info("Database %r prepared", db_name)
90
91		# Sanity check that we have actually configured all the required stores.
92		if not self.main:
93		raise Exception("No 'main' data store configured")
94
95		if not self.state:
96		raise Exception("No 'main' data store configured")

+0

-592

~~synapse/storage/data_stores/main/__init__.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2018 New Vector Ltd
3		# Copyright 2019 The Matrix.org Foundation C.I.C.
4		#
5		# Licensed under the Apache License, Version 2.0 (the "License");
6		# you may not use this file except in compliance with the License.
7		# You may obtain a copy of the License at
8		#
9		# http://www.apache.org/licenses/LICENSE-2.0
10		#
11		# Unless required by applicable law or agreed to in writing, software
12		# distributed under the License is distributed on an "AS IS" BASIS,
13		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14		# See the License for the specific language governing permissions and
15		# limitations under the License.
16
17		import calendar
18		import logging
19		import time
20
21		from synapse.api.constants import PresenceState
22		from synapse.config.homeserver import HomeServerConfig
23		from synapse.storage.database import Database
24		from synapse.storage.engines import PostgresEngine
25		from synapse.storage.util.id_generators import (
26		IdGenerator,
27		MultiWriterIdGenerator,
28		StreamIdGenerator,
29		)
30		from synapse.util.caches.stream_change_cache import StreamChangeCache
31
32		from .account_data import AccountDataStore
33		from .appservice import ApplicationServiceStore, ApplicationServiceTransactionStore
34		from .cache import CacheInvalidationWorkerStore
35		from .censor_events import CensorEventsStore
36		from .client_ips import ClientIpStore
37		from .deviceinbox import DeviceInboxStore
38		from .devices import DeviceStore
39		from .directory import DirectoryStore
40		from .e2e_room_keys import EndToEndRoomKeyStore
41		from .end_to_end_keys import EndToEndKeyStore
42		from .event_federation import EventFederationStore
43		from .event_push_actions import EventPushActionsStore
44		from .events_bg_updates import EventsBackgroundUpdatesStore
45		from .filtering import FilteringStore
46		from .group_server import GroupServerStore
47		from .keys import KeyStore
48		from .media_repository import MediaRepositoryStore
49		from .metrics import ServerMetricsStore
50		from .monthly_active_users import MonthlyActiveUsersStore
51		from .openid import OpenIdStore
52		from .presence import PresenceStore, UserPresenceState
53		from .profile import ProfileStore
54		from .purge_events import PurgeEventsStore
55		from .push_rule import PushRuleStore
56		from .pusher import PusherStore
57		from .receipts import ReceiptsStore
58		from .registration import RegistrationStore
59		from .rejections import RejectionsStore
60		from .relations import RelationsStore
61		from .room import RoomStore
62		from .roommember import RoomMemberStore
63		from .search import SearchStore
64		from .signatures import SignatureStore
65		from .state import StateStore
66		from .stats import StatsStore
67		from .stream import StreamStore
68		from .tags import TagsStore
69		from .transactions import TransactionStore
70		from .ui_auth import UIAuthStore
71		from .user_directory import UserDirectoryStore
72		from .user_erasure_store import UserErasureStore
73
74		logger = logging.getLogger(__name__)
75
76
77		class DataStore(
78		EventsBackgroundUpdatesStore,
79		RoomMemberStore,
80		RoomStore,
81		RegistrationStore,
82		StreamStore,
83		ProfileStore,
84		PresenceStore,
85		TransactionStore,
86		DirectoryStore,
87		KeyStore,
88		StateStore,
89		SignatureStore,
90		ApplicationServiceStore,
91		PurgeEventsStore,
92		EventFederationStore,
93		MediaRepositoryStore,
94		RejectionsStore,
95		FilteringStore,
96		PusherStore,
97		PushRuleStore,
98		ApplicationServiceTransactionStore,
99		ReceiptsStore,
100		EndToEndKeyStore,
101		EndToEndRoomKeyStore,
102		SearchStore,
103		TagsStore,
104		AccountDataStore,
105		EventPushActionsStore,
106		OpenIdStore,
107		ClientIpStore,
108		DeviceStore,
109		DeviceInboxStore,
110		UserDirectoryStore,
111		GroupServerStore,
112		UserErasureStore,
113		MonthlyActiveUsersStore,
114		StatsStore,
115		RelationsStore,
116		CensorEventsStore,
117		UIAuthStore,
118		CacheInvalidationWorkerStore,
119		ServerMetricsStore,
120		):
121		def __init__(self, database: Database, db_conn, hs):
122		self.hs = hs
123		self._clock = hs.get_clock()
124		self.database_engine = database.engine
125
126		self._presence_id_gen = StreamIdGenerator(
127		db_conn, "presence_stream", "stream_id"
128		)
129		self._device_inbox_id_gen = StreamIdGenerator(
130		db_conn, "device_inbox", "stream_id"
131		)
132		self._public_room_id_gen = StreamIdGenerator(
133		db_conn, "public_room_list_stream", "stream_id"
134		)
135		self._device_list_id_gen = StreamIdGenerator(
136		db_conn,
137		"device_lists_stream",
138		"stream_id",
139		extra_tables=[
140		("user_signature_stream", "stream_id"),
141		("device_lists_outbound_pokes", "stream_id"),
142		],
143		)
144		self._cross_signing_id_gen = StreamIdGenerator(
145		db_conn, "e2e_cross_signing_keys", "stream_id"
146		)
147
148		self._access_tokens_id_gen = IdGenerator(db_conn, "access_tokens", "id")
149		self._event_reports_id_gen = IdGenerator(db_conn, "event_reports", "id")
150		self._push_rule_id_gen = IdGenerator(db_conn, "push_rules", "id")
151		self._push_rules_enable_id_gen = IdGenerator(db_conn, "push_rules_enable", "id")
152		self._pushers_id_gen = StreamIdGenerator(
153		db_conn, "pushers", "id", extra_tables=[("deleted_pushers", "stream_id")]
154		)
155		self._group_updates_id_gen = StreamIdGenerator(
156		db_conn, "local_group_updates", "stream_id"
157		)
158
159		if isinstance(self.database_engine, PostgresEngine):
160		self._cache_id_gen = MultiWriterIdGenerator(
161		db_conn,
162		database,
163		instance_name="master",
164		table="cache_invalidation_stream_by_instance",
165		instance_column="instance_name",
166		id_column="stream_id",
167		sequence_name="cache_invalidation_stream_seq",
168		)
169		else:
170		self._cache_id_gen = None
171
172		super(DataStore, self).__init__(database, db_conn, hs)
173
174		self._presence_on_startup = self._get_active_presence(db_conn)
175
176		presence_cache_prefill, min_presence_val = self.db.get_cache_dict(
177		db_conn,
178		"presence_stream",
179		entity_column="user_id",
180		stream_column="stream_id",
181		max_value=self._presence_id_gen.get_current_token(),
182		)
183		self.presence_stream_cache = StreamChangeCache(
184		"PresenceStreamChangeCache",
185		min_presence_val,
186		prefilled_cache=presence_cache_prefill,
187		)
188
189		max_device_inbox_id = self._device_inbox_id_gen.get_current_token()
190		device_inbox_prefill, min_device_inbox_id = self.db.get_cache_dict(
191		db_conn,
192		"device_inbox",
193		entity_column="user_id",
194		stream_column="stream_id",
195		max_value=max_device_inbox_id,
196		limit=1000,
197		)
198		self._device_inbox_stream_cache = StreamChangeCache(
199		"DeviceInboxStreamChangeCache",
200		min_device_inbox_id,
201		prefilled_cache=device_inbox_prefill,
202		)
203		# The federation outbox and the local device inbox uses the same
204		# stream_id generator.
205		device_outbox_prefill, min_device_outbox_id = self.db.get_cache_dict(
206		db_conn,
207		"device_federation_outbox",
208		entity_column="destination",
209		stream_column="stream_id",
210		max_value=max_device_inbox_id,
211		limit=1000,
212		)
213		self._device_federation_outbox_stream_cache = StreamChangeCache(
214		"DeviceFederationOutboxStreamChangeCache",
215		min_device_outbox_id,
216		prefilled_cache=device_outbox_prefill,
217		)
218
219		device_list_max = self._device_list_id_gen.get_current_token()
220		self._device_list_stream_cache = StreamChangeCache(
221		"DeviceListStreamChangeCache", device_list_max
222		)
223		self._user_signature_stream_cache = StreamChangeCache(
224		"UserSignatureStreamChangeCache", device_list_max
225		)
226		self._device_list_federation_stream_cache = StreamChangeCache(
227		"DeviceListFederationStreamChangeCache", device_list_max
228		)
229
230		events_max = self._stream_id_gen.get_current_token()
231		curr_state_delta_prefill, min_curr_state_delta_id = self.db.get_cache_dict(
232		db_conn,
233		"current_state_delta_stream",
234		entity_column="room_id",
235		stream_column="stream_id",
236		max_value=events_max, # As we share the stream id with events token
237		limit=1000,
238		)
239		self._curr_state_delta_stream_cache = StreamChangeCache(
240		"_curr_state_delta_stream_cache",
241		min_curr_state_delta_id,
242		prefilled_cache=curr_state_delta_prefill,
243		)
244
245		_group_updates_prefill, min_group_updates_id = self.db.get_cache_dict(
246		db_conn,
247		"local_group_updates",
248		entity_column="user_id",
249		stream_column="stream_id",
250		max_value=self._group_updates_id_gen.get_current_token(),
251		limit=1000,
252		)
253		self._group_updates_stream_cache = StreamChangeCache(
254		"_group_updates_stream_cache",
255		min_group_updates_id,
256		prefilled_cache=_group_updates_prefill,
257		)
258
259		self._stream_order_on_start = self.get_room_max_stream_ordering()
260		self._min_stream_order_on_start = self.get_room_min_stream_ordering()
261
262		# Used in _generate_user_daily_visits to keep track of progress
263		self._last_user_visit_update = self._get_start_of_day()
264
265		def take_presence_startup_info(self):
266		active_on_startup = self._presence_on_startup
267		self._presence_on_startup = None
268		return active_on_startup
269
270		def _get_active_presence(self, db_conn):
271		"""Fetch non-offline presence from the database so that we can register
272		the appropriate time outs.
273		"""
274
275		sql = (
276		"SELECT user_id, state, last_active_ts, last_federation_update_ts,"
277		" last_user_sync_ts, status_msg, currently_active FROM presence_stream"
278		" WHERE state != ?"
279		)
280		sql = self.database_engine.convert_param_style(sql)
281
282		txn = db_conn.cursor()
283		txn.execute(sql, (PresenceState.OFFLINE,))
284		rows = self.db.cursor_to_dict(txn)
285		txn.close()
286
287		for row in rows:
288		row["currently_active"] = bool(row["currently_active"])
289
290		return [UserPresenceState(**row) for row in rows]
291
292		def count_daily_users(self):
293		"""
294		Counts the number of users who used this homeserver in the last 24 hours.
295		"""
296		yesterday = int(self._clock.time_msec()) - (1000 * 60 * 60 * 24)
297		return self.db.runInteraction("count_daily_users", self._count_users, yesterday)
298
299		def count_monthly_users(self):
300		"""
301		Counts the number of users who used this homeserver in the last 30 days.
302		Note this method is intended for phonehome metrics only and is different
303		from the mau figure in synapse.storage.monthly_active_users which,
304		amongst other things, includes a 3 day grace period before a user counts.
305		"""
306		thirty_days_ago = int(self._clock.time_msec()) - (1000 * 60 * 60 * 24 * 30)
307		return self.db.runInteraction(
308		"count_monthly_users", self._count_users, thirty_days_ago
309		)
310
311		def _count_users(self, txn, time_from):
312		"""
313		Returns number of users seen in the past time_from period
314		"""
315		sql = """
316		SELECT COALESCE(count(*), 0) FROM (
317		SELECT user_id FROM user_ips
318		WHERE last_seen > ?
319		GROUP BY user_id
320		) u
321		"""
322		txn.execute(sql, (time_from,))
323		(count,) = txn.fetchone()
324		return count
325
326		def count_r30_users(self):
327		"""
328		Counts the number of 30 day retained users, defined as:-
329		* Users who have created their accounts more than 30 days ago
330		* Where last seen at most 30 days ago
331		* Where account creation and last_seen are > 30 days apart
332
333		Returns counts globaly for a given user as well as breaking
334		by platform
335		"""
336
337		def _count_r30_users(txn):
338		thirty_days_in_secs = 86400 * 30
339		now = int(self._clock.time())
340		thirty_days_ago_in_secs = now - thirty_days_in_secs
341
342		sql = """
343		SELECT platform, COALESCE(count(*), 0) FROM (
344		SELECT
345		users.name, platform, users.creation_ts * 1000,
346		MAX(uip.last_seen)
347		FROM users
348		INNER JOIN (
349		SELECT
350		user_id,
351		last_seen,
352		CASE
353		WHEN user_agent LIKE '%%Android%%' THEN 'android'
354		WHEN user_agent LIKE '%%iOS%%' THEN 'ios'
355		WHEN user_agent LIKE '%%Electron%%' THEN 'electron'
356		WHEN user_agent LIKE '%%Mozilla%%' THEN 'web'
357		WHEN user_agent LIKE '%%Gecko%%' THEN 'web'
358		ELSE 'unknown'
359		END
360		AS platform
361		FROM user_ips
362		) uip
363		ON users.name = uip.user_id
364		AND users.appservice_id is NULL
365		AND users.creation_ts < ?
366		AND uip.last_seen/1000 > ?
367		AND (uip.last_seen/1000) - users.creation_ts > 86400 * 30
368		GROUP BY users.name, platform, users.creation_ts
369		) u GROUP BY platform
370		"""
371
372		results = {}
373		txn.execute(sql, (thirty_days_ago_in_secs, thirty_days_ago_in_secs))
374
375		for row in txn:
376		if row[0] == "unknown":
377		pass
378		results[row[0]] = row[1]
379
380		sql = """
381		SELECT COALESCE(count(*), 0) FROM (
382		SELECT users.name, users.creation_ts * 1000,
383		MAX(uip.last_seen)
384		FROM users
385		INNER JOIN (
386		SELECT
387		user_id,
388		last_seen
389		FROM user_ips
390		) uip
391		ON users.name = uip.user_id
392		AND appservice_id is NULL
393		AND users.creation_ts < ?
394		AND uip.last_seen/1000 > ?
395		AND (uip.last_seen/1000) - users.creation_ts > 86400 * 30
396		GROUP BY users.name, users.creation_ts
397		) u
398		"""
399
400		txn.execute(sql, (thirty_days_ago_in_secs, thirty_days_ago_in_secs))
401
402		(count,) = txn.fetchone()
403		results["all"] = count
404
405		return results
406
407		return self.db.runInteraction("count_r30_users", _count_r30_users)
408
409		def _get_start_of_day(self):
410		"""
411		Returns millisecond unixtime for start of UTC day.
412		"""
413		now = time.gmtime()
414		today_start = calendar.timegm((now.tm_year, now.tm_mon, now.tm_mday, 0, 0, 0))
415		return today_start * 1000
416
417		def generate_user_daily_visits(self):
418		"""
419		Generates daily visit data for use in cohort/ retention analysis
420		"""
421
422		def _generate_user_daily_visits(txn):
423		logger.info("Calling _generate_user_daily_visits")
424		today_start = self._get_start_of_day()
425		a_day_in_milliseconds = 24 * 60 * 60 * 1000
426		now = self.clock.time_msec()
427
428		sql = """
429		INSERT INTO user_daily_visits (user_id, device_id, timestamp)
430		SELECT u.user_id, u.device_id, ?
431		FROM user_ips AS u
432		LEFT JOIN (
433		SELECT user_id, device_id, timestamp FROM user_daily_visits
434		WHERE timestamp = ?
435		) udv
436		ON u.user_id = udv.user_id AND u.device_id=udv.device_id
437		INNER JOIN users ON users.name=u.user_id
438		WHERE last_seen > ? AND last_seen <= ?
439		AND udv.timestamp IS NULL AND users.is_guest=0
440		AND users.appservice_id IS NULL
441		GROUP BY u.user_id, u.device_id
442		"""
443
444		# This means that the day has rolled over but there could still
445		# be entries from the previous day. There is an edge case
446		# where if the user logs in at 23:59 and overwrites their
447		# last_seen at 00:01 then they will not be counted in the
448		# previous day's stats - it is important that the query is run
449		# often to minimise this case.
450		if today_start > self._last_user_visit_update:
451		yesterday_start = today_start - a_day_in_milliseconds
452		txn.execute(
453		sql,
454		(
455		yesterday_start,
456		yesterday_start,
457		self._last_user_visit_update,
458		today_start,
459		),
460		)
461		self._last_user_visit_update = today_start
462
463		txn.execute(
464		sql, (today_start, today_start, self._last_user_visit_update, now)
465		)
466		# Update _last_user_visit_update to now. The reason to do this
467		# rather just clamping to the beginning of the day is to limit
468		# the size of the join - meaning that the query can be run more
469		# frequently
470		self._last_user_visit_update = now
471
472		return self.db.runInteraction(
473		"generate_user_daily_visits", _generate_user_daily_visits
474		)
475
476		def get_users(self):
477		"""Function to retrieve a list of users in users table.
478
479		Args:
480		Returns:
481		defer.Deferred: resolves to list[dict[str, Any]]
482		"""
483		return self.db.simple_select_list(
484		table="users",
485		keyvalues={},
486		retcols=[
487		"name",
488		"password_hash",
489		"is_guest",
490		"admin",
491		"user_type",
492		"deactivated",
493		],
494		desc="get_users",
495		)
496
497		def get_users_paginate(
498		self, start, limit, name=None, guests=True, deactivated=False
499		):
500		"""Function to retrieve a paginated list of users from
501		users list. This will return a json list of users and the
502		total number of users matching the filter criteria.
503
504		Args:
505		start (int): start number to begin the query from
506		limit (int): number of rows to retrieve
507		name (string): filter for user names
508		guests (bool): whether to in include guest users
509		deactivated (bool): whether to include deactivated users
510		Returns:
511		defer.Deferred: resolves to list[dict[str, Any]], int
512		"""
513
514		def get_users_paginate_txn(txn):
515		filters = []
516		args = []
517
518		if name:
519		filters.append("name LIKE ?")
520		args.append("%" + name + "%")
521
522		if not guests:
523		filters.append("is_guest = 0")
524
525		if not deactivated:
526		filters.append("deactivated = 0")
527
528		where_clause = "WHERE " + " AND ".join(filters) if len(filters) > 0 else ""
529
530		sql = "SELECT COUNT(*) as total_users FROM users %s" % (where_clause)
531		txn.execute(sql, args)
532		count = txn.fetchone()[0]
533
534		args = [self.hs.config.server_name] + args + [limit, start]
535		sql = """
536		SELECT name, user_type, is_guest, admin, deactivated, displayname, avatar_url
537		FROM users as u
538		LEFT JOIN profiles AS p ON u.name = '@' \|\| p.user_id \|\| ':' \|\| ?
539		{}
540		ORDER BY u.name LIMIT ? OFFSET ?
541		""".format(
542		where_clause
543		)
544		txn.execute(sql, args)
545		users = self.db.cursor_to_dict(txn)
546		return users, count
547
548		return self.db.runInteraction("get_users_paginate_txn", get_users_paginate_txn)
549
550		def search_users(self, term):
551		"""Function to search users list for one or more users with
552		the matched term.
553
554		Args:
555		term (str): search term
556		col (str): column to query term should be matched to
557		Returns:
558		defer.Deferred: resolves to list[dict[str, Any]]
559		"""
560		return self.db.simple_search_list(
561		table="users",
562		term=term,
563		col="name",
564		retcols=["name", "password_hash", "is_guest", "admin", "user_type"],
565		desc="search_users",
566		)
567
568
569		def check_database_before_upgrade(cur, database_engine, config: HomeServerConfig):
570		"""Called before upgrading an existing database to check that it is broadly sane
571		compared with the configuration.
572		"""
573		domain = config.server_name
574
575		sql = database_engine.convert_param_style(
576		"SELECT COUNT(*) FROM users WHERE name NOT LIKE ?"
577		)
578		pat = "%:" + domain
579		cur.execute(sql, (pat,))
580		num_not_matching = cur.fetchall()[0][0]
581		if num_not_matching == 0:
582		return
583
584		raise Exception(
585		"Found users in database not native to %s!\n"
586		"You cannot changed a synapse server_name after it's been configured"
587		% (domain,)
588		)
589
590
591		__all__ = ["DataStore", "check_database_before_upgrade"]

+0

-430

~~synapse/storage/data_stores/main/account_data.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2018 New Vector Ltd
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		import abc
17		import logging
18		from typing import List, Tuple
19
20		from canonicaljson import json
21
22		from twisted.internet import defer
23
24		from synapse.storage._base import SQLBaseStore, db_to_json
25		from synapse.storage.database import Database
26		from synapse.storage.util.id_generators import StreamIdGenerator
27		from synapse.util.caches.descriptors import cached, cachedInlineCallbacks
28		from synapse.util.caches.stream_change_cache import StreamChangeCache
29
30		logger = logging.getLogger(__name__)
31
32
33		class AccountDataWorkerStore(SQLBaseStore):
34		"""This is an abstract base class where subclasses must implement
35		`get_max_account_data_stream_id` which can be called in the initializer.
36		"""
37
38		# This ABCMeta metaclass ensures that we cannot be instantiated without
39		# the abstract methods being implemented.
40		__metaclass__ = abc.ABCMeta
41
42		def __init__(self, database: Database, db_conn, hs):
43		account_max = self.get_max_account_data_stream_id()
44		self._account_data_stream_cache = StreamChangeCache(
45		"AccountDataAndTagsChangeCache", account_max
46		)
47
48		super(AccountDataWorkerStore, self).__init__(database, db_conn, hs)
49
50		@abc.abstractmethod
51		def get_max_account_data_stream_id(self):
52		"""Get the current max stream ID for account data stream
53
54		Returns:
55		int
56		"""
57		raise NotImplementedError()
58
59		@cached()
60		def get_account_data_for_user(self, user_id):
61		"""Get all the client account_data for a user.
62
63		Args:
64		user_id(str): The user to get the account_data for.
65		Returns:
66		A deferred pair of a dict of global account_data and a dict
67		mapping from room_id string to per room account_data dicts.
68		"""
69
70		def get_account_data_for_user_txn(txn):
71		rows = self.db.simple_select_list_txn(
72		txn,
73		"account_data",
74		{"user_id": user_id},
75		["account_data_type", "content"],
76		)
77
78		global_account_data = {
79		row["account_data_type"]: db_to_json(row["content"]) for row in rows
80		}
81
82		rows = self.db.simple_select_list_txn(
83		txn,
84		"room_account_data",
85		{"user_id": user_id},
86		["room_id", "account_data_type", "content"],
87		)
88
89		by_room = {}
90		for row in rows:
91		room_data = by_room.setdefault(row["room_id"], {})
92		room_data[row["account_data_type"]] = db_to_json(row["content"])
93
94		return global_account_data, by_room
95
96		return self.db.runInteraction(
97		"get_account_data_for_user", get_account_data_for_user_txn
98		)
99
100		@cachedInlineCallbacks(num_args=2, max_entries=5000)
101		def get_global_account_data_by_type_for_user(self, data_type, user_id):
102		"""
103		Returns:
104		Deferred: A dict
105		"""
106		result = yield self.db.simple_select_one_onecol(
107		table="account_data",
108		keyvalues={"user_id": user_id, "account_data_type": data_type},
109		retcol="content",
110		desc="get_global_account_data_by_type_for_user",
111		allow_none=True,
112		)
113
114		if result:
115		return db_to_json(result)
116		else:
117		return None
118
119		@cached(num_args=2)
120		def get_account_data_for_room(self, user_id, room_id):
121		"""Get all the client account_data for a user for a room.
122
123		Args:
124		user_id(str): The user to get the account_data for.
125		room_id(str): The room to get the account_data for.
126		Returns:
127		A deferred dict of the room account_data
128		"""
129
130		def get_account_data_for_room_txn(txn):
131		rows = self.db.simple_select_list_txn(
132		txn,
133		"room_account_data",
134		{"user_id": user_id, "room_id": room_id},
135		["account_data_type", "content"],
136		)
137
138		return {
139		row["account_data_type"]: db_to_json(row["content"]) for row in rows
140		}
141
142		return self.db.runInteraction(
143		"get_account_data_for_room", get_account_data_for_room_txn
144		)
145
146		@cached(num_args=3, max_entries=5000)
147		def get_account_data_for_room_and_type(self, user_id, room_id, account_data_type):
148		"""Get the client account_data of given type for a user for a room.
149
150		Args:
151		user_id(str): The user to get the account_data for.
152		room_id(str): The room to get the account_data for.
153		account_data_type (str): The account data type to get.
154		Returns:
155		A deferred of the room account_data for that type, or None if
156		there isn't any set.
157		"""
158
159		def get_account_data_for_room_and_type_txn(txn):
160		content_json = self.db.simple_select_one_onecol_txn(
161		txn,
162		table="room_account_data",
163		keyvalues={
164		"user_id": user_id,
165		"room_id": room_id,
166		"account_data_type": account_data_type,
167		},
168		retcol="content",
169		allow_none=True,
170		)
171
172		return db_to_json(content_json) if content_json else None
173
174		return self.db.runInteraction(
175		"get_account_data_for_room_and_type", get_account_data_for_room_and_type_txn
176		)
177
178		async def get_updated_global_account_data(
179		self, last_id: int, current_id: int, limit: int
180		) -> List[Tuple[int, str, str]]:
181		"""Get the global account_data that has changed, for the account_data stream
182
183		Args:
184		last_id: the last stream_id from the previous batch.
185		current_id: the maximum stream_id to return up to
186		limit: the maximum number of rows to return
187
188		Returns:
189		A list of tuples of stream_id int, user_id string,
190		and type string.
191		"""
192		if last_id == current_id:
193		return []
194
195		def get_updated_global_account_data_txn(txn):
196		sql = (
197		"SELECT stream_id, user_id, account_data_type"
198		" FROM account_data WHERE ? < stream_id AND stream_id <= ?"
199		" ORDER BY stream_id ASC LIMIT ?"
200		)
201		txn.execute(sql, (last_id, current_id, limit))
202		return txn.fetchall()
203
204		return await self.db.runInteraction(
205		"get_updated_global_account_data", get_updated_global_account_data_txn
206		)
207
208		async def get_updated_room_account_data(
209		self, last_id: int, current_id: int, limit: int
210		) -> List[Tuple[int, str, str, str]]:
211		"""Get the global account_data that has changed, for the account_data stream
212
213		Args:
214		last_id: the last stream_id from the previous batch.
215		current_id: the maximum stream_id to return up to
216		limit: the maximum number of rows to return
217
218		Returns:
219		A list of tuples of stream_id int, user_id string,
220		room_id string and type string.
221		"""
222		if last_id == current_id:
223		return []
224
225		def get_updated_room_account_data_txn(txn):
226		sql = (
227		"SELECT stream_id, user_id, room_id, account_data_type"
228		" FROM room_account_data WHERE ? < stream_id AND stream_id <= ?"
229		" ORDER BY stream_id ASC LIMIT ?"
230		)
231		txn.execute(sql, (last_id, current_id, limit))
232		return txn.fetchall()
233
234		return await self.db.runInteraction(
235		"get_updated_room_account_data", get_updated_room_account_data_txn
236		)
237
238		def get_updated_account_data_for_user(self, user_id, stream_id):
239		"""Get all the client account_data for a that's changed for a user
240
241		Args:
242		user_id(str): The user to get the account_data for.
243		stream_id(int): The point in the stream since which to get updates
244		Returns:
245		A deferred pair of a dict of global account_data and a dict
246		mapping from room_id string to per room account_data dicts.
247		"""
248
249		def get_updated_account_data_for_user_txn(txn):
250		sql = (
251		"SELECT account_data_type, content FROM account_data"
252		" WHERE user_id = ? AND stream_id > ?"
253		)
254
255		txn.execute(sql, (user_id, stream_id))
256
257		global_account_data = {row[0]: db_to_json(row[1]) for row in txn}
258
259		sql = (
260		"SELECT room_id, account_data_type, content FROM room_account_data"
261		" WHERE user_id = ? AND stream_id > ?"
262		)
263
264		txn.execute(sql, (user_id, stream_id))
265
266		account_data_by_room = {}
267		for row in txn:
268		room_account_data = account_data_by_room.setdefault(row[0], {})
269		room_account_data[row[1]] = db_to_json(row[2])
270
271		return global_account_data, account_data_by_room
272
273		changed = self._account_data_stream_cache.has_entity_changed(
274		user_id, int(stream_id)
275		)
276		if not changed:
277		return defer.succeed(({}, {}))
278
279		return self.db.runInteraction(
280		"get_updated_account_data_for_user", get_updated_account_data_for_user_txn
281		)
282
283		@cachedInlineCallbacks(num_args=2, cache_context=True, max_entries=5000)
284		def is_ignored_by(self, ignored_user_id, ignorer_user_id, cache_context):
285		ignored_account_data = yield self.get_global_account_data_by_type_for_user(
286		"m.ignored_user_list",
287		ignorer_user_id,
288		on_invalidate=cache_context.invalidate,
289		)
290		if not ignored_account_data:
291		return False
292
293		return ignored_user_id in ignored_account_data.get("ignored_users", {})
294
295
296		class AccountDataStore(AccountDataWorkerStore):
297		def __init__(self, database: Database, db_conn, hs):
298		self._account_data_id_gen = StreamIdGenerator(
299		db_conn,
300		"account_data_max_stream_id",
301		"stream_id",
302		extra_tables=[
303		("room_account_data", "stream_id"),
304		("room_tags_revisions", "stream_id"),
305		],
306		)
307
308		super(AccountDataStore, self).__init__(database, db_conn, hs)
309
310		def get_max_account_data_stream_id(self):
311		"""Get the current max stream id for the private user data stream
312
313		Returns:
314		A deferred int.
315		"""
316		return self._account_data_id_gen.get_current_token()
317
318		@defer.inlineCallbacks
319		def add_account_data_to_room(self, user_id, room_id, account_data_type, content):
320		"""Add some account_data to a room for a user.
321		Args:
322		user_id(str): The user to add a tag for.
323		room_id(str): The room to add a tag for.
324		account_data_type(str): The type of account_data to add.
325		content(dict): A json object to associate with the tag.
326		Returns:
327		A deferred that completes once the account_data has been added.
328		"""
329		content_json = json.dumps(content)
330
331		with self._account_data_id_gen.get_next() as next_id:
332		# no need to lock here as room_account_data has a unique constraint
333		# on (user_id, room_id, account_data_type) so simple_upsert will
334		# retry if there is a conflict.
335		yield self.db.simple_upsert(
336		desc="add_room_account_data",
337		table="room_account_data",
338		keyvalues={
339		"user_id": user_id,
340		"room_id": room_id,
341		"account_data_type": account_data_type,
342		},
343		values={"stream_id": next_id, "content": content_json},
344		lock=False,
345		)
346
347		# it's theoretically possible for the above to succeed and the
348		# below to fail - in which case we might reuse a stream id on
349		# restart, and the above update might not get propagated. That
350		# doesn't sound any worse than the whole update getting lost,
351		# which is what would happen if we combined the two into one
352		# transaction.
353		yield self._update_max_stream_id(next_id)
354
355		self._account_data_stream_cache.entity_has_changed(user_id, next_id)
356		self.get_account_data_for_user.invalidate((user_id,))
357		self.get_account_data_for_room.invalidate((user_id, room_id))
358		self.get_account_data_for_room_and_type.prefill(
359		(user_id, room_id, account_data_type), content
360		)
361
362		result = self._account_data_id_gen.get_current_token()
363		return result
364
365		@defer.inlineCallbacks
366		def add_account_data_for_user(self, user_id, account_data_type, content):
367		"""Add some account_data to a room for a user.
368		Args:
369		user_id(str): The user to add a tag for.
370		account_data_type(str): The type of account_data to add.
371		content(dict): A json object to associate with the tag.
372		Returns:
373		A deferred that completes once the account_data has been added.
374		"""
375		content_json = json.dumps(content)
376
377		with self._account_data_id_gen.get_next() as next_id:
378		# no need to lock here as account_data has a unique constraint on
379		# (user_id, account_data_type) so simple_upsert will retry if
380		# there is a conflict.
381		yield self.db.simple_upsert(
382		desc="add_user_account_data",
383		table="account_data",
384		keyvalues={"user_id": user_id, "account_data_type": account_data_type},
385		values={"stream_id": next_id, "content": content_json},
386		lock=False,
387		)
388
389		# it's theoretically possible for the above to succeed and the
390		# below to fail - in which case we might reuse a stream id on
391		# restart, and the above update might not get propagated. That
392		# doesn't sound any worse than the whole update getting lost,
393		# which is what would happen if we combined the two into one
394		# transaction.
395		#
396		# Note: This is only here for backwards compat to allow admins to
397		# roll back to a previous Synapse version. Next time we update the
398		# database version we can remove this table.
399		yield self._update_max_stream_id(next_id)
400
401		self._account_data_stream_cache.entity_has_changed(user_id, next_id)
402		self.get_account_data_for_user.invalidate((user_id,))
403		self.get_global_account_data_by_type_for_user.invalidate(
404		(account_data_type, user_id)
405		)
406
407		result = self._account_data_id_gen.get_current_token()
408		return result
409
410		def _update_max_stream_id(self, next_id):
411		"""Update the max stream_id
412
413		Args:
414		next_id(int): The the revision to advance to.
415		"""
416
417		# Note: This is only here for backwards compat to allow admins to
418		# roll back to a previous Synapse version. Next time we update the
419		# database version we can remove this table.
420
421		def _update(txn):
422		update_max_id_sql = (
423		"UPDATE account_data_max_stream_id"
424		" SET stream_id = ?"
425		" WHERE stream_id < ?"
426		)
427		txn.execute(update_max_id_sql, (next_id, next_id))
428
429		return self.db.runInteraction("update_account_data_max_stream_id", _update)

+0

-372

~~synapse/storage/data_stores/main/appservice.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2015, 2016 OpenMarket Ltd
2		# Copyright 2018 New Vector Ltd
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15		import logging
16		import re
17
18		from canonicaljson import json
19
20		from twisted.internet import defer
21
22		from synapse.appservice import AppServiceTransaction
23		from synapse.config.appservice import load_appservices
24		from synapse.storage._base import SQLBaseStore, db_to_json
25		from synapse.storage.data_stores.main.events_worker import EventsWorkerStore
26		from synapse.storage.database import Database
27
28		logger = logging.getLogger(__name__)
29
30
31		def _make_exclusive_regex(services_cache):
32		# We precompile a regex constructed from all the regexes that the AS's
33		# have registered for exclusive users.
34		exclusive_user_regexes = [
35		regex.pattern
36		for service in services_cache
37		for regex in service.get_exclusive_user_regexes()
38		]
39		if exclusive_user_regexes:
40		exclusive_user_regex = "\|".join("(" + r + ")" for r in exclusive_user_regexes)
41		exclusive_user_regex = re.compile(exclusive_user_regex)
42		else:
43		# We handle this case specially otherwise the constructed regex
44		# will always match
45		exclusive_user_regex = None
46
47		return exclusive_user_regex
48
49
50		class ApplicationServiceWorkerStore(SQLBaseStore):
51		def __init__(self, database: Database, db_conn, hs):
52		self.services_cache = load_appservices(
53		hs.hostname, hs.config.app_service_config_files
54		)
55		self.exclusive_user_regex = _make_exclusive_regex(self.services_cache)
56
57		super(ApplicationServiceWorkerStore, self).__init__(database, db_conn, hs)
58
59		def get_app_services(self):
60		return self.services_cache
61
62		def get_if_app_services_interested_in_user(self, user_id):
63		"""Check if the user is one associated with an app service (exclusively)
64		"""
65		if self.exclusive_user_regex:
66		return bool(self.exclusive_user_regex.match(user_id))
67		else:
68		return False
69
70		def get_app_service_by_user_id(self, user_id):
71		"""Retrieve an application service from their user ID.
72
73		All application services have associated with them a particular user ID.
74		There is no distinguishing feature on the user ID which indicates it
75		represents an application service. This function allows you to map from
76		a user ID to an application service.
77
78		Args:
79		user_id(str): The user ID to see if it is an application service.
80		Returns:
81		synapse.appservice.ApplicationService or None.
82		"""
83		for service in self.services_cache:
84		if service.sender == user_id:
85		return service
86		return None
87
88		def get_app_service_by_token(self, token):
89		"""Get the application service with the given appservice token.
90
91		Args:
92		token (str): The application service token.
93		Returns:
94		synapse.appservice.ApplicationService or None.
95		"""
96		for service in self.services_cache:
97		if service.token == token:
98		return service
99		return None
100
101		def get_app_service_by_id(self, as_id):
102		"""Get the application service with the given appservice ID.
103
104		Args:
105		as_id (str): The application service ID.
106		Returns:
107		synapse.appservice.ApplicationService or None.
108		"""
109		for service in self.services_cache:
110		if service.id == as_id:
111		return service
112		return None
113
114
115		class ApplicationServiceStore(ApplicationServiceWorkerStore):
116		# This is currently empty due to there not being any AS storage functions
117		# that can't be run on the workers. Since this may change in future, and
118		# to keep consistency with the other stores, we keep this empty class for
119		# now.
120		pass
121
122
123		class ApplicationServiceTransactionWorkerStore(
124		ApplicationServiceWorkerStore, EventsWorkerStore
125		):
126		@defer.inlineCallbacks
127		def get_appservices_by_state(self, state):
128		"""Get a list of application services based on their state.
129
130		Args:
131		state(ApplicationServiceState): The state to filter on.
132		Returns:
133		A Deferred which resolves to a list of ApplicationServices, which
134		may be empty.
135		"""
136		results = yield self.db.simple_select_list(
137		"application_services_state", {"state": state}, ["as_id"]
138		)
139		# NB: This assumes this class is linked with ApplicationServiceStore
140		as_list = self.get_app_services()
141		services = []
142
143		for res in results:
144		for service in as_list:
145		if service.id == res["as_id"]:
146		services.append(service)
147		return services
148
149		@defer.inlineCallbacks
150		def get_appservice_state(self, service):
151		"""Get the application service state.
152
153		Args:
154		service(ApplicationService): The service whose state to set.
155		Returns:
156		A Deferred which resolves to ApplicationServiceState.
157		"""
158		result = yield self.db.simple_select_one(
159		"application_services_state",
160		{"as_id": service.id},
161		["state"],
162		allow_none=True,
163		desc="get_appservice_state",
164		)
165		if result:
166		return result.get("state")
167		return None
168
169		def set_appservice_state(self, service, state):
170		"""Set the application service state.
171
172		Args:
173		service(ApplicationService): The service whose state to set.
174		state(ApplicationServiceState): The connectivity state to apply.
175		Returns:
176		A Deferred which resolves when the state was set successfully.
177		"""
178		return self.db.simple_upsert(
179		"application_services_state", {"as_id": service.id}, {"state": state}
180		)
181
182		def create_appservice_txn(self, service, events):
183		"""Atomically creates a new transaction for this application service
184		with the given list of events.
185
186		Args:
187		service(ApplicationService): The service who the transaction is for.
188		events(list<Event>): A list of events to put in the transaction.
189		Returns:
190		AppServiceTransaction: A new transaction.
191		"""
192
193		def _create_appservice_txn(txn):
194		# work out new txn id (highest txn id for this service += 1)
195		# The highest id may be the last one sent (in which case it is last_txn)
196		# or it may be the highest in the txns list (which are waiting to be/are
197		# being sent)
198		last_txn_id = self._get_last_txn(txn, service.id)
199
200		txn.execute(
201		"SELECT MAX(txn_id) FROM application_services_txns WHERE as_id=?",
202		(service.id,),
203		)
204		highest_txn_id = txn.fetchone()[0]
205		if highest_txn_id is None:
206		highest_txn_id = 0
207
208		new_txn_id = max(highest_txn_id, last_txn_id) + 1
209
210		# Insert new txn into txn table
211		event_ids = json.dumps([e.event_id for e in events])
212		txn.execute(
213		"INSERT INTO application_services_txns(as_id, txn_id, event_ids) "
214		"VALUES(?,?,?)",
215		(service.id, new_txn_id, event_ids),
216		)
217		return AppServiceTransaction(service=service, id=new_txn_id, events=events)
218
219		return self.db.runInteraction("create_appservice_txn", _create_appservice_txn)
220
221		def complete_appservice_txn(self, txn_id, service):
222		"""Completes an application service transaction.
223
224		Args:
225		txn_id(str): The transaction ID being completed.
226		service(ApplicationService): The application service which was sent
227		this transaction.
228		Returns:
229		A Deferred which resolves if this transaction was stored
230		successfully.
231		"""
232		txn_id = int(txn_id)
233
234		def _complete_appservice_txn(txn):
235		# Debugging query: Make sure the txn being completed is EXACTLY +1 from
236		# what was there before. If it isn't, we've got problems (e.g. the AS
237		# has probably missed some events), so whine loudly but still continue,
238		# since it shouldn't fail completion of the transaction.
239		last_txn_id = self._get_last_txn(txn, service.id)
240		if (last_txn_id + 1) != txn_id:
241		logger.error(
242		"appservice: Completing a transaction which has an ID > 1 from "
243		"the last ID sent to this AS. We've either dropped events or "
244		"sent it to the AS out of order. FIX ME. last_txn=%s "
245		"completing_txn=%s service_id=%s",
246		last_txn_id,
247		txn_id,
248		service.id,
249		)
250
251		# Set current txn_id for AS to 'txn_id'
252		self.db.simple_upsert_txn(
253		txn,
254		"application_services_state",
255		{"as_id": service.id},
256		{"last_txn": txn_id},
257		)
258
259		# Delete txn
260		self.db.simple_delete_txn(
261		txn,
262		"application_services_txns",
263		{"txn_id": txn_id, "as_id": service.id},
264		)
265
266		return self.db.runInteraction(
267		"complete_appservice_txn", _complete_appservice_txn
268		)
269
270		@defer.inlineCallbacks
271		def get_oldest_unsent_txn(self, service):
272		"""Get the oldest transaction which has not been sent for this
273		service.
274
275		Args:
276		service(ApplicationService): The app service to get the oldest txn.
277		Returns:
278		A Deferred which resolves to an AppServiceTransaction or
279		None.
280		"""
281
282		def _get_oldest_unsent_txn(txn):
283		# Monotonically increasing txn ids, so just select the smallest
284		# one in the txns table (we delete them when they are sent)
285		txn.execute(
286		"SELECT * FROM application_services_txns WHERE as_id=?"
287		" ORDER BY txn_id ASC LIMIT 1",
288		(service.id,),
289		)
290		rows = self.db.cursor_to_dict(txn)
291		if not rows:
292		return None
293
294		entry = rows[0]
295
296		return entry
297
298		entry = yield self.db.runInteraction(
299		"get_oldest_unsent_appservice_txn", _get_oldest_unsent_txn
300		)
301
302		if not entry:
303		return None
304
305		event_ids = db_to_json(entry["event_ids"])
306
307		events = yield self.get_events_as_list(event_ids)
308
309		return AppServiceTransaction(service=service, id=entry["txn_id"], events=events)
310
311		def _get_last_txn(self, txn, service_id):
312		txn.execute(
313		"SELECT last_txn FROM application_services_state WHERE as_id=?",
314		(service_id,),
315		)
316		last_txn_id = txn.fetchone()
317		if last_txn_id is None or last_txn_id[0] is None: # no row exists
318		return 0
319		else:
320		return int(last_txn_id[0]) # select 'last_txn' col
321
322		def set_appservice_last_pos(self, pos):
323		def set_appservice_last_pos_txn(txn):
324		txn.execute(
325		"UPDATE appservice_stream_position SET stream_ordering = ?", (pos,)
326		)
327
328		return self.db.runInteraction(
329		"set_appservice_last_pos", set_appservice_last_pos_txn
330		)
331
332		@defer.inlineCallbacks
333		def get_new_events_for_appservice(self, current_id, limit):
334		"""Get all new evnets"""
335
336		def get_new_events_for_appservice_txn(txn):
337		sql = (
338		"SELECT e.stream_ordering, e.event_id"
339		" FROM events AS e"
340		" WHERE"
341		" (SELECT stream_ordering FROM appservice_stream_position)"
342		" < e.stream_ordering"
343		" AND e.stream_ordering <= ?"
344		" ORDER BY e.stream_ordering ASC"
345		" LIMIT ?"
346		)
347
348		txn.execute(sql, (current_id, limit))
349		rows = txn.fetchall()
350
351		upper_bound = current_id
352		if len(rows) == limit:
353		upper_bound = rows[-1][0]
354
355		return upper_bound, [row[1] for row in rows]
356
357		upper_bound, event_ids = yield self.db.runInteraction(
358		"get_new_events_for_appservice", get_new_events_for_appservice_txn
359		)
360
361		events = yield self.get_events_as_list(event_ids)
362
363		return upper_bound, events
364
365
366		class ApplicationServiceTransactionStore(ApplicationServiceTransactionWorkerStore):
367		# This is currently empty due to there not being any AS storage functions
368		# that can't be run on the workers. Since this may change in future, and
369		# to keep consistency with the other stores, we keep this empty class for
370		# now.
371		pass

+0

-306

~~synapse/storage/data_stores/main/cache.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2019 The Matrix.org Foundation C.I.C.
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15
16		import itertools
17		import logging
18		from typing import Any, Iterable, List, Optional, Tuple
19
20		from synapse.api.constants import EventTypes
21		from synapse.replication.tcp.streams import BackfillStream, CachesStream
22		from synapse.replication.tcp.streams.events import (
23		EventsStream,
24		EventsStreamCurrentStateRow,
25		EventsStreamEventRow,
26		)
27		from synapse.storage._base import SQLBaseStore
28		from synapse.storage.database import Database
29		from synapse.storage.engines import PostgresEngine
30		from synapse.util.iterutils import batch_iter
31
32		logger = logging.getLogger(__name__)
33
34
35		# This is a special cache name we use to batch multiple invalidations of caches
36		# based on the current state when notifying workers over replication.
37		CURRENT_STATE_CACHE_NAME = "cs_cache_fake"
38
39
40		class CacheInvalidationWorkerStore(SQLBaseStore):
41		def __init__(self, database: Database, db_conn, hs):
42		super().__init__(database, db_conn, hs)
43
44		self._instance_name = hs.get_instance_name()
45
46		async def get_all_updated_caches(
47		self, instance_name: str, last_id: int, current_id: int, limit: int
48		) -> Tuple[List[Tuple[int, tuple]], int, bool]:
49		"""Get updates for caches replication stream.
50
51		Args:
52		instance_name: The writer we want to fetch updates from. Unused
53		here since there is only ever one writer.
54		last_id: The token to fetch updates from. Exclusive.
55		current_id: The token to fetch updates up to. Inclusive.
56		limit: The requested limit for the number of rows to return. The
57		function may return more or fewer rows.
58
59		Returns:
60		A tuple consisting of: the updates, a token to use to fetch
61		subsequent updates, and whether we returned fewer rows than exists
62		between the requested tokens due to the limit.
63
64		The token returned can be used in a subsequent call to this
65		function to get further updatees.
66
67		The updates are a list of 2-tuples of stream ID and the row data
68		"""
69
70		if last_id == current_id:
71		return [], current_id, False
72
73		def get_all_updated_caches_txn(txn):
74		# We purposefully don't bound by the current token, as we want to
75		# send across cache invalidations as quickly as possible. Cache
76		# invalidations are idempotent, so duplicates are fine.
77		sql = """
78		SELECT stream_id, cache_func, keys, invalidation_ts
79		FROM cache_invalidation_stream_by_instance
80		WHERE stream_id > ? AND instance_name = ?
81		ORDER BY stream_id ASC
82		LIMIT ?
83		"""
84		txn.execute(sql, (last_id, instance_name, limit))
85		updates = [(row[0], row[1:]) for row in txn]
86		limited = False
87		upto_token = current_id
88		if len(updates) >= limit:
89		upto_token = updates[-1][0]
90		limited = True
91
92		return updates, upto_token, limited
93
94		return await self.db.runInteraction(
95		"get_all_updated_caches", get_all_updated_caches_txn
96		)
97
98		def process_replication_rows(self, stream_name, instance_name, token, rows):
99		if stream_name == EventsStream.NAME:
100		for row in rows:
101		self._process_event_stream_row(token, row)
102		elif stream_name == BackfillStream.NAME:
103		for row in rows:
104		self._invalidate_caches_for_event(
105		-token,
106		row.event_id,
107		row.room_id,
108		row.type,
109		row.state_key,
110		row.redacts,
111		row.relates_to,
112		backfilled=True,
113		)
114		elif stream_name == CachesStream.NAME:
115		if self._cache_id_gen:
116		self._cache_id_gen.advance(instance_name, token)
117
118		for row in rows:
119		if row.cache_func == CURRENT_STATE_CACHE_NAME:
120		if row.keys is None:
121		raise Exception(
122		"Can't send an 'invalidate all' for current state cache"
123		)
124
125		room_id = row.keys[0]
126		members_changed = set(row.keys[1:])
127		self._invalidate_state_caches(room_id, members_changed)
128		else:
129		self._attempt_to_invalidate_cache(row.cache_func, row.keys)
130
131		super().process_replication_rows(stream_name, instance_name, token, rows)
132
133		def _process_event_stream_row(self, token, row):
134		data = row.data
135
136		if row.type == EventsStreamEventRow.TypeId:
137		self._invalidate_caches_for_event(
138		token,
139		data.event_id,
140		data.room_id,
141		data.type,
142		data.state_key,
143		data.redacts,
144		data.relates_to,
145		backfilled=False,
146		)
147		elif row.type == EventsStreamCurrentStateRow.TypeId:
148		self._curr_state_delta_stream_cache.entity_has_changed(
149		row.data.room_id, token
150		)
151
152		if data.type == EventTypes.Member:
153		self.get_rooms_for_user_with_stream_ordering.invalidate(
154		(data.state_key,)
155		)
156		else:
157		raise Exception("Unknown events stream row type %s" % (row.type,))
158
159		def _invalidate_caches_for_event(
160		self,
161		stream_ordering,
162		event_id,
163		room_id,
164		etype,
165		state_key,
166		redacts,
167		relates_to,
168		backfilled,
169		):
170		self._invalidate_get_event_cache(event_id)
171
172		self.get_latest_event_ids_in_room.invalidate((room_id,))
173
174		self.get_unread_event_push_actions_by_room_for_user.invalidate_many((room_id,))
175
176		if not backfilled:
177		self._events_stream_cache.entity_has_changed(room_id, stream_ordering)
178
179		if redacts:
180		self._invalidate_get_event_cache(redacts)
181
182		if etype == EventTypes.Member:
183		self._membership_stream_cache.entity_has_changed(state_key, stream_ordering)
184		self.get_invited_rooms_for_local_user.invalidate((state_key,))
185
186		if relates_to:
187		self.get_relations_for_event.invalidate_many((relates_to,))
188		self.get_aggregation_groups_for_event.invalidate_many((relates_to,))
189		self.get_applicable_edit.invalidate((relates_to,))
190
191		async def invalidate_cache_and_stream(self, cache_name: str, keys: Tuple[Any, ...]):
192		"""Invalidates the cache and adds it to the cache stream so slaves
193		will know to invalidate their caches.
194
195		This should only be used to invalidate caches where slaves won't
196		otherwise know from other replication streams that the cache should
197		be invalidated.
198		"""
199		cache_func = getattr(self, cache_name, None)
200		if not cache_func:
201		return
202
203		cache_func.invalidate(keys)
204		await self.db.runInteraction(
205		"invalidate_cache_and_stream",
206		self._send_invalidation_to_replication,
207		cache_func.__name__,
208		keys,
209		)
210
211		def _invalidate_cache_and_stream(self, txn, cache_func, keys):
212		"""Invalidates the cache and adds it to the cache stream so slaves
213		will know to invalidate their caches.
214
215		This should only be used to invalidate caches where slaves won't
216		otherwise know from other replication streams that the cache should
217		be invalidated.
218		"""
219		txn.call_after(cache_func.invalidate, keys)
220		self._send_invalidation_to_replication(txn, cache_func.__name__, keys)
221
222		def _invalidate_all_cache_and_stream(self, txn, cache_func):
223		"""Invalidates the entire cache and adds it to the cache stream so slaves
224		will know to invalidate their caches.
225		"""
226
227		txn.call_after(cache_func.invalidate_all)
228		self._send_invalidation_to_replication(txn, cache_func.__name__, None)
229
230		def _invalidate_state_caches_and_stream(self, txn, room_id, members_changed):
231		"""Special case invalidation of caches based on current state.
232
233		We special case this so that we can batch the cache invalidations into a
234		single replication poke.
235
236		Args:
237		txn
238		room_id (str): Room where state changed
239		members_changed (iterable[str]): The user_ids of members that have changed
240		"""
241		txn.call_after(self._invalidate_state_caches, room_id, members_changed)
242
243		if members_changed:
244		# We need to be careful that the size of the `members_changed` list
245		# isn't so large that it causes problems sending over replication, so we
246		# send them in chunks.
247		# Max line length is 16K, and max user ID length is 255, so 50 should
248		# be safe.
249		for chunk in batch_iter(members_changed, 50):
250		keys = itertools.chain([room_id], chunk)
251		self._send_invalidation_to_replication(
252		txn, CURRENT_STATE_CACHE_NAME, keys
253		)
254		else:
255		# if no members changed, we still need to invalidate the other caches.
256		self._send_invalidation_to_replication(
257		txn, CURRENT_STATE_CACHE_NAME, [room_id]
258		)
259
260		def _send_invalidation_to_replication(
261		self, txn, cache_name: str, keys: Optional[Iterable[Any]]
262		):
263		"""Notifies replication that given cache has been invalidated.
264
265		Note that this does not invalidate the cache locally.
266
267		Args:
268		txn
269		cache_name
270		keys: Entry to invalidate. If None will invalidate all.
271		"""
272
273		if cache_name == CURRENT_STATE_CACHE_NAME and keys is None:
274		raise Exception(
275		"Can't stream invalidate all with magic current state cache"
276		)
277
278		if isinstance(self.database_engine, PostgresEngine):
279		# get_next() returns a context manager which is designed to wrap
280		# the transaction. However, we want to only get an ID when we want
281		# to use it, here, so we need to call __enter__ manually, and have
282		# __exit__ called after the transaction finishes.
283		stream_id = self._cache_id_gen.get_next_txn(txn)
284		txn.call_after(self.hs.get_notifier().on_new_replication_data)
285
286		if keys is not None:
287		keys = list(keys)
288
289		self.db.simple_insert_txn(
290		txn,
291		table="cache_invalidation_stream_by_instance",
292		values={
293		"stream_id": stream_id,
294		"instance_name": self._instance_name,
295		"cache_func": cache_name,
296		"keys": keys,
297		"invalidation_ts": self.clock.time_msec(),
298		},
299		)
300
301		def get_cache_stream_token(self, instance_name):
302		if self._cache_id_gen:
303		return self._cache_id_gen.get_current_token(instance_name)
304		else:
305		return 0

+0

-208

~~synapse/storage/data_stores/main/censor_events.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2020 The Matrix.org Foundation C.I.C.
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16		from typing import TYPE_CHECKING
17
18		from twisted.internet import defer
19
20		from synapse.events.utils import prune_event_dict
21		from synapse.metrics.background_process_metrics import run_as_background_process
22		from synapse.storage._base import SQLBaseStore
23		from synapse.storage.data_stores.main.cache import CacheInvalidationWorkerStore
24		from synapse.storage.data_stores.main.events import encode_json
25		from synapse.storage.data_stores.main.events_worker import EventsWorkerStore
26		from synapse.storage.database import Database
27
28		if TYPE_CHECKING:
29		from synapse.server import HomeServer
30
31
32		logger = logging.getLogger(__name__)
33
34
35		class CensorEventsStore(EventsWorkerStore, CacheInvalidationWorkerStore, SQLBaseStore):
36		def __init__(self, database: Database, db_conn, hs: "HomeServer"):
37		super().__init__(database, db_conn, hs)
38
39		def _censor_redactions():
40		return run_as_background_process(
41		"_censor_redactions", self._censor_redactions
42		)
43
44		if self.hs.config.redaction_retention_period is not None:
45		hs.get_clock().looping_call(_censor_redactions, 5 * 60 * 1000)
46
47		async def _censor_redactions(self):
48		"""Censors all redactions older than the configured period that haven't
49		been censored yet.
50
51		By censor we mean update the event_json table with the redacted event.
52		"""
53
54		if self.hs.config.redaction_retention_period is None:
55		return
56
57		if not (
58		await self.db.updates.has_completed_background_update(
59		"redactions_have_censored_ts_idx"
60		)
61		):
62		# We don't want to run this until the appropriate index has been
63		# created.
64		return
65
66		before_ts = self._clock.time_msec() - self.hs.config.redaction_retention_period
67
68		# We fetch all redactions that:
69		# 1. point to an event we have,
70		# 2. has a received_ts from before the cut off, and
71		# 3. we haven't yet censored.
72		#
73		# This is limited to 100 events to ensure that we don't try and do too
74		# much at once. We'll get called again so this should eventually catch
75		# up.
76		sql = """
77		SELECT redactions.event_id, redacts FROM redactions
78		LEFT JOIN events AS original_event ON (
79		redacts = original_event.event_id
80		)
81		WHERE NOT have_censored
82		AND redactions.received_ts <= ?
83		ORDER BY redactions.received_ts ASC
84		LIMIT ?
85		"""
86
87		rows = await self.db.execute(
88		"_censor_redactions_fetch", None, sql, before_ts, 100
89		)
90
91		updates = []
92
93		for redaction_id, event_id in rows:
94		redaction_event = await self.get_event(redaction_id, allow_none=True)
95		original_event = await self.get_event(
96		event_id, allow_rejected=True, allow_none=True
97		)
98
99		# The SQL above ensures that we have both the redaction and
100		# original event, so if the `get_event` calls return None it
101		# means that the redaction wasn't allowed. Either way we know that
102		# the result won't change so we mark the fact that we've checked.
103		if (
104		redaction_event
105		and original_event
106		and original_event.internal_metadata.is_redacted()
107		):
108		# Redaction was allowed
109		pruned_json = encode_json(
110		prune_event_dict(
111		original_event.room_version, original_event.get_dict()
112		)
113		)
114		else:
115		# Redaction wasn't allowed
116		pruned_json = None
117
118		updates.append((redaction_id, event_id, pruned_json))
119
120		def _update_censor_txn(txn):
121		for redaction_id, event_id, pruned_json in updates:
122		if pruned_json:
123		self._censor_event_txn(txn, event_id, pruned_json)
124
125		self.db.simple_update_one_txn(
126		txn,
127		table="redactions",
128		keyvalues={"event_id": redaction_id},
129		updatevalues={"have_censored": True},
130		)
131
132		await self.db.runInteraction("_update_censor_txn", _update_censor_txn)
133
134		def _censor_event_txn(self, txn, event_id, pruned_json):
135		"""Censor an event by replacing its JSON in the event_json table with the
136		provided pruned JSON.
137
138		Args:
139		txn (LoggingTransaction): The database transaction.
140		event_id (str): The ID of the event to censor.
141		pruned_json (str): The pruned JSON
142		"""
143		self.db.simple_update_one_txn(
144		txn,
145		table="event_json",
146		keyvalues={"event_id": event_id},
147		updatevalues={"json": pruned_json},
148		)
149
150		@defer.inlineCallbacks
151		def expire_event(self, event_id):
152		"""Retrieve and expire an event that has expired, and delete its associated
153		expiry timestamp. If the event can't be retrieved, delete its associated
154		timestamp so we don't try to expire it again in the future.
155
156		Args:
157		event_id (str): The ID of the event to delete.
158		"""
159		# Try to retrieve the event's content from the database or the event cache.
160		event = yield self.get_event(event_id)
161
162		def delete_expired_event_txn(txn):
163		# Delete the expiry timestamp associated with this event from the database.
164		self._delete_event_expiry_txn(txn, event_id)
165
166		if not event:
167		# If we can't find the event, log a warning and delete the expiry date
168		# from the database so that we don't try to expire it again in the
169		# future.
170		logger.warning(
171		"Can't expire event %s because we don't have it.", event_id
172		)
173		return
174
175		# Prune the event's dict then convert it to JSON.
176		pruned_json = encode_json(
177		prune_event_dict(event.room_version, event.get_dict())
178		)
179
180		# Update the event_json table to replace the event's JSON with the pruned
181		# JSON.
182		self._censor_event_txn(txn, event.event_id, pruned_json)
183
184		# We need to invalidate the event cache entry for this event because we
185		# changed its content in the database. We can't call
186		# self._invalidate_cache_and_stream because self.get_event_cache isn't of the
187		# right type.
188		txn.call_after(self._get_event_cache.invalidate, (event.event_id,))
189		# Send that invalidation to replication so that other workers also invalidate
190		# the event cache.
191		self._send_invalidation_to_replication(
192		txn, "_get_event_cache", (event.event_id,)
193		)
194
195		yield self.db.runInteraction("delete_expired_event", delete_expired_event_txn)
196
197		def _delete_event_expiry_txn(self, txn, event_id):
198		"""Delete the expiry timestamp associated with an event ID without deleting the
199		actual event.
200
201		Args:
202		txn (LoggingTransaction): The transaction to use to perform the deletion.
203		event_id (str): The event ID to delete the associated expiry timestamp of.
204		"""
205		return self.db.simple_delete_txn(
206		txn=txn, table="event_expiry", keyvalues={"event_id": event_id}
207		)

+0

-576

~~synapse/storage/data_stores/main/client_ips.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16
17		from twisted.internet import defer
18
19		from synapse.metrics.background_process_metrics import wrap_as_background_process
20		from synapse.storage._base import SQLBaseStore
21		from synapse.storage.database import Database, make_tuple_comparison_clause
22		from synapse.util.caches.descriptors import Cache
23
24		logger = logging.getLogger(__name__)
25
26		# Number of msec of granularity to store the user IP 'last seen' time. Smaller
27		# times give more inserts into the database even for readonly API hits
28		# 120 seconds == 2 minutes
29		LAST_SEEN_GRANULARITY = 120 * 1000
30
31
32		class ClientIpBackgroundUpdateStore(SQLBaseStore):
33		def __init__(self, database: Database, db_conn, hs):
34		super(ClientIpBackgroundUpdateStore, self).__init__(database, db_conn, hs)
35
36		self.db.updates.register_background_index_update(
37		"user_ips_device_index",
38		index_name="user_ips_device_id",
39		table="user_ips",
40		columns=["user_id", "device_id", "last_seen"],
41		)
42
43		self.db.updates.register_background_index_update(
44		"user_ips_last_seen_index",
45		index_name="user_ips_last_seen",
46		table="user_ips",
47		columns=["user_id", "last_seen"],
48		)
49
50		self.db.updates.register_background_index_update(
51		"user_ips_last_seen_only_index",
52		index_name="user_ips_last_seen_only",
53		table="user_ips",
54		columns=["last_seen"],
55		)
56
57		self.db.updates.register_background_update_handler(
58		"user_ips_analyze", self._analyze_user_ip
59		)
60
61		self.db.updates.register_background_update_handler(
62		"user_ips_remove_dupes", self._remove_user_ip_dupes
63		)
64
65		# Register a unique index
66		self.db.updates.register_background_index_update(
67		"user_ips_device_unique_index",
68		index_name="user_ips_user_token_ip_unique_index",
69		table="user_ips",
70		columns=["user_id", "access_token", "ip"],
71		unique=True,
72		)
73
74		# Drop the old non-unique index
75		self.db.updates.register_background_update_handler(
76		"user_ips_drop_nonunique_index", self._remove_user_ip_nonunique
77		)
78
79		# Update the last seen info in devices.
80		self.db.updates.register_background_update_handler(
81		"devices_last_seen", self._devices_last_seen_update
82		)
83
84		@defer.inlineCallbacks
85		def _remove_user_ip_nonunique(self, progress, batch_size):
86		def f(conn):
87		txn = conn.cursor()
88		txn.execute("DROP INDEX IF EXISTS user_ips_user_ip")
89		txn.close()
90
91		yield self.db.runWithConnection(f)
92		yield self.db.updates._end_background_update("user_ips_drop_nonunique_index")
93		return 1
94
95		@defer.inlineCallbacks
96		def _analyze_user_ip(self, progress, batch_size):
97		# Background update to analyze user_ips table before we run the
98		# deduplication background update. The table may not have been analyzed
99		# for ages due to the table locks.
100		#
101		# This will lock out the naive upserts to user_ips while it happens, but
102		# the analyze should be quick (28GB table takes ~10s)
103		def user_ips_analyze(txn):
104		txn.execute("ANALYZE user_ips")
105
106		yield self.db.runInteraction("user_ips_analyze", user_ips_analyze)
107
108		yield self.db.updates._end_background_update("user_ips_analyze")
109
110		return 1
111
112		@defer.inlineCallbacks
113		def _remove_user_ip_dupes(self, progress, batch_size):
114		# This works function works by scanning the user_ips table in batches
115		# based on `last_seen`. For each row in a batch it searches the rest of
116		# the table to see if there are any duplicates, if there are then they
117		# are removed and replaced with a suitable row.
118
119		# Fetch the start of the batch
120		begin_last_seen = progress.get("last_seen", 0)
121
122		def get_last_seen(txn):
123		txn.execute(
124		"""
125		SELECT last_seen FROM user_ips
126		WHERE last_seen > ?
127		ORDER BY last_seen
128		LIMIT 1
129		OFFSET ?
130		""",
131		(begin_last_seen, batch_size),
132		)
133		row = txn.fetchone()
134		if row:
135		return row[0]
136		else:
137		return None
138
139		# Get a last seen that has roughly `batch_size` since `begin_last_seen`
140		end_last_seen = yield self.db.runInteraction(
141		"user_ips_dups_get_last_seen", get_last_seen
142		)
143
144		# If it returns None, then we're processing the last batch
145		last = end_last_seen is None
146
147		logger.info(
148		"Scanning for duplicate 'user_ips' rows in range: %s <= last_seen < %s",
149		begin_last_seen,
150		end_last_seen,
151		)
152
153		def remove(txn):
154		# This works by looking at all entries in the given time span, and
155		# then for each (user_id, access_token, ip) tuple in that range
156		# checking for any duplicates in the rest of the table (via a join).
157		# It then only returns entries which have duplicates, and the max
158		# last_seen across all duplicates, which can the be used to delete
159		# all other duplicates.
160		# It is efficient due to the existence of (user_id, access_token,
161		# ip) and (last_seen) indices.
162
163		# Define the search space, which requires handling the last batch in
164		# a different way
165		if last:
166		clause = "? <= last_seen"
167		args = (begin_last_seen,)
168		else:
169		clause = "? <= last_seen AND last_seen < ?"
170		args = (begin_last_seen, end_last_seen)
171
172		# (Note: The DISTINCT in the inner query is important to ensure that
173		# the COUNT(*) is accurate, otherwise double counting may happen due
174		# to the join effectively being a cross product)
175		txn.execute(
176		"""
177		SELECT user_id, access_token, ip,
178		MAX(device_id), MAX(user_agent), MAX(last_seen),
179		COUNT(*)
180		FROM (
181		SELECT DISTINCT user_id, access_token, ip
182		FROM user_ips
183		WHERE {}
184		) c
185		INNER JOIN user_ips USING (user_id, access_token, ip)
186		GROUP BY user_id, access_token, ip
187		HAVING count(*) > 1
188		""".format(
189		clause
190		),
191		args,
192		)
193		res = txn.fetchall()
194
195		# We've got some duplicates
196		for i in res:
197		user_id, access_token, ip, device_id, user_agent, last_seen, count = i
198
199		# We want to delete the duplicates so we end up with only a
200		# single row.
201		#
202		# The naive way of doing this would be just to delete all rows
203		# and reinsert a constructed row. However, if there are a lot of
204		# duplicate rows this can cause the table to grow a lot, which
205		# can be problematic in two ways:
206		# 1. If user_ips is already large then this can cause the
207		# table to rapidly grow, potentially filling the disk.
208		# 2. Reinserting a lot of rows can confuse the table
209		# statistics for postgres, causing it to not use the
210		# correct indices for the query above, resulting in a full
211		# table scan. This is incredibly slow for large tables and
212		# can kill database performance. (This seems to mainly
213		# happen for the last query where the clause is simply `? <
214		# last_seen`)
215		#
216		# So instead we want to delete all but one of the duplicate
217		# rows. That is hard to do reliably, so we cheat and do a two
218		# step process:
219		# 1. Delete all rows with a last_seen strictly less than the
220		# max last_seen. This hopefully results in deleting all but
221		# one row the majority of the time, but there may be
222		# duplicate last_seen
223		# 2. If multiple rows remain, we fall back to the naive method
224		# and simply delete all rows and reinsert.
225		#
226		# Note that this relies on no new duplicate rows being inserted,
227		# but if that is happening then this entire process is futile
228		# anyway.
229
230		# Do step 1:
231
232		txn.execute(
233		"""
234		DELETE FROM user_ips
235		WHERE user_id = ? AND access_token = ? AND ip = ? AND last_seen < ?
236		""",
237		(user_id, access_token, ip, last_seen),
238		)
239		if txn.rowcount == count - 1:
240		# We deleted all but one of the duplicate rows, i.e. there
241		# is exactly one remaining and so there is nothing left to
242		# do.
243		continue
244		elif txn.rowcount >= count:
245		raise Exception(
246		"We deleted more duplicate rows from 'user_ips' than expected"
247		)
248
249		# The previous step didn't delete enough rows, so we fallback to
250		# step 2:
251
252		# Drop all the duplicates
253		txn.execute(
254		"""
255		DELETE FROM user_ips
256		WHERE user_id = ? AND access_token = ? AND ip = ?
257		""",
258		(user_id, access_token, ip),
259		)
260
261		# Add in one to be the last_seen
262		txn.execute(
263		"""
264		INSERT INTO user_ips
265		(user_id, access_token, ip, device_id, user_agent, last_seen)
266		VALUES (?, ?, ?, ?, ?, ?)
267		""",
268		(user_id, access_token, ip, device_id, user_agent, last_seen),
269		)
270
271		self.db.updates._background_update_progress_txn(
272		txn, "user_ips_remove_dupes", {"last_seen": end_last_seen}
273		)
274
275		yield self.db.runInteraction("user_ips_dups_remove", remove)
276
277		if last:
278		yield self.db.updates._end_background_update("user_ips_remove_dupes")
279
280		return batch_size
281
282		@defer.inlineCallbacks
283		def _devices_last_seen_update(self, progress, batch_size):
284		"""Background update to insert last seen info into devices table
285		"""
286
287		last_user_id = progress.get("last_user_id", "")
288		last_device_id = progress.get("last_device_id", "")
289
290		def _devices_last_seen_update_txn(txn):
291		# This consists of two queries:
292		#
293		# 1. The sub-query searches for the next N devices and joins
294		# against user_ips to find the max last_seen associated with
295		# that device.
296		# 2. The outer query then joins again against user_ips on
297		# user/device/last_seen. This should hopefully only
298		# return one row, but if it does return more than one then
299		# we'll just end up updating the same device row multiple
300		# times, which is fine.
301
302		where_clause, where_args = make_tuple_comparison_clause(
303		self.database_engine,
304		[("user_id", last_user_id), ("device_id", last_device_id)],
305		)
306
307		sql = """
308		SELECT
309		last_seen, ip, user_agent, user_id, device_id
310		FROM (
311		SELECT
312		user_id, device_id, MAX(u.last_seen) AS last_seen
313		FROM devices
314		INNER JOIN user_ips AS u USING (user_id, device_id)
315		WHERE %(where_clause)s
316		GROUP BY user_id, device_id
317		ORDER BY user_id ASC, device_id ASC
318		LIMIT ?
319		) c
320		INNER JOIN user_ips AS u USING (user_id, device_id, last_seen)
321		""" % {
322		"where_clause": where_clause
323		}
324		txn.execute(sql, where_args + [batch_size])
325
326		rows = txn.fetchall()
327		if not rows:
328		return 0
329
330		sql = """
331		UPDATE devices
332		SET last_seen = ?, ip = ?, user_agent = ?
333		WHERE user_id = ? AND device_id = ?
334		"""
335		txn.execute_batch(sql, rows)
336
337		_, _, _, user_id, device_id = rows[-1]
338		self.db.updates._background_update_progress_txn(
339		txn,
340		"devices_last_seen",
341		{"last_user_id": user_id, "last_device_id": device_id},
342		)
343
344		return len(rows)
345
346		updated = yield self.db.runInteraction(
347		"_devices_last_seen_update", _devices_last_seen_update_txn
348		)
349
350		if not updated:
351		yield self.db.updates._end_background_update("devices_last_seen")
352
353		return updated
354
355
356		class ClientIpStore(ClientIpBackgroundUpdateStore):
357		def __init__(self, database: Database, db_conn, hs):
358
359		self.client_ip_last_seen = Cache(
360		name="client_ip_last_seen", keylen=4, max_entries=50000
361		)
362
363		super(ClientIpStore, self).__init__(database, db_conn, hs)
364
365		self.user_ips_max_age = hs.config.user_ips_max_age
366
367		# (user_id, access_token, ip,) -> (user_agent, device_id, last_seen)
368		self._batch_row_update = {}
369
370		self._client_ip_looper = self._clock.looping_call(
371		self._update_client_ips_batch, 5 * 1000
372		)
373		self.hs.get_reactor().addSystemEventTrigger(
374		"before", "shutdown", self._update_client_ips_batch
375		)
376
377		if self.user_ips_max_age:
378		self._clock.looping_call(self._prune_old_user_ips, 5 * 1000)
379
380		@defer.inlineCallbacks
381		def insert_client_ip(
382		self, user_id, access_token, ip, user_agent, device_id, now=None
383		):
384		if not now:
385		now = int(self._clock.time_msec())
386		key = (user_id, access_token, ip)
387
388		try:
389		last_seen = self.client_ip_last_seen.get(key)
390		except KeyError:
391		last_seen = None
392		yield self.populate_monthly_active_users(user_id)
393		# Rate-limited inserts
394		if last_seen is not None and (now - last_seen) < LAST_SEEN_GRANULARITY:
395		return
396
397		self.client_ip_last_seen.prefill(key, now)
398
399		self._batch_row_update[key] = (user_agent, device_id, now)
400
401		@wrap_as_background_process("update_client_ips")
402		def _update_client_ips_batch(self):
403
404		# If the DB pool has already terminated, don't try updating
405		if not self.db.is_running():
406		return
407
408		to_update = self._batch_row_update
409		self._batch_row_update = {}
410
411		return self.db.runInteraction(
412		"_update_client_ips_batch", self._update_client_ips_batch_txn, to_update
413		)
414
415		def _update_client_ips_batch_txn(self, txn, to_update):
416		if "user_ips" in self.db._unsafe_to_upsert_tables or (
417		not self.database_engine.can_native_upsert
418		):
419		self.database_engine.lock_table(txn, "user_ips")
420
421		for entry in to_update.items():
422		(user_id, access_token, ip), (user_agent, device_id, last_seen) = entry
423
424		try:
425		self.db.simple_upsert_txn(
426		txn,
427		table="user_ips",
428		keyvalues={
429		"user_id": user_id,
430		"access_token": access_token,
431		"ip": ip,
432		},
433		values={
434		"user_agent": user_agent,
435		"device_id": device_id,
436		"last_seen": last_seen,
437		},
438		lock=False,
439		)
440
441		# Technically an access token might not be associated with
442		# a device so we need to check.
443		if device_id:
444		# this is always an update rather than an upsert: the row should
445		# already exist, and if it doesn't, that may be because it has been
446		# deleted, and we don't want to re-create it.
447		self.db.simple_update_txn(
448		txn,
449		table="devices",
450		keyvalues={"user_id": user_id, "device_id": device_id},
451		updatevalues={
452		"user_agent": user_agent,
453		"last_seen": last_seen,
454		"ip": ip,
455		},
456		)
457		except Exception as e:
458		# Failed to upsert, log and continue
459		logger.error("Failed to insert client IP %r: %r", entry, e)
460
461		@defer.inlineCallbacks
462		def get_last_client_ip_by_device(self, user_id, device_id):
463		"""For each device_id listed, give the user_ip it was last seen on
464
465		Args:
466		user_id (str)
467		device_id (str): If None fetches all devices for the user
468
469		Returns:
470		defer.Deferred: resolves to a dict, where the keys
471		are (user_id, device_id) tuples. The values are also dicts, with
472		keys giving the column names
473		"""
474
475		keyvalues = {"user_id": user_id}
476		if device_id is not None:
477		keyvalues["device_id"] = device_id
478
479		res = yield self.db.simple_select_list(
480		table="devices",
481		keyvalues=keyvalues,
482		retcols=("user_id", "ip", "user_agent", "device_id", "last_seen"),
483		)
484
485		ret = {(d["user_id"], d["device_id"]): d for d in res}
486		for key in self._batch_row_update:
487		uid, access_token, ip = key
488		if uid == user_id:
489		user_agent, did, last_seen = self._batch_row_update[key]
490		if not device_id or did == device_id:
491		ret[(user_id, device_id)] = {
492		"user_id": user_id,
493		"access_token": access_token,
494		"ip": ip,
495		"user_agent": user_agent,
496		"device_id": did,
497		"last_seen": last_seen,
498		}
499		return ret
500
501		@defer.inlineCallbacks
502		def get_user_ip_and_agents(self, user):
503		user_id = user.to_string()
504		results = {}
505
506		for key in self._batch_row_update:
507		uid, access_token, ip, = key
508		if uid == user_id:
509		user_agent, _, last_seen = self._batch_row_update[key]
510		results[(access_token, ip)] = (user_agent, last_seen)
511
512		rows = yield self.db.simple_select_list(
513		table="user_ips",
514		keyvalues={"user_id": user_id},
515		retcols=["access_token", "ip", "user_agent", "last_seen"],
516		desc="get_user_ip_and_agents",
517		)
518
519		results.update(
520		((row["access_token"], row["ip"]), (row["user_agent"], row["last_seen"]))
521		for row in rows
522		)
523		return [
524		{
525		"access_token": access_token,
526		"ip": ip,
527		"user_agent": user_agent,
528		"last_seen": last_seen,
529		}
530		for (access_token, ip), (user_agent, last_seen) in results.items()
531		]
532
533		@wrap_as_background_process("prune_old_user_ips")
534		async def _prune_old_user_ips(self):
535		"""Removes entries in user IPs older than the configured period.
536		"""
537
538		if self.user_ips_max_age is None:
539		# Nothing to do
540		return
541
542		if not await self.db.updates.has_completed_background_update(
543		"devices_last_seen"
544		):
545		# Only start pruning if we have finished populating the devices
546		# last seen info.
547		return
548
549		# We do a slightly funky SQL delete to ensure we don't try and delete
550		# too much at once (as the table may be very large from before we
551		# started pruning).
552		#
553		# This works by finding the max last_seen that is less than the given
554		# time, but has no more than N rows before it, deleting all rows with
555		# a lesser last_seen time. (We COALESCE so that the sub-SELECT always
556		# returns exactly one row).
557		sql = """
558		DELETE FROM user_ips
559		WHERE last_seen <= (
560		SELECT COALESCE(MAX(last_seen), -1)
561		FROM (
562		SELECT last_seen FROM user_ips
563		WHERE last_seen <= ?
564		ORDER BY last_seen ASC
565		LIMIT 5000
566		) AS u
567		)
568		"""
569
570		timestamp = self.clock.time_msec() - self.user_ips_max_age
571
572		def _prune_old_user_ips_txn(txn):
573		txn.execute(sql, (timestamp,))
574
575		await self.db.runInteraction("_prune_old_user_ips", _prune_old_user_ips_txn)

+0

-476

~~synapse/storage/data_stores/main/deviceinbox.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16		from typing import List, Tuple
17
18		from canonicaljson import json
19
20		from twisted.internet import defer
21
22		from synapse.logging.opentracing import log_kv, set_tag, trace
23		from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
24		from synapse.storage.database import Database
25		from synapse.util.caches.expiringcache import ExpiringCache
26
27		logger = logging.getLogger(__name__)
28
29
30		class DeviceInboxWorkerStore(SQLBaseStore):
31		def get_to_device_stream_token(self):
32		return self._device_inbox_id_gen.get_current_token()
33
34		def get_new_messages_for_device(
35		self, user_id, device_id, last_stream_id, current_stream_id, limit=100
36		):
37		"""
38		Args:
39		user_id(str): The recipient user_id.
40		device_id(str): The recipient device_id.
41		current_stream_id(int): The current position of the to device
42		message stream.
43		Returns:
44		Deferred ([dict], int): List of messages for the device and where
45		in the stream the messages got to.
46		"""
47		has_changed = self._device_inbox_stream_cache.has_entity_changed(
48		user_id, last_stream_id
49		)
50		if not has_changed:
51		return defer.succeed(([], current_stream_id))
52
53		def get_new_messages_for_device_txn(txn):
54		sql = (
55		"SELECT stream_id, message_json FROM device_inbox"
56		" WHERE user_id = ? AND device_id = ?"
57		" AND ? < stream_id AND stream_id <= ?"
58		" ORDER BY stream_id ASC"
59		" LIMIT ?"
60		)
61		txn.execute(
62		sql, (user_id, device_id, last_stream_id, current_stream_id, limit)
63		)
64		messages = []
65		for row in txn:
66		stream_pos = row[0]
67		messages.append(db_to_json(row[1]))
68		if len(messages) < limit:
69		stream_pos = current_stream_id
70		return messages, stream_pos
71
72		return self.db.runInteraction(
73		"get_new_messages_for_device", get_new_messages_for_device_txn
74		)
75
76		@trace
77		@defer.inlineCallbacks
78		def delete_messages_for_device(self, user_id, device_id, up_to_stream_id):
79		"""
80		Args:
81		user_id(str): The recipient user_id.
82		device_id(str): The recipient device_id.
83		up_to_stream_id(int): Where to delete messages up to.
84		Returns:
85		A deferred that resolves to the number of messages deleted.
86		"""
87		# If we have cached the last stream id we've deleted up to, we can
88		# check if there is likely to be anything that needs deleting
89		last_deleted_stream_id = self._last_device_delete_cache.get(
90		(user_id, device_id), None
91		)
92
93		set_tag("last_deleted_stream_id", last_deleted_stream_id)
94
95		if last_deleted_stream_id:
96		has_changed = self._device_inbox_stream_cache.has_entity_changed(
97		user_id, last_deleted_stream_id
98		)
99		if not has_changed:
100		log_kv({"message": "No changes in cache since last check"})
101		return 0
102
103		def delete_messages_for_device_txn(txn):
104		sql = (
105		"DELETE FROM device_inbox"
106		" WHERE user_id = ? AND device_id = ?"
107		" AND stream_id <= ?"
108		)
109		txn.execute(sql, (user_id, device_id, up_to_stream_id))
110		return txn.rowcount
111
112		count = yield self.db.runInteraction(
113		"delete_messages_for_device", delete_messages_for_device_txn
114		)
115
116		log_kv(
117		{"message": "deleted {} messages for device".format(count), "count": count}
118		)
119
120		# Update the cache, ensuring that we only ever increase the value
121		last_deleted_stream_id = self._last_device_delete_cache.get(
122		(user_id, device_id), 0
123		)
124		self._last_device_delete_cache[(user_id, device_id)] = max(
125		last_deleted_stream_id, up_to_stream_id
126		)
127
128		return count
129
130		@trace
131		def get_new_device_msgs_for_remote(
132		self, destination, last_stream_id, current_stream_id, limit
133		):
134		"""
135		Args:
136		destination(str): The name of the remote server.
137		last_stream_id(int\|long): The last position of the device message stream
138		that the server sent up to.
139		current_stream_id(int\|long): The current position of the device
140		message stream.
141		Returns:
142		Deferred ([dict], int\|long): List of messages for the device and where
143		in the stream the messages got to.
144		"""
145
146		set_tag("destination", destination)
147		set_tag("last_stream_id", last_stream_id)
148		set_tag("current_stream_id", current_stream_id)
149		set_tag("limit", limit)
150
151		has_changed = self._device_federation_outbox_stream_cache.has_entity_changed(
152		destination, last_stream_id
153		)
154		if not has_changed or last_stream_id == current_stream_id:
155		log_kv({"message": "No new messages in stream"})
156		return defer.succeed(([], current_stream_id))
157
158		if limit <= 0:
159		# This can happen if we run out of room for EDUs in the transaction.
160		return defer.succeed(([], last_stream_id))
161
162		@trace
163		def get_new_messages_for_remote_destination_txn(txn):
164		sql = (
165		"SELECT stream_id, messages_json FROM device_federation_outbox"
166		" WHERE destination = ?"
167		" AND ? < stream_id AND stream_id <= ?"
168		" ORDER BY stream_id ASC"
169		" LIMIT ?"
170		)
171		txn.execute(sql, (destination, last_stream_id, current_stream_id, limit))
172		messages = []
173		for row in txn:
174		stream_pos = row[0]
175		messages.append(db_to_json(row[1]))
176		if len(messages) < limit:
177		log_kv({"message": "Set stream position to current position"})
178		stream_pos = current_stream_id
179		return messages, stream_pos
180
181		return self.db.runInteraction(
182		"get_new_device_msgs_for_remote",
183		get_new_messages_for_remote_destination_txn,
184		)
185
186		@trace
187		def delete_device_msgs_for_remote(self, destination, up_to_stream_id):
188		"""Used to delete messages when the remote destination acknowledges
189		their receipt.
190
191		Args:
192		destination(str): The destination server_name
193		up_to_stream_id(int): Where to delete messages up to.
194		Returns:
195		A deferred that resolves when the messages have been deleted.
196		"""
197
198		def delete_messages_for_remote_destination_txn(txn):
199		sql = (
200		"DELETE FROM device_federation_outbox"
201		" WHERE destination = ?"
202		" AND stream_id <= ?"
203		)
204		txn.execute(sql, (destination, up_to_stream_id))
205
206		return self.db.runInteraction(
207		"delete_device_msgs_for_remote", delete_messages_for_remote_destination_txn
208		)
209
210		async def get_all_new_device_messages(
211		self, instance_name: str, last_id: int, current_id: int, limit: int
212		) -> Tuple[List[Tuple[int, tuple]], int, bool]:
213		"""Get updates for to device replication stream.
214
215		Args:
216		instance_name: The writer we want to fetch updates from. Unused
217		here since there is only ever one writer.
218		last_id: The token to fetch updates from. Exclusive.
219		current_id: The token to fetch updates up to. Inclusive.
220		limit: The requested limit for the number of rows to return. The
221		function may return more or fewer rows.
222
223		Returns:
224		A tuple consisting of: the updates, a token to use to fetch
225		subsequent updates, and whether we returned fewer rows than exists
226		between the requested tokens due to the limit.
227
228		The token returned can be used in a subsequent call to this
229		function to get further updatees.
230
231		The updates are a list of 2-tuples of stream ID and the row data
232		"""
233
234		if last_id == current_id:
235		return [], current_id, False
236
237		def get_all_new_device_messages_txn(txn):
238		# We limit like this as we might have multiple rows per stream_id, and
239		# we want to make sure we always get all entries for any stream_id
240		# we return.
241		upper_pos = min(current_id, last_id + limit)
242		sql = (
243		"SELECT max(stream_id), user_id"
244		" FROM device_inbox"
245		" WHERE ? < stream_id AND stream_id <= ?"
246		" GROUP BY user_id"
247		)
248		txn.execute(sql, (last_id, upper_pos))
249		updates = [(row[0], row[1:]) for row in txn]
250
251		sql = (
252		"SELECT max(stream_id), destination"
253		" FROM device_federation_outbox"
254		" WHERE ? < stream_id AND stream_id <= ?"
255		" GROUP BY destination"
256		)
257		txn.execute(sql, (last_id, upper_pos))
258		updates.extend((row[0], row[1:]) for row in txn)
259
260		# Order by ascending stream ordering
261		updates.sort()
262
263		limited = False
264		upto_token = current_id
265		if len(updates) >= limit:
266		upto_token = updates[-1][0]
267		limited = True
268
269		return updates, upto_token, limited
270
271		return await self.db.runInteraction(
272		"get_all_new_device_messages", get_all_new_device_messages_txn
273		)
274
275
276		class DeviceInboxBackgroundUpdateStore(SQLBaseStore):
277		DEVICE_INBOX_STREAM_ID = "device_inbox_stream_drop"
278
279		def __init__(self, database: Database, db_conn, hs):
280		super(DeviceInboxBackgroundUpdateStore, self).__init__(database, db_conn, hs)
281
282		self.db.updates.register_background_index_update(
283		"device_inbox_stream_index",
284		index_name="device_inbox_stream_id_user_id",
285		table="device_inbox",
286		columns=["stream_id", "user_id"],
287		)
288
289		self.db.updates.register_background_update_handler(
290		self.DEVICE_INBOX_STREAM_ID, self._background_drop_index_device_inbox
291		)
292
293		@defer.inlineCallbacks
294		def _background_drop_index_device_inbox(self, progress, batch_size):
295		def reindex_txn(conn):
296		txn = conn.cursor()
297		txn.execute("DROP INDEX IF EXISTS device_inbox_stream_id")
298		txn.close()
299
300		yield self.db.runWithConnection(reindex_txn)
301
302		yield self.db.updates._end_background_update(self.DEVICE_INBOX_STREAM_ID)
303
304		return 1
305
306
307		class DeviceInboxStore(DeviceInboxWorkerStore, DeviceInboxBackgroundUpdateStore):
308		DEVICE_INBOX_STREAM_ID = "device_inbox_stream_drop"
309
310		def __init__(self, database: Database, db_conn, hs):
311		super(DeviceInboxStore, self).__init__(database, db_conn, hs)
312
313		# Map of (user_id, device_id) to the last stream_id that has been
314		# deleted up to. This is so that we can no op deletions.
315		self._last_device_delete_cache = ExpiringCache(
316		cache_name="last_device_delete_cache",
317		clock=self._clock,
318		max_len=10000,
319		expiry_ms=30 * 60 * 1000,
320		)
321
322		@trace
323		@defer.inlineCallbacks
324		def add_messages_to_device_inbox(
325		self, local_messages_by_user_then_device, remote_messages_by_destination
326		):
327		"""Used to send messages from this server.
328
329		Args:
330		sender_user_id(str): The ID of the user sending these messages.
331		local_messages_by_user_and_device(dict):
332		Dictionary of user_id to device_id to message.
333		remote_messages_by_destination(dict):
334		Dictionary of destination server_name to the EDU JSON to send.
335		Returns:
336		A deferred stream_id that resolves when the messages have been
337		inserted.
338		"""
339
340		def add_messages_txn(txn, now_ms, stream_id):
341		# Add the local messages directly to the local inbox.
342		self._add_messages_to_local_device_inbox_txn(
343		txn, stream_id, local_messages_by_user_then_device
344		)
345
346		# Add the remote messages to the federation outbox.
347		# We'll send them to a remote server when we next send a
348		# federation transaction to that destination.
349		sql = (
350		"INSERT INTO device_federation_outbox"
351		" (destination, stream_id, queued_ts, messages_json)"
352		" VALUES (?,?,?,?)"
353		)
354		rows = []
355		for destination, edu in remote_messages_by_destination.items():
356		edu_json = json.dumps(edu)
357		rows.append((destination, stream_id, now_ms, edu_json))
358		txn.executemany(sql, rows)
359
360		with self._device_inbox_id_gen.get_next() as stream_id:
361		now_ms = self.clock.time_msec()
362		yield self.db.runInteraction(
363		"add_messages_to_device_inbox", add_messages_txn, now_ms, stream_id
364		)
365		for user_id in local_messages_by_user_then_device.keys():
366		self._device_inbox_stream_cache.entity_has_changed(user_id, stream_id)
367		for destination in remote_messages_by_destination.keys():
368		self._device_federation_outbox_stream_cache.entity_has_changed(
369		destination, stream_id
370		)
371
372		return self._device_inbox_id_gen.get_current_token()
373
374		@defer.inlineCallbacks
375		def add_messages_from_remote_to_device_inbox(
376		self, origin, message_id, local_messages_by_user_then_device
377		):
378		def add_messages_txn(txn, now_ms, stream_id):
379		# Check if we've already inserted a matching message_id for that
380		# origin. This can happen if the origin doesn't receive our
381		# acknowledgement from the first time we received the message.
382		already_inserted = self.db.simple_select_one_txn(
383		txn,
384		table="device_federation_inbox",
385		keyvalues={"origin": origin, "message_id": message_id},
386		retcols=("message_id",),
387		allow_none=True,
388		)
389		if already_inserted is not None:
390		return
391
392		# Add an entry for this message_id so that we know we've processed
393		# it.
394		self.db.simple_insert_txn(
395		txn,
396		table="device_federation_inbox",
397		values={
398		"origin": origin,
399		"message_id": message_id,
400		"received_ts": now_ms,
401		},
402		)
403
404		# Add the messages to the approriate local device inboxes so that
405		# they'll be sent to the devices when they next sync.
406		self._add_messages_to_local_device_inbox_txn(
407		txn, stream_id, local_messages_by_user_then_device
408		)
409
410		with self._device_inbox_id_gen.get_next() as stream_id:
411		now_ms = self.clock.time_msec()
412		yield self.db.runInteraction(
413		"add_messages_from_remote_to_device_inbox",
414		add_messages_txn,
415		now_ms,
416		stream_id,
417		)
418		for user_id in local_messages_by_user_then_device.keys():
419		self._device_inbox_stream_cache.entity_has_changed(user_id, stream_id)
420
421		return stream_id
422
423		def _add_messages_to_local_device_inbox_txn(
424		self, txn, stream_id, messages_by_user_then_device
425		):
426		local_by_user_then_device = {}
427		for user_id, messages_by_device in messages_by_user_then_device.items():
428		messages_json_for_user = {}
429		devices = list(messages_by_device.keys())
430		if len(devices) == 1 and devices[0] == "*":
431		# Handle wildcard device_ids.
432		sql = "SELECT device_id FROM devices WHERE user_id = ?"
433		txn.execute(sql, (user_id,))
434		message_json = json.dumps(messages_by_device["*"])
435		for row in txn:
436		# Add the message for all devices for this user on this
437		# server.
438		device = row[0]
439		messages_json_for_user[device] = message_json
440		else:
441		if not devices:
442		continue
443
444		clause, args = make_in_list_sql_clause(
445		txn.database_engine, "device_id", devices
446		)
447		sql = "SELECT device_id FROM devices WHERE user_id = ? AND " + clause
448
449		# TODO: Maybe this needs to be done in batches if there are
450		# too many local devices for a given user.
451		txn.execute(sql, [user_id] + list(args))
452		for row in txn:
453		# Only insert into the local inbox if the device exists on
454		# this server
455		device = row[0]
456		message_json = json.dumps(messages_by_device[device])
457		messages_json_for_user[device] = message_json
458
459		if messages_json_for_user:
460		local_by_user_then_device[user_id] = messages_json_for_user
461
462		if not local_by_user_then_device:
463		return
464
465		sql = (
466		"INSERT INTO device_inbox"
467		" (user_id, device_id, stream_id, message_json)"
468		" VALUES (?,?,?,?)"
469		)
470		rows = []
471		for user_id, messages_by_device in local_by_user_then_device.items():
472		for device_id, message_json in messages_by_device.items():
473		rows.append((user_id, device_id, stream_id, message_json))
474
475		txn.executemany(sql, rows)

+0

-1309

~~synapse/storage/data_stores/main/devices.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2016 OpenMarket Ltd
2		# Copyright 2019 New Vector Ltd
3		# Copyright 2019 The Matrix.org Foundation C.I.C.
4		#
5		# Licensed under the Apache License, Version 2.0 (the "License");
6		# you may not use this file except in compliance with the License.
7		# You may obtain a copy of the License at
8		#
9		# http://www.apache.org/licenses/LICENSE-2.0
10		#
11		# Unless required by applicable law or agreed to in writing, software
12		# distributed under the License is distributed on an "AS IS" BASIS,
13		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14		# See the License for the specific language governing permissions and
15		# limitations under the License.
16		import logging
17		from typing import List, Optional, Set, Tuple
18
19		from canonicaljson import json
20
21		from twisted.internet import defer
22
23		from synapse.api.errors import Codes, StoreError
24		from synapse.logging.opentracing import (
25		get_active_span_text_map,
26		set_tag,
27		trace,
28		whitelisted_homeserver,
29		)
30		from synapse.metrics.background_process_metrics import run_as_background_process
31		from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
32		from synapse.storage.database import (
33		Database,
34		LoggingTransaction,
35		make_tuple_comparison_clause,
36		)
37		from synapse.types import Collection, get_verify_key_from_cross_signing_key
38		from synapse.util.caches.descriptors import (
39		Cache,
40		cached,
41		cachedInlineCallbacks,
42		cachedList,
43		)
44		from synapse.util.iterutils import batch_iter
45		from synapse.util.stringutils import shortstr
46
47		logger = logging.getLogger(__name__)
48
49		DROP_DEVICE_LIST_STREAMS_NON_UNIQUE_INDEXES = (
50		"drop_device_list_streams_non_unique_indexes"
51		)
52
53		BG_UPDATE_REMOVE_DUP_OUTBOUND_POKES = "remove_dup_outbound_pokes"
54
55
56		class DeviceWorkerStore(SQLBaseStore):
57		def get_device(self, user_id, device_id):
58		"""Retrieve a device. Only returns devices that are not marked as
59		hidden.
60
61		Args:
62		user_id (str): The ID of the user which owns the device
63		device_id (str): The ID of the device to retrieve
64		Returns:
65		defer.Deferred for a dict containing the device information
66		Raises:
67		StoreError: if the device is not found
68		"""
69		return self.db.simple_select_one(
70		table="devices",
71		keyvalues={"user_id": user_id, "device_id": device_id, "hidden": False},
72		retcols=("user_id", "device_id", "display_name"),
73		desc="get_device",
74		)
75
76		@defer.inlineCallbacks
77		def get_devices_by_user(self, user_id):
78		"""Retrieve all of a user's registered devices. Only returns devices
79		that are not marked as hidden.
80
81		Args:
82		user_id (str):
83		Returns:
84		defer.Deferred: resolves to a dict from device_id to a dict
85		containing "device_id", "user_id" and "display_name" for each
86		device.
87		"""
88		devices = yield self.db.simple_select_list(
89		table="devices",
90		keyvalues={"user_id": user_id, "hidden": False},
91		retcols=("user_id", "device_id", "display_name"),
92		desc="get_devices_by_user",
93		)
94
95		return {d["device_id"]: d for d in devices}
96
97		@trace
98		@defer.inlineCallbacks
99		def get_device_updates_by_remote(self, destination, from_stream_id, limit):
100		"""Get a stream of device updates to send to the given remote server.
101
102		Args:
103		destination (str): The host the device updates are intended for
104		from_stream_id (int): The minimum stream_id to filter updates by, exclusive
105		limit (int): Maximum number of device updates to return
106		Returns:
107		Deferred[tuple[int, list[tuple[string,dict]]]]:
108		current stream id (ie, the stream id of the last update included in the
109		response), and the list of updates, where each update is a pair of EDU
110		type and EDU contents
111		"""
112		now_stream_id = self._device_list_id_gen.get_current_token()
113
114		has_changed = self._device_list_federation_stream_cache.has_entity_changed(
115		destination, int(from_stream_id)
116		)
117		if not has_changed:
118		return now_stream_id, []
119
120		updates = yield self.db.runInteraction(
121		"get_device_updates_by_remote",
122		self._get_device_updates_by_remote_txn,
123		destination,
124		from_stream_id,
125		now_stream_id,
126		limit,
127		)
128
129		# Return an empty list if there are no updates
130		if not updates:
131		return now_stream_id, []
132
133		# get the cross-signing keys of the users in the list, so that we can
134		# determine which of the device changes were cross-signing keys
135		users = {r[0] for r in updates}
136		master_key_by_user = {}
137		self_signing_key_by_user = {}
138		for user in users:
139		cross_signing_key = yield self.get_e2e_cross_signing_key(user, "master")
140		if cross_signing_key:
141		key_id, verify_key = get_verify_key_from_cross_signing_key(
142		cross_signing_key
143		)
144		# verify_key is a VerifyKey from signedjson, which uses
145		# .version to denote the portion of the key ID after the
146		# algorithm and colon, which is the device ID
147		master_key_by_user[user] = {
148		"key_info": cross_signing_key,
149		"device_id": verify_key.version,
150		}
151
152		cross_signing_key = yield self.get_e2e_cross_signing_key(
153		user, "self_signing"
154		)
155		if cross_signing_key:
156		key_id, verify_key = get_verify_key_from_cross_signing_key(
157		cross_signing_key
158		)
159		self_signing_key_by_user[user] = {
160		"key_info": cross_signing_key,
161		"device_id": verify_key.version,
162		}
163
164		# Perform the equivalent of a GROUP BY
165		#
166		# Iterate through the updates list and copy non-duplicate
167		# (user_id, device_id) entries into a map, with the value being
168		# the max stream_id across each set of duplicate entries
169		#
170		# maps (user_id, device_id) -> (stream_id, opentracing_context)
171		#
172		# opentracing_context contains the opentracing metadata for the request
173		# that created the poke
174		#
175		# The most recent request's opentracing_context is used as the
176		# context which created the Edu.
177
178		query_map = {}
179		cross_signing_keys_by_user = {}
180		for user_id, device_id, update_stream_id, update_context in updates:
181		if (
182		user_id in master_key_by_user
183		and device_id == master_key_by_user[user_id]["device_id"]
184		):
185		result = cross_signing_keys_by_user.setdefault(user_id, {})
186		result["master_key"] = master_key_by_user[user_id]["key_info"]
187		elif (
188		user_id in self_signing_key_by_user
189		and device_id == self_signing_key_by_user[user_id]["device_id"]
190		):
191		result = cross_signing_keys_by_user.setdefault(user_id, {})
192		result["self_signing_key"] = self_signing_key_by_user[user_id][
193		"key_info"
194		]
195		else:
196		key = (user_id, device_id)
197
198		previous_update_stream_id, _ = query_map.get(key, (0, None))
199
200		if update_stream_id > previous_update_stream_id:
201		query_map[key] = (update_stream_id, update_context)
202
203		results = yield self._get_device_update_edus_by_remote(
204		destination, from_stream_id, query_map
205		)
206
207		# add the updated cross-signing keys to the results list
208		for user_id, result in cross_signing_keys_by_user.items():
209		result["user_id"] = user_id
210		# FIXME: switch to m.signing_key_update when MSC1756 is merged into the spec
211		results.append(("org.matrix.signing_key_update", result))
212
213		return now_stream_id, results
214
215		def _get_device_updates_by_remote_txn(
216		self, txn, destination, from_stream_id, now_stream_id, limit
217		):
218		"""Return device update information for a given remote destination
219
220		Args:
221		txn (LoggingTransaction): The transaction to execute
222		destination (str): The host the device updates are intended for
223		from_stream_id (int): The minimum stream_id to filter updates by, exclusive
224		now_stream_id (int): The maximum stream_id to filter updates by, inclusive
225		limit (int): Maximum number of device updates to return
226
227		Returns:
228		List: List of device updates
229		"""
230		# get the list of device updates that need to be sent
231		sql = """
232		SELECT user_id, device_id, stream_id, opentracing_context FROM device_lists_outbound_pokes
233		WHERE destination = ? AND ? < stream_id AND stream_id <= ?
234		ORDER BY stream_id
235		LIMIT ?
236		"""
237		txn.execute(sql, (destination, from_stream_id, now_stream_id, limit))
238
239		return list(txn)
240
241		@defer.inlineCallbacks
242		def _get_device_update_edus_by_remote(self, destination, from_stream_id, query_map):
243		"""Returns a list of device update EDUs as well as E2EE keys
244
245		Args:
246		destination (str): The host the device updates are intended for
247		from_stream_id (int): The minimum stream_id to filter updates by, exclusive
248		query_map (Dict[(str, str): (int, str\|None)]): Dictionary mapping
249		user_id/device_id to update stream_id and the relevent json-encoded
250		opentracing context
251
252		Returns:
253		List[Dict]: List of objects representing an device update EDU
254
255		"""
256		devices = (
257		yield self.db.runInteraction(
258		"_get_e2e_device_keys_txn",
259		self._get_e2e_device_keys_txn,
260		query_map.keys(),
261		include_all_devices=True,
262		include_deleted_devices=True,
263		)
264		if query_map
265		else {}
266		)
267
268		results = []
269		for user_id, user_devices in devices.items():
270		# The prev_id for the first row is always the last row before
271		# `from_stream_id`
272		prev_id = yield self._get_last_device_update_for_remote_user(
273		destination, user_id, from_stream_id
274		)
275
276		# make sure we go through the devices in stream order
277		device_ids = sorted(
278		user_devices.keys(), key=lambda i: query_map[(user_id, i)][0],
279		)
280
281		for device_id in device_ids:
282		device = user_devices[device_id]
283		stream_id, opentracing_context = query_map[(user_id, device_id)]
284		result = {
285		"user_id": user_id,
286		"device_id": device_id,
287		"prev_id": [prev_id] if prev_id else [],
288		"stream_id": stream_id,
289		"org.matrix.opentracing_context": opentracing_context,
290		}
291
292		prev_id = stream_id
293
294		if device is not None:
295		key_json = device.get("key_json", None)
296		if key_json:
297		result["keys"] = db_to_json(key_json)
298
299		if "signatures" in device:
300		for sig_user_id, sigs in device["signatures"].items():
301		result["keys"].setdefault("signatures", {}).setdefault(
302		sig_user_id, {}
303		).update(sigs)
304
305		device_display_name = device.get("device_display_name", None)
306		if device_display_name:
307		result["device_display_name"] = device_display_name
308		else:
309		result["deleted"] = True
310
311		results.append(("m.device_list_update", result))
312
313		return results
314
315		def _get_last_device_update_for_remote_user(
316		self, destination, user_id, from_stream_id
317		):
318		def f(txn):
319		prev_sent_id_sql = """
320		SELECT coalesce(max(stream_id), 0) as stream_id
321		FROM device_lists_outbound_last_success
322		WHERE destination = ? AND user_id = ? AND stream_id <= ?
323		"""
324		txn.execute(prev_sent_id_sql, (destination, user_id, from_stream_id))
325		rows = txn.fetchall()
326		return rows[0][0]
327
328		return self.db.runInteraction("get_last_device_update_for_remote_user", f)
329
330		def mark_as_sent_devices_by_remote(self, destination, stream_id):
331		"""Mark that updates have successfully been sent to the destination.
332		"""
333		return self.db.runInteraction(
334		"mark_as_sent_devices_by_remote",
335		self._mark_as_sent_devices_by_remote_txn,
336		destination,
337		stream_id,
338		)
339
340		def _mark_as_sent_devices_by_remote_txn(self, txn, destination, stream_id):
341		# We update the device_lists_outbound_last_success with the successfully
342		# poked users.
343		sql = """
344		SELECT user_id, coalesce(max(o.stream_id), 0)
345		FROM device_lists_outbound_pokes as o
346		WHERE destination = ? AND o.stream_id <= ?
347		GROUP BY user_id
348		"""
349		txn.execute(sql, (destination, stream_id))
350		rows = txn.fetchall()
351
352		self.db.simple_upsert_many_txn(
353		txn=txn,
354		table="device_lists_outbound_last_success",
355		key_names=("destination", "user_id"),
356		key_values=((destination, user_id) for user_id, _ in rows),
357		value_names=("stream_id",),
358		value_values=((stream_id,) for _, stream_id in rows),
359		)
360
361		# Delete all sent outbound pokes
362		sql = """
363		DELETE FROM device_lists_outbound_pokes
364		WHERE destination = ? AND stream_id <= ?
365		"""
366		txn.execute(sql, (destination, stream_id))
367
368		@defer.inlineCallbacks
369		def add_user_signature_change_to_streams(self, from_user_id, user_ids):
370		"""Persist that a user has made new signatures
371
372		Args:
373		from_user_id (str): the user who made the signatures
374		user_ids (list[str]): the users who were signed
375		"""
376
377		with self._device_list_id_gen.get_next() as stream_id:
378		yield self.db.runInteraction(
379		"add_user_sig_change_to_streams",
380		self._add_user_signature_change_txn,
381		from_user_id,
382		user_ids,
383		stream_id,
384		)
385		return stream_id
386
387		def _add_user_signature_change_txn(self, txn, from_user_id, user_ids, stream_id):
388		txn.call_after(
389		self._user_signature_stream_cache.entity_has_changed,
390		from_user_id,
391		stream_id,
392		)
393		self.db.simple_insert_txn(
394		txn,
395		"user_signature_stream",
396		values={
397		"stream_id": stream_id,
398		"from_user_id": from_user_id,
399		"user_ids": json.dumps(user_ids),
400		},
401		)
402
403		def get_device_stream_token(self):
404		return self._device_list_id_gen.get_current_token()
405
406		@trace
407		@defer.inlineCallbacks
408		def get_user_devices_from_cache(self, query_list):
409		"""Get the devices (and keys if any) for remote users from the cache.
410
411		Args:
412		query_list(list): List of (user_id, device_ids), if device_ids is
413		falsey then return all device ids for that user.
414
415		Returns:
416		(user_ids_not_in_cache, results_map), where user_ids_not_in_cache is
417		a set of user_ids and results_map is a mapping of
418		user_id -> device_id -> device_info
419		"""
420		user_ids = {user_id for user_id, _ in query_list}
421		user_map = yield self.get_device_list_last_stream_id_for_remotes(list(user_ids))
422
423		# We go and check if any of the users need to have their device lists
424		# resynced. If they do then we remove them from the cached list.
425		users_needing_resync = yield self.get_user_ids_requiring_device_list_resync(
426		user_ids
427		)
428		user_ids_in_cache = {
429		user_id for user_id, stream_id in user_map.items() if stream_id
430		} - users_needing_resync
431		user_ids_not_in_cache = user_ids - user_ids_in_cache
432
433		results = {}
434		for user_id, device_id in query_list:
435		if user_id not in user_ids_in_cache:
436		continue
437
438		if device_id:
439		device = yield self._get_cached_user_device(user_id, device_id)
440		results.setdefault(user_id, {})[device_id] = device
441		else:
442		results[user_id] = yield self.get_cached_devices_for_user(user_id)
443
444		set_tag("in_cache", results)
445		set_tag("not_in_cache", user_ids_not_in_cache)
446
447		return user_ids_not_in_cache, results
448
449		@cachedInlineCallbacks(num_args=2, tree=True)
450		def _get_cached_user_device(self, user_id, device_id):
451		content = yield self.db.simple_select_one_onecol(
452		table="device_lists_remote_cache",
453		keyvalues={"user_id": user_id, "device_id": device_id},
454		retcol="content",
455		desc="_get_cached_user_device",
456		)
457		return db_to_json(content)
458
459		@cachedInlineCallbacks()
460		def get_cached_devices_for_user(self, user_id):
461		devices = yield self.db.simple_select_list(
462		table="device_lists_remote_cache",
463		keyvalues={"user_id": user_id},
464		retcols=("device_id", "content"),
465		desc="get_cached_devices_for_user",
466		)
467		return {
468		device["device_id"]: db_to_json(device["content"]) for device in devices
469		}
470
471		def get_devices_with_keys_by_user(self, user_id):
472		"""Get all devices (with any device keys) for a user
473
474		Returns:
475		(stream_id, devices)
476		"""
477		return self.db.runInteraction(
478		"get_devices_with_keys_by_user",
479		self._get_devices_with_keys_by_user_txn,
480		user_id,
481		)
482
483		def _get_devices_with_keys_by_user_txn(self, txn, user_id):
484		now_stream_id = self._device_list_id_gen.get_current_token()
485
486		devices = self._get_e2e_device_keys_txn(
487		txn, [(user_id, None)], include_all_devices=True
488		)
489
490		if devices:
491		user_devices = devices[user_id]
492		results = []
493		for device_id, device in user_devices.items():
494		result = {"device_id": device_id}
495
496		key_json = device.get("key_json", None)
497		if key_json:
498		result["keys"] = db_to_json(key_json)
499
500		if "signatures" in device:
501		for sig_user_id, sigs in device["signatures"].items():
502		result["keys"].setdefault("signatures", {}).setdefault(
503		sig_user_id, {}
504		).update(sigs)
505
506		device_display_name = device.get("device_display_name", None)
507		if device_display_name:
508		result["device_display_name"] = device_display_name
509
510		results.append(result)
511
512		return now_stream_id, results
513
514		return now_stream_id, []
515
516		def get_users_whose_devices_changed(self, from_key, user_ids):
517		"""Get set of users whose devices have changed since `from_key` that
518		are in the given list of user_ids.
519
520		Args:
521		from_key (str): The device lists stream token
522		user_ids (Iterable[str])
523
524		Returns:
525		Deferred[set[str]]: The set of user_ids whose devices have changed
526		since `from_key`
527		"""
528		from_key = int(from_key)
529
530		# Get set of users who may have changed. Users not in the returned
531		# list have definitely not changed.
532		to_check = self._device_list_stream_cache.get_entities_changed(
533		user_ids, from_key
534		)
535
536		if not to_check:
537		return defer.succeed(set())
538
539		def _get_users_whose_devices_changed_txn(txn):
540		changes = set()
541
542		sql = """
543		SELECT DISTINCT user_id FROM device_lists_stream
544		WHERE stream_id > ?
545		AND
546		"""
547
548		for chunk in batch_iter(to_check, 100):
549		clause, args = make_in_list_sql_clause(
550		txn.database_engine, "user_id", chunk
551		)
552		txn.execute(sql + clause, (from_key,) + tuple(args))
553		changes.update(user_id for user_id, in txn)
554
555		return changes
556
557		return self.db.runInteraction(
558		"get_users_whose_devices_changed", _get_users_whose_devices_changed_txn
559		)
560
561		@defer.inlineCallbacks
562		def get_users_whose_signatures_changed(self, user_id, from_key):
563		"""Get the users who have new cross-signing signatures made by `user_id` since
564		`from_key`.
565
566		Args:
567		user_id (str): the user who made the signatures
568		from_key (str): The device lists stream token
569		"""
570		from_key = int(from_key)
571		if self._user_signature_stream_cache.has_entity_changed(user_id, from_key):
572		sql = """
573		SELECT DISTINCT user_ids FROM user_signature_stream
574		WHERE from_user_id = ? AND stream_id > ?
575		"""
576		rows = yield self.db.execute(
577		"get_users_whose_signatures_changed", None, sql, user_id, from_key
578		)
579		return {user for row in rows for user in db_to_json(row[0])}
580		else:
581		return set()
582
583		async def get_all_device_list_changes_for_remotes(
584		self, instance_name: str, last_id: int, current_id: int, limit: int
585		) -> Tuple[List[Tuple[int, tuple]], int, bool]:
586		"""Get updates for device lists replication stream.
587
588		Args:
589		instance_name: The writer we want to fetch updates from. Unused
590		here since there is only ever one writer.
591		last_id: The token to fetch updates from. Exclusive.
592		current_id: The token to fetch updates up to. Inclusive.
593		limit: The requested limit for the number of rows to return. The
594		function may return more or fewer rows.
595
596		Returns:
597		A tuple consisting of: the updates, a token to use to fetch
598		subsequent updates, and whether we returned fewer rows than exists
599		between the requested tokens due to the limit.
600
601		The token returned can be used in a subsequent call to this
602		function to get further updatees.
603
604		The updates are a list of 2-tuples of stream ID and the row data
605		"""
606
607		if last_id == current_id:
608		return [], current_id, False
609
610		def _get_all_device_list_changes_for_remotes(txn):
611		# This query Does The Right Thing where it'll correctly apply the
612		# bounds to the inner queries.
613		sql = """
614		SELECT stream_id, entity FROM (
615		SELECT stream_id, user_id AS entity FROM device_lists_stream
616		UNION ALL
617		SELECT stream_id, destination AS entity FROM device_lists_outbound_pokes
618		) AS e
619		WHERE ? < stream_id AND stream_id <= ?
620		LIMIT ?
621		"""
622
623		txn.execute(sql, (last_id, current_id, limit))
624		updates = [(row[0], row[1:]) for row in txn]
625		limited = False
626		upto_token = current_id
627		if len(updates) >= limit:
628		upto_token = updates[-1][0]
629		limited = True
630
631		return updates, upto_token, limited
632
633		return await self.db.runInteraction(
634		"get_all_device_list_changes_for_remotes",
635		_get_all_device_list_changes_for_remotes,
636		)
637
638		@cached(max_entries=10000)
639		def get_device_list_last_stream_id_for_remote(self, user_id):
640		"""Get the last stream_id we got for a user. May be None if we haven't
641		got any information for them.
642		"""
643		return self.db.simple_select_one_onecol(
644		table="device_lists_remote_extremeties",
645		keyvalues={"user_id": user_id},
646		retcol="stream_id",
647		desc="get_device_list_last_stream_id_for_remote",
648		allow_none=True,
649		)
650
651		@cachedList(
652		cached_method_name="get_device_list_last_stream_id_for_remote",
653		list_name="user_ids",
654		inlineCallbacks=True,
655		)
656		def get_device_list_last_stream_id_for_remotes(self, user_ids):
657		rows = yield self.db.simple_select_many_batch(
658		table="device_lists_remote_extremeties",
659		column="user_id",
660		iterable=user_ids,
661		retcols=("user_id", "stream_id"),
662		desc="get_device_list_last_stream_id_for_remotes",
663		)
664
665		results = {user_id: None for user_id in user_ids}
666		results.update({row["user_id"]: row["stream_id"] for row in rows})
667
668		return results
669
670		@defer.inlineCallbacks
671		def get_user_ids_requiring_device_list_resync(
672		self, user_ids: Optional[Collection[str]] = None,
673		) -> Set[str]:
674		"""Given a list of remote users return the list of users that we
675		should resync the device lists for. If None is given instead of a list,
676		return every user that we should resync the device lists for.
677
678		Returns:
679		The IDs of users whose device lists need resync.
680		"""
681		if user_ids:
682		rows = yield self.db.simple_select_many_batch(
683		table="device_lists_remote_resync",
684		column="user_id",
685		iterable=user_ids,
686		retcols=("user_id",),
687		desc="get_user_ids_requiring_device_list_resync_with_iterable",
688		)
689		else:
690		rows = yield self.db.simple_select_list(
691		table="device_lists_remote_resync",
692		keyvalues=None,
693		retcols=("user_id",),
694		desc="get_user_ids_requiring_device_list_resync",
695		)
696
697		return {row["user_id"] for row in rows}
698
699		def mark_remote_user_device_cache_as_stale(self, user_id: str):
700		"""Records that the server has reason to believe the cache of the devices
701		for the remote users is out of date.
702		"""
703		return self.db.simple_upsert(
704		table="device_lists_remote_resync",
705		keyvalues={"user_id": user_id},
706		values={},
707		insertion_values={"added_ts": self._clock.time_msec()},
708		desc="make_remote_user_device_cache_as_stale",
709		)
710
711		def mark_remote_user_device_list_as_unsubscribed(self, user_id):
712		"""Mark that we no longer track device lists for remote user.
713		"""
714
715		def _mark_remote_user_device_list_as_unsubscribed_txn(txn):
716		self.db.simple_delete_txn(
717		txn,
718		table="device_lists_remote_extremeties",
719		keyvalues={"user_id": user_id},
720		)
721		self._invalidate_cache_and_stream(
722		txn, self.get_device_list_last_stream_id_for_remote, (user_id,)
723		)
724
725		return self.db.runInteraction(
726		"mark_remote_user_device_list_as_unsubscribed",
727		_mark_remote_user_device_list_as_unsubscribed_txn,
728		)
729
730
731		class DeviceBackgroundUpdateStore(SQLBaseStore):
732		def __init__(self, database: Database, db_conn, hs):
733		super(DeviceBackgroundUpdateStore, self).__init__(database, db_conn, hs)
734
735		self.db.updates.register_background_index_update(
736		"device_lists_stream_idx",
737		index_name="device_lists_stream_user_id",
738		table="device_lists_stream",
739		columns=["user_id", "device_id"],
740		)
741
742		# create a unique index on device_lists_remote_cache
743		self.db.updates.register_background_index_update(
744		"device_lists_remote_cache_unique_idx",
745		index_name="device_lists_remote_cache_unique_id",
746		table="device_lists_remote_cache",
747		columns=["user_id", "device_id"],
748		unique=True,
749		)
750
751		# And one on device_lists_remote_extremeties
752		self.db.updates.register_background_index_update(
753		"device_lists_remote_extremeties_unique_idx",
754		index_name="device_lists_remote_extremeties_unique_idx",
755		table="device_lists_remote_extremeties",
756		columns=["user_id"],
757		unique=True,
758		)
759
760		# once they complete, we can remove the old non-unique indexes.
761		self.db.updates.register_background_update_handler(
762		DROP_DEVICE_LIST_STREAMS_NON_UNIQUE_INDEXES,
763		self._drop_device_list_streams_non_unique_indexes,
764		)
765
766		# clear out duplicate device list outbound pokes
767		self.db.updates.register_background_update_handler(
768		BG_UPDATE_REMOVE_DUP_OUTBOUND_POKES, self._remove_duplicate_outbound_pokes,
769		)
770
771		# a pair of background updates that were added during the 1.14 release cycle,
772		# but replaced with 58/06dlols_unique_idx.py
773		self.db.updates.register_noop_background_update(
774		"device_lists_outbound_last_success_unique_idx",
775		)
776		self.db.updates.register_noop_background_update(
777		"drop_device_lists_outbound_last_success_non_unique_idx",
778		)
779
780		@defer.inlineCallbacks
781		def _drop_device_list_streams_non_unique_indexes(self, progress, batch_size):
782		def f(conn):
783		txn = conn.cursor()
784		txn.execute("DROP INDEX IF EXISTS device_lists_remote_cache_id")
785		txn.execute("DROP INDEX IF EXISTS device_lists_remote_extremeties_id")
786		txn.close()
787
788		yield self.db.runWithConnection(f)
789		yield self.db.updates._end_background_update(
790		DROP_DEVICE_LIST_STREAMS_NON_UNIQUE_INDEXES
791		)
792		return 1
793
794		async def _remove_duplicate_outbound_pokes(self, progress, batch_size):
795		# for some reason, we have accumulated duplicate entries in
796		# device_lists_outbound_pokes, which makes prune_outbound_device_list_pokes less
797		# efficient.
798		#
799		# For each duplicate, we delete all the existing rows and put one back.
800
801		KEY_COLS = ["stream_id", "destination", "user_id", "device_id"]
802		last_row = progress.get(
803		"last_row",
804		{"stream_id": 0, "destination": "", "user_id": "", "device_id": ""},
805		)
806
807		def _txn(txn):
808		clause, args = make_tuple_comparison_clause(
809		self.db.engine, [(x, last_row[x]) for x in KEY_COLS]
810		)
811		sql = """
812		SELECT stream_id, destination, user_id, device_id, MAX(ts) AS ts
813		FROM device_lists_outbound_pokes
814		WHERE %s
815		GROUP BY %s
816		HAVING count(*) > 1
817		ORDER BY %s
818		LIMIT ?
819		""" % (
820		clause, # WHERE
821		",".join(KEY_COLS), # GROUP BY
822		",".join(KEY_COLS), # ORDER BY
823		)
824		txn.execute(sql, args + [batch_size])
825		rows = self.db.cursor_to_dict(txn)
826
827		row = None
828		for row in rows:
829		self.db.simple_delete_txn(
830		txn, "device_lists_outbound_pokes", {x: row[x] for x in KEY_COLS},
831		)
832
833		row["sent"] = False
834		self.db.simple_insert_txn(
835		txn, "device_lists_outbound_pokes", row,
836		)
837
838		if row:
839		self.db.updates._background_update_progress_txn(
840		txn, BG_UPDATE_REMOVE_DUP_OUTBOUND_POKES, {"last_row": row},
841		)
842
843		return len(rows)
844
845		rows = await self.db.runInteraction(BG_UPDATE_REMOVE_DUP_OUTBOUND_POKES, _txn)
846
847		if not rows:
848		await self.db.updates._end_background_update(
849		BG_UPDATE_REMOVE_DUP_OUTBOUND_POKES
850		)
851
852		return rows
853
854
855		class DeviceStore(DeviceWorkerStore, DeviceBackgroundUpdateStore):
856		def __init__(self, database: Database, db_conn, hs):
857		super(DeviceStore, self).__init__(database, db_conn, hs)
858
859		# Map of (user_id, device_id) -> bool. If there is an entry that implies
860		# the device exists.
861		self.device_id_exists_cache = Cache(
862		name="device_id_exists", keylen=2, max_entries=10000
863		)
864
865		self._clock.looping_call(self._prune_old_outbound_device_pokes, 60 * 60 * 1000)
866
867		@defer.inlineCallbacks
868		def store_device(self, user_id, device_id, initial_device_display_name):
869		"""Ensure the given device is known; add it to the store if not
870
871		Args:
872		user_id (str): id of user associated with the device
873		device_id (str): id of device
874		initial_device_display_name (str): initial displayname of the
875		device. Ignored if device exists.
876		Returns:
877		defer.Deferred: boolean whether the device was inserted or an
878		existing device existed with that ID.
879		Raises:
880		StoreError: if the device is already in use
881		"""
882		key = (user_id, device_id)
883		if self.device_id_exists_cache.get(key, None):
884		return False
885
886		try:
887		inserted = yield self.db.simple_insert(
888		"devices",
889		values={
890		"user_id": user_id,
891		"device_id": device_id,
892		"display_name": initial_device_display_name,
893		"hidden": False,
894		},
895		desc="store_device",
896		or_ignore=True,
897		)
898		if not inserted:
899		# if the device already exists, check if it's a real device, or
900		# if the device ID is reserved by something else
901		hidden = yield self.db.simple_select_one_onecol(
902		"devices",
903		keyvalues={"user_id": user_id, "device_id": device_id},
904		retcol="hidden",
905		)
906		if hidden:
907		raise StoreError(400, "The device ID is in use", Codes.FORBIDDEN)
908		self.device_id_exists_cache.prefill(key, True)
909		return inserted
910		except StoreError:
911		raise
912		except Exception as e:
913		logger.error(
914		"store_device with device_id=%s(%r) user_id=%s(%r)"
915		" display_name=%s(%r) failed: %s",
916		type(device_id).__name__,
917		device_id,
918		type(user_id).__name__,
919		user_id,
920		type(initial_device_display_name).__name__,
921		initial_device_display_name,
922		e,
923		)
924		raise StoreError(500, "Problem storing device.")
925
926		@defer.inlineCallbacks
927		def delete_device(self, user_id, device_id):
928		"""Delete a device.
929
930		Args:
931		user_id (str): The ID of the user which owns the device
932		device_id (str): The ID of the device to delete
933		Returns:
934		defer.Deferred
935		"""
936		yield self.db.simple_delete_one(
937		table="devices",
938		keyvalues={"user_id": user_id, "device_id": device_id, "hidden": False},
939		desc="delete_device",
940		)
941
942		self.device_id_exists_cache.invalidate((user_id, device_id))
943
944		@defer.inlineCallbacks
945		def delete_devices(self, user_id, device_ids):
946		"""Deletes several devices.
947
948		Args:
949		user_id (str): The ID of the user which owns the devices
950		device_ids (list): The IDs of the devices to delete
951		Returns:
952		defer.Deferred
953		"""
954		yield self.db.simple_delete_many(
955		table="devices",
956		column="device_id",
957		iterable=device_ids,
958		keyvalues={"user_id": user_id, "hidden": False},
959		desc="delete_devices",
960		)
961		for device_id in device_ids:
962		self.device_id_exists_cache.invalidate((user_id, device_id))
963
964		def update_device(self, user_id, device_id, new_display_name=None):
965		"""Update a device. Only updates the device if it is not marked as
966		hidden.
967
968		Args:
969		user_id (str): The ID of the user which owns the device
970		device_id (str): The ID of the device to update
971		new_display_name (str\|None): new displayname for device; None
972		to leave unchanged
973		Raises:
974		StoreError: if the device is not found
975		Returns:
976		defer.Deferred
977		"""
978		updates = {}
979		if new_display_name is not None:
980		updates["display_name"] = new_display_name
981		if not updates:
982		return defer.succeed(None)
983		return self.db.simple_update_one(
984		table="devices",
985		keyvalues={"user_id": user_id, "device_id": device_id, "hidden": False},
986		updatevalues=updates,
987		desc="update_device",
988		)
989
990		def update_remote_device_list_cache_entry(
991		self, user_id, device_id, content, stream_id
992		):
993		"""Updates a single device in the cache of a remote user's devicelist.
994
995		Note: assumes that we are the only thread that can be updating this user's
996		device list.
997
998		Args:
999		user_id (str): User to update device list for
1000		device_id (str): ID of decivice being updated
1001		content (dict): new data on this device
1002		stream_id (int): the version of the device list
1003
1004		Returns:
1005		Deferred[None]
1006		"""
1007		return self.db.runInteraction(
1008		"update_remote_device_list_cache_entry",
1009		self._update_remote_device_list_cache_entry_txn,
1010		user_id,
1011		device_id,
1012		content,
1013		stream_id,
1014		)
1015
1016		def _update_remote_device_list_cache_entry_txn(
1017		self, txn, user_id, device_id, content, stream_id
1018		):
1019		if content.get("deleted"):
1020		self.db.simple_delete_txn(
1021		txn,
1022		table="device_lists_remote_cache",
1023		keyvalues={"user_id": user_id, "device_id": device_id},
1024		)
1025
1026		txn.call_after(self.device_id_exists_cache.invalidate, (user_id, device_id))
1027		else:
1028		self.db.simple_upsert_txn(
1029		txn,
1030		table="device_lists_remote_cache",
1031		keyvalues={"user_id": user_id, "device_id": device_id},
1032		values={"content": json.dumps(content)},
1033		# we don't need to lock, because we assume we are the only thread
1034		# updating this user's devices.
1035		lock=False,
1036		)
1037
1038		txn.call_after(self._get_cached_user_device.invalidate, (user_id, device_id))
1039		txn.call_after(self.get_cached_devices_for_user.invalidate, (user_id,))
1040		txn.call_after(
1041		self.get_device_list_last_stream_id_for_remote.invalidate, (user_id,)
1042		)
1043
1044		self.db.simple_upsert_txn(
1045		txn,
1046		table="device_lists_remote_extremeties",
1047		keyvalues={"user_id": user_id},
1048		values={"stream_id": stream_id},
1049		# again, we can assume we are the only thread updating this user's
1050		# extremity.
1051		lock=False,
1052		)
1053
1054		def update_remote_device_list_cache(self, user_id, devices, stream_id):
1055		"""Replace the entire cache of the remote user's devices.
1056
1057		Note: assumes that we are the only thread that can be updating this user's
1058		device list.
1059
1060		Args:
1061		user_id (str): User to update device list for
1062		devices (list[dict]): list of device objects supplied over federation
1063		stream_id (int): the version of the device list
1064
1065		Returns:
1066		Deferred[None]
1067		"""
1068		return self.db.runInteraction(
1069		"update_remote_device_list_cache",
1070		self._update_remote_device_list_cache_txn,
1071		user_id,
1072		devices,
1073		stream_id,
1074		)
1075
1076		def _update_remote_device_list_cache_txn(self, txn, user_id, devices, stream_id):
1077		self.db.simple_delete_txn(
1078		txn, table="device_lists_remote_cache", keyvalues={"user_id": user_id}
1079		)
1080
1081		self.db.simple_insert_many_txn(
1082		txn,
1083		table="device_lists_remote_cache",
1084		values=[
1085		{
1086		"user_id": user_id,
1087		"device_id": content["device_id"],
1088		"content": json.dumps(content),
1089		}
1090		for content in devices
1091		],
1092		)
1093
1094		txn.call_after(self.get_cached_devices_for_user.invalidate, (user_id,))
1095		txn.call_after(self._get_cached_user_device.invalidate_many, (user_id,))
1096		txn.call_after(
1097		self.get_device_list_last_stream_id_for_remote.invalidate, (user_id,)
1098		)
1099
1100		self.db.simple_upsert_txn(
1101		txn,
1102		table="device_lists_remote_extremeties",
1103		keyvalues={"user_id": user_id},
1104		values={"stream_id": stream_id},
1105		# we don't need to lock, because we can assume we are the only thread
1106		# updating this user's extremity.
1107		lock=False,
1108		)
1109
1110		# If we're replacing the remote user's device list cache presumably
1111		# we've done a full resync, so we remove the entry that says we need
1112		# to resync
1113		self.db.simple_delete_txn(
1114		txn, table="device_lists_remote_resync", keyvalues={"user_id": user_id},
1115		)
1116
1117		@defer.inlineCallbacks
1118		def add_device_change_to_streams(self, user_id, device_ids, hosts):
1119		"""Persist that a user's devices have been updated, and which hosts
1120		(if any) should be poked.
1121		"""
1122		if not device_ids:
1123		return
1124
1125		with self._device_list_id_gen.get_next_mult(len(device_ids)) as stream_ids:
1126		yield self.db.runInteraction(
1127		"add_device_change_to_stream",
1128		self._add_device_change_to_stream_txn,
1129		user_id,
1130		device_ids,
1131		stream_ids,
1132		)
1133
1134		if not hosts:
1135		return stream_ids[-1]
1136
1137		context = get_active_span_text_map()
1138		with self._device_list_id_gen.get_next_mult(
1139		len(hosts) * len(device_ids)
1140		) as stream_ids:
1141		yield self.db.runInteraction(
1142		"add_device_outbound_poke_to_stream",
1143		self._add_device_outbound_poke_to_stream_txn,
1144		user_id,
1145		device_ids,
1146		hosts,
1147		stream_ids,
1148		context,
1149		)
1150
1151		return stream_ids[-1]
1152
1153		def _add_device_change_to_stream_txn(
1154		self,
1155		txn: LoggingTransaction,
1156		user_id: str,
1157		device_ids: Collection[str],
1158		stream_ids: List[str],
1159		):
1160		txn.call_after(
1161		self._device_list_stream_cache.entity_has_changed, user_id, stream_ids[-1],
1162		)
1163
1164		min_stream_id = stream_ids[0]
1165
1166		# Delete older entries in the table, as we really only care about
1167		# when the latest change happened.
1168		txn.executemany(
1169		"""
1170		DELETE FROM device_lists_stream
1171		WHERE user_id = ? AND device_id = ? AND stream_id < ?
1172		""",
1173		[(user_id, device_id, min_stream_id) for device_id in device_ids],
1174		)
1175
1176		self.db.simple_insert_many_txn(
1177		txn,
1178		table="device_lists_stream",
1179		values=[
1180		{"stream_id": stream_id, "user_id": user_id, "device_id": device_id}
1181		for stream_id, device_id in zip(stream_ids, device_ids)
1182		],
1183		)
1184
1185		def _add_device_outbound_poke_to_stream_txn(
1186		self, txn, user_id, device_ids, hosts, stream_ids, context,
1187		):
1188		for host in hosts:
1189		txn.call_after(
1190		self._device_list_federation_stream_cache.entity_has_changed,
1191		host,
1192		stream_ids[-1],
1193		)
1194
1195		now = self._clock.time_msec()
1196		next_stream_id = iter(stream_ids)
1197
1198		self.db.simple_insert_many_txn(
1199		txn,
1200		table="device_lists_outbound_pokes",
1201		values=[
1202		{
1203		"destination": destination,
1204		"stream_id": next(next_stream_id),
1205		"user_id": user_id,
1206		"device_id": device_id,
1207		"sent": False,
1208		"ts": now,
1209		"opentracing_context": json.dumps(context)
1210		if whitelisted_homeserver(destination)
1211		else "{}",
1212		}
1213		for destination in hosts
1214		for device_id in device_ids
1215		],
1216		)
1217
1218		def _prune_old_outbound_device_pokes(self, prune_age=24 * 60 * 60 * 1000):
1219		"""Delete old entries out of the device_lists_outbound_pokes to ensure
1220		that we don't fill up due to dead servers.
1221
1222		Normally, we try to send device updates as a delta since a previous known point:
1223		this is done by setting the prev_id in the m.device_list_update EDU. However,
1224		for that to work, we have to have a complete record of each change to
1225		each device, which can add up to quite a lot of data.
1226
1227		An alternative mechanism is that, if the remote server sees that it has missed
1228		an entry in the stream_id sequence for a given user, it will request a full
1229		list of that user's devices. Hence, we can reduce the amount of data we have to
1230		store (and transmit in some future transaction), by clearing almost everything
1231		for a given destination out of the database, and having the remote server
1232		resync.
1233
1234		All we need to do is make sure we keep at least one row for each
1235		(user, destination) pair, to remind us to send a m.device_list_update EDU for
1236		that user when the destination comes back. It doesn't matter which device
1237		we keep.
1238		"""
1239		yesterday = self._clock.time_msec() - prune_age
1240
1241		def _prune_txn(txn):
1242		# look for (user, destination) pairs which have an update older than
1243		# the cutoff.
1244		#
1245		# For each pair, we also need to know the most recent stream_id, and
1246		# an arbitrary device_id at that stream_id.
1247		select_sql = """
1248		SELECT
1249		dlop1.destination,
1250		dlop1.user_id,
1251		MAX(dlop1.stream_id) AS stream_id,
1252		(SELECT MIN(dlop2.device_id) AS device_id FROM
1253		device_lists_outbound_pokes dlop2
1254		WHERE dlop2.destination = dlop1.destination AND
1255		dlop2.user_id=dlop1.user_id AND
1256		dlop2.stream_id=MAX(dlop1.stream_id)
1257		)
1258		FROM device_lists_outbound_pokes dlop1
1259		GROUP BY destination, user_id
1260		HAVING min(ts) < ? AND count(*) > 1
1261		"""
1262
1263		txn.execute(select_sql, (yesterday,))
1264		rows = txn.fetchall()
1265
1266		if not rows:
1267		return
1268
1269		logger.info(
1270		"Pruning old outbound device list updates for %i users/destinations: %s",
1271		len(rows),
1272		shortstr((row[0], row[1]) for row in rows),
1273		)
1274
1275		# we want to keep the update with the highest stream_id for each user.
1276		#
1277		# there might be more than one update (with different device_ids) with the
1278		# same stream_id, so we also delete all but one rows with the max stream id.
1279		delete_sql = """
1280		DELETE FROM device_lists_outbound_pokes
1281		WHERE destination = ? AND user_id = ? AND (
1282		stream_id < ? OR
1283		(stream_id = ? AND device_id != ?)
1284		)
1285		"""
1286		count = 0
1287		for (destination, user_id, stream_id, device_id) in rows:
1288		txn.execute(
1289		delete_sql, (destination, user_id, stream_id, stream_id, device_id)
1290		)
1291		count += txn.rowcount
1292
1293		# Since we've deleted unsent deltas, we need to remove the entry
1294		# of last successful sent so that the prev_ids are correctly set.
1295		sql = """
1296		DELETE FROM device_lists_outbound_last_success
1297		WHERE destination = ? AND user_id = ?
1298		"""
1299		txn.executemany(sql, ((row[0], row[1]) for row in rows))
1300
1301		logger.info("Pruned %d device list outbound pokes", count)
1302
1303		return run_as_background_process(
1304		"prune_old_outbound_device_pokes",
1305		self.db.runInteraction,
1306		"_prune_old_outbound_device_pokes",
1307		_prune_txn,
1308		)

+0

-195

~~synapse/storage/data_stores/main/directory.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		from collections import namedtuple
16		from typing import Optional
17
18		from twisted.internet import defer
19
20		from synapse.api.errors import SynapseError
21		from synapse.storage._base import SQLBaseStore
22		from synapse.util.caches.descriptors import cached
23
24		RoomAliasMapping = namedtuple("RoomAliasMapping", ("room_id", "room_alias", "servers"))
25
26
27		class DirectoryWorkerStore(SQLBaseStore):
28		@defer.inlineCallbacks
29		def get_association_from_room_alias(self, room_alias):
30		""" Get's the room_id and server list for a given room_alias
31
32		Args:
33		room_alias (RoomAlias)
34
35		Returns:
36		Deferred: results in namedtuple with keys "room_id" and
37		"servers" or None if no association can be found
38		"""
39		room_id = yield self.db.simple_select_one_onecol(
40		"room_aliases",
41		{"room_alias": room_alias.to_string()},
42		"room_id",
43		allow_none=True,
44		desc="get_association_from_room_alias",
45		)
46
47		if not room_id:
48		return None
49
50		servers = yield self.db.simple_select_onecol(
51		"room_alias_servers",
52		{"room_alias": room_alias.to_string()},
53		"server",
54		desc="get_association_from_room_alias",
55		)
56
57		if not servers:
58		return None
59
60		return RoomAliasMapping(room_id, room_alias.to_string(), servers)
61
62		def get_room_alias_creator(self, room_alias):
63		return self.db.simple_select_one_onecol(
64		table="room_aliases",
65		keyvalues={"room_alias": room_alias},
66		retcol="creator",
67		desc="get_room_alias_creator",
68		)
69
70		@cached(max_entries=5000)
71		def get_aliases_for_room(self, room_id):
72		return self.db.simple_select_onecol(
73		"room_aliases",
74		{"room_id": room_id},
75		"room_alias",
76		desc="get_aliases_for_room",
77		)
78
79
80		class DirectoryStore(DirectoryWorkerStore):
81		@defer.inlineCallbacks
82		def create_room_alias_association(self, room_alias, room_id, servers, creator=None):
83		""" Creates an association between a room alias and room_id/servers
84
85		Args:
86		room_alias (RoomAlias)
87		room_id (str)
88		servers (list)
89		creator (str): Optional user_id of creator.
90
91		Returns:
92		Deferred
93		"""
94
95		def alias_txn(txn):
96		self.db.simple_insert_txn(
97		txn,
98		"room_aliases",
99		{
100		"room_alias": room_alias.to_string(),
101		"room_id": room_id,
102		"creator": creator,
103		},
104		)
105
106		self.db.simple_insert_many_txn(
107		txn,
108		table="room_alias_servers",
109		values=[
110		{"room_alias": room_alias.to_string(), "server": server}
111		for server in servers
112		],
113		)
114
115		self._invalidate_cache_and_stream(
116		txn, self.get_aliases_for_room, (room_id,)
117		)
118
119		try:
120		ret = yield self.db.runInteraction(
121		"create_room_alias_association", alias_txn
122		)
123		except self.database_engine.module.IntegrityError:
124		raise SynapseError(
125		409, "Room alias %s already exists" % room_alias.to_string()
126		)
127		return ret
128
129		@defer.inlineCallbacks
130		def delete_room_alias(self, room_alias):
131		room_id = yield self.db.runInteraction(
132		"delete_room_alias", self._delete_room_alias_txn, room_alias
133		)
134
135		return room_id
136
137		def _delete_room_alias_txn(self, txn, room_alias):
138		txn.execute(
139		"SELECT room_id FROM room_aliases WHERE room_alias = ?",
140		(room_alias.to_string(),),
141		)
142
143		res = txn.fetchone()
144		if res:
145		room_id = res[0]
146		else:
147		return None
148
149		txn.execute(
150		"DELETE FROM room_aliases WHERE room_alias = ?", (room_alias.to_string(),)
151		)
152
153		txn.execute(
154		"DELETE FROM room_alias_servers WHERE room_alias = ?",
155		(room_alias.to_string(),),
156		)
157
158		self._invalidate_cache_and_stream(txn, self.get_aliases_for_room, (room_id,))
159
160		return room_id
161
162		def update_aliases_for_room(
163		self, old_room_id: str, new_room_id: str, creator: Optional[str] = None,
164		):
165		"""Repoint all of the aliases for a given room, to a different room.
166
167		Args:
168		old_room_id:
169		new_room_id:
170		creator: The user to record as the creator of the new mapping.
171		If None, the creator will be left unchanged.
172		"""
173
174		def _update_aliases_for_room_txn(txn):
175		update_creator_sql = ""
176		sql_params = (new_room_id, old_room_id)
177		if creator:
178		update_creator_sql = ", creator = ?"
179		sql_params = (new_room_id, creator, old_room_id)
180
181		sql = "UPDATE room_aliases SET room_id = ? %s WHERE room_id = ?" % (
182		update_creator_sql,
183		)
184		txn.execute(sql, sql_params)
185		self._invalidate_cache_and_stream(
186		txn, self.get_aliases_for_room, (old_room_id,)
187		)
188		self._invalidate_cache_and_stream(
189		txn, self.get_aliases_for_room, (new_room_id,)
190		)
191
192		return self.db.runInteraction(
193		"_update_aliases_for_room_txn", _update_aliases_for_room_txn
194		)

+0

-439

~~synapse/storage/data_stores/main/e2e_room_keys.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2017 New Vector Ltd
2		# Copyright 2019 Matrix.org Foundation C.I.C.
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		from canonicaljson import json
17
18		from twisted.internet import defer
19
20		from synapse.api.errors import StoreError
21		from synapse.logging.opentracing import log_kv, trace
22		from synapse.storage._base import SQLBaseStore, db_to_json
23
24
25		class EndToEndRoomKeyStore(SQLBaseStore):
26		@defer.inlineCallbacks
27		def update_e2e_room_key(self, user_id, version, room_id, session_id, room_key):
28		"""Replaces the encrypted E2E room key for a given session in a given backup
29
30		Args:
31		user_id(str): the user whose backup we're setting
32		version(str): the version ID of the backup we're updating
33		room_id(str): the ID of the room whose keys we're setting
34		session_id(str): the session whose room_key we're setting
35		room_key(dict): the room_key being set
36		Raises:
37		StoreError
38		"""
39
40		yield self.db.simple_update_one(
41		table="e2e_room_keys",
42		keyvalues={
43		"user_id": user_id,
44		"version": version,
45		"room_id": room_id,
46		"session_id": session_id,
47		},
48		updatevalues={
49		"first_message_index": room_key["first_message_index"],
50		"forwarded_count": room_key["forwarded_count"],
51		"is_verified": room_key["is_verified"],
52		"session_data": json.dumps(room_key["session_data"]),
53		},
54		desc="update_e2e_room_key",
55		)
56
57		@defer.inlineCallbacks
58		def add_e2e_room_keys(self, user_id, version, room_keys):
59		"""Bulk add room keys to a given backup.
60
61		Args:
62		user_id (str): the user whose backup we're adding to
63		version (str): the version ID of the backup for the set of keys we're adding to
64		room_keys (iterable[(str, str, dict)]): the keys to add, in the form
65		(roomID, sessionID, keyData)
66		"""
67
68		values = []
69		for (room_id, session_id, room_key) in room_keys:
70		values.append(
71		{
72		"user_id": user_id,
73		"version": version,
74		"room_id": room_id,
75		"session_id": session_id,
76		"first_message_index": room_key["first_message_index"],
77		"forwarded_count": room_key["forwarded_count"],
78		"is_verified": room_key["is_verified"],
79		"session_data": json.dumps(room_key["session_data"]),
80		}
81		)
82		log_kv(
83		{
84		"message": "Set room key",
85		"room_id": room_id,
86		"session_id": session_id,
87		"room_key": room_key,
88		}
89		)
90
91		yield self.db.simple_insert_many(
92		table="e2e_room_keys", values=values, desc="add_e2e_room_keys"
93		)
94
95		@trace
96		@defer.inlineCallbacks
97		def get_e2e_room_keys(self, user_id, version, room_id=None, session_id=None):
98		"""Bulk get the E2E room keys for a given backup, optionally filtered to a given
99		room, or a given session.
100
101		Args:
102		user_id (str): the user whose backup we're querying
103		version (str): the version ID of the backup for the set of keys we're querying
104		room_id (str): Optional. the ID of the room whose keys we're querying, if any.
105		If not specified, we return the keys for all the rooms in the backup.
106		session_id (str): Optional. the session whose room_key we're querying, if any.
107		If specified, we also require the room_id to be specified.
108		If not specified, we return all the keys in this version of
109		the backup (or for the specified room)
110
111		Returns:
112		A deferred list of dicts giving the session_data and message metadata for
113		these room keys.
114		"""
115
116		try:
117		version = int(version)
118		except ValueError:
119		return {"rooms": {}}
120
121		keyvalues = {"user_id": user_id, "version": version}
122		if room_id:
123		keyvalues["room_id"] = room_id
124		if session_id:
125		keyvalues["session_id"] = session_id
126
127		rows = yield self.db.simple_select_list(
128		table="e2e_room_keys",
129		keyvalues=keyvalues,
130		retcols=(
131		"user_id",
132		"room_id",
133		"session_id",
134		"first_message_index",
135		"forwarded_count",
136		"is_verified",
137		"session_data",
138		),
139		desc="get_e2e_room_keys",
140		)
141
142		sessions = {"rooms": {}}
143		for row in rows:
144		room_entry = sessions["rooms"].setdefault(row["room_id"], {"sessions": {}})
145		room_entry["sessions"][row["session_id"]] = {
146		"first_message_index": row["first_message_index"],
147		"forwarded_count": row["forwarded_count"],
148		# is_verified must be returned to the client as a boolean
149		"is_verified": bool(row["is_verified"]),
150		"session_data": db_to_json(row["session_data"]),
151		}
152
153		return sessions
154
155		def get_e2e_room_keys_multi(self, user_id, version, room_keys):
156		"""Get multiple room keys at a time. The difference between this function and
157		get_e2e_room_keys is that this function can be used to retrieve
158		multiple specific keys at a time, whereas get_e2e_room_keys is used for
159		getting all the keys in a backup version, all the keys for a room, or a
160		specific key.
161
162		Args:
163		user_id (str): the user whose backup we're querying
164		version (str): the version ID of the backup we're querying about
165		room_keys (dict[str, dict[str, iterable[str]]]): a map from
166		room ID -> {"session": [session ids]} indicating the session IDs
167		that we want to query
168
169		Returns:
170		Deferred[dict[str, dict[str, dict]]]: a map of room IDs to session IDs to room key
171		"""
172
173		return self.db.runInteraction(
174		"get_e2e_room_keys_multi",
175		self._get_e2e_room_keys_multi_txn,
176		user_id,
177		version,
178		room_keys,
179		)
180
181		@staticmethod
182		def _get_e2e_room_keys_multi_txn(txn, user_id, version, room_keys):
183		if not room_keys:
184		return {}
185
186		where_clauses = []
187		params = [user_id, version]
188		for room_id, room in room_keys.items():
189		sessions = list(room["sessions"])
190		if not sessions:
191		continue
192		params.append(room_id)
193		params.extend(sessions)
194		where_clauses.append(
195		"(room_id = ? AND session_id IN (%s))"
196		% (",".join(["?" for _ in sessions]),)
197		)
198
199		# check if we're actually querying something
200		if not where_clauses:
201		return {}
202
203		sql = """
204		SELECT room_id, session_id, first_message_index, forwarded_count,
205		is_verified, session_data
206		FROM e2e_room_keys
207		WHERE user_id = ? AND version = ? AND (%s)
208		""" % (
209		" OR ".join(where_clauses)
210		)
211
212		txn.execute(sql, params)
213
214		ret = {}
215
216		for row in txn:
217		room_id = row[0]
218		session_id = row[1]
219		ret.setdefault(room_id, {})
220		ret[room_id][session_id] = {
221		"first_message_index": row[2],
222		"forwarded_count": row[3],
223		"is_verified": row[4],
224		"session_data": db_to_json(row[5]),
225		}
226
227		return ret
228
229		def count_e2e_room_keys(self, user_id, version):
230		"""Get the number of keys in a backup version.
231
232		Args:
233		user_id (str): the user whose backup we're querying
234		version (str): the version ID of the backup we're querying about
235		"""
236
237		return self.db.simple_select_one_onecol(
238		table="e2e_room_keys",
239		keyvalues={"user_id": user_id, "version": version},
240		retcol="COUNT(*)",
241		desc="count_e2e_room_keys",
242		)
243
244		@trace
245		@defer.inlineCallbacks
246		def delete_e2e_room_keys(self, user_id, version, room_id=None, session_id=None):
247		"""Bulk delete the E2E room keys for a given backup, optionally filtered to a given
248		room or a given session.
249
250		Args:
251		user_id(str): the user whose backup we're deleting from
252		version(str): the version ID of the backup for the set of keys we're deleting
253		room_id(str): Optional. the ID of the room whose keys we're deleting, if any.
254		If not specified, we delete the keys for all the rooms in the backup.
255		session_id(str): Optional. the session whose room_key we're querying, if any.
256		If specified, we also require the room_id to be specified.
257		If not specified, we delete all the keys in this version of
258		the backup (or for the specified room)
259
260		Returns:
261		A deferred of the deletion transaction
262		"""
263
264		keyvalues = {"user_id": user_id, "version": int(version)}
265		if room_id:
266		keyvalues["room_id"] = room_id
267		if session_id:
268		keyvalues["session_id"] = session_id
269
270		yield self.db.simple_delete(
271		table="e2e_room_keys", keyvalues=keyvalues, desc="delete_e2e_room_keys"
272		)
273
274		@staticmethod
275		def _get_current_version(txn, user_id):
276		txn.execute(
277		"SELECT MAX(version) FROM e2e_room_keys_versions "
278		"WHERE user_id=? AND deleted=0",
279		(user_id,),
280		)
281		row = txn.fetchone()
282		if not row:
283		raise StoreError(404, "No current backup version")
284		return row[0]
285
286		def get_e2e_room_keys_version_info(self, user_id, version=None):
287		"""Get info metadata about a version of our room_keys backup.
288
289		Args:
290		user_id(str): the user whose backup we're querying
291		version(str): Optional. the version ID of the backup we're querying about
292		If missing, we return the information about the current version.
293		Raises:
294		StoreError: with code 404 if there are no e2e_room_keys_versions present
295		Returns:
296		A deferred dict giving the info metadata for this backup version, with
297		fields including:
298		version(str)
299		algorithm(str)
300		auth_data(object): opaque dict supplied by the client
301		etag(int): tag of the keys in the backup
302		"""
303
304		def _get_e2e_room_keys_version_info_txn(txn):
305		if version is None:
306		this_version = self._get_current_version(txn, user_id)
307		else:
308		try:
309		this_version = int(version)
310		except ValueError:
311		# Our versions are all ints so if we can't convert it to an integer,
312		# it isn't there.
313		raise StoreError(404, "No row found")
314
315		result = self.db.simple_select_one_txn(
316		txn,
317		table="e2e_room_keys_versions",
318		keyvalues={"user_id": user_id, "version": this_version, "deleted": 0},
319		retcols=("version", "algorithm", "auth_data", "etag"),
320		)
321		result["auth_data"] = db_to_json(result["auth_data"])
322		result["version"] = str(result["version"])
323		if result["etag"] is None:
324		result["etag"] = 0
325		return result
326
327		return self.db.runInteraction(
328		"get_e2e_room_keys_version_info", _get_e2e_room_keys_version_info_txn
329		)
330
331		@trace
332		def create_e2e_room_keys_version(self, user_id, info):
333		"""Atomically creates a new version of this user's e2e_room_keys store
334		with the given version info.
335
336		Args:
337		user_id(str): the user whose backup we're creating a version
338		info(dict): the info about the backup version to be created
339
340		Returns:
341		A deferred string for the newly created version ID
342		"""
343
344		def _create_e2e_room_keys_version_txn(txn):
345		txn.execute(
346		"SELECT MAX(version) FROM e2e_room_keys_versions WHERE user_id=?",
347		(user_id,),
348		)
349		current_version = txn.fetchone()[0]
350		if current_version is None:
351		current_version = "0"
352
353		new_version = str(int(current_version) + 1)
354
355		self.db.simple_insert_txn(
356		txn,
357		table="e2e_room_keys_versions",
358		values={
359		"user_id": user_id,
360		"version": new_version,
361		"algorithm": info["algorithm"],
362		"auth_data": json.dumps(info["auth_data"]),
363		},
364		)
365
366		return new_version
367
368		return self.db.runInteraction(
369		"create_e2e_room_keys_version_txn", _create_e2e_room_keys_version_txn
370		)
371
372		@trace
373		def update_e2e_room_keys_version(
374		self, user_id, version, info=None, version_etag=None
375		):
376		"""Update a given backup version
377
378		Args:
379		user_id(str): the user whose backup version we're updating
380		version(str): the version ID of the backup version we're updating
381		info (dict): the new backup version info to store. If None, then
382		the backup version info is not updated
383		version_etag (Optional[int]): etag of the keys in the backup. If
384		None, then the etag is not updated
385		"""
386		updatevalues = {}
387
388		if info is not None and "auth_data" in info:
389		updatevalues["auth_data"] = json.dumps(info["auth_data"])
390		if version_etag is not None:
391		updatevalues["etag"] = version_etag
392
393		if updatevalues:
394		return self.db.simple_update(
395		table="e2e_room_keys_versions",
396		keyvalues={"user_id": user_id, "version": version},
397		updatevalues=updatevalues,
398		desc="update_e2e_room_keys_version",
399		)
400
401		@trace
402		def delete_e2e_room_keys_version(self, user_id, version=None):
403		"""Delete a given backup version of the user's room keys.
404		Doesn't delete their actual key data.
405
406		Args:
407		user_id(str): the user whose backup version we're deleting
408		version(str): Optional. the version ID of the backup version we're deleting
409		If missing, we delete the current backup version info.
410		Raises:
411		StoreError: with code 404 if there are no e2e_room_keys_versions present,
412		or if the version requested doesn't exist.
413		"""
414
415		def _delete_e2e_room_keys_version_txn(txn):
416		if version is None:
417		this_version = self._get_current_version(txn, user_id)
418		if this_version is None:
419		raise StoreError(404, "No current backup version")
420		else:
421		this_version = version
422
423		self.db.simple_delete_txn(
424		txn,
425		table="e2e_room_keys",
426		keyvalues={"user_id": user_id, "version": this_version},
427		)
428
429		return self.db.simple_update_one_txn(
430		txn,
431		table="e2e_room_keys_versions",
432		keyvalues={"user_id": user_id, "version": this_version},
433		updatevalues={"deleted": 1},
434		)
435
436		return self.db.runInteraction(
437		"delete_e2e_room_keys_version", _delete_e2e_room_keys_version_txn
438		)

+0

-746

~~synapse/storage/data_stores/main/end_to_end_keys.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2015, 2016 OpenMarket Ltd
2		# Copyright 2019 New Vector Ltd
3		# Copyright 2019 The Matrix.org Foundation C.I.C.
4		#
5		# Licensed under the Apache License, Version 2.0 (the "License");
6		# you may not use this file except in compliance with the License.
7		# You may obtain a copy of the License at
8		#
9		# http://www.apache.org/licenses/LICENSE-2.0
10		#
11		# Unless required by applicable law or agreed to in writing, software
12		# distributed under the License is distributed on an "AS IS" BASIS,
13		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14		# See the License for the specific language governing permissions and
15		# limitations under the License.
16		from typing import Dict, List, Tuple
17
18		from canonicaljson import encode_canonical_json, json
19
20		from twisted.enterprise.adbapi import Connection
21		from twisted.internet import defer
22
23		from synapse.logging.opentracing import log_kv, set_tag, trace
24		from synapse.storage._base import SQLBaseStore, db_to_json
25		from synapse.storage.database import make_in_list_sql_clause
26		from synapse.util.caches.descriptors import cached, cachedList
27		from synapse.util.iterutils import batch_iter
28
29
30		class EndToEndKeyWorkerStore(SQLBaseStore):
31		@trace
32		@defer.inlineCallbacks
33		def get_e2e_device_keys(
34		self, query_list, include_all_devices=False, include_deleted_devices=False
35		):
36		"""Fetch a list of device keys.
37		Args:
38		query_list(list): List of pairs of user_ids and device_ids.
39		include_all_devices (bool): whether to include entries for devices
40		that don't have device keys
41		include_deleted_devices (bool): whether to include null entries for
42		devices which no longer exist (but were in the query_list).
43		This option only takes effect if include_all_devices is true.
44		Returns:
45		Dict mapping from user-id to dict mapping from device_id to
46		key data. The key data will be a dict in the same format as the
47		DeviceKeys type returned by POST /_matrix/client/r0/keys/query.
48		"""
49		set_tag("query_list", query_list)
50		if not query_list:
51		return {}
52
53		results = yield self.db.runInteraction(
54		"get_e2e_device_keys",
55		self._get_e2e_device_keys_txn,
56		query_list,
57		include_all_devices,
58		include_deleted_devices,
59		)
60
61		# Build the result structure, un-jsonify the results, and add the
62		# "unsigned" section
63		rv = {}
64		for user_id, device_keys in results.items():
65		rv[user_id] = {}
66		for device_id, device_info in device_keys.items():
67		r = db_to_json(device_info.pop("key_json"))
68		r["unsigned"] = {}
69		display_name = device_info["device_display_name"]
70		if display_name is not None:
71		r["unsigned"]["device_display_name"] = display_name
72		if "signatures" in device_info:
73		for sig_user_id, sigs in device_info["signatures"].items():
74		r.setdefault("signatures", {}).setdefault(
75		sig_user_id, {}
76		).update(sigs)
77		rv[user_id][device_id] = r
78
79		return rv
80
81		@trace
82		def _get_e2e_device_keys_txn(
83		self, txn, query_list, include_all_devices=False, include_deleted_devices=False
84		):
85		set_tag("include_all_devices", include_all_devices)
86		set_tag("include_deleted_devices", include_deleted_devices)
87
88		query_clauses = []
89		query_params = []
90		signature_query_clauses = []
91		signature_query_params = []
92
93		if include_all_devices is False:
94		include_deleted_devices = False
95
96		if include_deleted_devices:
97		deleted_devices = set(query_list)
98
99		for (user_id, device_id) in query_list:
100		query_clause = "user_id = ?"
101		query_params.append(user_id)
102		signature_query_clause = "target_user_id = ?"
103		signature_query_params.append(user_id)
104
105		if device_id is not None:
106		query_clause += " AND device_id = ?"
107		query_params.append(device_id)
108		signature_query_clause += " AND target_device_id = ?"
109		signature_query_params.append(device_id)
110
111		signature_query_clause += " AND user_id = ?"
112		signature_query_params.append(user_id)
113
114		query_clauses.append(query_clause)
115		signature_query_clauses.append(signature_query_clause)
116
117		sql = (
118		"SELECT user_id, device_id, "
119		" d.display_name AS device_display_name, "
120		" k.key_json"
121		" FROM devices d"
122		" %s JOIN e2e_device_keys_json k USING (user_id, device_id)"
123		" WHERE %s AND NOT d.hidden"
124		) % (
125		"LEFT" if include_all_devices else "INNER",
126		" OR ".join("(" + q + ")" for q in query_clauses),
127		)
128
129		txn.execute(sql, query_params)
130		rows = self.db.cursor_to_dict(txn)
131
132		result = {}
133		for row in rows:
134		if include_deleted_devices:
135		deleted_devices.remove((row["user_id"], row["device_id"]))
136		result.setdefault(row["user_id"], {})[row["device_id"]] = row
137
138		if include_deleted_devices:
139		for user_id, device_id in deleted_devices:
140		result.setdefault(user_id, {})[device_id] = None
141
142		# get signatures on the device
143		signature_sql = ("SELECT * FROM e2e_cross_signing_signatures WHERE %s") % (
144		" OR ".join("(" + q + ")" for q in signature_query_clauses)
145		)
146
147		txn.execute(signature_sql, signature_query_params)
148		rows = self.db.cursor_to_dict(txn)
149
150		# add each cross-signing signature to the correct device in the result dict.
151		for row in rows:
152		signing_user_id = row["user_id"]
153		signing_key_id = row["key_id"]
154		target_user_id = row["target_user_id"]
155		target_device_id = row["target_device_id"]
156		signature = row["signature"]
157
158		target_user_result = result.get(target_user_id)
159		if not target_user_result:
160		continue
161
162		target_device_result = target_user_result.get(target_device_id)
163		if not target_device_result:
164		# note that target_device_result will be None for deleted devices.
165		continue
166
167		target_device_signatures = target_device_result.setdefault("signatures", {})
168		signing_user_signatures = target_device_signatures.setdefault(
169		signing_user_id, {}
170		)
171		signing_user_signatures[signing_key_id] = signature
172
173		log_kv(result)
174		return result
175
176		@defer.inlineCallbacks
177		def get_e2e_one_time_keys(self, user_id, device_id, key_ids):
178		"""Retrieve a number of one-time keys for a user
179
180		Args:
181		user_id(str): id of user to get keys for
182		device_id(str): id of device to get keys for
183		key_ids(list[str]): list of key ids (excluding algorithm) to
184		retrieve
185
186		Returns:
187		deferred resolving to Dict[(str, str), str]: map from (algorithm,
188		key_id) to json string for key
189		"""
190
191		rows = yield self.db.simple_select_many_batch(
192		table="e2e_one_time_keys_json",
193		column="key_id",
194		iterable=key_ids,
195		retcols=("algorithm", "key_id", "key_json"),
196		keyvalues={"user_id": user_id, "device_id": device_id},
197		desc="add_e2e_one_time_keys_check",
198		)
199		result = {(row["algorithm"], row["key_id"]): row["key_json"] for row in rows}
200		log_kv({"message": "Fetched one time keys for user", "one_time_keys": result})
201		return result
202
203		@defer.inlineCallbacks
204		def add_e2e_one_time_keys(self, user_id, device_id, time_now, new_keys):
205		"""Insert some new one time keys for a device. Errors if any of the
206		keys already exist.
207
208		Args:
209		user_id(str): id of user to get keys for
210		device_id(str): id of device to get keys for
211		time_now(long): insertion time to record (ms since epoch)
212		new_keys(iterable[(str, str, str)]: keys to add - each a tuple of
213		(algorithm, key_id, key json)
214		"""
215
216		def _add_e2e_one_time_keys(txn):
217		set_tag("user_id", user_id)
218		set_tag("device_id", device_id)
219		set_tag("new_keys", new_keys)
220		# We are protected from race between lookup and insertion due to
221		# a unique constraint. If there is a race of two calls to
222		# `add_e2e_one_time_keys` then they'll conflict and we will only
223		# insert one set.
224		self.db.simple_insert_many_txn(
225		txn,
226		table="e2e_one_time_keys_json",
227		values=[
228		{
229		"user_id": user_id,
230		"device_id": device_id,
231		"algorithm": algorithm,
232		"key_id": key_id,
233		"ts_added_ms": time_now,
234		"key_json": json_bytes,
235		}
236		for algorithm, key_id, json_bytes in new_keys
237		],
238		)
239		self._invalidate_cache_and_stream(
240		txn, self.count_e2e_one_time_keys, (user_id, device_id)
241		)
242
243		yield self.db.runInteraction(
244		"add_e2e_one_time_keys_insert", _add_e2e_one_time_keys
245		)
246
247		@cached(max_entries=10000)
248		def count_e2e_one_time_keys(self, user_id, device_id):
249		""" Count the number of one time keys the server has for a device
250		Returns:
251		Dict mapping from algorithm to number of keys for that algorithm.
252		"""
253
254		def _count_e2e_one_time_keys(txn):
255		sql = (
256		"SELECT algorithm, COUNT(key_id) FROM e2e_one_time_keys_json"
257		" WHERE user_id = ? AND device_id = ?"
258		" GROUP BY algorithm"
259		)
260		txn.execute(sql, (user_id, device_id))
261		result = {}
262		for algorithm, key_count in txn:
263		result[algorithm] = key_count
264		return result
265
266		return self.db.runInteraction(
267		"count_e2e_one_time_keys", _count_e2e_one_time_keys
268		)
269
270		@defer.inlineCallbacks
271		def get_e2e_cross_signing_key(self, user_id, key_type, from_user_id=None):
272		"""Returns a user's cross-signing key.
273
274		Args:
275		user_id (str): the user whose key is being requested
276		key_type (str): the type of key that is being requested: either 'master'
277		for a master key, 'self_signing' for a self-signing key, or
278		'user_signing' for a user-signing key
279		from_user_id (str): if specified, signatures made by this user on
280		the self-signing key will be included in the result
281
282		Returns:
283		dict of the key data or None if not found
284		"""
285		res = yield self.get_e2e_cross_signing_keys_bulk([user_id], from_user_id)
286		user_keys = res.get(user_id)
287		if not user_keys:
288		return None
289		return user_keys.get(key_type)
290
291		@cached(num_args=1)
292		def _get_bare_e2e_cross_signing_keys(self, user_id):
293		"""Dummy function. Only used to make a cache for
294		_get_bare_e2e_cross_signing_keys_bulk.
295		"""
296		raise NotImplementedError()
297
298		@cachedList(
299		cached_method_name="_get_bare_e2e_cross_signing_keys",
300		list_name="user_ids",
301		num_args=1,
302		)
303		def _get_bare_e2e_cross_signing_keys_bulk(
304		self, user_ids: List[str]
305		) -> Dict[str, Dict[str, dict]]:
306		"""Returns the cross-signing keys for a set of users. The output of this
307		function should be passed to _get_e2e_cross_signing_signatures_txn if
308		the signatures for the calling user need to be fetched.
309
310		Args:
311		user_ids (list[str]): the users whose keys are being requested
312
313		Returns:
314		dict[str, dict[str, dict]]: mapping from user ID to key type to key
315		data. If a user's cross-signing keys were not found, either
316		their user ID will not be in the dict, or their user ID will map
317		to None.
318
319		"""
320		return self.db.runInteraction(
321		"get_bare_e2e_cross_signing_keys_bulk",
322		self._get_bare_e2e_cross_signing_keys_bulk_txn,
323		user_ids,
324		)
325
326		def _get_bare_e2e_cross_signing_keys_bulk_txn(
327		self, txn: Connection, user_ids: List[str],
328		) -> Dict[str, Dict[str, dict]]:
329		"""Returns the cross-signing keys for a set of users. The output of this
330		function should be passed to _get_e2e_cross_signing_signatures_txn if
331		the signatures for the calling user need to be fetched.
332
333		Args:
334		txn (twisted.enterprise.adbapi.Connection): db connection
335		user_ids (list[str]): the users whose keys are being requested
336
337		Returns:
338		dict[str, dict[str, dict]]: mapping from user ID to key type to key
339		data. If a user's cross-signing keys were not found, their user
340		ID will not be in the dict.
341
342		"""
343		result = {}
344
345		for user_chunk in batch_iter(user_ids, 100):
346		clause, params = make_in_list_sql_clause(
347		txn.database_engine, "k.user_id", user_chunk
348		)
349		sql = (
350		"""
351		SELECT k.user_id, k.keytype, k.keydata, k.stream_id
352		FROM e2e_cross_signing_keys k
353		INNER JOIN (SELECT user_id, keytype, MAX(stream_id) AS stream_id
354		FROM e2e_cross_signing_keys
355		GROUP BY user_id, keytype) s
356		USING (user_id, stream_id, keytype)
357		WHERE
358		"""
359		+ clause
360		)
361
362		txn.execute(sql, params)
363		rows = self.db.cursor_to_dict(txn)
364
365		for row in rows:
366		user_id = row["user_id"]
367		key_type = row["keytype"]
368		key = db_to_json(row["keydata"])
369		user_info = result.setdefault(user_id, {})
370		user_info[key_type] = key
371
372		return result
373
374		def _get_e2e_cross_signing_signatures_txn(
375		self, txn: Connection, keys: Dict[str, Dict[str, dict]], from_user_id: str,
376		) -> Dict[str, Dict[str, dict]]:
377		"""Returns the cross-signing signatures made by a user on a set of keys.
378
379		Args:
380		txn (twisted.enterprise.adbapi.Connection): db connection
381		keys (dict[str, dict[str, dict]]): a map of user ID to key type to
382		key data. This dict will be modified to add signatures.
383		from_user_id (str): fetch the signatures made by this user
384
385		Returns:
386		dict[str, dict[str, dict]]: mapping from user ID to key type to key
387		data. The return value will be the same as the keys argument,
388		with the modifications included.
389		"""
390
391		# find out what cross-signing keys (a.k.a. devices) we need to get
392		# signatures for. This is a map of (user_id, device_id) to key type
393		# (device_id is the key's public part).
394		devices = {}
395
396		for user_id, user_info in keys.items():
397		if user_info is None:
398		continue
399		for key_type, key in user_info.items():
400		device_id = None
401		for k in key["keys"].values():
402		device_id = k
403		devices[(user_id, device_id)] = key_type
404
405		for batch in batch_iter(devices.keys(), size=100):
406		sql = """
407		SELECT target_user_id, target_device_id, key_id, signature
408		FROM e2e_cross_signing_signatures
409		WHERE user_id = ?
410		AND (%s)
411		""" % (
412		" OR ".join(
413		"(target_user_id = ? AND target_device_id = ?)" for _ in batch
414		)
415		)
416		query_params = [from_user_id]
417		for item in batch:
418		# item is a (user_id, device_id) tuple
419		query_params.extend(item)
420
421		txn.execute(sql, query_params)
422		rows = self.db.cursor_to_dict(txn)
423
424		# and add the signatures to the appropriate keys
425		for row in rows:
426		key_id = row["key_id"]
427		target_user_id = row["target_user_id"]
428		target_device_id = row["target_device_id"]
429		key_type = devices[(target_user_id, target_device_id)]
430		# We need to copy everything, because the result may have come
431		# from the cache. dict.copy only does a shallow copy, so we
432		# need to recursively copy the dicts that will be modified.
433		user_info = keys[target_user_id] = keys[target_user_id].copy()
434		target_user_key = user_info[key_type] = user_info[key_type].copy()
435		if "signatures" in target_user_key:
436		signatures = target_user_key["signatures"] = target_user_key[
437		"signatures"
438		].copy()
439		if from_user_id in signatures:
440		user_sigs = signatures[from_user_id] = signatures[from_user_id]
441		user_sigs[key_id] = row["signature"]
442		else:
443		signatures[from_user_id] = {key_id: row["signature"]}
444		else:
445		target_user_key["signatures"] = {
446		from_user_id: {key_id: row["signature"]}
447		}
448
449		return keys
450
451		@defer.inlineCallbacks
452		def get_e2e_cross_signing_keys_bulk(
453		self, user_ids: List[str], from_user_id: str = None
454		) -> defer.Deferred:
455		"""Returns the cross-signing keys for a set of users.
456
457		Args:
458		user_ids (list[str]): the users whose keys are being requested
459		from_user_id (str): if specified, signatures made by this user on
460		the self-signing keys will be included in the result
461
462		Returns:
463		Deferred[dict[str, dict[str, dict]]]: map of user ID to key type to
464		key data. If a user's cross-signing keys were not found, either
465		their user ID will not be in the dict, or their user ID will map
466		to None.
467		"""
468
469		result = yield self._get_bare_e2e_cross_signing_keys_bulk(user_ids)
470
471		if from_user_id:
472		result = yield self.db.runInteraction(
473		"get_e2e_cross_signing_signatures",
474		self._get_e2e_cross_signing_signatures_txn,
475		result,
476		from_user_id,
477		)
478
479		return result
480
481		async def get_all_user_signature_changes_for_remotes(
482		self, instance_name: str, last_id: int, current_id: int, limit: int
483		) -> Tuple[List[Tuple[int, tuple]], int, bool]:
484		"""Get updates for groups replication stream.
485
486		Note that the user signature stream represents when a user signs their
487		device with their user-signing key, which is not published to other
488		users or servers, so no `destination` is needed in the returned
489		list. However, this is needed to poke workers.
490
491		Args:
492		instance_name: The writer we want to fetch updates from. Unused
493		here since there is only ever one writer.
494		last_id: The token to fetch updates from. Exclusive.
495		current_id: The token to fetch updates up to. Inclusive.
496		limit: The requested limit for the number of rows to return. The
497		function may return more or fewer rows.
498
499		Returns:
500		A tuple consisting of: the updates, a token to use to fetch
501		subsequent updates, and whether we returned fewer rows than exists
502		between the requested tokens due to the limit.
503
504		The token returned can be used in a subsequent call to this
505		function to get further updatees.
506
507		The updates are a list of 2-tuples of stream ID and the row data
508		"""
509
510		if last_id == current_id:
511		return [], current_id, False
512
513		def _get_all_user_signature_changes_for_remotes_txn(txn):
514		sql = """
515		SELECT stream_id, from_user_id AS user_id
516		FROM user_signature_stream
517		WHERE ? < stream_id AND stream_id <= ?
518		ORDER BY stream_id ASC
519		LIMIT ?
520		"""
521		txn.execute(sql, (last_id, current_id, limit))
522
523		updates = [(row[0], (row[1:])) for row in txn]
524
525		limited = False
526		upto_token = current_id
527		if len(updates) >= limit:
528		upto_token = updates[-1][0]
529		limited = True
530
531		return updates, upto_token, limited
532
533		return await self.db.runInteraction(
534		"get_all_user_signature_changes_for_remotes",
535		_get_all_user_signature_changes_for_remotes_txn,
536		)
537
538
539		class EndToEndKeyStore(EndToEndKeyWorkerStore, SQLBaseStore):
540		def set_e2e_device_keys(self, user_id, device_id, time_now, device_keys):
541		"""Stores device keys for a device. Returns whether there was a change
542		or the keys were already in the database.
543		"""
544
545		def _set_e2e_device_keys_txn(txn):
546		set_tag("user_id", user_id)
547		set_tag("device_id", device_id)
548		set_tag("time_now", time_now)
549		set_tag("device_keys", device_keys)
550
551		old_key_json = self.db.simple_select_one_onecol_txn(
552		txn,
553		table="e2e_device_keys_json",
554		keyvalues={"user_id": user_id, "device_id": device_id},
555		retcol="key_json",
556		allow_none=True,
557		)
558
559		# In py3 we need old_key_json to match new_key_json type. The DB
560		# returns unicode while encode_canonical_json returns bytes.
561		new_key_json = encode_canonical_json(device_keys).decode("utf-8")
562
563		if old_key_json == new_key_json:
564		log_kv({"Message": "Device key already stored."})
565		return False
566
567		self.db.simple_upsert_txn(
568		txn,
569		table="e2e_device_keys_json",
570		keyvalues={"user_id": user_id, "device_id": device_id},
571		values={"ts_added_ms": time_now, "key_json": new_key_json},
572		)
573		log_kv({"message": "Device keys stored."})
574		return True
575
576		return self.db.runInteraction("set_e2e_device_keys", _set_e2e_device_keys_txn)
577
578		def claim_e2e_one_time_keys(self, query_list):
579		"""Take a list of one time keys out of the database"""
580
581		@trace
582		def _claim_e2e_one_time_keys(txn):
583		sql = (
584		"SELECT key_id, key_json FROM e2e_one_time_keys_json"
585		" WHERE user_id = ? AND device_id = ? AND algorithm = ?"
586		" LIMIT 1"
587		)
588		result = {}
589		delete = []
590		for user_id, device_id, algorithm in query_list:
591		user_result = result.setdefault(user_id, {})
592		device_result = user_result.setdefault(device_id, {})
593		txn.execute(sql, (user_id, device_id, algorithm))
594		for key_id, key_json in txn:
595		device_result[algorithm + ":" + key_id] = key_json
596		delete.append((user_id, device_id, algorithm, key_id))
597		sql = (
598		"DELETE FROM e2e_one_time_keys_json"
599		" WHERE user_id = ? AND device_id = ? AND algorithm = ?"
600		" AND key_id = ?"
601		)
602		for user_id, device_id, algorithm, key_id in delete:
603		log_kv(
604		{
605		"message": "Executing claim e2e_one_time_keys transaction on database."
606		}
607		)
608		txn.execute(sql, (user_id, device_id, algorithm, key_id))
609		log_kv({"message": "finished executing and invalidating cache"})
610		self._invalidate_cache_and_stream(
611		txn, self.count_e2e_one_time_keys, (user_id, device_id)
612		)
613		return result
614
615		return self.db.runInteraction(
616		"claim_e2e_one_time_keys", _claim_e2e_one_time_keys
617		)
618
619		def delete_e2e_keys_by_device(self, user_id, device_id):
620		def delete_e2e_keys_by_device_txn(txn):
621		log_kv(
622		{
623		"message": "Deleting keys for device",
624		"device_id": device_id,
625		"user_id": user_id,
626		}
627		)
628		self.db.simple_delete_txn(
629		txn,
630		table="e2e_device_keys_json",
631		keyvalues={"user_id": user_id, "device_id": device_id},
632		)
633		self.db.simple_delete_txn(
634		txn,
635		table="e2e_one_time_keys_json",
636		keyvalues={"user_id": user_id, "device_id": device_id},
637		)
638		self._invalidate_cache_and_stream(
639		txn, self.count_e2e_one_time_keys, (user_id, device_id)
640		)
641
642		return self.db.runInteraction(
643		"delete_e2e_keys_by_device", delete_e2e_keys_by_device_txn
644		)
645
646		def _set_e2e_cross_signing_key_txn(self, txn, user_id, key_type, key):
647		"""Set a user's cross-signing key.
648
649		Args:
650		txn (twisted.enterprise.adbapi.Connection): db connection
651		user_id (str): the user to set the signing key for
652		key_type (str): the type of key that is being set: either 'master'
653		for a master key, 'self_signing' for a self-signing key, or
654		'user_signing' for a user-signing key
655		key (dict): the key data
656		"""
657		# the 'key' dict will look something like:
658		# {
659		# "user_id": "@alice:example.com",
660		# "usage": ["self_signing"],
661		# "keys": {
662		# "ed25519:base64+self+signing+public+key": "base64+self+signing+public+key",
663		# },
664		# "signatures": {
665		# "@alice:example.com": {
666		# "ed25519:base64+master+public+key": "base64+signature"
667		# }
668		# }
669		# }
670		# The "keys" property must only have one entry, which will be the public
671		# key, so we just grab the first value in there
672		pubkey = next(iter(key["keys"].values()))
673
674		# The cross-signing keys need to occupy the same namespace as devices,
675		# since signatures are identified by device ID. So add an entry to the
676		# device table to make sure that we don't have a collision with device
677		# IDs.
678		# We only need to do this for local users, since remote servers should be
679		# responsible for checking this for their own users.
680		if self.hs.is_mine_id(user_id):
681		self.db.simple_insert_txn(
682		txn,
683		"devices",
684		values={
685		"user_id": user_id,
686		"device_id": pubkey,
687		"display_name": key_type + " signing key",
688		"hidden": True,
689		},
690		)
691
692		# and finally, store the key itself
693		with self._cross_signing_id_gen.get_next() as stream_id:
694		self.db.simple_insert_txn(
695		txn,
696		"e2e_cross_signing_keys",
697		values={
698		"user_id": user_id,
699		"keytype": key_type,
700		"keydata": json.dumps(key),
701		"stream_id": stream_id,
702		},
703		)
704
705		self._invalidate_cache_and_stream(
706		txn, self._get_bare_e2e_cross_signing_keys, (user_id,)
707		)
708
709		def set_e2e_cross_signing_key(self, user_id, key_type, key):
710		"""Set a user's cross-signing key.
711
712		Args:
713		user_id (str): the user to set the user-signing key for
714		key_type (str): the type of cross-signing key to set
715		key (dict): the key data
716		"""
717		return self.db.runInteraction(
718		"add_e2e_cross_signing_key",
719		self._set_e2e_cross_signing_key_txn,
720		user_id,
721		key_type,
722		key,
723		)
724
725		def store_e2e_cross_signing_signatures(self, user_id, signatures):
726		"""Stores cross-signing signatures.
727
728		Args:
729		user_id (str): the user who made the signatures
730		signatures (iterable[SignatureListItem]): signatures to add
731		"""
732		return self.db.simple_insert_many(
733		"e2e_cross_signing_signatures",
734		[
735		{
736		"user_id": user_id,
737		"key_id": item.signing_key_id,
738		"target_user_id": item.target_user_id,
739		"target_device_id": item.target_device_id,
740		"signature": item.signature,
741		}
742		for item in signatures
743		],
744		"add_e2e_signing_key",
745		)

+0

-724

~~synapse/storage/data_stores/main/event_federation.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14		import itertools
15		import logging
16		from queue import Empty, PriorityQueue
17		from typing import Dict, List, Optional, Set, Tuple
18
19		from twisted.internet import defer
20
21		from synapse.api.errors import StoreError
22		from synapse.metrics.background_process_metrics import run_as_background_process
23		from synapse.storage._base import SQLBaseStore, make_in_list_sql_clause
24		from synapse.storage.data_stores.main.events_worker import EventsWorkerStore
25		from synapse.storage.data_stores.main.signatures import SignatureWorkerStore
26		from synapse.storage.database import Database
27		from synapse.util.caches.descriptors import cached
28		from synapse.util.iterutils import batch_iter
29
30		logger = logging.getLogger(__name__)
31
32
33		class EventFederationWorkerStore(EventsWorkerStore, SignatureWorkerStore, SQLBaseStore):
34		def get_auth_chain(self, event_ids, include_given=False):
35		"""Get auth events for given event_ids. The events must be state events.
36
37		Args:
38		event_ids (list): state events
39		include_given (bool): include the given events in result
40
41		Returns:
42		list of events
43		"""
44		return self.get_auth_chain_ids(
45		event_ids, include_given=include_given
46		).addCallback(self.get_events_as_list)
47
48		def get_auth_chain_ids(
49		self,
50		event_ids: List[str],
51		include_given: bool = False,
52		ignore_events: Optional[Set[str]] = None,
53		):
54		"""Get auth events for given event_ids. The events must be state events.
55
56		Args:
57		event_ids: state events
58		include_given: include the given events in result
59		ignore_events: Set of events to exclude from the returned auth
60		chain. This is useful if the caller will just discard the
61		given events anyway, and saves us from figuring out their auth
62		chains if not required.
63
64		Returns:
65		list of event_ids
66		"""
67		return self.db.runInteraction(
68		"get_auth_chain_ids",
69		self._get_auth_chain_ids_txn,
70		event_ids,
71		include_given,
72		ignore_events,
73		)
74
75		def _get_auth_chain_ids_txn(self, txn, event_ids, include_given, ignore_events):
76		if ignore_events is None:
77		ignore_events = set()
78
79		if include_given:
80		results = set(event_ids)
81		else:
82		results = set()
83
84		base_sql = "SELECT auth_id FROM event_auth WHERE "
85
86		front = set(event_ids)
87		while front:
88		new_front = set()
89		for chunk in batch_iter(front, 100):
90		clause, args = make_in_list_sql_clause(
91		txn.database_engine, "event_id", chunk
92		)
93		txn.execute(base_sql + clause, args)
94		new_front.update(r[0] for r in txn)
95
96		new_front -= ignore_events
97		new_front -= results
98
99		front = new_front
100		results.update(front)
101
102		return list(results)
103
104		def get_auth_chain_difference(self, state_sets: List[Set[str]]):
105		"""Given sets of state events figure out the auth chain difference (as
106		per state res v2 algorithm).
107
108		This equivalent to fetching the full auth chain for each set of state
109		and returning the events that don't appear in each and every auth
110		chain.
111
112		Returns:
113		Deferred[Set[str]]
114		"""
115
116		return self.db.runInteraction(
117		"get_auth_chain_difference",
118		self._get_auth_chain_difference_txn,
119		state_sets,
120		)
121
122		def _get_auth_chain_difference_txn(
123		self, txn, state_sets: List[Set[str]]
124		) -> Set[str]:
125
126		# Algorithm Description
127		# ~~~~~~~~~~~~~~~~~~~~~
128		#
129		# The idea here is to basically walk the auth graph of each state set in
130		# tandem, keeping track of which auth events are reachable by each state
131		# set. If we reach an auth event we've already visited (via a different
132		# state set) then we mark that auth event and all ancestors as reachable
133		# by the state set. This requires that we keep track of the auth chains
134		# in memory.
135		#
136		# Doing it in a such a way means that we can stop early if all auth
137		# events we're currently walking are reachable by all state sets.
138		#
139		# Note: We can't stop walking an event's auth chain if it is reachable
140		# by all state sets. This is because other auth chains we're walking
141		# might be reachable only via the original auth chain. For example,
142		# given the following auth chain:
143		#
144		# A -> C -> D -> E
145		# / /
146		# B -´---------´
147		#
148		# and state sets {A} and {B} then walking the auth chains of A and B
149		# would immediately show that C is reachable by both. However, if we
150		# stopped at C then we'd only reach E via the auth chain of B and so E
151		# would errornously get included in the returned difference.
152		#
153		# The other thing that we do is limit the number of auth chains we walk
154		# at once, due to practical limits (i.e. we can only query the database
155		# with a limited set of parameters). We pick the auth chains we walk
156		# each iteration based on their depth, in the hope that events with a
157		# lower depth are likely reachable by those with higher depths.
158		#
159		# We could use any ordering that we believe would give a rough
160		# topological ordering, e.g. origin server timestamp. If the ordering
161		# chosen is not topological then the algorithm still produces the right
162		# result, but perhaps a bit more inefficiently. This is why it is safe
163		# to use "depth" here.
164
165		initial_events = set(state_sets[0]).union(*state_sets[1:])
166
167		# Dict from events in auth chains to which sets cannot reach them.
168		# I.e. if the set is empty then all sets can reach the event.
169		event_to_missing_sets = {
170		event_id: {i for i, a in enumerate(state_sets) if event_id not in a}
171		for event_id in initial_events
172		}
173
174		# The sorted list of events whose auth chains we should walk.
175		search = [] # type: List[Tuple[int, str]]
176
177		# We need to get the depth of the initial events for sorting purposes.
178		sql = """
179		SELECT depth, event_id FROM events
180		WHERE %s
181		"""
182		# the list can be huge, so let's avoid looking them all up in one massive
183		# query.
184		for batch in batch_iter(initial_events, 1000):
185		clause, args = make_in_list_sql_clause(
186		txn.database_engine, "event_id", batch
187		)
188		txn.execute(sql % (clause,), args)
189
190		# I think building a temporary list with fetchall is more efficient than
191		# just `search.extend(txn)`, but this is unconfirmed
192		search.extend(txn.fetchall())
193
194		# sort by depth
195		search.sort()
196
197		# Map from event to its auth events
198		event_to_auth_events = {} # type: Dict[str, Set[str]]
199
200		base_sql = """
201		SELECT a.event_id, auth_id, depth
202		FROM event_auth AS a
203		INNER JOIN events AS e ON (e.event_id = a.auth_id)
204		WHERE
205		"""
206
207		while search:
208		# Check whether all our current walks are reachable by all state
209		# sets. If so we can bail.
210		if all(not event_to_missing_sets[eid] for _, eid in search):
211		break
212
213		# Fetch the auth events and their depths of the N last events we're
214		# currently walking
215		search, chunk = search[:-100], search[-100:]
216		clause, args = make_in_list_sql_clause(
217		txn.database_engine, "a.event_id", [e_id for _, e_id in chunk]
218		)
219		txn.execute(base_sql + clause, args)
220
221		for event_id, auth_event_id, auth_event_depth in txn:
222		event_to_auth_events.setdefault(event_id, set()).add(auth_event_id)
223
224		sets = event_to_missing_sets.get(auth_event_id)
225		if sets is None:
226		# First time we're seeing this event, so we add it to the
227		# queue of things to fetch.
228		search.append((auth_event_depth, auth_event_id))
229
230		# Assume that this event is unreachable from any of the
231		# state sets until proven otherwise
232		sets = event_to_missing_sets[auth_event_id] = set(
233		range(len(state_sets))
234		)
235		else:
236		# We've previously seen this event, so look up its auth
237		# events and recursively mark all ancestors as reachable
238		# by the current event's state set.
239		a_ids = event_to_auth_events.get(auth_event_id)
240		while a_ids:
241		new_aids = set()
242		for a_id in a_ids:
243		event_to_missing_sets[a_id].intersection_update(
244		event_to_missing_sets[event_id]
245		)
246
247		b = event_to_auth_events.get(a_id)
248		if b:
249		new_aids.update(b)
250
251		a_ids = new_aids
252
253		# Mark that the auth event is reachable by the approriate sets.
254		sets.intersection_update(event_to_missing_sets[event_id])
255
256		search.sort()
257
258		# Return all events where not all sets can reach them.
259		return {eid for eid, n in event_to_missing_sets.items() if n}
260
261		def get_oldest_events_in_room(self, room_id):
262		return self.db.runInteraction(
263		"get_oldest_events_in_room", self._get_oldest_events_in_room_txn, room_id
264		)
265
266		def get_oldest_events_with_depth_in_room(self, room_id):
267		return self.db.runInteraction(
268		"get_oldest_events_with_depth_in_room",
269		self.get_oldest_events_with_depth_in_room_txn,
270		room_id,
271		)
272
273		def get_oldest_events_with_depth_in_room_txn(self, txn, room_id):
274		sql = (
275		"SELECT b.event_id, MAX(e.depth) FROM events as e"
276		" INNER JOIN event_edges as g"
277		" ON g.event_id = e.event_id"
278		" INNER JOIN event_backward_extremities as b"
279		" ON g.prev_event_id = b.event_id"
280		" WHERE b.room_id = ? AND g.is_state is ?"
281		" GROUP BY b.event_id"
282		)
283
284		txn.execute(sql, (room_id, False))
285
286		return dict(txn)
287
288		@defer.inlineCallbacks
289		def get_max_depth_of(self, event_ids):
290		"""Returns the max depth of a set of event IDs
291
292		Args:
293		event_ids (list[str])
294
295		Returns
296		Deferred[int]
297		"""
298		rows = yield self.db.simple_select_many_batch(
299		table="events",
300		column="event_id",
301		iterable=event_ids,
302		retcols=("depth",),
303		desc="get_max_depth_of",
304		)
305
306		if not rows:
307		return 0
308		else:
309		return max(row["depth"] for row in rows)
310
311		def _get_oldest_events_in_room_txn(self, txn, room_id):
312		return self.db.simple_select_onecol_txn(
313		txn,
314		table="event_backward_extremities",
315		keyvalues={"room_id": room_id},
316		retcol="event_id",
317		)
318
319		def get_prev_events_for_room(self, room_id: str):
320		"""
321		Gets a subset of the current forward extremities in the given room.
322
323		Limits the result to 10 extremities, so that we can avoid creating
324		events which refer to hundreds of prev_events.
325
326		Args:
327		room_id (str): room_id
328
329		Returns:
330		Deferred[List[str]]: the event ids of the forward extremites
331
332		"""
333
334		return self.db.runInteraction(
335		"get_prev_events_for_room", self._get_prev_events_for_room_txn, room_id
336		)
337
338		def _get_prev_events_for_room_txn(self, txn, room_id: str):
339		# we just use the 10 newest events. Older events will become
340		# prev_events of future events.
341
342		sql = """
343		SELECT e.event_id FROM event_forward_extremities AS f
344		INNER JOIN events AS e USING (event_id)
345		WHERE f.room_id = ?
346		ORDER BY e.depth DESC
347		LIMIT 10
348		"""
349
350		txn.execute(sql, (room_id,))
351
352		return [row[0] for row in txn]
353
354		def get_rooms_with_many_extremities(self, min_count, limit, room_id_filter):
355		"""Get the top rooms with at least N extremities.
356
357		Args:
358		min_count (int): The minimum number of extremities
359		limit (int): The maximum number of rooms to return.
360		room_id_filter (iterable[str]): room_ids to exclude from the results
361
362		Returns:
363		Deferred[list]: At most `limit` room IDs that have at least
364		`min_count` extremities, sorted by extremity count.
365		"""
366
367		def _get_rooms_with_many_extremities_txn(txn):
368		where_clause = "1=1"
369		if room_id_filter:
370		where_clause = "room_id NOT IN (%s)" % (
371		",".join("?" for _ in room_id_filter),
372		)
373
374		sql = """
375		SELECT room_id FROM event_forward_extremities
376		WHERE %s
377		GROUP BY room_id
378		HAVING count(*) > ?
379		ORDER BY count(*) DESC
380		LIMIT ?
381		""" % (
382		where_clause,
383		)
384
385		query_args = list(itertools.chain(room_id_filter, [min_count, limit]))
386		txn.execute(sql, query_args)
387		return [room_id for room_id, in txn]
388
389		return self.db.runInteraction(
390		"get_rooms_with_many_extremities", _get_rooms_with_many_extremities_txn
391		)
392
393		@cached(max_entries=5000, iterable=True)
394		def get_latest_event_ids_in_room(self, room_id):
395		return self.db.simple_select_onecol(
396		table="event_forward_extremities",
397		keyvalues={"room_id": room_id},
398		retcol="event_id",
399		desc="get_latest_event_ids_in_room",
400		)
401
402		def get_min_depth(self, room_id):
403		""" For hte given room, get the minimum depth we have seen for it.
404		"""
405		return self.db.runInteraction(
406		"get_min_depth", self._get_min_depth_interaction, room_id
407		)
408
409		def _get_min_depth_interaction(self, txn, room_id):
410		min_depth = self.db.simple_select_one_onecol_txn(
411		txn,
412		table="room_depth",
413		keyvalues={"room_id": room_id},
414		retcol="min_depth",
415		allow_none=True,
416		)
417
418		return int(min_depth) if min_depth is not None else None
419
420		def get_forward_extremeties_for_room(self, room_id, stream_ordering):
421		"""For a given room_id and stream_ordering, return the forward
422		extremeties of the room at that point in "time".
423
424		Throws a StoreError if we have since purged the index for
425		stream_orderings from that point.
426
427		Args:
428		room_id (str):
429		stream_ordering (int):
430
431		Returns:
432		deferred, which resolves to a list of event_ids
433		"""
434		# We want to make the cache more effective, so we clamp to the last
435		# change before the given ordering.
436		last_change = self._events_stream_cache.get_max_pos_of_last_change(room_id)
437
438		# We don't always have a full stream_to_exterm_id table, e.g. after
439		# the upgrade that introduced it, so we make sure we never ask for a
440		# stream_ordering from before a restart
441		last_change = max(self._stream_order_on_start, last_change)
442
443		# provided the last_change is recent enough, we now clamp the requested
444		# stream_ordering to it.
445		if last_change > self.stream_ordering_month_ago:
446		stream_ordering = min(last_change, stream_ordering)
447
448		return self._get_forward_extremeties_for_room(room_id, stream_ordering)
449
450		@cached(max_entries=5000, num_args=2)
451		def _get_forward_extremeties_for_room(self, room_id, stream_ordering):
452		"""For a given room_id and stream_ordering, return the forward
453		extremeties of the room at that point in "time".
454
455		Throws a StoreError if we have since purged the index for
456		stream_orderings from that point.
457		"""
458
459		if stream_ordering <= self.stream_ordering_month_ago:
460		raise StoreError(400, "stream_ordering too old")
461
462		sql = """
463		SELECT event_id FROM stream_ordering_to_exterm
464		INNER JOIN (
465		SELECT room_id, MAX(stream_ordering) AS stream_ordering
466		FROM stream_ordering_to_exterm
467		WHERE stream_ordering <= ? GROUP BY room_id
468		) AS rms USING (room_id, stream_ordering)
469		WHERE room_id = ?
470		"""
471
472		def get_forward_extremeties_for_room_txn(txn):
473		txn.execute(sql, (stream_ordering, room_id))
474		return [event_id for event_id, in txn]
475
476		return self.db.runInteraction(
477		"get_forward_extremeties_for_room", get_forward_extremeties_for_room_txn
478		)
479
480		def get_backfill_events(self, room_id, event_list, limit):
481		"""Get a list of Events for a given topic that occurred before (and
482		including) the events in event_list. Return a list of max size `limit`
483
484		Args:
485		txn
486		room_id (str)
487		event_list (list)
488		limit (int)
489		"""
490		return (
491		self.db.runInteraction(
492		"get_backfill_events",
493		self._get_backfill_events,
494		room_id,
495		event_list,
496		limit,
497		)
498		.addCallback(self.get_events_as_list)
499		.addCallback(lambda l: sorted(l, key=lambda e: -e.depth))
500		)
501
502		def _get_backfill_events(self, txn, room_id, event_list, limit):
503		logger.debug("_get_backfill_events: %s, %r, %s", room_id, event_list, limit)
504
505		event_results = set()
506
507		# We want to make sure that we do a breadth-first, "depth" ordered
508		# search.
509
510		query = (
511		"SELECT depth, prev_event_id FROM event_edges"
512		" INNER JOIN events"
513		" ON prev_event_id = events.event_id"
514		" WHERE event_edges.event_id = ?"
515		" AND event_edges.is_state = ?"
516		" LIMIT ?"
517		)
518
519		queue = PriorityQueue()
520
521		for event_id in event_list:
522		depth = self.db.simple_select_one_onecol_txn(
523		txn,
524		table="events",
525		keyvalues={"event_id": event_id, "room_id": room_id},
526		retcol="depth",
527		allow_none=True,
528		)
529
530		if depth:
531		queue.put((-depth, event_id))
532
533		while not queue.empty() and len(event_results) < limit:
534		try:
535		_, event_id = queue.get_nowait()
536		except Empty:
537		break
538
539		if event_id in event_results:
540		continue
541
542		event_results.add(event_id)
543
544		txn.execute(query, (event_id, False, limit - len(event_results)))
545
546		for row in txn:
547		if row[1] not in event_results:
548		queue.put((-row[0], row[1]))
549
550		return event_results
551
552		@defer.inlineCallbacks
553		def get_missing_events(self, room_id, earliest_events, latest_events, limit):
554		ids = yield self.db.runInteraction(
555		"get_missing_events",
556		self._get_missing_events,
557		room_id,
558		earliest_events,
559		latest_events,
560		limit,
561		)
562		events = yield self.get_events_as_list(ids)
563		return events
564
565		def _get_missing_events(self, txn, room_id, earliest_events, latest_events, limit):
566
567		seen_events = set(earliest_events)
568		front = set(latest_events) - seen_events
569		event_results = []
570
571		query = (
572		"SELECT prev_event_id FROM event_edges "
573		"WHERE room_id = ? AND event_id = ? AND is_state = ? "
574		"LIMIT ?"
575		)
576
577		while front and len(event_results) < limit:
578		new_front = set()
579		for event_id in front:
580		txn.execute(
581		query, (room_id, event_id, False, limit - len(event_results))
582		)
583
584		new_results = {t[0] for t in txn} - seen_events
585
586		new_front \|= new_results
587		seen_events \|= new_results
588		event_results.extend(new_results)
589
590		front = new_front
591
592		# we built the list working backwards from latest_events; we now need to
593		# reverse it so that the events are approximately chronological.
594		event_results.reverse()
595		return event_results
596
597		@defer.inlineCallbacks
598		def get_successor_events(self, event_ids):
599		"""Fetch all events that have the given events as a prev event
600
601		Args:
602		event_ids (iterable[str])
603
604		Returns:
605		Deferred[list[str]]
606		"""
607		rows = yield self.db.simple_select_many_batch(
608		table="event_edges",
609		column="prev_event_id",
610		iterable=event_ids,
611		retcols=("event_id",),
612		desc="get_successor_events",
613		)
614
615		return [row["event_id"] for row in rows]
616
617
618		class EventFederationStore(EventFederationWorkerStore):
619		""" Responsible for storing and serving up the various graphs associated
620		with an event. Including the main event graph and the auth chains for an
621		event.
622
623		Also has methods for getting the front (latest) and back (oldest) edges
624		of the event graphs. These are used to generate the parents for new events
625		and backfilling from another server respectively.
626		"""
627
628		EVENT_AUTH_STATE_ONLY = "event_auth_state_only"
629
630		def __init__(self, database: Database, db_conn, hs):
631		super(EventFederationStore, self).__init__(database, db_conn, hs)
632
633		self.db.updates.register_background_update_handler(
634		self.EVENT_AUTH_STATE_ONLY, self._background_delete_non_state_event_auth
635		)
636
637		hs.get_clock().looping_call(
638		self._delete_old_forward_extrem_cache, 60 * 60 * 1000
639		)
640
641		def _delete_old_forward_extrem_cache(self):
642		def _delete_old_forward_extrem_cache_txn(txn):
643		# Delete entries older than a month, while making sure we don't delete
644		# the only entries for a room.
645		sql = """
646		DELETE FROM stream_ordering_to_exterm
647		WHERE
648		room_id IN (
649		SELECT room_id
650		FROM stream_ordering_to_exterm
651		WHERE stream_ordering > ?
652		) AND stream_ordering < ?
653		"""
654		txn.execute(
655		sql, (self.stream_ordering_month_ago, self.stream_ordering_month_ago)
656		)
657
658		return run_as_background_process(
659		"delete_old_forward_extrem_cache",
660		self.db.runInteraction,
661		"_delete_old_forward_extrem_cache",
662		_delete_old_forward_extrem_cache_txn,
663		)
664
665		def clean_room_for_join(self, room_id):
666		return self.db.runInteraction(
667		"clean_room_for_join", self._clean_room_for_join_txn, room_id
668		)
669
670		def _clean_room_for_join_txn(self, txn, room_id):
671		query = "DELETE FROM event_forward_extremities WHERE room_id = ?"
672
673		txn.execute(query, (room_id,))
674		txn.call_after(self.get_latest_event_ids_in_room.invalidate, (room_id,))
675
676		@defer.inlineCallbacks
677		def _background_delete_non_state_event_auth(self, progress, batch_size):
678		def delete_event_auth(txn):
679		target_min_stream_id = progress.get("target_min_stream_id_inclusive")
680		max_stream_id = progress.get("max_stream_id_exclusive")
681
682		if not target_min_stream_id or not max_stream_id:
683		txn.execute("SELECT COALESCE(MIN(stream_ordering), 0) FROM events")
684		rows = txn.fetchall()
685		target_min_stream_id = rows[0][0]
686
687		txn.execute("SELECT COALESCE(MAX(stream_ordering), 0) FROM events")
688		rows = txn.fetchall()
689		max_stream_id = rows[0][0]
690
691		min_stream_id = max_stream_id - batch_size
692
693		sql = """
694		DELETE FROM event_auth
695		WHERE event_id IN (
696		SELECT event_id FROM events
697		LEFT JOIN state_events USING (room_id, event_id)
698		WHERE ? <= stream_ordering AND stream_ordering < ?
699		AND state_key IS null
700		)
701		"""
702
703		txn.execute(sql, (min_stream_id, max_stream_id))
704
705		new_progress = {
706		"target_min_stream_id_inclusive": target_min_stream_id,
707		"max_stream_id_exclusive": min_stream_id,
708		}
709
710		self.db.updates._background_update_progress_txn(
711		txn, self.EVENT_AUTH_STATE_ONLY, new_progress
712		)
713
714		return min_stream_id >= target_min_stream_id
715
716		result = yield self.db.runInteraction(
717		self.EVENT_AUTH_STATE_ONLY, delete_event_auth
718		)
719
720		if not result:
721		yield self.db.updates._end_background_update(self.EVENT_AUTH_STATE_ONLY)
722
723		return batch_size

+0

-887

~~synapse/storage/data_stores/main/event_push_actions.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2015 OpenMarket Ltd
2		# Copyright 2018 New Vector Ltd
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		import logging
17
18		from canonicaljson import json
19
20		from twisted.internet import defer
21
22		from synapse.metrics.background_process_metrics import run_as_background_process
23		from synapse.storage._base import LoggingTransaction, SQLBaseStore, db_to_json
24		from synapse.storage.database import Database
25		from synapse.util.caches.descriptors import cachedInlineCallbacks
26
27		logger = logging.getLogger(__name__)
28
29
30		DEFAULT_NOTIF_ACTION = ["notify", {"set_tweak": "highlight", "value": False}]
31		DEFAULT_HIGHLIGHT_ACTION = [
32		"notify",
33		{"set_tweak": "sound", "value": "default"},
34		{"set_tweak": "highlight"},
35		]
36
37
38		def _serialize_action(actions, is_highlight):
39		"""Custom serializer for actions. This allows us to "compress" common actions.
40
41		We use the fact that most users have the same actions for notifs (and for
42		highlights).
43		We store these default actions as the empty string rather than the full JSON.
44		Since the empty string isn't valid JSON there is no risk of this clashing with
45		any real JSON actions
46		"""
47		if is_highlight:
48		if actions == DEFAULT_HIGHLIGHT_ACTION:
49		return "" # We use empty string as the column is non-NULL
50		else:
51		if actions == DEFAULT_NOTIF_ACTION:
52		return ""
53		return json.dumps(actions)
54
55
56		def _deserialize_action(actions, is_highlight):
57		"""Custom deserializer for actions. This allows us to "compress" common actions
58		"""
59		if actions:
60		return db_to_json(actions)
61
62		if is_highlight:
63		return DEFAULT_HIGHLIGHT_ACTION
64		else:
65		return DEFAULT_NOTIF_ACTION
66
67
68		class EventPushActionsWorkerStore(SQLBaseStore):
69		def __init__(self, database: Database, db_conn, hs):
70		super(EventPushActionsWorkerStore, self).__init__(database, db_conn, hs)
71
72		# These get correctly set by _find_stream_orderings_for_times_txn
73		self.stream_ordering_month_ago = None
74		self.stream_ordering_day_ago = None
75
76		cur = LoggingTransaction(
77		db_conn.cursor(),
78		name="_find_stream_orderings_for_times_txn",
79		database_engine=self.database_engine,
80		)
81		self._find_stream_orderings_for_times_txn(cur)
82		cur.close()
83
84		self.find_stream_orderings_looping_call = self._clock.looping_call(
85		self._find_stream_orderings_for_times, 10 * 60 * 1000
86		)
87		self._rotate_delay = 3
88		self._rotate_count = 10000
89
90		@cachedInlineCallbacks(num_args=3, tree=True, max_entries=5000)
91		def get_unread_event_push_actions_by_room_for_user(
92		self, room_id, user_id, last_read_event_id
93		):
94		ret = yield self.db.runInteraction(
95		"get_unread_event_push_actions_by_room",
96		self._get_unread_counts_by_receipt_txn,
97		room_id,
98		user_id,
99		last_read_event_id,
100		)
101		return ret
102
103		def _get_unread_counts_by_receipt_txn(
104		self, txn, room_id, user_id, last_read_event_id
105		):
106		sql = (
107		"SELECT stream_ordering"
108		" FROM events"
109		" WHERE room_id = ? AND event_id = ?"
110		)
111		txn.execute(sql, (room_id, last_read_event_id))
112		results = txn.fetchall()
113		if len(results) == 0:
114		return {"notify_count": 0, "highlight_count": 0}
115
116		stream_ordering = results[0][0]
117
118		return self._get_unread_counts_by_pos_txn(
119		txn, room_id, user_id, stream_ordering
120		)
121
122		def _get_unread_counts_by_pos_txn(self, txn, room_id, user_id, stream_ordering):
123
124		# First get number of notifications.
125		# We don't need to put a notif=1 clause as all rows always have
126		# notif=1
127		sql = (
128		"SELECT count(*)"
129		" FROM event_push_actions ea"
130		" WHERE"
131		" user_id = ?"
132		" AND room_id = ?"
133		" AND stream_ordering > ?"
134		)
135
136		txn.execute(sql, (user_id, room_id, stream_ordering))
137		row = txn.fetchone()
138		notify_count = row[0] if row else 0
139
140		txn.execute(
141		"""
142		SELECT notif_count FROM event_push_summary
143		WHERE room_id = ? AND user_id = ? AND stream_ordering > ?
144		""",
145		(room_id, user_id, stream_ordering),
146		)
147		rows = txn.fetchall()
148		if rows:
149		notify_count += rows[0][0]
150
151		# Now get the number of highlights
152		sql = (
153		"SELECT count(*)"
154		" FROM event_push_actions ea"
155		" WHERE"
156		" highlight = 1"
157		" AND user_id = ?"
158		" AND room_id = ?"
159		" AND stream_ordering > ?"
160		)
161
162		txn.execute(sql, (user_id, room_id, stream_ordering))
163		row = txn.fetchone()
164		highlight_count = row[0] if row else 0
165
166		return {"notify_count": notify_count, "highlight_count": highlight_count}
167
168		@defer.inlineCallbacks
169		def get_push_action_users_in_range(self, min_stream_ordering, max_stream_ordering):
170		def f(txn):
171		sql = (
172		"SELECT DISTINCT(user_id) FROM event_push_actions WHERE"
173		" stream_ordering >= ? AND stream_ordering <= ?"
174		)
175		txn.execute(sql, (min_stream_ordering, max_stream_ordering))
176		return [r[0] for r in txn]
177
178		ret = yield self.db.runInteraction("get_push_action_users_in_range", f)
179		return ret
180
181		@defer.inlineCallbacks
182		def get_unread_push_actions_for_user_in_range_for_http(
183		self, user_id, min_stream_ordering, max_stream_ordering, limit=20
184		):
185		"""Get a list of the most recent unread push actions for a given user,
186		within the given stream ordering range. Called by the httppusher.
187
188		Args:
189		user_id (str): The user to fetch push actions for.
190		min_stream_ordering(int): The exclusive lower bound on the
191		stream ordering of event push actions to fetch.
192		max_stream_ordering(int): The inclusive upper bound on the
193		stream ordering of event push actions to fetch.
194		limit (int): The maximum number of rows to return.
195		Returns:
196		A promise which resolves to a list of dicts with the keys "event_id",
197		"room_id", "stream_ordering", "actions".
198		The list will be ordered by ascending stream_ordering.
199		The list will have between 0~limit entries.
200		"""
201		# find rooms that have a read receipt in them and return the next
202		# push actions
203		def get_after_receipt(txn):
204		# find rooms that have a read receipt in them and return the next
205		# push actions
206		sql = (
207		"SELECT ep.event_id, ep.room_id, ep.stream_ordering, ep.actions,"
208		" ep.highlight "
209		" FROM ("
210		" SELECT room_id,"
211		" MAX(stream_ordering) as stream_ordering"
212		" FROM events"
213		" INNER JOIN receipts_linearized USING (room_id, event_id)"
214		" WHERE receipt_type = 'm.read' AND user_id = ?"
215		" GROUP BY room_id"
216		") AS rl,"
217		" event_push_actions AS ep"
218		" WHERE"
219		" ep.room_id = rl.room_id"
220		" AND ep.stream_ordering > rl.stream_ordering"
221		" AND ep.user_id = ?"
222		" AND ep.stream_ordering > ?"
223		" AND ep.stream_ordering <= ?"
224		" ORDER BY ep.stream_ordering ASC LIMIT ?"
225		)
226		args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]
227		txn.execute(sql, args)
228		return txn.fetchall()
229
230		after_read_receipt = yield self.db.runInteraction(
231		"get_unread_push_actions_for_user_in_range_http_arr", get_after_receipt
232		)
233
234		# There are rooms with push actions in them but you don't have a read receipt in
235		# them e.g. rooms you've been invited to, so get push actions for rooms which do
236		# not have read receipts in them too.
237		def get_no_receipt(txn):
238		sql = (
239		"SELECT ep.event_id, ep.room_id, ep.stream_ordering, ep.actions,"
240		" ep.highlight "
241		" FROM event_push_actions AS ep"
242		" INNER JOIN events AS e USING (room_id, event_id)"
243		" WHERE"
244		" ep.room_id NOT IN ("
245		" SELECT room_id FROM receipts_linearized"
246		" WHERE receipt_type = 'm.read' AND user_id = ?"
247		" GROUP BY room_id"
248		" )"
249		" AND ep.user_id = ?"
250		" AND ep.stream_ordering > ?"
251		" AND ep.stream_ordering <= ?"
252		" ORDER BY ep.stream_ordering ASC LIMIT ?"
253		)
254		args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]
255		txn.execute(sql, args)
256		return txn.fetchall()
257
258		no_read_receipt = yield self.db.runInteraction(
259		"get_unread_push_actions_for_user_in_range_http_nrr", get_no_receipt
260		)
261
262		notifs = [
263		{
264		"event_id": row[0],
265		"room_id": row[1],
266		"stream_ordering": row[2],
267		"actions": _deserialize_action(row[3], row[4]),
268		}
269		for row in after_read_receipt + no_read_receipt
270		]
271
272		# Now sort it so it's ordered correctly, since currently it will
273		# contain results from the first query, correctly ordered, followed
274		# by results from the second query, but we want them all ordered
275		# by stream_ordering, oldest first.
276		notifs.sort(key=lambda r: r["stream_ordering"])
277
278		# Take only up to the limit. We have to stop at the limit because
279		# one of the subqueries may have hit the limit.
280		return notifs[:limit]
281
282		@defer.inlineCallbacks
283		def get_unread_push_actions_for_user_in_range_for_email(
284		self, user_id, min_stream_ordering, max_stream_ordering, limit=20
285		):
286		"""Get a list of the most recent unread push actions for a given user,
287		within the given stream ordering range. Called by the emailpusher
288
289		Args:
290		user_id (str): The user to fetch push actions for.
291		min_stream_ordering(int): The exclusive lower bound on the
292		stream ordering of event push actions to fetch.
293		max_stream_ordering(int): The inclusive upper bound on the
294		stream ordering of event push actions to fetch.
295		limit (int): The maximum number of rows to return.
296		Returns:
297		A promise which resolves to a list of dicts with the keys "event_id",
298		"room_id", "stream_ordering", "actions", "received_ts".
299		The list will be ordered by descending received_ts.
300		The list will have between 0~limit entries.
301		"""
302		# find rooms that have a read receipt in them and return the most recent
303		# push actions
304		def get_after_receipt(txn):
305		sql = (
306		"SELECT ep.event_id, ep.room_id, ep.stream_ordering, ep.actions,"
307		" ep.highlight, e.received_ts"
308		" FROM ("
309		" SELECT room_id,"
310		" MAX(stream_ordering) as stream_ordering"
311		" FROM events"
312		" INNER JOIN receipts_linearized USING (room_id, event_id)"
313		" WHERE receipt_type = 'm.read' AND user_id = ?"
314		" GROUP BY room_id"
315		") AS rl,"
316		" event_push_actions AS ep"
317		" INNER JOIN events AS e USING (room_id, event_id)"
318		" WHERE"
319		" ep.room_id = rl.room_id"
320		" AND ep.stream_ordering > rl.stream_ordering"
321		" AND ep.user_id = ?"
322		" AND ep.stream_ordering > ?"
323		" AND ep.stream_ordering <= ?"
324		" ORDER BY ep.stream_ordering DESC LIMIT ?"
325		)
326		args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]
327		txn.execute(sql, args)
328		return txn.fetchall()
329
330		after_read_receipt = yield self.db.runInteraction(
331		"get_unread_push_actions_for_user_in_range_email_arr", get_after_receipt
332		)
333
334		# There are rooms with push actions in them but you don't have a read receipt in
335		# them e.g. rooms you've been invited to, so get push actions for rooms which do
336		# not have read receipts in them too.
337		def get_no_receipt(txn):
338		sql = (
339		"SELECT ep.event_id, ep.room_id, ep.stream_ordering, ep.actions,"
340		" ep.highlight, e.received_ts"
341		" FROM event_push_actions AS ep"
342		" INNER JOIN events AS e USING (room_id, event_id)"
343		" WHERE"
344		" ep.room_id NOT IN ("
345		" SELECT room_id FROM receipts_linearized"
346		" WHERE receipt_type = 'm.read' AND user_id = ?"
347		" GROUP BY room_id"
348		" )"
349		" AND ep.user_id = ?"
350		" AND ep.stream_ordering > ?"
351		" AND ep.stream_ordering <= ?"
352		" ORDER BY ep.stream_ordering DESC LIMIT ?"
353		)
354		args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]
355		txn.execute(sql, args)
356		return txn.fetchall()
357
358		no_read_receipt = yield self.db.runInteraction(
359		"get_unread_push_actions_for_user_in_range_email_nrr", get_no_receipt
360		)
361
362		# Make a list of dicts from the two sets of results.
363		notifs = [
364		{
365		"event_id": row[0],
366		"room_id": row[1],
367		"stream_ordering": row[2],
368		"actions": _deserialize_action(row[3], row[4]),
369		"received_ts": row[5],
370		}
371		for row in after_read_receipt + no_read_receipt
372		]
373
374		# Now sort it so it's ordered correctly, since currently it will
375		# contain results from the first query, correctly ordered, followed
376		# by results from the second query, but we want them all ordered
377		# by received_ts (most recent first)
378		notifs.sort(key=lambda r: -(r["received_ts"] or 0))
379
380		# Now return the first `limit`
381		return notifs[:limit]
382
383		def get_if_maybe_push_in_range_for_user(self, user_id, min_stream_ordering):
384		"""A fast check to see if there might be something to push for the
385		user since the given stream ordering. May return false positives.
386
387		Useful to know whether to bother starting a pusher on start up or not.
388
389		Args:
390		user_id (str)
391		min_stream_ordering (int)
392
393		Returns:
394		Deferred[bool]: True if there may be push to process, False if
395		there definitely isn't.
396		"""
397
398		def _get_if_maybe_push_in_range_for_user_txn(txn):
399		sql = """
400		SELECT 1 FROM event_push_actions
401		WHERE user_id = ? AND stream_ordering > ?
402		LIMIT 1
403		"""
404
405		txn.execute(sql, (user_id, min_stream_ordering))
406		return bool(txn.fetchone())
407
408		return self.db.runInteraction(
409		"get_if_maybe_push_in_range_for_user",
410		_get_if_maybe_push_in_range_for_user_txn,
411		)
412
413		def add_push_actions_to_staging(self, event_id, user_id_actions):
414		"""Add the push actions for the event to the push action staging area.
415
416		Args:
417		event_id (str)
418		user_id_actions (dict[str, list[dict\|str])]): A dictionary mapping
419		user_id to list of push actions, where an action can either be
420		a string or dict.
421
422		Returns:
423		Deferred
424		"""
425
426		if not user_id_actions:
427		return
428
429		# This is a helper function for generating the necessary tuple that
430		# can be used to inert into the `event_push_actions_staging` table.
431		def _gen_entry(user_id, actions):
432		is_highlight = 1 if _action_has_highlight(actions) else 0
433		return (
434		event_id, # event_id column
435		user_id, # user_id column
436		_serialize_action(actions, is_highlight), # actions column
437		1, # notif column
438		is_highlight, # highlight column
439		)
440
441		def _add_push_actions_to_staging_txn(txn):
442		# We don't use simple_insert_many here to avoid the overhead
443		# of generating lists of dicts.
444
445		sql = """
446		INSERT INTO event_push_actions_staging
447		(event_id, user_id, actions, notif, highlight)
448		VALUES (?, ?, ?, ?, ?)
449		"""
450
451		txn.executemany(
452		sql,
453		(
454		_gen_entry(user_id, actions)
455		for user_id, actions in user_id_actions.items()
456		),
457		)
458
459		return self.db.runInteraction(
460		"add_push_actions_to_staging", _add_push_actions_to_staging_txn
461		)
462
463		@defer.inlineCallbacks
464		def remove_push_actions_from_staging(self, event_id):
465		"""Called if we failed to persist the event to ensure that stale push
466		actions don't build up in the DB
467
468		Args:
469		event_id (str)
470		"""
471
472		try:
473		res = yield self.db.simple_delete(
474		table="event_push_actions_staging",
475		keyvalues={"event_id": event_id},
476		desc="remove_push_actions_from_staging",
477		)
478		return res
479		except Exception:
480		# this method is called from an exception handler, so propagating
481		# another exception here really isn't helpful - there's nothing
482		# the caller can do about it. Just log the exception and move on.
483		logger.exception(
484		"Error removing push actions after event persistence failure"
485		)
486
487		def _find_stream_orderings_for_times(self):
488		return run_as_background_process(
489		"event_push_action_stream_orderings",
490		self.db.runInteraction,
491		"_find_stream_orderings_for_times",
492		self._find_stream_orderings_for_times_txn,
493		)
494
495		def _find_stream_orderings_for_times_txn(self, txn):
496		logger.info("Searching for stream ordering 1 month ago")
497		self.stream_ordering_month_ago = self._find_first_stream_ordering_after_ts_txn(
498		txn, self._clock.time_msec() - 30 * 24 * 60 * 60 * 1000
499		)
500		logger.info(
501		"Found stream ordering 1 month ago: it's %d", self.stream_ordering_month_ago
502		)
503		logger.info("Searching for stream ordering 1 day ago")
504		self.stream_ordering_day_ago = self._find_first_stream_ordering_after_ts_txn(
505		txn, self._clock.time_msec() - 24 * 60 * 60 * 1000
506		)
507		logger.info(
508		"Found stream ordering 1 day ago: it's %d", self.stream_ordering_day_ago
509		)
510
511		def find_first_stream_ordering_after_ts(self, ts):
512		"""Gets the stream ordering corresponding to a given timestamp.
513
514		Specifically, finds the stream_ordering of the first event that was
515		received on or after the timestamp. This is done by a binary search on
516		the events table, since there is no index on received_ts, so is
517		relatively slow.
518
519		Args:
520		ts (int): timestamp in millis
521
522		Returns:
523		Deferred[int]: stream ordering of the first event received on/after
524		the timestamp
525		"""
526		return self.db.runInteraction(
527		"_find_first_stream_ordering_after_ts_txn",
528		self._find_first_stream_ordering_after_ts_txn,
529		ts,
530		)
531
532		@staticmethod
533		def _find_first_stream_ordering_after_ts_txn(txn, ts):
534		"""
535		Find the stream_ordering of the first event that was received on or
536		after a given timestamp. This is relatively slow as there is no index
537		on received_ts but we can then use this to delete push actions before
538		this.
539
540		received_ts must necessarily be in the same order as stream_ordering
541		and stream_ordering is indexed, so we manually binary search using
542		stream_ordering
543
544		Args:
545		txn (twisted.enterprise.adbapi.Transaction):
546		ts (int): timestamp to search for
547
548		Returns:
549		int: stream ordering
550		"""
551		txn.execute("SELECT MAX(stream_ordering) FROM events")
552		max_stream_ordering = txn.fetchone()[0]
553
554		if max_stream_ordering is None:
555		return 0
556
557		# We want the first stream_ordering in which received_ts is greater
558		# than or equal to ts. Call this point X.
559		#
560		# We maintain the invariants:
561		#
562		# range_start <= X <= range_end
563		#
564		range_start = 0
565		range_end = max_stream_ordering + 1
566
567		# Given a stream_ordering, look up the timestamp at that
568		# stream_ordering.
569		#
570		# The array may be sparse (we may be missing some stream_orderings).
571		# We treat the gaps as the same as having the same value as the
572		# preceding entry, because we will pick the lowest stream_ordering
573		# which satisfies our requirement of received_ts >= ts.
574		#
575		# For example, if our array of events indexed by stream_ordering is
576		# [10, <none>, 20], we should treat this as being equivalent to
577		# [10, 10, 20].
578		#
579		sql = (
580		"SELECT received_ts FROM events"
581		" WHERE stream_ordering <= ?"
582		" ORDER BY stream_ordering DESC"
583		" LIMIT 1"
584		)
585
586		while range_end - range_start > 0:
587		middle = (range_end + range_start) // 2
588		txn.execute(sql, (middle,))
589		row = txn.fetchone()
590		if row is None:
591		# no rows with stream_ordering<=middle
592		range_start = middle + 1
593		continue
594
595		middle_ts = row[0]
596		if ts > middle_ts:
597		# we got a timestamp lower than the one we were looking for.
598		# definitely need to look higher: X > middle.
599		range_start = middle + 1
600		else:
601		# we got a timestamp higher than (or the same as) the one we
602		# were looking for. We aren't yet sure about the point we
603		# looked up, but we can be sure that X <= middle.
604		range_end = middle
605
606		return range_end
607
608		@defer.inlineCallbacks
609		def get_time_of_last_push_action_before(self, stream_ordering):
610		def f(txn):
611		sql = (
612		"SELECT e.received_ts"
613		" FROM event_push_actions AS ep"
614		" JOIN events e ON ep.room_id = e.room_id AND ep.event_id = e.event_id"
615		" WHERE ep.stream_ordering > ?"
616		" ORDER BY ep.stream_ordering ASC"
617		" LIMIT 1"
618		)
619		txn.execute(sql, (stream_ordering,))
620		return txn.fetchone()
621
622		result = yield self.db.runInteraction("get_time_of_last_push_action_before", f)
623		return result[0] if result else None
624
625
626		class EventPushActionsStore(EventPushActionsWorkerStore):
627		EPA_HIGHLIGHT_INDEX = "epa_highlight_index"
628
629		def __init__(self, database: Database, db_conn, hs):
630		super(EventPushActionsStore, self).__init__(database, db_conn, hs)
631
632		self.db.updates.register_background_index_update(
633		self.EPA_HIGHLIGHT_INDEX,
634		index_name="event_push_actions_u_highlight",
635		table="event_push_actions",
636		columns=["user_id", "stream_ordering"],
637		)
638
639		self.db.updates.register_background_index_update(
640		"event_push_actions_highlights_index",
641		index_name="event_push_actions_highlights_index",
642		table="event_push_actions",
643		columns=["user_id", "room_id", "topological_ordering", "stream_ordering"],
644		where_clause="highlight=1",
645		)
646
647		self._doing_notif_rotation = False
648		self._rotate_notif_loop = self._clock.looping_call(
649		self._start_rotate_notifs, 30 * 60 * 1000
650		)
651
652		@defer.inlineCallbacks
653		def get_push_actions_for_user(
654		self, user_id, before=None, limit=50, only_highlight=False
655		):
656		def f(txn):
657		before_clause = ""
658		if before:
659		before_clause = "AND epa.stream_ordering < ?"
660		args = [user_id, before, limit]
661		else:
662		args = [user_id, limit]
663
664		if only_highlight:
665		if len(before_clause) > 0:
666		before_clause += " "
667		before_clause += "AND epa.highlight = 1"
668
669		# NB. This assumes event_ids are globally unique since
670		# it makes the query easier to index
671		sql = (
672		"SELECT epa.event_id, epa.room_id,"
673		" epa.stream_ordering, epa.topological_ordering,"
674		" epa.actions, epa.highlight, epa.profile_tag, e.received_ts"
675		" FROM event_push_actions epa, events e"
676		" WHERE epa.event_id = e.event_id"
677		" AND epa.user_id = ? %s"
678		" ORDER BY epa.stream_ordering DESC"
679		" LIMIT ?" % (before_clause,)
680		)
681		txn.execute(sql, args)
682		return self.db.cursor_to_dict(txn)
683
684		push_actions = yield self.db.runInteraction("get_push_actions_for_user", f)
685		for pa in push_actions:
686		pa["actions"] = _deserialize_action(pa["actions"], pa["highlight"])
687		return push_actions
688
689		@defer.inlineCallbacks
690		def get_latest_push_action_stream_ordering(self):
691		def f(txn):
692		txn.execute("SELECT MAX(stream_ordering) FROM event_push_actions")
693		return txn.fetchone()
694
695		result = yield self.db.runInteraction(
696		"get_latest_push_action_stream_ordering", f
697		)
698		return result[0] or 0
699
700		def _remove_old_push_actions_before_txn(
701		self, txn, room_id, user_id, stream_ordering
702		):
703		"""
704		Purges old push actions for a user and room before a given
705		stream_ordering.
706
707		We however keep a months worth of highlighted notifications, so that
708		users can still get a list of recent highlights.
709
710		Args:
711		txn: The transcation
712		room_id: Room ID to delete from
713		user_id: user ID to delete for
714		stream_ordering: The lowest stream ordering which will
715		not be deleted.
716		"""
717		txn.call_after(
718		self.get_unread_event_push_actions_by_room_for_user.invalidate_many,
719		(room_id, user_id),
720		)
721
722		# We need to join on the events table to get the received_ts for
723		# event_push_actions and sqlite won't let us use a join in a delete so
724		# we can't just delete where received_ts < x. Furthermore we can
725		# only identify event_push_actions by a tuple of room_id, event_id
726		# we we can't use a subquery.
727		# Instead, we look up the stream ordering for the last event in that
728		# room received before the threshold time and delete event_push_actions
729		# in the room with a stream_odering before that.
730		txn.execute(
731		"DELETE FROM event_push_actions "
732		" WHERE user_id = ? AND room_id = ? AND "
733		" stream_ordering <= ?"
734		" AND ((stream_ordering < ? AND highlight = 1) or highlight = 0)",
735		(user_id, room_id, stream_ordering, self.stream_ordering_month_ago),
736		)
737
738		txn.execute(
739		"""
740		DELETE FROM event_push_summary
741		WHERE room_id = ? AND user_id = ? AND stream_ordering <= ?
742		""",
743		(room_id, user_id, stream_ordering),
744		)
745
746		def _start_rotate_notifs(self):
747		return run_as_background_process("rotate_notifs", self._rotate_notifs)
748
749		@defer.inlineCallbacks
750		def _rotate_notifs(self):
751		if self._doing_notif_rotation or self.stream_ordering_day_ago is None:
752		return
753		self._doing_notif_rotation = True
754
755		try:
756		while True:
757		logger.info("Rotating notifications")
758
759		caught_up = yield self.db.runInteraction(
760		"_rotate_notifs", self._rotate_notifs_txn
761		)
762		if caught_up:
763		break
764		yield self.hs.get_clock().sleep(self._rotate_delay)
765		finally:
766		self._doing_notif_rotation = False
767
768		def _rotate_notifs_txn(self, txn):
769		"""Archives older notifications into event_push_summary. Returns whether
770		the archiving process has caught up or not.
771		"""
772
773		old_rotate_stream_ordering = self.db.simple_select_one_onecol_txn(
774		txn,
775		table="event_push_summary_stream_ordering",
776		keyvalues={},
777		retcol="stream_ordering",
778		)
779
780		# We don't to try and rotate millions of rows at once, so we cap the
781		# maximum stream ordering we'll rotate before.
782		txn.execute(
783		"""
784		SELECT stream_ordering FROM event_push_actions
785		WHERE stream_ordering > ?
786		ORDER BY stream_ordering ASC LIMIT 1 OFFSET ?
787		""",
788		(old_rotate_stream_ordering, self._rotate_count),
789		)
790		stream_row = txn.fetchone()
791		if stream_row:
792		(offset_stream_ordering,) = stream_row
793		rotate_to_stream_ordering = min(
794		self.stream_ordering_day_ago, offset_stream_ordering
795		)
796		caught_up = offset_stream_ordering >= self.stream_ordering_day_ago
797		else:
798		rotate_to_stream_ordering = self.stream_ordering_day_ago
799		caught_up = True
800
801		logger.info("Rotating notifications up to: %s", rotate_to_stream_ordering)
802
803		self._rotate_notifs_before_txn(txn, rotate_to_stream_ordering)
804
805		# We have caught up iff we were limited by `stream_ordering_day_ago`
806		return caught_up
807
808		def _rotate_notifs_before_txn(self, txn, rotate_to_stream_ordering):
809		old_rotate_stream_ordering = self.db.simple_select_one_onecol_txn(
810		txn,
811		table="event_push_summary_stream_ordering",
812		keyvalues={},
813		retcol="stream_ordering",
814		)
815
816		# Calculate the new counts that should be upserted into event_push_summary
817		sql = """
818		SELECT user_id, room_id,
819		coalesce(old.notif_count, 0) + upd.notif_count,
820		upd.stream_ordering,
821		old.user_id
822		FROM (
823		SELECT user_id, room_id, count(*) as notif_count,
824		max(stream_ordering) as stream_ordering
825		FROM event_push_actions
826		WHERE ? <= stream_ordering AND stream_ordering < ?
827		AND highlight = 0
828		GROUP BY user_id, room_id
829		) AS upd
830		LEFT JOIN event_push_summary AS old USING (user_id, room_id)
831		"""
832
833		txn.execute(sql, (old_rotate_stream_ordering, rotate_to_stream_ordering))
834		rows = txn.fetchall()
835
836		logger.info("Rotating notifications, handling %d rows", len(rows))
837
838		# If the `old.user_id` above is NULL then we know there isn't already an
839		# entry in the table, so we simply insert it. Otherwise we update the
840		# existing table.
841		self.db.simple_insert_many_txn(
842		txn,
843		table="event_push_summary",
844		values=[
845		{
846		"user_id": row[0],
847		"room_id": row[1],
848		"notif_count": row[2],
849		"stream_ordering": row[3],
850		}
851		for row in rows
852		if row[4] is None
853		],
854		)
855
856		txn.executemany(
857		"""
858		UPDATE event_push_summary SET notif_count = ?, stream_ordering = ?
859		WHERE user_id = ? AND room_id = ?
860		""",
861		((row[2], row[3], row[0], row[1]) for row in rows if row[4] is not None),
862		)
863
864		txn.execute(
865		"DELETE FROM event_push_actions"
866		" WHERE ? <= stream_ordering AND stream_ordering < ? AND highlight = 0",
867		(old_rotate_stream_ordering, rotate_to_stream_ordering),
868		)
869
870		logger.info("Rotating notifications, deleted %s push actions", txn.rowcount)
871
872		txn.execute(
873		"UPDATE event_push_summary_stream_ordering SET stream_ordering = ?",
874		(rotate_to_stream_ordering,),
875		)
876
877
878		def _action_has_highlight(actions):
879		for action in actions:
880		try:
881		if action.get("set_tweak", None) == "highlight":
882		return action.get("value", True)
883		except AttributeError:
884		pass
885
886		return False

+0

-1475

~~synapse/storage/data_stores/main/events.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2018-2019 New Vector Ltd
3		# Copyright 2019 The Matrix.org Foundation C.I.C.
4		#
5		# Licensed under the Apache License, Version 2.0 (the "License");
6		# you may not use this file except in compliance with the License.
7		# You may obtain a copy of the License at
8		#
9		# http://www.apache.org/licenses/LICENSE-2.0
10		#
11		# Unless required by applicable law or agreed to in writing, software
12		# distributed under the License is distributed on an "AS IS" BASIS,
13		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14		# See the License for the specific language governing permissions and
15		# limitations under the License.
16		import itertools
17		import logging
18		from collections import OrderedDict, namedtuple
19		from typing import TYPE_CHECKING, Dict, Iterable, List, Tuple
20
21		import attr
22		from prometheus_client import Counter
23
24		from twisted.internet import defer
25
26		import synapse.metrics
27		from synapse.api.constants import EventContentFields, EventTypes, RelationTypes
28		from synapse.api.room_versions import RoomVersions
29		from synapse.crypto.event_signing import compute_event_reference_hash
30		from synapse.events import EventBase # noqa: F401
31		from synapse.events.snapshot import EventContext # noqa: F401
32		from synapse.logging.utils import log_function
33		from synapse.storage._base import db_to_json, make_in_list_sql_clause
34		from synapse.storage.data_stores.main.search import SearchEntry
35		from synapse.storage.database import Database, LoggingTransaction
36		from synapse.storage.util.id_generators import StreamIdGenerator
37		from synapse.types import StateMap, get_domain_from_id
38		from synapse.util.frozenutils import frozendict_json_encoder
39		from synapse.util.iterutils import batch_iter
40
41		if TYPE_CHECKING:
42		from synapse.server import HomeServer
43		from synapse.storage.data_stores.main import DataStore
44
45
46		logger = logging.getLogger(__name__)
47
48		persist_event_counter = Counter("synapse_storage_events_persisted_events", "")
49		event_counter = Counter(
50		"synapse_storage_events_persisted_events_sep",
51		"",
52		["type", "origin_type", "origin_entity"],
53		)
54
55
56		def encode_json(json_object):
57		"""
58		Encode a Python object as JSON and return it in a Unicode string.
59		"""
60		out = frozendict_json_encoder.encode(json_object)
61		if isinstance(out, bytes):
62		out = out.decode("utf8")
63		return out
64
65
66		_EventCacheEntry = namedtuple("_EventCacheEntry", ("event", "redacted_event"))
67
68
69		@attr.s(slots=True)
70		class DeltaState:
71		"""Deltas to use to update the `current_state_events` table.
72
73		Attributes:
74		to_delete: List of type/state_keys to delete from current state
75		to_insert: Map of state to upsert into current state
76		no_longer_in_room: The server is not longer in the room, so the room
77		should e.g. be removed from `current_state_events` table.
78		"""
79
80		to_delete = attr.ib(type=List[Tuple[str, str]])
81		to_insert = attr.ib(type=StateMap[str])
82		no_longer_in_room = attr.ib(type=bool, default=False)
83
84
85		class PersistEventsStore:
86		"""Contains all the functions for writing events to the database.
87
88		Should only be instantiated on one process (when using a worker mode setup).
89
90		Note: This is not part of the `DataStore` mixin.
91		"""
92
93		def __init__(self, hs: "HomeServer", db: Database, main_data_store: "DataStore"):
94		self.hs = hs
95		self.db = db
96		self.store = main_data_store
97		self.database_engine = db.engine
98		self._clock = hs.get_clock()
99
100		self._ephemeral_messages_enabled = hs.config.enable_ephemeral_messages
101		self.is_mine_id = hs.is_mine_id
102
103		# Ideally we'd move these ID gens here, unfortunately some other ID
104		# generators are chained off them so doing so is a bit of a PITA.
105		self._backfill_id_gen = self.store._backfill_id_gen # type: StreamIdGenerator
106		self._stream_id_gen = self.store._stream_id_gen # type: StreamIdGenerator
107
108		# This should only exist on instances that are configured to write
109		assert (
110		hs.config.worker.writers.events == hs.get_instance_name()
111		), "Can only instantiate EventsStore on master"
112
113		@defer.inlineCallbacks
114		def _persist_events_and_state_updates(
115		self,
116		events_and_contexts: List[Tuple[EventBase, EventContext]],
117		current_state_for_room: Dict[str, StateMap[str]],
118		state_delta_for_room: Dict[str, DeltaState],
119		new_forward_extremeties: Dict[str, List[str]],
120		backfilled: bool = False,
121		):
122		"""Persist a set of events alongside updates to the current state and
123		forward extremities tables.
124
125		Args:
126		events_and_contexts:
127		current_state_for_room: Map from room_id to the current state of
128		the room based on forward extremities
129		state_delta_for_room: Map from room_id to the delta to apply to
130		room state
131		new_forward_extremities: Map from room_id to list of event IDs
132		that are the new forward extremities of the room.
133		backfilled
134
135		Returns:
136		Deferred: resolves when the events have been persisted
137		"""
138
139		# We want to calculate the stream orderings as late as possible, as
140		# we only notify after all events with a lesser stream ordering have
141		# been persisted. I.e. if we spend 10s inside the with block then
142		# that will delay all subsequent events from being notified about.
143		# Hence why we do it down here rather than wrapping the entire
144		# function.
145		#
146		# Its safe to do this after calculating the state deltas etc as we
147		# only need to protect the persistence of the events. This is to
148		# ensure that queries of the form "fetch events since X" don't
149		# return events and stream positions after events that are still in
150		# flight, as otherwise subsequent requests "fetch event since Y"
151		# will not return those events.
152		#
153		# Note: Multiple instances of this function cannot be in flight at
154		# the same time for the same room.
155		if backfilled:
156		stream_ordering_manager = self._backfill_id_gen.get_next_mult(
157		len(events_and_contexts)
158		)
159		else:
160		stream_ordering_manager = self._stream_id_gen.get_next_mult(
161		len(events_and_contexts)
162		)
163
164		with stream_ordering_manager as stream_orderings:
165		for (event, context), stream in zip(events_and_contexts, stream_orderings):
166		event.internal_metadata.stream_ordering = stream
167
168		yield self.db.runInteraction(
169		"persist_events",
170		self._persist_events_txn,
171		events_and_contexts=events_and_contexts,
172		backfilled=backfilled,
173		state_delta_for_room=state_delta_for_room,
174		new_forward_extremeties=new_forward_extremeties,
175		)
176		persist_event_counter.inc(len(events_and_contexts))
177
178		if not backfilled:
179		# backfilled events have negative stream orderings, so we don't
180		# want to set the event_persisted_position to that.
181		synapse.metrics.event_persisted_position.set(
182		events_and_contexts[-1][0].internal_metadata.stream_ordering
183		)
184
185		for event, context in events_and_contexts:
186		if context.app_service:
187		origin_type = "local"
188		origin_entity = context.app_service.id
189		elif self.hs.is_mine_id(event.sender):
190		origin_type = "local"
191		origin_entity = "client"
192		else:
193		origin_type = "remote"
194		origin_entity = get_domain_from_id(event.sender)
195
196		event_counter.labels(event.type, origin_type, origin_entity).inc()
197
198		for room_id, new_state in current_state_for_room.items():
199		self.store.get_current_state_ids.prefill((room_id,), new_state)
200
201		for room_id, latest_event_ids in new_forward_extremeties.items():
202		self.store.get_latest_event_ids_in_room.prefill(
203		(room_id,), list(latest_event_ids)
204		)
205
206		@defer.inlineCallbacks
207		def _get_events_which_are_prevs(self, event_ids):
208		"""Filter the supplied list of event_ids to get those which are prev_events of
209		existing (non-outlier/rejected) events.
210
211		Args:
212		event_ids (Iterable[str]): event ids to filter
213
214		Returns:
215		Deferred[List[str]]: filtered event ids
216		"""
217		results = []
218
219		def _get_events_which_are_prevs_txn(txn, batch):
220		sql = """
221		SELECT prev_event_id, internal_metadata
222		FROM event_edges
223		INNER JOIN events USING (event_id)
224		LEFT JOIN rejections USING (event_id)
225		LEFT JOIN event_json USING (event_id)
226		WHERE
227		NOT events.outlier
228		AND rejections.event_id IS NULL
229		AND
230		"""
231
232		clause, args = make_in_list_sql_clause(
233		self.database_engine, "prev_event_id", batch
234		)
235
236		txn.execute(sql + clause, args)
237		results.extend(r[0] for r in txn if not db_to_json(r[1]).get("soft_failed"))
238
239		for chunk in batch_iter(event_ids, 100):
240		yield self.db.runInteraction(
241		"_get_events_which_are_prevs", _get_events_which_are_prevs_txn, chunk
242		)
243
244		return results
245
246		@defer.inlineCallbacks
247		def _get_prevs_before_rejected(self, event_ids):
248		"""Get soft-failed ancestors to remove from the extremities.
249
250		Given a set of events, find all those that have been soft-failed or
251		rejected. Returns those soft failed/rejected events and their prev
252		events (whether soft-failed/rejected or not), and recurses up the
253		prev-event graph until it finds no more soft-failed/rejected events.
254
255		This is used to find extremities that are ancestors of new events, but
256		are separated by soft failed events.
257
258		Args:
259		event_ids (Iterable[str]): Events to find prev events for. Note
260		that these must have already been persisted.
261
262		Returns:
263		Deferred[set[str]]
264		"""
265
266		# The set of event_ids to return. This includes all soft-failed events
267		# and their prev events.
268		existing_prevs = set()
269
270		def _get_prevs_before_rejected_txn(txn, batch):
271		to_recursively_check = batch
272
273		while to_recursively_check:
274		sql = """
275		SELECT
276		event_id, prev_event_id, internal_metadata,
277		rejections.event_id IS NOT NULL
278		FROM event_edges
279		INNER JOIN events USING (event_id)
280		LEFT JOIN rejections USING (event_id)
281		LEFT JOIN event_json USING (event_id)
282		WHERE
283		NOT events.outlier
284		AND
285		"""
286
287		clause, args = make_in_list_sql_clause(
288		self.database_engine, "event_id", to_recursively_check
289		)
290
291		txn.execute(sql + clause, args)
292		to_recursively_check = []
293
294		for event_id, prev_event_id, metadata, rejected in txn:
295		if prev_event_id in existing_prevs:
296		continue
297
298		soft_failed = db_to_json(metadata).get("soft_failed")
299		if soft_failed or rejected:
300		to_recursively_check.append(prev_event_id)
301		existing_prevs.add(prev_event_id)
302
303		for chunk in batch_iter(event_ids, 100):
304		yield self.db.runInteraction(
305		"_get_prevs_before_rejected", _get_prevs_before_rejected_txn, chunk
306		)
307
308		return existing_prevs
309
310		@log_function
311		def _persist_events_txn(
312		self,
313		txn: LoggingTransaction,
314		events_and_contexts: List[Tuple[EventBase, EventContext]],
315		backfilled: bool,
316		state_delta_for_room: Dict[str, DeltaState] = {},
317		new_forward_extremeties: Dict[str, List[str]] = {},
318		):
319		"""Insert some number of room events into the necessary database tables.
320
321		Rejected events are only inserted into the events table, the events_json table,
322		and the rejections table. Things reading from those table will need to check
323		whether the event was rejected.
324
325		Args:
326		txn
327		events_and_contexts: events to persist
328		backfilled: True if the events were backfilled
329		delete_existing True to purge existing table rows for the events
330		from the database. This is useful when retrying due to
331		IntegrityError.
332		state_delta_for_room: The current-state delta for each room.
333		new_forward_extremetie: The new forward extremities for each room.
334		For each room, a list of the event ids which are the forward
335		extremities.
336
337		"""
338		all_events_and_contexts = events_and_contexts
339
340		min_stream_order = events_and_contexts[0][0].internal_metadata.stream_ordering
341		max_stream_order = events_and_contexts[-1][0].internal_metadata.stream_ordering
342
343		self._update_forward_extremities_txn(
344		txn,
345		new_forward_extremities=new_forward_extremeties,
346		max_stream_order=max_stream_order,
347		)
348
349		# Ensure that we don't have the same event twice.
350		events_and_contexts = self._filter_events_and_contexts_for_duplicates(
351		events_and_contexts
352		)
353
354		self._update_room_depths_txn(
355		txn, events_and_contexts=events_and_contexts, backfilled=backfilled
356		)
357
358		# _update_outliers_txn filters out any events which have already been
359		# persisted, and returns the filtered list.
360		events_and_contexts = self._update_outliers_txn(
361		txn, events_and_contexts=events_and_contexts
362		)
363
364		# From this point onwards the events are only events that we haven't
365		# seen before.
366
367		self._store_event_txn(txn, events_and_contexts=events_and_contexts)
368
369		# Insert into event_to_state_groups.
370		self._store_event_state_mappings_txn(txn, events_and_contexts)
371
372		# We want to store event_auth mappings for rejected events, as they're
373		# used in state res v2.
374		# This is only necessary if the rejected event appears in an accepted
375		# event's auth chain, but its easier for now just to store them (and
376		# it doesn't take much storage compared to storing the entire event
377		# anyway).
378		self.db.simple_insert_many_txn(
379		txn,
380		table="event_auth",
381		values=[
382		{
383		"event_id": event.event_id,
384		"room_id": event.room_id,
385		"auth_id": auth_id,
386		}
387		for event, _ in events_and_contexts
388		for auth_id in event.auth_event_ids()
389		if event.is_state()
390		],
391		)
392
393		# _store_rejected_events_txn filters out any events which were
394		# rejected, and returns the filtered list.
395		events_and_contexts = self._store_rejected_events_txn(
396		txn, events_and_contexts=events_and_contexts
397		)
398
399		# From this point onwards the events are only ones that weren't
400		# rejected.
401
402		self._update_metadata_tables_txn(
403		txn,
404		events_and_contexts=events_and_contexts,
405		all_events_and_contexts=all_events_and_contexts,
406		backfilled=backfilled,
407		)
408
409		# We call this last as it assumes we've inserted the events into
410		# room_memberships, where applicable.
411		self._update_current_state_txn(txn, state_delta_for_room, min_stream_order)
412
413		def _update_current_state_txn(
414		self,
415		txn: LoggingTransaction,
416		state_delta_by_room: Dict[str, DeltaState],
417		stream_id: int,
418		):
419		for room_id, delta_state in state_delta_by_room.items():
420		to_delete = delta_state.to_delete
421		to_insert = delta_state.to_insert
422
423		if delta_state.no_longer_in_room:
424		# Server is no longer in the room so we delete the room from
425		# current_state_events, being careful we've already updated the
426		# rooms.room_version column (which gets populated in a
427		# background task).
428		self._upsert_room_version_txn(txn, room_id)
429
430		# Before deleting we populate the current_state_delta_stream
431		# so that async background tasks get told what happened.
432		sql = """
433		INSERT INTO current_state_delta_stream
434		(stream_id, room_id, type, state_key, event_id, prev_event_id)
435		SELECT ?, room_id, type, state_key, null, event_id
436		FROM current_state_events
437		WHERE room_id = ?
438		"""
439		txn.execute(sql, (stream_id, room_id))
440
441		self.db.simple_delete_txn(
442		txn, table="current_state_events", keyvalues={"room_id": room_id},
443		)
444		else:
445		# We're still in the room, so we update the current state as normal.
446
447		# First we add entries to the current_state_delta_stream. We
448		# do this before updating the current_state_events table so
449		# that we can use it to calculate the `prev_event_id`. (This
450		# allows us to not have to pull out the existing state
451		# unnecessarily).
452		#
453		# The stream_id for the update is chosen to be the minimum of the stream_ids
454		# for the batch of the events that we are persisting; that means we do not
455		# end up in a situation where workers see events before the
456		# current_state_delta updates.
457		#
458		sql = """
459		INSERT INTO current_state_delta_stream
460		(stream_id, room_id, type, state_key, event_id, prev_event_id)
461		SELECT ?, ?, ?, ?, ?, (
462		SELECT event_id FROM current_state_events
463		WHERE room_id = ? AND type = ? AND state_key = ?
464		)
465		"""
466		txn.executemany(
467		sql,
468		(
469		(
470		stream_id,
471		room_id,
472		etype,
473		state_key,
474		to_insert.get((etype, state_key)),
475		room_id,
476		etype,
477		state_key,
478		)
479		for etype, state_key in itertools.chain(to_delete, to_insert)
480		),
481		)
482		# Now we actually update the current_state_events table
483
484		txn.executemany(
485		"DELETE FROM current_state_events"
486		" WHERE room_id = ? AND type = ? AND state_key = ?",
487		(
488		(room_id, etype, state_key)
489		for etype, state_key in itertools.chain(to_delete, to_insert)
490		),
491		)
492
493		# We include the membership in the current state table, hence we do
494		# a lookup when we insert. This assumes that all events have already
495		# been inserted into room_memberships.
496		txn.executemany(
497		"""INSERT INTO current_state_events
498		(room_id, type, state_key, event_id, membership)
499		VALUES (?, ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?))
500		""",
501		[
502		(room_id, key[0], key[1], ev_id, ev_id)
503		for key, ev_id in to_insert.items()
504		],
505		)
506
507		# We now update `local_current_membership`. We do this regardless
508		# of whether we're still in the room or not to handle the case where
509		# e.g. we just got banned (where we need to record that fact here).
510
511		# Note: Do we really want to delete rows here (that we do not
512		# subsequently reinsert below)? While technically correct it means
513		# we have no record of the fact the user was a member of the
514		# room but got, say, state reset out of it.
515		if to_delete or to_insert:
516		txn.executemany(
517		"DELETE FROM local_current_membership"
518		" WHERE room_id = ? AND user_id = ?",
519		(
520		(room_id, state_key)
521		for etype, state_key in itertools.chain(to_delete, to_insert)
522		if etype == EventTypes.Member and self.is_mine_id(state_key)
523		),
524		)
525
526		if to_insert:
527		txn.executemany(
528		"""INSERT INTO local_current_membership
529		(room_id, user_id, event_id, membership)
530		VALUES (?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?))
531		""",
532		[
533		(room_id, key[1], ev_id, ev_id)
534		for key, ev_id in to_insert.items()
535		if key[0] == EventTypes.Member and self.is_mine_id(key[1])
536		],
537		)
538
539		txn.call_after(
540		self.store._curr_state_delta_stream_cache.entity_has_changed,
541		room_id,
542		stream_id,
543		)
544
545		# Invalidate the various caches
546
547		# Figure out the changes of membership to invalidate the
548		# `get_rooms_for_user` cache.
549		# We find out which membership events we may have deleted
550		# and which we have added, then we invlidate the caches for all
551		# those users.
552		members_changed = {
553		state_key
554		for ev_type, state_key in itertools.chain(to_delete, to_insert)
555		if ev_type == EventTypes.Member
556		}
557
558		for member in members_changed:
559		txn.call_after(
560		self.store.get_rooms_for_user_with_stream_ordering.invalidate,
561		(member,),
562		)
563
564		self.store._invalidate_state_caches_and_stream(
565		txn, room_id, members_changed
566		)
567
568		def _upsert_room_version_txn(self, txn: LoggingTransaction, room_id: str):
569		"""Update the room version in the database based off current state
570		events.
571
572		This is used when we're about to delete current state and we want to
573		ensure that the `rooms.room_version` column is up to date.
574		"""
575
576		sql = """
577		SELECT json FROM event_json
578		INNER JOIN current_state_events USING (room_id, event_id)
579		WHERE room_id = ? AND type = ? AND state_key = ?
580		"""
581		txn.execute(sql, (room_id, EventTypes.Create, ""))
582		row = txn.fetchone()
583		if row:
584		event_json = db_to_json(row[0])
585		content = event_json.get("content", {})
586		creator = content.get("creator")
587		room_version_id = content.get("room_version", RoomVersions.V1.identifier)
588
589		self.db.simple_upsert_txn(
590		txn,
591		table="rooms",
592		keyvalues={"room_id": room_id},
593		values={"room_version": room_version_id},
594		insertion_values={"is_public": False, "creator": creator},
595		)
596
597		def _update_forward_extremities_txn(
598		self, txn, new_forward_extremities, max_stream_order
599		):
600		for room_id, new_extrem in new_forward_extremities.items():
601		self.db.simple_delete_txn(
602		txn, table="event_forward_extremities", keyvalues={"room_id": room_id}
603		)
604		txn.call_after(
605		self.store.get_latest_event_ids_in_room.invalidate, (room_id,)
606		)
607
608		self.db.simple_insert_many_txn(
609		txn,
610		table="event_forward_extremities",
611		values=[
612		{"event_id": ev_id, "room_id": room_id}
613		for room_id, new_extrem in new_forward_extremities.items()
614		for ev_id in new_extrem
615		],
616		)
617		# We now insert into stream_ordering_to_exterm a mapping from room_id,
618		# new stream_ordering to new forward extremeties in the room.
619		# This allows us to later efficiently look up the forward extremeties
620		# for a room before a given stream_ordering
621		self.db.simple_insert_many_txn(
622		txn,
623		table="stream_ordering_to_exterm",
624		values=[
625		{
626		"room_id": room_id,
627		"event_id": event_id,
628		"stream_ordering": max_stream_order,
629		}
630		for room_id, new_extrem in new_forward_extremities.items()
631		for event_id in new_extrem
632		],
633		)
634
635		@classmethod
636		def _filter_events_and_contexts_for_duplicates(cls, events_and_contexts):
637		"""Ensure that we don't have the same event twice.
638
639		Pick the earliest non-outlier if there is one, else the earliest one.
640
641		Args:
642		events_and_contexts (list[(EventBase, EventContext)]):
643		Returns:
644		list[(EventBase, EventContext)]: filtered list
645		"""
646		new_events_and_contexts = OrderedDict()
647		for event, context in events_and_contexts:
648		prev_event_context = new_events_and_contexts.get(event.event_id)
649		if prev_event_context:
650		if not event.internal_metadata.is_outlier():
651		if prev_event_context[0].internal_metadata.is_outlier():
652		# To ensure correct ordering we pop, as OrderedDict is
653		# ordered by first insertion.
654		new_events_and_contexts.pop(event.event_id, None)
655		new_events_and_contexts[event.event_id] = (event, context)
656		else:
657		new_events_and_contexts[event.event_id] = (event, context)
658		return list(new_events_and_contexts.values())
659
660		def _update_room_depths_txn(self, txn, events_and_contexts, backfilled):
661		"""Update min_depth for each room
662
663		Args:
664		txn (twisted.enterprise.adbapi.Connection): db connection
665		events_and_contexts (list[(EventBase, EventContext)]): events
666		we are persisting
667		backfilled (bool): True if the events were backfilled
668		"""
669		depth_updates = {}
670		for event, context in events_and_contexts:
671		# Remove the any existing cache entries for the event_ids
672		txn.call_after(self.store._invalidate_get_event_cache, event.event_id)
673		if not backfilled:
674		txn.call_after(
675		self.store._events_stream_cache.entity_has_changed,
676		event.room_id,
677		event.internal_metadata.stream_ordering,
678		)
679
680		if not event.internal_metadata.is_outlier() and not context.rejected:
681		depth_updates[event.room_id] = max(
682		event.depth, depth_updates.get(event.room_id, event.depth)
683		)
684
685		for room_id, depth in depth_updates.items():
686		self._update_min_depth_for_room_txn(txn, room_id, depth)
687
688		def _update_outliers_txn(self, txn, events_and_contexts):
689		"""Update any outliers with new event info.
690
691		This turns outliers into ex-outliers (unless the new event was
692		rejected).
693
694		Args:
695		txn (twisted.enterprise.adbapi.Connection): db connection
696		events_and_contexts (list[(EventBase, EventContext)]): events
697		we are persisting
698
699		Returns:
700		list[(EventBase, EventContext)] new list, without events which
701		are already in the events table.
702		"""
703		txn.execute(
704		"SELECT event_id, outlier FROM events WHERE event_id in (%s)"
705		% (",".join(["?"] * len(events_and_contexts)),),
706		[event.event_id for event, _ in events_and_contexts],
707		)
708
709		have_persisted = {event_id: outlier for event_id, outlier in txn}
710
711		to_remove = set()
712		for event, context in events_and_contexts:
713		if event.event_id not in have_persisted:
714		continue
715
716		to_remove.add(event)
717
718		if context.rejected:
719		# If the event is rejected then we don't care if the event
720		# was an outlier or not.
721		continue
722
723		outlier_persisted = have_persisted[event.event_id]
724		if not event.internal_metadata.is_outlier() and outlier_persisted:
725		# We received a copy of an event that we had already stored as
726		# an outlier in the database. We now have some state at that
727		# so we need to update the state_groups table with that state.
728
729		# insert into event_to_state_groups.
730		try:
731		self._store_event_state_mappings_txn(txn, ((event, context),))
732		except Exception:
733		logger.exception("")
734		raise
735
736		metadata_json = encode_json(event.internal_metadata.get_dict())
737
738		sql = "UPDATE event_json SET internal_metadata = ? WHERE event_id = ?"
739		txn.execute(sql, (metadata_json, event.event_id))
740
741		# Add an entry to the ex_outlier_stream table to replicate the
742		# change in outlier status to our workers.
743		stream_order = event.internal_metadata.stream_ordering
744		state_group_id = context.state_group
745		self.db.simple_insert_txn(
746		txn,
747		table="ex_outlier_stream",
748		values={
749		"event_stream_ordering": stream_order,
750		"event_id": event.event_id,
751		"state_group": state_group_id,
752		},
753		)
754
755		sql = "UPDATE events SET outlier = ? WHERE event_id = ?"
756		txn.execute(sql, (False, event.event_id))
757
758		# Update the event_backward_extremities table now that this
759		# event isn't an outlier any more.
760		self._update_backward_extremeties(txn, [event])
761
762		return [ec for ec in events_and_contexts if ec[0] not in to_remove]
763
764		def _store_event_txn(self, txn, events_and_contexts):
765		"""Insert new events into the event and event_json tables
766
767		Args:
768		txn (twisted.enterprise.adbapi.Connection): db connection
769		events_and_contexts (list[(EventBase, EventContext)]): events
770		we are persisting
771		"""
772
773		if not events_and_contexts:
774		# nothing to do here
775		return
776
777		def event_dict(event):
778		d = event.get_dict()
779		d.pop("redacted", None)
780		d.pop("redacted_because", None)
781		return d
782
783		self.db.simple_insert_many_txn(
784		txn,
785		table="event_json",
786		values=[
787		{
788		"event_id": event.event_id,
789		"room_id": event.room_id,
790		"internal_metadata": encode_json(
791		event.internal_metadata.get_dict()
792		),
793		"json": encode_json(event_dict(event)),
794		"format_version": event.format_version,
795		}
796		for event, _ in events_and_contexts
797		],
798		)
799
800		self.db.simple_insert_many_txn(
801		txn,
802		table="events",
803		values=[
804		{
805		"stream_ordering": event.internal_metadata.stream_ordering,
806		"topological_ordering": event.depth,
807		"depth": event.depth,
808		"event_id": event.event_id,
809		"room_id": event.room_id,
810		"type": event.type,
811		"processed": True,
812		"outlier": event.internal_metadata.is_outlier(),
813		"origin_server_ts": int(event.origin_server_ts),
814		"received_ts": self._clock.time_msec(),
815		"sender": event.sender,
816		"contains_url": (
817		"url" in event.content and isinstance(event.content["url"], str)
818		),
819		}
820		for event, _ in events_and_contexts
821		],
822		)
823
824		for event, _ in events_and_contexts:
825		if not event.internal_metadata.is_redacted():
826		# If we're persisting an unredacted event we go and ensure
827		# that we mark any redactions that reference this event as
828		# requiring censoring.
829		self.db.simple_update_txn(
830		txn,
831		table="redactions",
832		keyvalues={"redacts": event.event_id},
833		updatevalues={"have_censored": False},
834		)
835
836		def _store_rejected_events_txn(self, txn, events_and_contexts):
837		"""Add rows to the 'rejections' table for received events which were
838		rejected
839
840		Args:
841		txn (twisted.enterprise.adbapi.Connection): db connection
842		events_and_contexts (list[(EventBase, EventContext)]): events
843		we are persisting
844
845		Returns:
846		list[(EventBase, EventContext)] new list, without the rejected
847		events.
848		"""
849		# Remove the rejected events from the list now that we've added them
850		# to the events table and the events_json table.
851		to_remove = set()
852		for event, context in events_and_contexts:
853		if context.rejected:
854		# Insert the event_id into the rejections table
855		self._store_rejections_txn(txn, event.event_id, context.rejected)
856		to_remove.add(event)
857
858		return [ec for ec in events_and_contexts if ec[0] not in to_remove]
859
860		def _update_metadata_tables_txn(
861		self, txn, events_and_contexts, all_events_and_contexts, backfilled
862		):
863		"""Update all the miscellaneous tables for new events
864
865		Args:
866		txn (twisted.enterprise.adbapi.Connection): db connection
867		events_and_contexts (list[(EventBase, EventContext)]): events
868		we are persisting
869		all_events_and_contexts (list[(EventBase, EventContext)]): all
870		events that we were going to persist. This includes events
871		we've already persisted, etc, that wouldn't appear in
872		events_and_context.
873		backfilled (bool): True if the events were backfilled
874		"""
875
876		# Insert all the push actions into the event_push_actions table.
877		self._set_push_actions_for_event_and_users_txn(
878		txn,
879		events_and_contexts=events_and_contexts,
880		all_events_and_contexts=all_events_and_contexts,
881		)
882
883		if not events_and_contexts:
884		# nothing to do here
885		return
886
887		for event, context in events_and_contexts:
888		if event.type == EventTypes.Redaction and event.redacts is not None:
889		# Remove the entries in the event_push_actions table for the
890		# redacted event.
891		self._remove_push_actions_for_event_id_txn(
892		txn, event.room_id, event.redacts
893		)
894
895		# Remove from relations table.
896		self._handle_redaction(txn, event.redacts)
897
898		# Update the event_forward_extremities, event_backward_extremities and
899		# event_edges tables.
900		self._handle_mult_prev_events(
901		txn, events=[event for event, _ in events_and_contexts]
902		)
903
904		for event, _ in events_and_contexts:
905		if event.type == EventTypes.Name:
906		# Insert into the event_search table.
907		self._store_room_name_txn(txn, event)
908		elif event.type == EventTypes.Topic:
909		# Insert into the event_search table.
910		self._store_room_topic_txn(txn, event)
911		elif event.type == EventTypes.Message:
912		# Insert into the event_search table.
913		self._store_room_message_txn(txn, event)
914		elif event.type == EventTypes.Redaction and event.redacts is not None:
915		# Insert into the redactions table.
916		self._store_redaction(txn, event)
917		elif event.type == EventTypes.Retention:
918		# Update the room_retention table.
919		self._store_retention_policy_for_room_txn(txn, event)
920
921		self._handle_event_relations(txn, event)
922
923		# Store the labels for this event.
924		labels = event.content.get(EventContentFields.LABELS)
925		if labels:
926		self.insert_labels_for_event_txn(
927		txn, event.event_id, labels, event.room_id, event.depth
928		)
929
930		if self._ephemeral_messages_enabled:
931		# If there's an expiry timestamp on the event, store it.
932		expiry_ts = event.content.get(EventContentFields.SELF_DESTRUCT_AFTER)
933		if isinstance(expiry_ts, int) and not event.is_state():
934		self._insert_event_expiry_txn(txn, event.event_id, expiry_ts)
935
936		# Insert into the room_memberships table.
937		self._store_room_members_txn(
938		txn,
939		[
940		event
941		for event, _ in events_and_contexts
942		if event.type == EventTypes.Member
943		],
944		backfilled=backfilled,
945		)
946
947		# Insert event_reference_hashes table.
948		self._store_event_reference_hashes_txn(
949		txn, [event for event, _ in events_and_contexts]
950		)
951
952		state_events_and_contexts = [
953		ec for ec in events_and_contexts if ec[0].is_state()
954		]
955
956		state_values = []
957		for event, context in state_events_and_contexts:
958		vals = {
959		"event_id": event.event_id,
960		"room_id": event.room_id,
961		"type": event.type,
962		"state_key": event.state_key,
963		}
964
965		# TODO: How does this work with backfilling?
966		if hasattr(event, "replaces_state"):
967		vals["prev_state"] = event.replaces_state
968
969		state_values.append(vals)
970
971		self.db.simple_insert_many_txn(txn, table="state_events", values=state_values)
972
973		# Prefill the event cache
974		self._add_to_cache(txn, events_and_contexts)
975
976		def _add_to_cache(self, txn, events_and_contexts):
977		to_prefill = []
978
979		rows = []
980		N = 200
981		for i in range(0, len(events_and_contexts), N):
982		ev_map = {e[0].event_id: e[0] for e in events_and_contexts[i : i + N]}
983		if not ev_map:
984		break
985
986		sql = (
987		"SELECT "
988		" e.event_id as event_id, "
989		" r.redacts as redacts,"
990		" rej.event_id as rejects "
991		" FROM events as e"
992		" LEFT JOIN rejections as rej USING (event_id)"
993		" LEFT JOIN redactions as r ON e.event_id = r.redacts"
994		" WHERE "
995		)
996
997		clause, args = make_in_list_sql_clause(
998		self.database_engine, "e.event_id", list(ev_map)
999		)
1000
1001		txn.execute(sql + clause, args)
1002		rows = self.db.cursor_to_dict(txn)
1003		for row in rows:
1004		event = ev_map[row["event_id"]]
1005		if not row["rejects"] and not row["redacts"]:
1006		to_prefill.append(
1007		_EventCacheEntry(event=event, redacted_event=None)
1008		)
1009
1010		def prefill():
1011		for cache_entry in to_prefill:
1012		self.store._get_event_cache.prefill(
1013		(cache_entry[0].event_id,), cache_entry
1014		)
1015
1016		txn.call_after(prefill)
1017
1018		def _store_redaction(self, txn, event):
1019		# invalidate the cache for the redacted event
1020		txn.call_after(self.store._invalidate_get_event_cache, event.redacts)
1021
1022		self.db.simple_insert_txn(
1023		txn,
1024		table="redactions",
1025		values={
1026		"event_id": event.event_id,
1027		"redacts": event.redacts,
1028		"received_ts": self._clock.time_msec(),
1029		},
1030		)
1031
1032		def insert_labels_for_event_txn(
1033		self, txn, event_id, labels, room_id, topological_ordering
1034		):
1035		"""Store the mapping between an event's ID and its labels, with one row per
1036		(event_id, label) tuple.
1037
1038		Args:
1039		txn (LoggingTransaction): The transaction to execute.
1040		event_id (str): The event's ID.
1041		labels (list[str]): A list of text labels.
1042		room_id (str): The ID of the room the event was sent to.
1043		topological_ordering (int): The position of the event in the room's topology.
1044		"""
1045		return self.db.simple_insert_many_txn(
1046		txn=txn,
1047		table="event_labels",
1048		values=[
1049		{
1050		"event_id": event_id,
1051		"label": label,
1052		"room_id": room_id,
1053		"topological_ordering": topological_ordering,
1054		}
1055		for label in labels
1056		],
1057		)
1058
1059		def _insert_event_expiry_txn(self, txn, event_id, expiry_ts):
1060		"""Save the expiry timestamp associated with a given event ID.
1061
1062		Args:
1063		txn (LoggingTransaction): The database transaction to use.
1064		event_id (str): The event ID the expiry timestamp is associated with.
1065		expiry_ts (int): The timestamp at which to expire (delete) the event.
1066		"""
1067		return self.db.simple_insert_txn(
1068		txn=txn,
1069		table="event_expiry",
1070		values={"event_id": event_id, "expiry_ts": expiry_ts},
1071		)
1072
1073		def _store_event_reference_hashes_txn(self, txn, events):
1074		"""Store a hash for a PDU
1075		Args:
1076		txn (cursor):
1077		events (list): list of Events.
1078		"""
1079
1080		vals = []
1081		for event in events:
1082		ref_alg, ref_hash_bytes = compute_event_reference_hash(event)
1083		vals.append(
1084		{
1085		"event_id": event.event_id,
1086		"algorithm": ref_alg,
1087		"hash": memoryview(ref_hash_bytes),
1088		}
1089		)
1090
1091		self.db.simple_insert_many_txn(txn, table="event_reference_hashes", values=vals)
1092
1093		def _store_room_members_txn(self, txn, events, backfilled):
1094		"""Store a room member in the database.
1095		"""
1096		self.db.simple_insert_many_txn(
1097		txn,
1098		table="room_memberships",
1099		values=[
1100		{
1101		"event_id": event.event_id,
1102		"user_id": event.state_key,
1103		"sender": event.user_id,
1104		"room_id": event.room_id,
1105		"membership": event.membership,
1106		"display_name": event.content.get("displayname", None),
1107		"avatar_url": event.content.get("avatar_url", None),
1108		}
1109		for event in events
1110		],
1111		)
1112
1113		for event in events:
1114		txn.call_after(
1115		self.store._membership_stream_cache.entity_has_changed,
1116		event.state_key,
1117		event.internal_metadata.stream_ordering,
1118		)
1119		txn.call_after(
1120		self.store.get_invited_rooms_for_local_user.invalidate,
1121		(event.state_key,),
1122		)
1123
1124		# We update the local_current_membership table only if the event is
1125		# "current", i.e., its something that has just happened.
1126		#
1127		# This will usually get updated by the `current_state_events` handling,
1128		# unless its an outlier, and an outlier is only "current" if it's an "out of
1129		# band membership", like a remote invite or a rejection of a remote invite.
1130		if (
1131		self.is_mine_id(event.state_key)
1132		and not backfilled
1133		and event.internal_metadata.is_outlier()
1134		and event.internal_metadata.is_out_of_band_membership()
1135		):
1136		self.db.simple_upsert_txn(
1137		txn,
1138		table="local_current_membership",
1139		keyvalues={"room_id": event.room_id, "user_id": event.state_key},
1140		values={
1141		"event_id": event.event_id,
1142		"membership": event.membership,
1143		},
1144		)
1145
1146		def _handle_event_relations(self, txn, event):
1147		"""Handles inserting relation data during peristence of events
1148
1149		Args:
1150		txn
1151		event (EventBase)
1152		"""
1153		relation = event.content.get("m.relates_to")
1154		if not relation:
1155		# No relations
1156		return
1157
1158		rel_type = relation.get("rel_type")
1159		if rel_type not in (
1160		RelationTypes.ANNOTATION,
1161		RelationTypes.REFERENCE,
1162		RelationTypes.REPLACE,
1163		):
1164		# Unknown relation type
1165		return
1166
1167		parent_id = relation.get("event_id")
1168		if not parent_id:
1169		# Invalid relation
1170		return
1171
1172		aggregation_key = relation.get("key")
1173
1174		self.db.simple_insert_txn(
1175		txn,
1176		table="event_relations",
1177		values={
1178		"event_id": event.event_id,
1179		"relates_to_id": parent_id,
1180		"relation_type": rel_type,
1181		"aggregation_key": aggregation_key,
1182		},
1183		)
1184
1185		txn.call_after(self.store.get_relations_for_event.invalidate_many, (parent_id,))
1186		txn.call_after(
1187		self.store.get_aggregation_groups_for_event.invalidate_many, (parent_id,)
1188		)
1189
1190		if rel_type == RelationTypes.REPLACE:
1191		txn.call_after(self.store.get_applicable_edit.invalidate, (parent_id,))
1192
1193		def _handle_redaction(self, txn, redacted_event_id):
1194		"""Handles receiving a redaction and checking whether we need to remove
1195		any redacted relations from the database.
1196
1197		Args:
1198		txn
1199		redacted_event_id (str): The event that was redacted.
1200		"""
1201
1202		self.db.simple_delete_txn(
1203		txn, table="event_relations", keyvalues={"event_id": redacted_event_id}
1204		)
1205
1206		def _store_room_topic_txn(self, txn, event):
1207		if hasattr(event, "content") and "topic" in event.content:
1208		self.store_event_search_txn(
1209		txn, event, "content.topic", event.content["topic"]
1210		)
1211
1212		def _store_room_name_txn(self, txn, event):
1213		if hasattr(event, "content") and "name" in event.content:
1214		self.store_event_search_txn(
1215		txn, event, "content.name", event.content["name"]
1216		)
1217
1218		def _store_room_message_txn(self, txn, event):
1219		if hasattr(event, "content") and "body" in event.content:
1220		self.store_event_search_txn(
1221		txn, event, "content.body", event.content["body"]
1222		)
1223
1224		def _store_retention_policy_for_room_txn(self, txn, event):
1225		if hasattr(event, "content") and (
1226		"min_lifetime" in event.content or "max_lifetime" in event.content
1227		):
1228		if (
1229		"min_lifetime" in event.content
1230		and not isinstance(event.content.get("min_lifetime"), int)
1231		) or (
1232		"max_lifetime" in event.content
1233		and not isinstance(event.content.get("max_lifetime"), int)
1234		):
1235		# Ignore the event if one of the value isn't an integer.
1236		return
1237
1238		self.db.simple_insert_txn(
1239		txn=txn,
1240		table="room_retention",
1241		values={
1242		"room_id": event.room_id,
1243		"event_id": event.event_id,
1244		"min_lifetime": event.content.get("min_lifetime"),
1245		"max_lifetime": event.content.get("max_lifetime"),
1246		},
1247		)
1248
1249		self.store._invalidate_cache_and_stream(
1250		txn, self.store.get_retention_policy_for_room, (event.room_id,)
1251		)
1252
1253		def store_event_search_txn(self, txn, event, key, value):
1254		"""Add event to the search table
1255
1256		Args:
1257		txn (cursor):
1258		event (EventBase):
1259		key (str):
1260		value (str):
1261		"""
1262		self.store.store_search_entries_txn(
1263		txn,
1264		(
1265		SearchEntry(
1266		key=key,
1267		value=value,
1268		event_id=event.event_id,
1269		room_id=event.room_id,
1270		stream_ordering=event.internal_metadata.stream_ordering,
1271		origin_server_ts=event.origin_server_ts,
1272		),
1273		),
1274		)
1275
1276		def _set_push_actions_for_event_and_users_txn(
1277		self, txn, events_and_contexts, all_events_and_contexts
1278		):
1279		"""Handles moving push actions from staging table to main
1280		event_push_actions table for all events in `events_and_contexts`.
1281
1282		Also ensures that all events in `all_events_and_contexts` are removed
1283		from the push action staging area.
1284
1285		Args:
1286		events_and_contexts (list[(EventBase, EventContext)]): events
1287		we are persisting
1288		all_events_and_contexts (list[(EventBase, EventContext)]): all
1289		events that we were going to persist. This includes events
1290		we've already persisted, etc, that wouldn't appear in
1291		events_and_context.
1292		"""
1293
1294		sql = """
1295		INSERT INTO event_push_actions (
1296		room_id, event_id, user_id, actions, stream_ordering,
1297		topological_ordering, notif, highlight
1298		)
1299		SELECT ?, event_id, user_id, actions, ?, ?, notif, highlight
1300		FROM event_push_actions_staging
1301		WHERE event_id = ?
1302		"""
1303
1304		if events_and_contexts:
1305		txn.executemany(
1306		sql,
1307		(
1308		(
1309		event.room_id,
1310		event.internal_metadata.stream_ordering,
1311		event.depth,
1312		event.event_id,
1313		)
1314		for event, _ in events_and_contexts
1315		),
1316		)
1317
1318		for event, _ in events_and_contexts:
1319		user_ids = self.db.simple_select_onecol_txn(
1320		txn,
1321		table="event_push_actions_staging",
1322		keyvalues={"event_id": event.event_id},
1323		retcol="user_id",
1324		)
1325
1326		for uid in user_ids:
1327		txn.call_after(
1328		self.store.get_unread_event_push_actions_by_room_for_user.invalidate_many,
1329		(event.room_id, uid),
1330		)
1331
1332		# Now we delete the staging area for all events that were being
1333		# persisted.
1334		txn.executemany(
1335		"DELETE FROM event_push_actions_staging WHERE event_id = ?",
1336		((event.event_id,) for event, _ in all_events_and_contexts),
1337		)
1338
1339		def _remove_push_actions_for_event_id_txn(self, txn, room_id, event_id):
1340		# Sad that we have to blow away the cache for the whole room here
1341		txn.call_after(
1342		self.store.get_unread_event_push_actions_by_room_for_user.invalidate_many,
1343		(room_id,),
1344		)
1345		txn.execute(
1346		"DELETE FROM event_push_actions WHERE room_id = ? AND event_id = ?",
1347		(room_id, event_id),
1348		)
1349
1350		def _store_rejections_txn(self, txn, event_id, reason):
1351		self.db.simple_insert_txn(
1352		txn,
1353		table="rejections",
1354		values={
1355		"event_id": event_id,
1356		"reason": reason,
1357		"last_check": self._clock.time_msec(),
1358		},
1359		)
1360
1361		def _store_event_state_mappings_txn(
1362		self, txn, events_and_contexts: Iterable[Tuple[EventBase, EventContext]]
1363		):
1364		state_groups = {}
1365		for event, context in events_and_contexts:
1366		if event.internal_metadata.is_outlier():
1367		continue
1368
1369		# if the event was rejected, just give it the same state as its
1370		# predecessor.
1371		if context.rejected:
1372		state_groups[event.event_id] = context.state_group_before_event
1373		continue
1374
1375		state_groups[event.event_id] = context.state_group
1376
1377		self.db.simple_insert_many_txn(
1378		txn,
1379		table="event_to_state_groups",
1380		values=[
1381		{"state_group": state_group_id, "event_id": event_id}
1382		for event_id, state_group_id in state_groups.items()
1383		],
1384		)
1385
1386		for event_id, state_group_id in state_groups.items():
1387		txn.call_after(
1388		self.store._get_state_group_for_event.prefill,
1389		(event_id,),
1390		state_group_id,
1391		)
1392
1393		def _update_min_depth_for_room_txn(self, txn, room_id, depth):
1394		min_depth = self.store._get_min_depth_interaction(txn, room_id)
1395
1396		if min_depth is not None and depth >= min_depth:
1397		return
1398
1399		self.db.simple_upsert_txn(
1400		txn,
1401		table="room_depth",
1402		keyvalues={"room_id": room_id},
1403		values={"min_depth": depth},
1404		)
1405
1406		def _handle_mult_prev_events(self, txn, events):
1407		"""
1408		For the given event, update the event edges table and forward and
1409		backward extremities tables.
1410		"""
1411		self.db.simple_insert_many_txn(
1412		txn,
1413		table="event_edges",
1414		values=[
1415		{
1416		"event_id": ev.event_id,
1417		"prev_event_id": e_id,
1418		"room_id": ev.room_id,
1419		"is_state": False,
1420		}
1421		for ev in events
1422		for e_id in ev.prev_event_ids()
1423		],
1424		)
1425
1426		self._update_backward_extremeties(txn, events)
1427
1428		def _update_backward_extremeties(self, txn, events):
1429		"""Updates the event_backward_extremities tables based on the new/updated
1430		events being persisted.
1431
1432		This is called for new events and for events that were outliers, but
1433		are now being persisted as non-outliers.
1434
1435		Forward extremities are handled when we first start persisting the events.
1436		"""
1437		events_by_room = {}
1438		for ev in events:
1439		events_by_room.setdefault(ev.room_id, []).append(ev)
1440
1441		query = (
1442		"INSERT INTO event_backward_extremities (event_id, room_id)"
1443		" SELECT ?, ? WHERE NOT EXISTS ("
1444		" SELECT 1 FROM event_backward_extremities"
1445		" WHERE event_id = ? AND room_id = ?"
1446		" )"
1447		" AND NOT EXISTS ("
1448		" SELECT 1 FROM events WHERE event_id = ? AND room_id = ? "
1449		" AND outlier = ?"
1450		" )"
1451		)
1452
1453		txn.executemany(
1454		query,
1455		[
1456		(e_id, ev.room_id, e_id, ev.room_id, e_id, ev.room_id, False)
1457		for ev in events
1458		for e_id in ev.prev_event_ids()
1459		if not ev.internal_metadata.is_outlier()
1460		],
1461		)
1462
1463		query = (
1464		"DELETE FROM event_backward_extremities"
1465		" WHERE event_id = ? AND room_id = ?"
1466		)
1467		txn.executemany(
1468		query,
1469		[
1470		(ev.event_id, ev.room_id)
1471		for ev in events
1472		if not ev.internal_metadata.is_outlier()
1473		],
1474		)

+0

-585

~~synapse/storage/data_stores/main/events_bg_updates.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2019 The Matrix.org Foundation C.I.C.
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16
17		from twisted.internet import defer
18
19		from synapse.api.constants import EventContentFields
20		from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
21		from synapse.storage.database import Database
22
23		logger = logging.getLogger(__name__)
24
25
26		class EventsBackgroundUpdatesStore(SQLBaseStore):
27
28		EVENT_ORIGIN_SERVER_TS_NAME = "event_origin_server_ts"
29		EVENT_FIELDS_SENDER_URL_UPDATE_NAME = "event_fields_sender_url"
30		DELETE_SOFT_FAILED_EXTREMITIES = "delete_soft_failed_extremities"
31
32		def __init__(self, database: Database, db_conn, hs):
33		super(EventsBackgroundUpdatesStore, self).__init__(database, db_conn, hs)
34
35		self.db.updates.register_background_update_handler(
36		self.EVENT_ORIGIN_SERVER_TS_NAME, self._background_reindex_origin_server_ts
37		)
38		self.db.updates.register_background_update_handler(
39		self.EVENT_FIELDS_SENDER_URL_UPDATE_NAME,
40		self._background_reindex_fields_sender,
41		)
42
43		self.db.updates.register_background_index_update(
44		"event_contains_url_index",
45		index_name="event_contains_url_index",
46		table="events",
47		columns=["room_id", "topological_ordering", "stream_ordering"],
48		where_clause="contains_url = true AND outlier = false",
49		)
50
51		# an event_id index on event_search is useful for the purge_history
52		# api. Plus it means we get to enforce some integrity with a UNIQUE
53		# clause
54		self.db.updates.register_background_index_update(
55		"event_search_event_id_idx",
56		index_name="event_search_event_id_idx",
57		table="event_search",
58		columns=["event_id"],
59		unique=True,
60		psql_only=True,
61		)
62
63		self.db.updates.register_background_update_handler(
64		self.DELETE_SOFT_FAILED_EXTREMITIES, self._cleanup_extremities_bg_update
65		)
66
67		self.db.updates.register_background_update_handler(
68		"redactions_received_ts", self._redactions_received_ts
69		)
70
71		# This index gets deleted in `event_fix_redactions_bytes` update
72		self.db.updates.register_background_index_update(
73		"event_fix_redactions_bytes_create_index",
74		index_name="redactions_censored_redacts",
75		table="redactions",
76		columns=["redacts"],
77		where_clause="have_censored",
78		)
79
80		self.db.updates.register_background_update_handler(
81		"event_fix_redactions_bytes", self._event_fix_redactions_bytes
82		)
83
84		self.db.updates.register_background_update_handler(
85		"event_store_labels", self._event_store_labels
86		)
87
88		self.db.updates.register_background_index_update(
89		"redactions_have_censored_ts_idx",
90		index_name="redactions_have_censored_ts",
91		table="redactions",
92		columns=["received_ts"],
93		where_clause="NOT have_censored",
94		)
95
96		@defer.inlineCallbacks
97		def _background_reindex_fields_sender(self, progress, batch_size):
98		target_min_stream_id = progress["target_min_stream_id_inclusive"]
99		max_stream_id = progress["max_stream_id_exclusive"]
100		rows_inserted = progress.get("rows_inserted", 0)
101
102		INSERT_CLUMP_SIZE = 1000
103
104		def reindex_txn(txn):
105		sql = (
106		"SELECT stream_ordering, event_id, json FROM events"
107		" INNER JOIN event_json USING (event_id)"
108		" WHERE ? <= stream_ordering AND stream_ordering < ?"
109		" ORDER BY stream_ordering DESC"
110		" LIMIT ?"
111		)
112
113		txn.execute(sql, (target_min_stream_id, max_stream_id, batch_size))
114
115		rows = txn.fetchall()
116		if not rows:
117		return 0
118
119		min_stream_id = rows[-1][0]
120
121		update_rows = []
122		for row in rows:
123		try:
124		event_id = row[1]
125		event_json = db_to_json(row[2])
126		sender = event_json["sender"]
127		content = event_json["content"]
128
129		contains_url = "url" in content
130		if contains_url:
131		contains_url &= isinstance(content["url"], str)
132		except (KeyError, AttributeError):
133		# If the event is missing a necessary field then
134		# skip over it.
135		continue
136
137		update_rows.append((sender, contains_url, event_id))
138
139		sql = "UPDATE events SET sender = ?, contains_url = ? WHERE event_id = ?"
140
141		for index in range(0, len(update_rows), INSERT_CLUMP_SIZE):
142		clump = update_rows[index : index + INSERT_CLUMP_SIZE]
143		txn.executemany(sql, clump)
144
145		progress = {
146		"target_min_stream_id_inclusive": target_min_stream_id,
147		"max_stream_id_exclusive": min_stream_id,
148		"rows_inserted": rows_inserted + len(rows),
149		}
150
151		self.db.updates._background_update_progress_txn(
152		txn, self.EVENT_FIELDS_SENDER_URL_UPDATE_NAME, progress
153		)
154
155		return len(rows)
156
157		result = yield self.db.runInteraction(
158		self.EVENT_FIELDS_SENDER_URL_UPDATE_NAME, reindex_txn
159		)
160
161		if not result:
162		yield self.db.updates._end_background_update(
163		self.EVENT_FIELDS_SENDER_URL_UPDATE_NAME
164		)
165
166		return result
167
168		@defer.inlineCallbacks
169		def _background_reindex_origin_server_ts(self, progress, batch_size):
170		target_min_stream_id = progress["target_min_stream_id_inclusive"]
171		max_stream_id = progress["max_stream_id_exclusive"]
172		rows_inserted = progress.get("rows_inserted", 0)
173
174		INSERT_CLUMP_SIZE = 1000
175
176		def reindex_search_txn(txn):
177		sql = (
178		"SELECT stream_ordering, event_id FROM events"
179		" WHERE ? <= stream_ordering AND stream_ordering < ?"
180		" ORDER BY stream_ordering DESC"
181		" LIMIT ?"
182		)
183
184		txn.execute(sql, (target_min_stream_id, max_stream_id, batch_size))
185
186		rows = txn.fetchall()
187		if not rows:
188		return 0
189
190		min_stream_id = rows[-1][0]
191		event_ids = [row[1] for row in rows]
192
193		rows_to_update = []
194
195		chunks = [event_ids[i : i + 100] for i in range(0, len(event_ids), 100)]
196		for chunk in chunks:
197		ev_rows = self.db.simple_select_many_txn(
198		txn,
199		table="event_json",
200		column="event_id",
201		iterable=chunk,
202		retcols=["event_id", "json"],
203		keyvalues={},
204		)
205
206		for row in ev_rows:
207		event_id = row["event_id"]
208		event_json = db_to_json(row["json"])
209		try:
210		origin_server_ts = event_json["origin_server_ts"]
211		except (KeyError, AttributeError):
212		# If the event is missing a necessary field then
213		# skip over it.
214		continue
215
216		rows_to_update.append((origin_server_ts, event_id))
217
218		sql = "UPDATE events SET origin_server_ts = ? WHERE event_id = ?"
219
220		for index in range(0, len(rows_to_update), INSERT_CLUMP_SIZE):
221		clump = rows_to_update[index : index + INSERT_CLUMP_SIZE]
222		txn.executemany(sql, clump)
223
224		progress = {
225		"target_min_stream_id_inclusive": target_min_stream_id,
226		"max_stream_id_exclusive": min_stream_id,
227		"rows_inserted": rows_inserted + len(rows_to_update),
228		}
229
230		self.db.updates._background_update_progress_txn(
231		txn, self.EVENT_ORIGIN_SERVER_TS_NAME, progress
232		)
233
234		return len(rows_to_update)
235
236		result = yield self.db.runInteraction(
237		self.EVENT_ORIGIN_SERVER_TS_NAME, reindex_search_txn
238		)
239
240		if not result:
241		yield self.db.updates._end_background_update(
242		self.EVENT_ORIGIN_SERVER_TS_NAME
243		)
244
245		return result
246
247		@defer.inlineCallbacks
248		def _cleanup_extremities_bg_update(self, progress, batch_size):
249		"""Background update to clean out extremities that should have been
250		deleted previously.
251
252		Mainly used to deal with the aftermath of #5269.
253		"""
254
255		# This works by first copying all existing forward extremities into the
256		# `_extremities_to_check` table at start up, and then checking each
257		# event in that table whether we have any descendants that are not
258		# soft-failed/rejected. If that is the case then we delete that event
259		# from the forward extremities table.
260		#
261		# For efficiency, we do this in batches by recursively pulling out all
262		# descendants of a batch until we find the non soft-failed/rejected
263		# events, i.e. the set of descendants whose chain of prev events back
264		# to the batch of extremities are all soft-failed or rejected.
265		# Typically, we won't find any such events as extremities will rarely
266		# have any descendants, but if they do then we should delete those
267		# extremities.
268
269		def _cleanup_extremities_bg_update_txn(txn):
270		# The set of extremity event IDs that we're checking this round
271		original_set = set()
272
273		# A dict[str, set[str]] of event ID to their prev events.
274		graph = {}
275
276		# The set of descendants of the original set that are not rejected
277		# nor soft-failed. Ancestors of these events should be removed
278		# from the forward extremities table.
279		non_rejected_leaves = set()
280
281		# Set of event IDs that have been soft failed, and for which we
282		# should check if they have descendants which haven't been soft
283		# failed.
284		soft_failed_events_to_lookup = set()
285
286		# First, we get `batch_size` events from the table, pulling out
287		# their successor events, if any, and the successor events'
288		# rejection status.
289		txn.execute(
290		"""SELECT prev_event_id, event_id, internal_metadata,
291		rejections.event_id IS NOT NULL, events.outlier
292		FROM (
293		SELECT event_id AS prev_event_id
294		FROM _extremities_to_check
295		LIMIT ?
296		) AS f
297		LEFT JOIN event_edges USING (prev_event_id)
298		LEFT JOIN events USING (event_id)
299		LEFT JOIN event_json USING (event_id)
300		LEFT JOIN rejections USING (event_id)
301		""",
302		(batch_size,),
303		)
304
305		for prev_event_id, event_id, metadata, rejected, outlier in txn:
306		original_set.add(prev_event_id)
307
308		if not event_id or outlier:
309		# Common case where the forward extremity doesn't have any
310		# descendants.
311		continue
312
313		graph.setdefault(event_id, set()).add(prev_event_id)
314
315		soft_failed = False
316		if metadata:
317		soft_failed = db_to_json(metadata).get("soft_failed")
318
319		if soft_failed or rejected:
320		soft_failed_events_to_lookup.add(event_id)
321		else:
322		non_rejected_leaves.add(event_id)
323
324		# Now we recursively check all the soft-failed descendants we
325		# found above in the same way, until we have nothing left to
326		# check.
327		while soft_failed_events_to_lookup:
328		# We only want to do 100 at a time, so we split given list
329		# into two.
330		batch = list(soft_failed_events_to_lookup)
331		to_check, to_defer = batch[:100], batch[100:]
332		soft_failed_events_to_lookup = set(to_defer)
333
334		sql = """SELECT prev_event_id, event_id, internal_metadata,
335		rejections.event_id IS NOT NULL
336		FROM event_edges
337		INNER JOIN events USING (event_id)
338		INNER JOIN event_json USING (event_id)
339		LEFT JOIN rejections USING (event_id)
340		WHERE
341		NOT events.outlier
342		AND
343		"""
344		clause, args = make_in_list_sql_clause(
345		self.database_engine, "prev_event_id", to_check
346		)
347		txn.execute(sql + clause, list(args))
348
349		for prev_event_id, event_id, metadata, rejected in txn:
350		if event_id in graph:
351		# Already handled this event previously, but we still
352		# want to record the edge.
353		graph[event_id].add(prev_event_id)
354		continue
355
356		graph[event_id] = {prev_event_id}
357
358		soft_failed = db_to_json(metadata).get("soft_failed")
359		if soft_failed or rejected:
360		soft_failed_events_to_lookup.add(event_id)
361		else:
362		non_rejected_leaves.add(event_id)
363
364		# We have a set of non-soft-failed descendants, so we recurse up
365		# the graph to find all ancestors and add them to the set of event
366		# IDs that we can delete from forward extremities table.
367		to_delete = set()
368		while non_rejected_leaves:
369		event_id = non_rejected_leaves.pop()
370		prev_event_ids = graph.get(event_id, set())
371		non_rejected_leaves.update(prev_event_ids)
372		to_delete.update(prev_event_ids)
373
374		to_delete.intersection_update(original_set)
375
376		deleted = self.db.simple_delete_many_txn(
377		txn=txn,
378		table="event_forward_extremities",
379		column="event_id",
380		iterable=to_delete,
381		keyvalues={},
382		)
383
384		logger.info(
385		"Deleted %d forward extremities of %d checked, to clean up #5269",
386		deleted,
387		len(original_set),
388		)
389
390		if deleted:
391		# We now need to invalidate the caches of these rooms
392		rows = self.db.simple_select_many_txn(
393		txn,
394		table="events",
395		column="event_id",
396		iterable=to_delete,
397		keyvalues={},
398		retcols=("room_id",),
399		)
400		room_ids = {row["room_id"] for row in rows}
401		for room_id in room_ids:
402		txn.call_after(
403		self.get_latest_event_ids_in_room.invalidate, (room_id,)
404		)
405
406		self.db.simple_delete_many_txn(
407		txn=txn,
408		table="_extremities_to_check",
409		column="event_id",
410		iterable=original_set,
411		keyvalues={},
412		)
413
414		return len(original_set)
415
416		num_handled = yield self.db.runInteraction(
417		"_cleanup_extremities_bg_update", _cleanup_extremities_bg_update_txn
418		)
419
420		if not num_handled:
421		yield self.db.updates._end_background_update(
422		self.DELETE_SOFT_FAILED_EXTREMITIES
423		)
424
425		def _drop_table_txn(txn):
426		txn.execute("DROP TABLE _extremities_to_check")
427
428		yield self.db.runInteraction(
429		"_cleanup_extremities_bg_update_drop_table", _drop_table_txn
430		)
431
432		return num_handled
433
434		@defer.inlineCallbacks
435		def _redactions_received_ts(self, progress, batch_size):
436		"""Handles filling out the `received_ts` column in redactions.
437		"""
438		last_event_id = progress.get("last_event_id", "")
439
440		def _redactions_received_ts_txn(txn):
441		# Fetch the set of event IDs that we want to update
442		sql = """
443		SELECT event_id FROM redactions
444		WHERE event_id > ?
445		ORDER BY event_id ASC
446		LIMIT ?
447		"""
448
449		txn.execute(sql, (last_event_id, batch_size))
450
451		rows = txn.fetchall()
452		if not rows:
453		return 0
454
455		(upper_event_id,) = rows[-1]
456
457		# Update the redactions with the received_ts.
458		#
459		# Note: Not all events have an associated received_ts, so we
460		# fallback to using origin_server_ts. If we for some reason don't
461		# have an origin_server_ts, lets just use the current timestamp.
462		#
463		# We don't want to leave it null, as then we'll never try and
464		# censor those redactions.
465		sql = """
466		UPDATE redactions
467		SET received_ts = (
468		SELECT COALESCE(received_ts, origin_server_ts, ?) FROM events
469		WHERE events.event_id = redactions.event_id
470		)
471		WHERE ? <= event_id AND event_id <= ?
472		"""
473
474		txn.execute(sql, (self._clock.time_msec(), last_event_id, upper_event_id))
475
476		self.db.updates._background_update_progress_txn(
477		txn, "redactions_received_ts", {"last_event_id": upper_event_id}
478		)
479
480		return len(rows)
481
482		count = yield self.db.runInteraction(
483		"_redactions_received_ts", _redactions_received_ts_txn
484		)
485
486		if not count:
487		yield self.db.updates._end_background_update("redactions_received_ts")
488
489		return count
490
491		@defer.inlineCallbacks
492		def _event_fix_redactions_bytes(self, progress, batch_size):
493		"""Undoes hex encoded censored redacted event JSON.
494		"""
495
496		def _event_fix_redactions_bytes_txn(txn):
497		# This update is quite fast due to new index.
498		txn.execute(
499		"""
500		UPDATE event_json
501		SET
502		json = convert_from(json::bytea, 'utf8')
503		FROM redactions
504		WHERE
505		redactions.have_censored
506		AND event_json.event_id = redactions.redacts
507		AND json NOT LIKE '{%';
508		"""
509		)
510
511		txn.execute("DROP INDEX redactions_censored_redacts")
512
513		yield self.db.runInteraction(
514		"_event_fix_redactions_bytes", _event_fix_redactions_bytes_txn
515		)
516
517		yield self.db.updates._end_background_update("event_fix_redactions_bytes")
518
519		return 1
520
521		@defer.inlineCallbacks
522		def _event_store_labels(self, progress, batch_size):
523		"""Background update handler which will store labels for existing events."""
524		last_event_id = progress.get("last_event_id", "")
525
526		def _event_store_labels_txn(txn):
527		txn.execute(
528		"""
529		SELECT event_id, json FROM event_json
530		LEFT JOIN event_labels USING (event_id)
531		WHERE event_id > ? AND label IS NULL
532		ORDER BY event_id LIMIT ?
533		""",
534		(last_event_id, batch_size),
535		)
536
537		results = list(txn)
538
539		nbrows = 0
540		last_row_event_id = ""
541		for (event_id, event_json_raw) in results:
542		try:
543		event_json = db_to_json(event_json_raw)
544
545		self.db.simple_insert_many_txn(
546		txn=txn,
547		table="event_labels",
548		values=[
549		{
550		"event_id": event_id,
551		"label": label,
552		"room_id": event_json["room_id"],
553		"topological_ordering": event_json["depth"],
554		}
555		for label in event_json["content"].get(
556		EventContentFields.LABELS, []
557		)
558		if isinstance(label, str)
559		],
560		)
561		except Exception as e:
562		logger.warning(
563		"Unable to load event %s (no labels will be imported): %s",
564		event_id,
565		e,
566		)
567
568		nbrows += 1
569		last_row_event_id = event_id
570
571		self.db.updates._background_update_progress_txn(
572		txn, "event_store_labels", {"last_event_id": last_row_event_id}
573		)
574
575		return nbrows
576
577		num_rows = yield self.db.runInteraction(
578		desc="event_store_labels", func=_event_store_labels_txn
579		)
580
581		if not num_rows:
582		yield self.db.updates._end_background_update("event_store_labels")
583
584		return num_rows

+0

-1370

~~synapse/storage/data_stores/main/events_worker.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2018 New Vector Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		from __future__ import division
16
17		import itertools
18		import logging
19		import threading
20		from collections import namedtuple
21		from typing import List, Optional, Tuple
22
23		from constantly import NamedConstant, Names
24
25		from twisted.internet import defer
26
27		from synapse.api.constants import EventTypes
28		from synapse.api.errors import NotFoundError, SynapseError
29		from synapse.api.room_versions import (
30		KNOWN_ROOM_VERSIONS,
31		EventFormatVersions,
32		RoomVersions,
33		)
34		from synapse.events import make_event_from_dict
35		from synapse.events.utils import prune_event
36		from synapse.logging.context import PreserveLoggingContext, current_context
37		from synapse.metrics.background_process_metrics import run_as_background_process
38		from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
39		from synapse.replication.tcp.streams import BackfillStream
40		from synapse.replication.tcp.streams.events import EventsStream
41		from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
42		from synapse.storage.database import Database
43		from synapse.storage.util.id_generators import StreamIdGenerator
44		from synapse.types import get_domain_from_id
45		from synapse.util.caches.descriptors import Cache, cached, cachedInlineCallbacks
46		from synapse.util.iterutils import batch_iter
47		from synapse.util.metrics import Measure
48
49		logger = logging.getLogger(__name__)
50
51
52		# These values are used in the `enqueus_event` and `_do_fetch` methods to
53		# control how we batch/bulk fetch events from the database.
54		# The values are plucked out of thing air to make initial sync run faster
55		# on jki.re
56		# TODO: Make these configurable.
57		EVENT_QUEUE_THREADS = 3 # Max number of threads that will fetch events
58		EVENT_QUEUE_ITERATIONS = 3 # No. times we block waiting for requests for events
59		EVENT_QUEUE_TIMEOUT_S = 0.1 # Timeout when waiting for requests for events
60
61
62		_EventCacheEntry = namedtuple("_EventCacheEntry", ("event", "redacted_event"))
63
64
65		class EventRedactBehaviour(Names):
66		"""
67		What to do when retrieving a redacted event from the database.
68		"""
69
70		AS_IS = NamedConstant()
71		REDACT = NamedConstant()
72		BLOCK = NamedConstant()
73
74
75		class EventsWorkerStore(SQLBaseStore):
76		def __init__(self, database: Database, db_conn, hs):
77		super(EventsWorkerStore, self).__init__(database, db_conn, hs)
78
79		if hs.config.worker.writers.events == hs.get_instance_name():
80		# We are the process in charge of generating stream ids for events,
81		# so instantiate ID generators based on the database
82		self._stream_id_gen = StreamIdGenerator(
83		db_conn, "events", "stream_ordering",
84		)
85		self._backfill_id_gen = StreamIdGenerator(
86		db_conn,
87		"events",
88		"stream_ordering",
89		step=-1,
90		extra_tables=[("ex_outlier_stream", "event_stream_ordering")],
91		)
92		else:
93		# Another process is in charge of persisting events and generating
94		# stream IDs: rely on the replication streams to let us know which
95		# IDs we can process.
96		self._stream_id_gen = SlavedIdTracker(db_conn, "events", "stream_ordering")
97		self._backfill_id_gen = SlavedIdTracker(
98		db_conn, "events", "stream_ordering", step=-1
99		)
100
101		self._get_event_cache = Cache(
102		"getEvent",
103		keylen=3,
104		max_entries=hs.config.caches.event_cache_size,
105		apply_cache_factor_from_config=False,
106		)
107
108		self._event_fetch_lock = threading.Condition()
109		self._event_fetch_list = []
110		self._event_fetch_ongoing = 0
111
112		def process_replication_rows(self, stream_name, instance_name, token, rows):
113		if stream_name == EventsStream.NAME:
114		self._stream_id_gen.advance(token)
115		elif stream_name == BackfillStream.NAME:
116		self._backfill_id_gen.advance(-token)
117
118		super().process_replication_rows(stream_name, instance_name, token, rows)
119
120		def get_received_ts(self, event_id):
121		"""Get received_ts (when it was persisted) for the event.
122
123		Raises an exception for unknown events.
124
125		Args:
126		event_id (str)
127
128		Returns:
129		Deferred[int\|None]: Timestamp in milliseconds, or None for events
130		that were persisted before received_ts was implemented.
131		"""
132		return self.db.simple_select_one_onecol(
133		table="events",
134		keyvalues={"event_id": event_id},
135		retcol="received_ts",
136		desc="get_received_ts",
137		)
138
139		def get_received_ts_by_stream_pos(self, stream_ordering):
140		"""Given a stream ordering get an approximate timestamp of when it
141		happened.
142
143		This is done by simply taking the received ts of the first event that
144		has a stream ordering greater than or equal to the given stream pos.
145		If none exists returns the current time, on the assumption that it must
146		have happened recently.
147
148		Args:
149		stream_ordering (int)
150
151		Returns:
152		Deferred[int]
153		"""
154
155		def _get_approximate_received_ts_txn(txn):
156		sql = """
157		SELECT received_ts FROM events
158		WHERE stream_ordering >= ?
159		LIMIT 1
160		"""
161
162		txn.execute(sql, (stream_ordering,))
163		row = txn.fetchone()
164		if row and row[0]:
165		ts = row[0]
166		else:
167		ts = self.clock.time_msec()
168
169		return ts
170
171		return self.db.runInteraction(
172		"get_approximate_received_ts", _get_approximate_received_ts_txn
173		)
174
175		@defer.inlineCallbacks
176		def get_event(
177		self,
178		event_id: str,
179		redact_behaviour: EventRedactBehaviour = EventRedactBehaviour.REDACT,
180		get_prev_content: bool = False,
181		allow_rejected: bool = False,
182		allow_none: bool = False,
183		check_room_id: Optional[str] = None,
184		):
185		"""Get an event from the database by event_id.
186
187		Args:
188		event_id: The event_id of the event to fetch
189
190		redact_behaviour: Determine what to do with a redacted event. Possible values:
191		* AS_IS - Return the full event body with no redacted content
192		* REDACT - Return the event but with a redacted body
193		* DISALLOW - Do not return redacted events (behave as per allow_none
194		if the event is redacted)
195
196		get_prev_content: If True and event is a state event,
197		include the previous states content in the unsigned field.
198
199		allow_rejected: If True, return rejected events. Otherwise,
200		behave as per allow_none.
201
202		allow_none: If True, return None if no event found, if
203		False throw a NotFoundError
204
205		check_room_id: if not None, check the room of the found event.
206		If there is a mismatch, behave as per allow_none.
207
208		Returns:
209		Deferred[EventBase\|None]
210		"""
211		if not isinstance(event_id, str):
212		raise TypeError("Invalid event event_id %r" % (event_id,))
213
214		events = yield self.get_events_as_list(
215		[event_id],
216		redact_behaviour=redact_behaviour,
217		get_prev_content=get_prev_content,
218		allow_rejected=allow_rejected,
219		)
220
221		event = events[0] if events else None
222
223		if event is not None and check_room_id is not None:
224		if event.room_id != check_room_id:
225		event = None
226
227		if event is None and not allow_none:
228		raise NotFoundError("Could not find event %s" % (event_id,))
229
230		return event
231
232		@defer.inlineCallbacks
233		def get_events(
234		self,
235		event_ids: List[str],
236		redact_behaviour: EventRedactBehaviour = EventRedactBehaviour.REDACT,
237		get_prev_content: bool = False,
238		allow_rejected: bool = False,
239		):
240		"""Get events from the database
241
242		Args:
243		event_ids: The event_ids of the events to fetch
244
245		redact_behaviour: Determine what to do with a redacted event. Possible
246		values:
247		* AS_IS - Return the full event body with no redacted content
248		* REDACT - Return the event but with a redacted body
249		* DISALLOW - Do not return redacted events (omit them from the response)
250
251		get_prev_content: If True and event is a state event,
252		include the previous states content in the unsigned field.
253
254		allow_rejected: If True, return rejected events. Otherwise,
255		omits rejeted events from the response.
256
257		Returns:
258		Deferred : Dict from event_id to event.
259		"""
260		events = yield self.get_events_as_list(
261		event_ids,
262		redact_behaviour=redact_behaviour,
263		get_prev_content=get_prev_content,
264		allow_rejected=allow_rejected,
265		)
266
267		return {e.event_id: e for e in events}
268
269		@defer.inlineCallbacks
270		def get_events_as_list(
271		self,
272		event_ids: List[str],
273		redact_behaviour: EventRedactBehaviour = EventRedactBehaviour.REDACT,
274		get_prev_content: bool = False,
275		allow_rejected: bool = False,
276		):
277		"""Get events from the database and return in a list in the same order
278		as given by `event_ids` arg.
279
280		Unknown events will be omitted from the response.
281
282		Args:
283		event_ids: The event_ids of the events to fetch
284
285		redact_behaviour: Determine what to do with a redacted event. Possible values:
286		* AS_IS - Return the full event body with no redacted content
287		* REDACT - Return the event but with a redacted body
288		* DISALLOW - Do not return redacted events (omit them from the response)
289
290		get_prev_content: If True and event is a state event,
291		include the previous states content in the unsigned field.
292
293		allow_rejected: If True, return rejected events. Otherwise,
294		omits rejected events from the response.
295
296		Returns:
297		Deferred[list[EventBase]]: List of events fetched from the database. The
298		events are in the same order as `event_ids` arg.
299
300		Note that the returned list may be smaller than the list of event
301		IDs if not all events could be fetched.
302		"""
303
304		if not event_ids:
305		return []
306
307		# there may be duplicates so we cast the list to a set
308		event_entry_map = yield self._get_events_from_cache_or_db(
309		set(event_ids), allow_rejected=allow_rejected
310		)
311
312		events = []
313		for event_id in event_ids:
314		entry = event_entry_map.get(event_id, None)
315		if not entry:
316		continue
317
318		if not allow_rejected:
319		assert not entry.event.rejected_reason, (
320		"rejected event returned from _get_events_from_cache_or_db despite "
321		"allow_rejected=False"
322		)
323
324		# We may not have had the original event when we received a redaction, so
325		# we have to recheck auth now.
326
327		if not allow_rejected and entry.event.type == EventTypes.Redaction:
328		if entry.event.redacts is None:
329		# A redacted redaction doesn't have a `redacts` key, in
330		# which case lets just withhold the event.
331		#
332		# Note: Most of the time if the redactions has been
333		# redacted we still have the un-redacted event in the DB
334		# and so we'll still see the `redacts` key. However, this
335		# isn't always true e.g. if we have censored the event.
336		logger.debug(
337		"Withholding redaction event %s as we don't have redacts key",
338		event_id,
339		)
340		continue
341
342		redacted_event_id = entry.event.redacts
343		event_map = yield self._get_events_from_cache_or_db([redacted_event_id])
344		original_event_entry = event_map.get(redacted_event_id)
345		if not original_event_entry:
346		# we don't have the redacted event (or it was rejected).
347		#
348		# We assume that the redaction isn't authorized for now; if the
349		# redacted event later turns up, the redaction will be re-checked,
350		# and if it is found valid, the original will get redacted before it
351		# is served to the client.
352		logger.debug(
353		"Withholding redaction event %s since we don't (yet) have the "
354		"original %s",
355		event_id,
356		redacted_event_id,
357		)
358		continue
359
360		original_event = original_event_entry.event
361		if original_event.type == EventTypes.Create:
362		# we never serve redactions of Creates to clients.
363		logger.info(
364		"Withholding redaction %s of create event %s",
365		event_id,
366		redacted_event_id,
367		)
368		continue
369
370		if original_event.room_id != entry.event.room_id:
371		logger.info(
372		"Withholding redaction %s of event %s from a different room",
373		event_id,
374		redacted_event_id,
375		)
376		continue
377
378		if entry.event.internal_metadata.need_to_check_redaction():
379		original_domain = get_domain_from_id(original_event.sender)
380		redaction_domain = get_domain_from_id(entry.event.sender)
381		if original_domain != redaction_domain:
382		# the senders don't match, so this is forbidden
383		logger.info(
384		"Withholding redaction %s whose sender domain %s doesn't "
385		"match that of redacted event %s %s",
386		event_id,
387		redaction_domain,
388		redacted_event_id,
389		original_domain,
390		)
391		continue
392
393		# Update the cache to save doing the checks again.
394		entry.event.internal_metadata.recheck_redaction = False
395
396		event = entry.event
397
398		if entry.redacted_event:
399		if redact_behaviour == EventRedactBehaviour.BLOCK:
400		# Skip this event
401		continue
402		elif redact_behaviour == EventRedactBehaviour.REDACT:
403		event = entry.redacted_event
404
405		events.append(event)
406
407		if get_prev_content:
408		if "replaces_state" in event.unsigned:
409		prev = yield self.get_event(
410		event.unsigned["replaces_state"],
411		get_prev_content=False,
412		allow_none=True,
413		)
414		if prev:
415		event.unsigned = dict(event.unsigned)
416		event.unsigned["prev_content"] = prev.content
417		event.unsigned["prev_sender"] = prev.sender
418
419		return events
420
421		@defer.inlineCallbacks
422		def _get_events_from_cache_or_db(self, event_ids, allow_rejected=False):
423		"""Fetch a bunch of events from the cache or the database.
424
425		If events are pulled from the database, they will be cached for future lookups.
426
427		Unknown events are omitted from the response.
428
429		Args:
430
431		event_ids (Iterable[str]): The event_ids of the events to fetch
432
433		allow_rejected (bool): Whether to include rejected events. If False,
434		rejected events are omitted from the response.
435
436		Returns:
437		Deferred[Dict[str, _EventCacheEntry]]:
438		map from event id to result
439		"""
440		event_entry_map = self._get_events_from_cache(
441		event_ids, allow_rejected=allow_rejected
442		)
443
444		missing_events_ids = [e for e in event_ids if e not in event_entry_map]
445
446		if missing_events_ids:
447		log_ctx = current_context()
448		log_ctx.record_event_fetch(len(missing_events_ids))
449
450		# Note that _get_events_from_db is also responsible for turning db rows
451		# into FrozenEvents (via _get_event_from_row), which involves seeing if
452		# the events have been redacted, and if so pulling the redaction event out
453		# of the database to check it.
454		#
455		missing_events = yield self._get_events_from_db(
456		missing_events_ids, allow_rejected=allow_rejected
457		)
458
459		event_entry_map.update(missing_events)
460
461		return event_entry_map
462
463		def _invalidate_get_event_cache(self, event_id):
464		self._get_event_cache.invalidate((event_id,))
465
466		def _get_events_from_cache(self, events, allow_rejected, update_metrics=True):
467		"""Fetch events from the caches
468
469		Args:
470		events (Iterable[str]): list of event_ids to fetch
471		allow_rejected (bool): Whether to return events that were rejected
472		update_metrics (bool): Whether to update the cache hit ratio metrics
473
474		Returns:
475		dict of event_id -> _EventCacheEntry for each event_id in cache. If
476		allow_rejected is `False` then there will still be an entry but it
477		will be `None`
478		"""
479		event_map = {}
480
481		for event_id in events:
482		ret = self._get_event_cache.get(
483		(event_id,), None, update_metrics=update_metrics
484		)
485		if not ret:
486		continue
487
488		if allow_rejected or not ret.event.rejected_reason:
489		event_map[event_id] = ret
490		else:
491		event_map[event_id] = None
492
493		return event_map
494
495		def _do_fetch(self, conn):
496		"""Takes a database connection and waits for requests for events from
497		the _event_fetch_list queue.
498		"""
499		i = 0
500		while True:
501		with self._event_fetch_lock:
502		event_list = self._event_fetch_list
503		self._event_fetch_list = []
504
505		if not event_list:
506		single_threaded = self.database_engine.single_threaded
507		if single_threaded or i > EVENT_QUEUE_ITERATIONS:
508		self._event_fetch_ongoing -= 1
509		return
510		else:
511		self._event_fetch_lock.wait(EVENT_QUEUE_TIMEOUT_S)
512		i += 1
513		continue
514		i = 0
515
516		self._fetch_event_list(conn, event_list)
517
518		def _fetch_event_list(self, conn, event_list):
519		"""Handle a load of requests from the _event_fetch_list queue
520
521		Args:
522		conn (twisted.enterprise.adbapi.Connection): database connection
523
524		event_list (list[Tuple[list[str], Deferred]]):
525		The fetch requests. Each entry consists of a list of event
526		ids to be fetched, and a deferred to be completed once the
527		events have been fetched.
528
529		The deferreds are callbacked with a dictionary mapping from event id
530		to event row. Note that it may well contain additional events that
531		were not part of this request.
532		"""
533		with Measure(self._clock, "_fetch_event_list"):
534		try:
535		events_to_fetch = {
536		event_id for events, _ in event_list for event_id in events
537		}
538
539		row_dict = self.db.new_transaction(
540		conn, "do_fetch", [], [], self._fetch_event_rows, events_to_fetch
541		)
542
543		# We only want to resolve deferreds from the main thread
544		def fire():
545		for _, d in event_list:
546		d.callback(row_dict)
547
548		with PreserveLoggingContext():
549		self.hs.get_reactor().callFromThread(fire)
550		except Exception as e:
551		logger.exception("do_fetch")
552
553		# We only want to resolve deferreds from the main thread
554		def fire(evs, exc):
555		for _, d in evs:
556		if not d.called:
557		with PreserveLoggingContext():
558		d.errback(exc)
559
560		with PreserveLoggingContext():
561		self.hs.get_reactor().callFromThread(fire, event_list, e)
562
563		@defer.inlineCallbacks
564		def _get_events_from_db(self, event_ids, allow_rejected=False):
565		"""Fetch a bunch of events from the database.
566
567		Returned events will be added to the cache for future lookups.
568
569		Unknown events are omitted from the response.
570
571		Args:
572		event_ids (Iterable[str]): The event_ids of the events to fetch
573
574		allow_rejected (bool): Whether to include rejected events. If False,
575		rejected events are omitted from the response.
576
577		Returns:
578		Deferred[Dict[str, _EventCacheEntry]]:
579		map from event id to result. May return extra events which
580		weren't asked for.
581		"""
582		fetched_events = {}
583		events_to_fetch = event_ids
584
585		while events_to_fetch:
586		row_map = yield self._enqueue_events(events_to_fetch)
587
588		# we need to recursively fetch any redactions of those events
589		redaction_ids = set()
590		for event_id in events_to_fetch:
591		row = row_map.get(event_id)
592		fetched_events[event_id] = row
593		if row:
594		redaction_ids.update(row["redactions"])
595
596		events_to_fetch = redaction_ids.difference(fetched_events.keys())
597		if events_to_fetch:
598		logger.debug("Also fetching redaction events %s", events_to_fetch)
599
600		# build a map from event_id to EventBase
601		event_map = {}
602		for event_id, row in fetched_events.items():
603		if not row:
604		continue
605		assert row["event_id"] == event_id
606
607		rejected_reason = row["rejected_reason"]
608
609		if not allow_rejected and rejected_reason:
610		continue
611
612		d = db_to_json(row["json"])
613		internal_metadata = db_to_json(row["internal_metadata"])
614
615		format_version = row["format_version"]
616		if format_version is None:
617		# This means that we stored the event before we had the concept
618		# of a event format version, so it must be a V1 event.
619		format_version = EventFormatVersions.V1
620
621		room_version_id = row["room_version_id"]
622
623		if not room_version_id:
624		# this should only happen for out-of-band membership events
625		if not internal_metadata.get("out_of_band_membership"):
626		logger.warning(
627		"Room %s for event %s is unknown", d["room_id"], event_id
628		)
629		continue
630
631		# take a wild stab at the room version based on the event format
632		if format_version == EventFormatVersions.V1:
633		room_version = RoomVersions.V1
634		elif format_version == EventFormatVersions.V2:
635		room_version = RoomVersions.V3
636		else:
637		room_version = RoomVersions.V5
638		else:
639		room_version = KNOWN_ROOM_VERSIONS.get(room_version_id)
640		if not room_version:
641		logger.warning(
642		"Event %s in room %s has unknown room version %s",
643		event_id,
644		d["room_id"],
645		room_version_id,
646		)
647		continue
648
649		if room_version.event_format != format_version:
650		logger.error(
651		"Event %s in room %s with version %s has wrong format: "
652		"expected %s, was %s",
653		event_id,
654		d["room_id"],
655		room_version_id,
656		room_version.event_format,
657		format_version,
658		)
659		continue
660
661		original_ev = make_event_from_dict(
662		event_dict=d,
663		room_version=room_version,
664		internal_metadata_dict=internal_metadata,
665		rejected_reason=rejected_reason,
666		)
667
668		event_map[event_id] = original_ev
669
670		# finally, we can decide whether each one needs redacting, and build
671		# the cache entries.
672		result_map = {}
673		for event_id, original_ev in event_map.items():
674		redactions = fetched_events[event_id]["redactions"]
675		redacted_event = self._maybe_redact_event_row(
676		original_ev, redactions, event_map
677		)
678
679		cache_entry = _EventCacheEntry(
680		event=original_ev, redacted_event=redacted_event
681		)
682
683		self._get_event_cache.prefill((event_id,), cache_entry)
684		result_map[event_id] = cache_entry
685
686		return result_map
687
688		@defer.inlineCallbacks
689		def _enqueue_events(self, events):
690		"""Fetches events from the database using the _event_fetch_list. This
691		allows batch and bulk fetching of events - it allows us to fetch events
692		without having to create a new transaction for each request for events.
693
694		Args:
695		events (Iterable[str]): events to be fetched.
696
697		Returns:
698		Deferred[Dict[str, Dict]]: map from event id to row data from the database.
699		May contain events that weren't requested.
700		"""
701
702		events_d = defer.Deferred()
703		with self._event_fetch_lock:
704		self._event_fetch_list.append((events, events_d))
705
706		self._event_fetch_lock.notify()
707
708		if self._event_fetch_ongoing < EVENT_QUEUE_THREADS:
709		self._event_fetch_ongoing += 1
710		should_start = True
711		else:
712		should_start = False
713
714		if should_start:
715		run_as_background_process(
716		"fetch_events", self.db.runWithConnection, self._do_fetch
717		)
718
719		logger.debug("Loading %d events: %s", len(events), events)
720		with PreserveLoggingContext():
721		row_map = yield events_d
722		logger.debug("Loaded %d events (%d rows)", len(events), len(row_map))
723
724		return row_map
725
726		def _fetch_event_rows(self, txn, event_ids):
727		"""Fetch event rows from the database
728
729		Events which are not found are omitted from the result.
730
731		The returned per-event dicts contain the following keys:
732
733		* event_id (str)
734
735		* json (str): json-encoded event structure
736
737		* internal_metadata (str): json-encoded internal metadata dict
738
739		* format_version (int\|None): The format of the event. Hopefully one
740		of EventFormatVersions. 'None' means the event predates
741		EventFormatVersions (so the event is format V1).
742
743		* room_version_id (str\|None): The version of the room which contains the event.
744		Hopefully one of RoomVersions.
745
746		Due to historical reasons, there may be a few events in the database which
747		do not have an associated room; in this case None will be returned here.
748
749		* rejected_reason (str\|None): if the event was rejected, the reason
750		why.
751
752		* redactions (List[str]): a list of event-ids which (claim to) redact
753		this event.
754
755		Args:
756		txn (twisted.enterprise.adbapi.Connection):
757		event_ids (Iterable[str]): event IDs to fetch
758
759		Returns:
760		Dict[str, Dict]: a map from event id to event info.
761		"""
762		event_dict = {}
763		for evs in batch_iter(event_ids, 200):
764		sql = """\
765		SELECT
766		e.event_id,
767		e.internal_metadata,
768		e.json,
769		e.format_version,
770		r.room_version,
771		rej.reason
772		FROM event_json as e
773		LEFT JOIN rooms r USING (room_id)
774		LEFT JOIN rejections as rej USING (event_id)
775		WHERE """
776
777		clause, args = make_in_list_sql_clause(
778		txn.database_engine, "e.event_id", evs
779		)
780
781		txn.execute(sql + clause, args)
782
783		for row in txn:
784		event_id = row[0]
785		event_dict[event_id] = {
786		"event_id": event_id,
787		"internal_metadata": row[1],
788		"json": row[2],
789		"format_version": row[3],
790		"room_version_id": row[4],
791		"rejected_reason": row[5],
792		"redactions": [],
793		}
794
795		# check for redactions
796		redactions_sql = "SELECT event_id, redacts FROM redactions WHERE "
797
798		clause, args = make_in_list_sql_clause(txn.database_engine, "redacts", evs)
799
800		txn.execute(redactions_sql + clause, args)
801
802		for (redacter, redacted) in txn:
803		d = event_dict.get(redacted)
804		if d:
805		d["redactions"].append(redacter)
806
807		return event_dict
808
809		def _maybe_redact_event_row(self, original_ev, redactions, event_map):
810		"""Given an event object and a list of possible redacting event ids,
811		determine whether to honour any of those redactions and if so return a redacted
812		event.
813
814		Args:
815		original_ev (EventBase):
816		redactions (iterable[str]): list of event ids of potential redaction events
817		event_map (dict[str, EventBase]): other events which have been fetched, in
818		which we can look up the redaaction events. Map from event id to event.
819
820		Returns:
821		Deferred[EventBase\|None]: if the event should be redacted, a pruned
822		event object. Otherwise, None.
823		"""
824		if original_ev.type == "m.room.create":
825		# we choose to ignore redactions of m.room.create events.
826		return None
827
828		for redaction_id in redactions:
829		redaction_event = event_map.get(redaction_id)
830		if not redaction_event or redaction_event.rejected_reason:
831		# we don't have the redaction event, or the redaction event was not
832		# authorized.
833		logger.debug(
834		"%s was redacted by %s but redaction not found/authed",
835		original_ev.event_id,
836		redaction_id,
837		)
838		continue
839
840		if redaction_event.room_id != original_ev.room_id:
841		logger.debug(
842		"%s was redacted by %s but redaction was in a different room!",
843		original_ev.event_id,
844		redaction_id,
845		)
846		continue
847
848		# Starting in room version v3, some redactions need to be
849		# rechecked if we didn't have the redacted event at the
850		# time, so we recheck on read instead.
851		if redaction_event.internal_metadata.need_to_check_redaction():
852		expected_domain = get_domain_from_id(original_ev.sender)
853		if get_domain_from_id(redaction_event.sender) == expected_domain:
854		# This redaction event is allowed. Mark as not needing a recheck.
855		redaction_event.internal_metadata.recheck_redaction = False
856		else:
857		# Senders don't match, so the event isn't actually redacted
858		logger.debug(
859		"%s was redacted by %s but the senders don't match",
860		original_ev.event_id,
861		redaction_id,
862		)
863		continue
864
865		logger.debug("Redacting %s due to %s", original_ev.event_id, redaction_id)
866
867		# we found a good redaction event. Redact!
868		redacted_event = prune_event(original_ev)
869		redacted_event.unsigned["redacted_by"] = redaction_id
870
871		# It's fine to add the event directly, since get_pdu_json
872		# will serialise this field correctly
873		redacted_event.unsigned["redacted_because"] = redaction_event
874
875		return redacted_event
876
877		# no valid redaction found for this event
878		return None
879
880		@defer.inlineCallbacks
881		def have_events_in_timeline(self, event_ids):
882		"""Given a list of event ids, check if we have already processed and
883		stored them as non outliers.
884		"""
885		rows = yield self.db.simple_select_many_batch(
886		table="events",
887		retcols=("event_id",),
888		column="event_id",
889		iterable=list(event_ids),
890		keyvalues={"outlier": False},
891		desc="have_events_in_timeline",
892		)
893
894		return {r["event_id"] for r in rows}
895
896		@defer.inlineCallbacks
897		def have_seen_events(self, event_ids):
898		"""Given a list of event ids, check if we have already processed them.
899
900		Args:
901		event_ids (iterable[str]):
902
903		Returns:
904		Deferred[set[str]]: The events we have already seen.
905		"""
906		results = set()
907
908		def have_seen_events_txn(txn, chunk):
909		sql = "SELECT event_id FROM events as e WHERE "
910		clause, args = make_in_list_sql_clause(
911		txn.database_engine, "e.event_id", chunk
912		)
913		txn.execute(sql + clause, args)
914		for (event_id,) in txn:
915		results.add(event_id)
916
917		# break the input up into chunks of 100
918		input_iterator = iter(event_ids)
919		for chunk in iter(lambda: list(itertools.islice(input_iterator, 100)), []):
920		yield self.db.runInteraction(
921		"have_seen_events", have_seen_events_txn, chunk
922		)
923		return results
924
925		def _get_total_state_event_counts_txn(self, txn, room_id):
926		"""
927		See get_total_state_event_counts.
928		"""
929		# We join against the events table as that has an index on room_id
930		sql = """
931		SELECT COUNT(*) FROM state_events
932		INNER JOIN events USING (room_id, event_id)
933		WHERE room_id=?
934		"""
935		txn.execute(sql, (room_id,))
936		row = txn.fetchone()
937		return row[0] if row else 0
938
939		def get_total_state_event_counts(self, room_id):
940		"""
941		Gets the total number of state events in a room.
942
943		Args:
944		room_id (str)
945
946		Returns:
947		Deferred[int]
948		"""
949		return self.db.runInteraction(
950		"get_total_state_event_counts",
951		self._get_total_state_event_counts_txn,
952		room_id,
953		)
954
955		def _get_current_state_event_counts_txn(self, txn, room_id):
956		"""
957		See get_current_state_event_counts.
958		"""
959		sql = "SELECT COUNT(*) FROM current_state_events WHERE room_id=?"
960		txn.execute(sql, (room_id,))
961		row = txn.fetchone()
962		return row[0] if row else 0
963
964		def get_current_state_event_counts(self, room_id):
965		"""
966		Gets the current number of state events in a room.
967
968		Args:
969		room_id (str)
970
971		Returns:
972		Deferred[int]
973		"""
974		return self.db.runInteraction(
975		"get_current_state_event_counts",
976		self._get_current_state_event_counts_txn,
977		room_id,
978		)
979
980		@defer.inlineCallbacks
981		def get_room_complexity(self, room_id):
982		"""
983		Get a rough approximation of the complexity of the room. This is used by
984		remote servers to decide whether they wish to join the room or not.
985		Higher complexity value indicates that being in the room will consume
986		more resources.
987
988		Args:
989		room_id (str)
990
991		Returns:
992		Deferred[dict[str:int]] of complexity version to complexity.
993		"""
994		state_events = yield self.get_current_state_event_counts(room_id)
995
996		# Call this one "v1", so we can introduce new ones as we want to develop
997		# it.
998		complexity_v1 = round(state_events / 500, 2)
999
1000		return {"v1": complexity_v1}
1001
1002		def get_current_backfill_token(self):
1003		"""The current minimum token that backfilled events have reached"""
1004		return -self._backfill_id_gen.get_current_token()
1005
1006		def get_current_events_token(self):
1007		"""The current maximum token that events have reached"""
1008		return self._stream_id_gen.get_current_token()
1009
1010		def get_all_new_forward_event_rows(self, last_id, current_id, limit):
1011		"""Returns new events, for the Events replication stream
1012
1013		Args:
1014		last_id: the last stream_id from the previous batch.
1015		current_id: the maximum stream_id to return up to
1016		limit: the maximum number of rows to return
1017
1018		Returns: Deferred[List[Tuple]]
1019		a list of events stream rows. Each tuple consists of a stream id as
1020		the first element, followed by fields suitable for casting into an
1021		EventsStreamRow.
1022		"""
1023
1024		def get_all_new_forward_event_rows(txn):
1025		sql = (
1026		"SELECT e.stream_ordering, e.event_id, e.room_id, e.type,"
1027		" state_key, redacts, relates_to_id"
1028		" FROM events AS e"
1029		" LEFT JOIN redactions USING (event_id)"
1030		" LEFT JOIN state_events USING (event_id)"
1031		" LEFT JOIN event_relations USING (event_id)"
1032		" WHERE ? < stream_ordering AND stream_ordering <= ?"
1033		" ORDER BY stream_ordering ASC"
1034		" LIMIT ?"
1035		)
1036		txn.execute(sql, (last_id, current_id, limit))
1037		return txn.fetchall()
1038
1039		return self.db.runInteraction(
1040		"get_all_new_forward_event_rows", get_all_new_forward_event_rows
1041		)
1042
1043		def get_ex_outlier_stream_rows(self, last_id, current_id):
1044		"""Returns de-outliered events, for the Events replication stream
1045
1046		Args:
1047		last_id: the last stream_id from the previous batch.
1048		current_id: the maximum stream_id to return up to
1049
1050		Returns: Deferred[List[Tuple]]
1051		a list of events stream rows. Each tuple consists of a stream id as
1052		the first element, followed by fields suitable for casting into an
1053		EventsStreamRow.
1054		"""
1055
1056		def get_ex_outlier_stream_rows_txn(txn):
1057		sql = (
1058		"SELECT event_stream_ordering, e.event_id, e.room_id, e.type,"
1059		" state_key, redacts, relates_to_id"
1060		" FROM events AS e"
1061		" INNER JOIN ex_outlier_stream USING (event_id)"
1062		" LEFT JOIN redactions USING (event_id)"
1063		" LEFT JOIN state_events USING (event_id)"
1064		" LEFT JOIN event_relations USING (event_id)"
1065		" WHERE ? < event_stream_ordering"
1066		" AND event_stream_ordering <= ?"
1067		" ORDER BY event_stream_ordering ASC"
1068		)
1069
1070		txn.execute(sql, (last_id, current_id))
1071		return txn.fetchall()
1072
1073		return self.db.runInteraction(
1074		"get_ex_outlier_stream_rows", get_ex_outlier_stream_rows_txn
1075		)
1076
1077		async def get_all_new_backfill_event_rows(
1078		self, instance_name: str, last_id: int, current_id: int, limit: int
1079		) -> Tuple[List[Tuple[int, list]], int, bool]:
1080		"""Get updates for backfill replication stream, including all new
1081		backfilled events and events that have gone from being outliers to not.
1082
1083		Args:
1084		instance_name: The writer we want to fetch updates from. Unused
1085		here since there is only ever one writer.
1086		last_id: The token to fetch updates from. Exclusive.
1087		current_id: The token to fetch updates up to. Inclusive.
1088		limit: The requested limit for the number of rows to return. The
1089		function may return more or fewer rows.
1090
1091		Returns:
1092		A tuple consisting of: the updates, a token to use to fetch
1093		subsequent updates, and whether we returned fewer rows than exists
1094		between the requested tokens due to the limit.
1095
1096		The token returned can be used in a subsequent call to this
1097		function to get further updatees.
1098
1099		The updates are a list of 2-tuples of stream ID and the row data
1100		"""
1101		if last_id == current_id:
1102		return [], current_id, False
1103
1104		def get_all_new_backfill_event_rows(txn):
1105		sql = (
1106		"SELECT -e.stream_ordering, e.event_id, e.room_id, e.type,"
1107		" state_key, redacts, relates_to_id"
1108		" FROM events AS e"
1109		" LEFT JOIN redactions USING (event_id)"
1110		" LEFT JOIN state_events USING (event_id)"
1111		" LEFT JOIN event_relations USING (event_id)"
1112		" WHERE ? > stream_ordering AND stream_ordering >= ?"
1113		" ORDER BY stream_ordering ASC"
1114		" LIMIT ?"
1115		)
1116		txn.execute(sql, (-last_id, -current_id, limit))
1117		new_event_updates = [(row[0], row[1:]) for row in txn]
1118
1119		limited = False
1120		if len(new_event_updates) == limit:
1121		upper_bound = new_event_updates[-1][0]
1122		limited = True
1123		else:
1124		upper_bound = current_id
1125
1126		sql = (
1127		"SELECT -event_stream_ordering, e.event_id, e.room_id, e.type,"
1128		" state_key, redacts, relates_to_id"
1129		" FROM events AS e"
1130		" INNER JOIN ex_outlier_stream USING (event_id)"
1131		" LEFT JOIN redactions USING (event_id)"
1132		" LEFT JOIN state_events USING (event_id)"
1133		" LEFT JOIN event_relations USING (event_id)"
1134		" WHERE ? > event_stream_ordering"
1135		" AND event_stream_ordering >= ?"
1136		" ORDER BY event_stream_ordering DESC"
1137		)
1138		txn.execute(sql, (-last_id, -upper_bound))
1139		new_event_updates.extend((row[0], row[1:]) for row in txn)
1140
1141		if len(new_event_updates) >= limit:
1142		upper_bound = new_event_updates[-1][0]
1143		limited = True
1144
1145		return new_event_updates, upper_bound, limited
1146
1147		return await self.db.runInteraction(
1148		"get_all_new_backfill_event_rows", get_all_new_backfill_event_rows
1149		)
1150
1151		async def get_all_updated_current_state_deltas(
1152		self, from_token: int, to_token: int, target_row_count: int
1153		) -> Tuple[List[Tuple], int, bool]:
1154		"""Fetch updates from current_state_delta_stream
1155
1156		Args:
1157		from_token: The previous stream token. Updates from this stream id will
1158		be excluded.
1159
1160		to_token: The current stream token (ie the upper limit). Updates up to this
1161		stream id will be included (modulo the 'limit' param)
1162
1163		target_row_count: The number of rows to try to return. If more rows are
1164		available, we will set 'limited' in the result. In the event of a large
1165		batch, we may return more rows than this.
1166		Returns:
1167		A triplet `(updates, new_last_token, limited)`, where:
1168		* `updates` is a list of database tuples.
1169		* `new_last_token` is the new position in stream.
1170		* `limited` is whether there are more updates to fetch.
1171		"""
1172
1173		def get_all_updated_current_state_deltas_txn(txn):
1174		sql = """
1175		SELECT stream_id, room_id, type, state_key, event_id
1176		FROM current_state_delta_stream
1177		WHERE ? < stream_id AND stream_id <= ?
1178		ORDER BY stream_id ASC LIMIT ?
1179		"""
1180		txn.execute(sql, (from_token, to_token, target_row_count))
1181		return txn.fetchall()
1182
1183		def get_deltas_for_stream_id_txn(txn, stream_id):
1184		sql = """
1185		SELECT stream_id, room_id, type, state_key, event_id
1186		FROM current_state_delta_stream
1187		WHERE stream_id = ?
1188		"""
1189		txn.execute(sql, [stream_id])
1190		return txn.fetchall()
1191
1192		# we need to make sure that, for every stream id in the results, we get all
1193		# the rows with that stream id.
1194
1195		rows = await self.db.runInteraction(
1196		"get_all_updated_current_state_deltas",
1197		get_all_updated_current_state_deltas_txn,
1198		) # type: List[Tuple]
1199
1200		# if we've got fewer rows than the limit, we're good
1201		if len(rows) < target_row_count:
1202		return rows, to_token, False
1203
1204		# we hit the limit, so reduce the upper limit so that we exclude the stream id
1205		# of the last row in the result.
1206		assert rows[-1][0] <= to_token
1207		to_token = rows[-1][0] - 1
1208
1209		# search backwards through the list for the point to truncate
1210		for idx in range(len(rows) - 1, 0, -1):
1211		if rows[idx - 1][0] <= to_token:
1212		return rows[:idx], to_token, True
1213
1214		# bother. We didn't get a full set of changes for even a single
1215		# stream id. let's run the query again, without a row limit, but for
1216		# just one stream id.
1217		to_token += 1
1218		rows = await self.db.runInteraction(
1219		"get_deltas_for_stream_id", get_deltas_for_stream_id_txn, to_token
1220		)
1221
1222		return rows, to_token, True
1223
1224		@cached(num_args=5, max_entries=10)
1225		def get_all_new_events(
1226		self,
1227		last_backfill_id,
1228		last_forward_id,
1229		current_backfill_id,
1230		current_forward_id,
1231		limit,
1232		):
1233		"""Get all the new events that have arrived at the server either as
1234		new events or as backfilled events"""
1235		have_backfill_events = last_backfill_id != current_backfill_id
1236		have_forward_events = last_forward_id != current_forward_id
1237
1238		if not have_backfill_events and not have_forward_events:
1239		return defer.succeed(AllNewEventsResult([], [], [], [], []))
1240
1241		def get_all_new_events_txn(txn):
1242		sql = (
1243		"SELECT e.stream_ordering, e.event_id, e.room_id, e.type,"
1244		" state_key, redacts"
1245		" FROM events AS e"
1246		" LEFT JOIN redactions USING (event_id)"
1247		" LEFT JOIN state_events USING (event_id)"
1248		" WHERE ? < stream_ordering AND stream_ordering <= ?"
1249		" ORDER BY stream_ordering ASC"
1250		" LIMIT ?"
1251		)
1252		if have_forward_events:
1253		txn.execute(sql, (last_forward_id, current_forward_id, limit))
1254		new_forward_events = txn.fetchall()
1255
1256		if len(new_forward_events) == limit:
1257		upper_bound = new_forward_events[-1][0]
1258		else:
1259		upper_bound = current_forward_id
1260
1261		sql = (
1262		"SELECT event_stream_ordering, event_id, state_group"
1263		" FROM ex_outlier_stream"
1264		" WHERE ? > event_stream_ordering"
1265		" AND event_stream_ordering >= ?"
1266		" ORDER BY event_stream_ordering DESC"
1267		)
1268		txn.execute(sql, (last_forward_id, upper_bound))
1269		forward_ex_outliers = txn.fetchall()
1270		else:
1271		new_forward_events = []
1272		forward_ex_outliers = []
1273
1274		sql = (
1275		"SELECT -e.stream_ordering, e.event_id, e.room_id, e.type,"
1276		" state_key, redacts"
1277		" FROM events AS e"
1278		" LEFT JOIN redactions USING (event_id)"
1279		" LEFT JOIN state_events USING (event_id)"
1280		" WHERE ? > stream_ordering AND stream_ordering >= ?"
1281		" ORDER BY stream_ordering DESC"
1282		" LIMIT ?"
1283		)
1284		if have_backfill_events:
1285		txn.execute(sql, (-last_backfill_id, -current_backfill_id, limit))
1286		new_backfill_events = txn.fetchall()
1287
1288		if len(new_backfill_events) == limit:
1289		upper_bound = new_backfill_events[-1][0]
1290		else:
1291		upper_bound = current_backfill_id
1292
1293		sql = (
1294		"SELECT -event_stream_ordering, event_id, state_group"
1295		" FROM ex_outlier_stream"
1296		" WHERE ? > event_stream_ordering"
1297		" AND event_stream_ordering >= ?"
1298		" ORDER BY event_stream_ordering DESC"
1299		)
1300		txn.execute(sql, (-last_backfill_id, -upper_bound))
1301		backward_ex_outliers = txn.fetchall()
1302		else:
1303		new_backfill_events = []
1304		backward_ex_outliers = []
1305
1306		return AllNewEventsResult(
1307		new_forward_events,
1308		new_backfill_events,
1309		forward_ex_outliers,
1310		backward_ex_outliers,
1311		)
1312
1313		return self.db.runInteraction("get_all_new_events", get_all_new_events_txn)
1314
1315		async def is_event_after(self, event_id1, event_id2):
1316		"""Returns True if event_id1 is after event_id2 in the stream
1317		"""
1318		to_1, so_1 = await self.get_event_ordering(event_id1)
1319		to_2, so_2 = await self.get_event_ordering(event_id2)
1320		return (to_1, so_1) > (to_2, so_2)
1321
1322		@cachedInlineCallbacks(max_entries=5000)
1323		def get_event_ordering(self, event_id):
1324		res = yield self.db.simple_select_one(
1325		table="events",
1326		retcols=["topological_ordering", "stream_ordering"],
1327		keyvalues={"event_id": event_id},
1328		allow_none=True,
1329		)
1330
1331		if not res:
1332		raise SynapseError(404, "Could not find event %s" % (event_id,))
1333
1334		return (int(res["topological_ordering"]), int(res["stream_ordering"]))
1335
1336		def get_next_event_to_expire(self):
1337		"""Retrieve the entry with the lowest expiry timestamp in the event_expiry
1338		table, or None if there's no more event to expire.
1339
1340		Returns: Deferred[Optional[Tuple[str, int]]]
1341		A tuple containing the event ID as its first element and an expiry timestamp
1342		as its second one, if there's at least one row in the event_expiry table.
1343		None otherwise.
1344		"""
1345
1346		def get_next_event_to_expire_txn(txn):
1347		txn.execute(
1348		"""
1349		SELECT event_id, expiry_ts FROM event_expiry
1350		ORDER BY expiry_ts ASC LIMIT 1
1351		"""
1352		)
1353
1354		return txn.fetchone()
1355
1356		return self.db.runInteraction(
1357		desc="get_next_event_to_expire", func=get_next_event_to_expire_txn
1358		)
1359
1360
1361		AllNewEventsResult = namedtuple(
1362		"AllNewEventsResult",
1363		[
1364		"new_forward_events",
1365		"new_backfill_events",
1366		"forward_ex_outliers",
1367		"backward_ex_outliers",
1368		],
1369		)

+0

-74

~~synapse/storage/data_stores/main/filtering.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2015, 2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		from canonicaljson import encode_canonical_json
16
17		from synapse.api.errors import Codes, SynapseError
18		from synapse.storage._base import SQLBaseStore, db_to_json
19		from synapse.util.caches.descriptors import cachedInlineCallbacks
20
21
22		class FilteringStore(SQLBaseStore):
23		@cachedInlineCallbacks(num_args=2)
24		def get_user_filter(self, user_localpart, filter_id):
25		# filter_id is BIGINT UNSIGNED, so if it isn't a number, fail
26		# with a coherent error message rather than 500 M_UNKNOWN.
27		try:
28		int(filter_id)
29		except ValueError:
30		raise SynapseError(400, "Invalid filter ID", Codes.INVALID_PARAM)
31
32		def_json = yield self.db.simple_select_one_onecol(
33		table="user_filters",
34		keyvalues={"user_id": user_localpart, "filter_id": filter_id},
35		retcol="filter_json",
36		allow_none=False,
37		desc="get_user_filter",
38		)
39
40		return db_to_json(def_json)
41
42		def add_user_filter(self, user_localpart, user_filter):
43		def_json = encode_canonical_json(user_filter)
44
45		# Need an atomic transaction to SELECT the maximal ID so far then
46		# INSERT a new one
47		def _do_txn(txn):
48		sql = (
49		"SELECT filter_id FROM user_filters "
50		"WHERE user_id = ? AND filter_json = ?"
51		)
52		txn.execute(sql, (user_localpart, bytearray(def_json)))
53		filter_id_response = txn.fetchone()
54		if filter_id_response is not None:
55		return filter_id_response[0]
56
57		sql = "SELECT MAX(filter_id) FROM user_filters WHERE user_id = ?"
58		txn.execute(sql, (user_localpart,))
59		max_id = txn.fetchone()[0]
60		if max_id is None:
61		filter_id = 0
62		else:
63		filter_id = max_id + 1
64
65		sql = (
66		"INSERT INTO user_filters (user_id, filter_id, filter_json)"
67		"VALUES(?, ?, ?)"
68		)
69		txn.execute(sql, (user_localpart, filter_id, bytearray(def_json)))
70
71		return filter_id
72
73		return self.db.runInteraction("add_user_filter", _do_txn)

+0

-1295

~~synapse/storage/data_stores/main/group_server.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2017 Vector Creations Ltd
2		# Copyright 2018 New Vector Ltd
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		from typing import List, Tuple
17
18		from canonicaljson import json
19
20		from twisted.internet import defer
21
22		from synapse.api.errors import SynapseError
23		from synapse.storage._base import SQLBaseStore, db_to_json
24
25		# The category ID for the "default" category. We don't store as null in the
26		# database to avoid the fun of null != null
27		_DEFAULT_CATEGORY_ID = ""
28		_DEFAULT_ROLE_ID = ""
29
30
31		class GroupServerWorkerStore(SQLBaseStore):
32		def get_group(self, group_id):
33		return self.db.simple_select_one(
34		table="groups",
35		keyvalues={"group_id": group_id},
36		retcols=(
37		"name",
38		"short_description",
39		"long_description",
40		"avatar_url",
41		"is_public",
42		"join_policy",
43		),
44		allow_none=True,
45		desc="get_group",
46		)
47
48		def get_users_in_group(self, group_id, include_private=False):
49		# TODO: Pagination
50
51		keyvalues = {"group_id": group_id}
52		if not include_private:
53		keyvalues["is_public"] = True
54
55		return self.db.simple_select_list(
56		table="group_users",
57		keyvalues=keyvalues,
58		retcols=("user_id", "is_public", "is_admin"),
59		desc="get_users_in_group",
60		)
61
62		def get_invited_users_in_group(self, group_id):
63		# TODO: Pagination
64
65		return self.db.simple_select_onecol(
66		table="group_invites",
67		keyvalues={"group_id": group_id},
68		retcol="user_id",
69		desc="get_invited_users_in_group",
70		)
71
72		def get_rooms_in_group(self, group_id: str, include_private: bool = False):
73		"""Retrieve the rooms that belong to a given group. Does not return rooms that
74		lack members.
75
76		Args:
77		group_id: The ID of the group to query for rooms
78		include_private: Whether to return private rooms in results
79
80		Returns:
81		Deferred[List[Dict[str, str\|bool]]]: A list of dictionaries, each in the
82		form of:
83
84		{
85		"room_id": "!a_room_id:example.com", # The ID of the room
86		"is_public": False # Whether this is a public room or not
87		}
88		"""
89		# TODO: Pagination
90
91		def _get_rooms_in_group_txn(txn):
92		sql = """
93		SELECT room_id, is_public FROM group_rooms
94		WHERE group_id = ?
95		AND room_id IN (
96		SELECT group_rooms.room_id FROM group_rooms
97		LEFT JOIN room_stats_current ON
98		group_rooms.room_id = room_stats_current.room_id
99		AND joined_members > 0
100		AND local_users_in_room > 0
101		LEFT JOIN rooms ON
102		group_rooms.room_id = rooms.room_id
103		AND (room_version <> '') = ?
104		)
105		"""
106		args = [group_id, False]
107
108		if not include_private:
109		sql += " AND is_public = ?"
110		args += [True]
111
112		txn.execute(sql, args)
113
114		return [
115		{"room_id": room_id, "is_public": is_public}
116		for room_id, is_public in txn
117		]
118
119		return self.db.runInteraction("get_rooms_in_group", _get_rooms_in_group_txn)
120
121		def get_rooms_for_summary_by_category(
122		self, group_id: str, include_private: bool = False,
123		):
124		"""Get the rooms and categories that should be included in a summary request
125
126		Args:
127		group_id: The ID of the group to query the summary for
128		include_private: Whether to return private rooms in results
129
130		Returns:
131		Deferred[Tuple[List, Dict]]: A tuple containing:
132
133		* A list of dictionaries with the keys:
134		* "room_id": str, the room ID
135		* "is_public": bool, whether the room is public
136		* "category_id": str\|None, the category ID if set, else None
137		* "order": int, the sort order of rooms
138
139		* A dictionary with the key:
140		* category_id (str): a dictionary with the keys:
141		* "is_public": bool, whether the category is public
142		* "profile": str, the category profile
143		* "order": int, the sort order of rooms in this category
144		"""
145
146		def _get_rooms_for_summary_txn(txn):
147		keyvalues = {"group_id": group_id}
148		if not include_private:
149		keyvalues["is_public"] = True
150
151		sql = """
152		SELECT room_id, is_public, category_id, room_order
153		FROM group_summary_rooms
154		WHERE group_id = ?
155		AND room_id IN (
156		SELECT group_rooms.room_id FROM group_rooms
157		LEFT JOIN room_stats_current ON
158		group_rooms.room_id = room_stats_current.room_id
159		AND joined_members > 0
160		AND local_users_in_room > 0
161		LEFT JOIN rooms ON
162		group_rooms.room_id = rooms.room_id
163		AND (room_version <> '') = ?
164		)
165		"""
166
167		if not include_private:
168		sql += " AND is_public = ?"
169		txn.execute(sql, (group_id, False, True))
170		else:
171		txn.execute(sql, (group_id, False))
172
173		rooms = [
174		{
175		"room_id": row[0],
176		"is_public": row[1],
177		"category_id": row[2] if row[2] != _DEFAULT_CATEGORY_ID else None,
178		"order": row[3],
179		}
180		for row in txn
181		]
182
183		sql = """
184		SELECT category_id, is_public, profile, cat_order
185		FROM group_summary_room_categories
186		INNER JOIN group_room_categories USING (group_id, category_id)
187		WHERE group_id = ?
188		"""
189
190		if not include_private:
191		sql += " AND is_public = ?"
192		txn.execute(sql, (group_id, True))
193		else:
194		txn.execute(sql, (group_id,))
195
196		categories = {
197		row[0]: {
198		"is_public": row[1],
199		"profile": db_to_json(row[2]),
200		"order": row[3],
201		}
202		for row in txn
203		}
204
205		return rooms, categories
206
207		return self.db.runInteraction(
208		"get_rooms_for_summary", _get_rooms_for_summary_txn
209		)
210
211		@defer.inlineCallbacks
212		def get_group_categories(self, group_id):
213		rows = yield self.db.simple_select_list(
214		table="group_room_categories",
215		keyvalues={"group_id": group_id},
216		retcols=("category_id", "is_public", "profile"),
217		desc="get_group_categories",
218		)
219
220		return {
221		row["category_id"]: {
222		"is_public": row["is_public"],
223		"profile": db_to_json(row["profile"]),
224		}
225		for row in rows
226		}
227
228		@defer.inlineCallbacks
229		def get_group_category(self, group_id, category_id):
230		category = yield self.db.simple_select_one(
231		table="group_room_categories",
232		keyvalues={"group_id": group_id, "category_id": category_id},
233		retcols=("is_public", "profile"),
234		desc="get_group_category",
235		)
236
237		category["profile"] = db_to_json(category["profile"])
238
239		return category
240
241		@defer.inlineCallbacks
242		def get_group_roles(self, group_id):
243		rows = yield self.db.simple_select_list(
244		table="group_roles",
245		keyvalues={"group_id": group_id},
246		retcols=("role_id", "is_public", "profile"),
247		desc="get_group_roles",
248		)
249
250		return {
251		row["role_id"]: {
252		"is_public": row["is_public"],
253		"profile": db_to_json(row["profile"]),
254		}
255		for row in rows
256		}
257
258		@defer.inlineCallbacks
259		def get_group_role(self, group_id, role_id):
260		role = yield self.db.simple_select_one(
261		table="group_roles",
262		keyvalues={"group_id": group_id, "role_id": role_id},
263		retcols=("is_public", "profile"),
264		desc="get_group_role",
265		)
266
267		role["profile"] = db_to_json(role["profile"])
268
269		return role
270
271		def get_local_groups_for_room(self, room_id):
272		"""Get all of the local group that contain a given room
273		Args:
274		room_id (str): The ID of a room
275		Returns:
276		Deferred[list[str]]: A twisted.Deferred containing a list of group ids
277		containing this room
278		"""
279		return self.db.simple_select_onecol(
280		table="group_rooms",
281		keyvalues={"room_id": room_id},
282		retcol="group_id",
283		desc="get_local_groups_for_room",
284		)
285
286		def get_users_for_summary_by_role(self, group_id, include_private=False):
287		"""Get the users and roles that should be included in a summary request
288
289		Returns ([users], [roles])
290		"""
291
292		def _get_users_for_summary_txn(txn):
293		keyvalues = {"group_id": group_id}
294		if not include_private:
295		keyvalues["is_public"] = True
296
297		sql = """
298		SELECT user_id, is_public, role_id, user_order
299		FROM group_summary_users
300		WHERE group_id = ?
301		"""
302
303		if not include_private:
304		sql += " AND is_public = ?"
305		txn.execute(sql, (group_id, True))
306		else:
307		txn.execute(sql, (group_id,))
308
309		users = [
310		{
311		"user_id": row[0],
312		"is_public": row[1],
313		"role_id": row[2] if row[2] != _DEFAULT_ROLE_ID else None,
314		"order": row[3],
315		}
316		for row in txn
317		]
318
319		sql = """
320		SELECT role_id, is_public, profile, role_order
321		FROM group_summary_roles
322		INNER JOIN group_roles USING (group_id, role_id)
323		WHERE group_id = ?
324		"""
325
326		if not include_private:
327		sql += " AND is_public = ?"
328		txn.execute(sql, (group_id, True))
329		else:
330		txn.execute(sql, (group_id,))
331
332		roles = {
333		row[0]: {
334		"is_public": row[1],
335		"profile": db_to_json(row[2]),
336		"order": row[3],
337		}
338		for row in txn
339		}
340
341		return users, roles
342
343		return self.db.runInteraction(
344		"get_users_for_summary_by_role", _get_users_for_summary_txn
345		)
346
347		def is_user_in_group(self, user_id, group_id):
348		return self.db.simple_select_one_onecol(
349		table="group_users",
350		keyvalues={"group_id": group_id, "user_id": user_id},
351		retcol="user_id",
352		allow_none=True,
353		desc="is_user_in_group",
354		).addCallback(lambda r: bool(r))
355
356		def is_user_admin_in_group(self, group_id, user_id):
357		return self.db.simple_select_one_onecol(
358		table="group_users",
359		keyvalues={"group_id": group_id, "user_id": user_id},
360		retcol="is_admin",
361		allow_none=True,
362		desc="is_user_admin_in_group",
363		)
364
365		def is_user_invited_to_local_group(self, group_id, user_id):
366		"""Has the group server invited a user?
367		"""
368		return self.db.simple_select_one_onecol(
369		table="group_invites",
370		keyvalues={"group_id": group_id, "user_id": user_id},
371		retcol="user_id",
372		desc="is_user_invited_to_local_group",
373		allow_none=True,
374		)
375
376		def get_users_membership_info_in_group(self, group_id, user_id):
377		"""Get a dict describing the membership of a user in a group.
378
379		Example if joined:
380
381		{
382		"membership": "join",
383		"is_public": True,
384		"is_privileged": False,
385		}
386
387		Returns an empty dict if the user is not join/invite/etc
388		"""
389
390		def _get_users_membership_in_group_txn(txn):
391		row = self.db.simple_select_one_txn(
392		txn,
393		table="group_users",
394		keyvalues={"group_id": group_id, "user_id": user_id},
395		retcols=("is_admin", "is_public"),
396		allow_none=True,
397		)
398
399		if row:
400		return {
401		"membership": "join",
402		"is_public": row["is_public"],
403		"is_privileged": row["is_admin"],
404		}
405
406		row = self.db.simple_select_one_onecol_txn(
407		txn,
408		table="group_invites",
409		keyvalues={"group_id": group_id, "user_id": user_id},
410		retcol="user_id",
411		allow_none=True,
412		)
413
414		if row:
415		return {"membership": "invite"}
416
417		return {}
418
419		return self.db.runInteraction(
420		"get_users_membership_info_in_group", _get_users_membership_in_group_txn
421		)
422
423		def get_publicised_groups_for_user(self, user_id):
424		"""Get all groups a user is publicising
425		"""
426		return self.db.simple_select_onecol(
427		table="local_group_membership",
428		keyvalues={"user_id": user_id, "membership": "join", "is_publicised": True},
429		retcol="group_id",
430		desc="get_publicised_groups_for_user",
431		)
432
433		def get_attestations_need_renewals(self, valid_until_ms):
434		"""Get all attestations that need to be renewed until givent time
435		"""
436
437		def _get_attestations_need_renewals_txn(txn):
438		sql = """
439		SELECT group_id, user_id FROM group_attestations_renewals
440		WHERE valid_until_ms <= ?
441		"""
442		txn.execute(sql, (valid_until_ms,))
443		return self.db.cursor_to_dict(txn)
444
445		return self.db.runInteraction(
446		"get_attestations_need_renewals", _get_attestations_need_renewals_txn
447		)
448
449		@defer.inlineCallbacks
450		def get_remote_attestation(self, group_id, user_id):
451		"""Get the attestation that proves the remote agrees that the user is
452		in the group.
453		"""
454		row = yield self.db.simple_select_one(
455		table="group_attestations_remote",
456		keyvalues={"group_id": group_id, "user_id": user_id},
457		retcols=("valid_until_ms", "attestation_json"),
458		desc="get_remote_attestation",
459		allow_none=True,
460		)
461
462		now = int(self._clock.time_msec())
463		if row and now < row["valid_until_ms"]:
464		return db_to_json(row["attestation_json"])
465
466		return None
467
468		def get_joined_groups(self, user_id):
469		return self.db.simple_select_onecol(
470		table="local_group_membership",
471		keyvalues={"user_id": user_id, "membership": "join"},
472		retcol="group_id",
473		desc="get_joined_groups",
474		)
475
476		def get_all_groups_for_user(self, user_id, now_token):
477		def _get_all_groups_for_user_txn(txn):
478		sql = """
479		SELECT group_id, type, membership, u.content
480		FROM local_group_updates AS u
481		INNER JOIN local_group_membership USING (group_id, user_id)
482		WHERE user_id = ? AND membership != 'leave'
483		AND stream_id <= ?
484		"""
485		txn.execute(sql, (user_id, now_token))
486		return [
487		{
488		"group_id": row[0],
489		"type": row[1],
490		"membership": row[2],
491		"content": db_to_json(row[3]),
492		}
493		for row in txn
494		]
495
496		return self.db.runInteraction(
497		"get_all_groups_for_user", _get_all_groups_for_user_txn
498		)
499
500		def get_groups_changes_for_user(self, user_id, from_token, to_token):
501		from_token = int(from_token)
502		has_changed = self._group_updates_stream_cache.has_entity_changed(
503		user_id, from_token
504		)
505		if not has_changed:
506		return defer.succeed([])
507
508		def _get_groups_changes_for_user_txn(txn):
509		sql = """
510		SELECT group_id, membership, type, u.content
511		FROM local_group_updates AS u
512		INNER JOIN local_group_membership USING (group_id, user_id)
513		WHERE user_id = ? AND ? < stream_id AND stream_id <= ?
514		"""
515		txn.execute(sql, (user_id, from_token, to_token))
516		return [
517		{
518		"group_id": group_id,
519		"membership": membership,
520		"type": gtype,
521		"content": db_to_json(content_json),
522		}
523		for group_id, membership, gtype, content_json in txn
524		]
525
526		return self.db.runInteraction(
527		"get_groups_changes_for_user", _get_groups_changes_for_user_txn
528		)
529
530		async def get_all_groups_changes(
531		self, instance_name: str, last_id: int, current_id: int, limit: int
532		) -> Tuple[List[Tuple[int, tuple]], int, bool]:
533		"""Get updates for groups replication stream.
534
535		Args:
536		instance_name: The writer we want to fetch updates from. Unused
537		here since there is only ever one writer.
538		last_id: The token to fetch updates from. Exclusive.
539		current_id: The token to fetch updates up to. Inclusive.
540		limit: The requested limit for the number of rows to return. The
541		function may return more or fewer rows.
542
543		Returns:
544		A tuple consisting of: the updates, a token to use to fetch
545		subsequent updates, and whether we returned fewer rows than exists
546		between the requested tokens due to the limit.
547
548		The token returned can be used in a subsequent call to this
549		function to get further updatees.
550
551		The updates are a list of 2-tuples of stream ID and the row data
552		"""
553
554		last_id = int(last_id)
555		has_changed = self._group_updates_stream_cache.has_any_entity_changed(last_id)
556
557		if not has_changed:
558		return [], current_id, False
559
560		def _get_all_groups_changes_txn(txn):
561		sql = """
562		SELECT stream_id, group_id, user_id, type, content
563		FROM local_group_updates
564		WHERE ? < stream_id AND stream_id <= ?
565		LIMIT ?
566		"""
567		txn.execute(sql, (last_id, current_id, limit))
568		updates = [
569		(stream_id, (group_id, user_id, gtype, db_to_json(content_json)))
570		for stream_id, group_id, user_id, gtype, content_json in txn
571		]
572
573		limited = False
574		upto_token = current_id
575		if len(updates) >= limit:
576		upto_token = updates[-1][0]
577		limited = True
578
579		return updates, upto_token, limited
580
581		return await self.db.runInteraction(
582		"get_all_groups_changes", _get_all_groups_changes_txn
583		)
584
585
586		class GroupServerStore(GroupServerWorkerStore):
587		def set_group_join_policy(self, group_id, join_policy):
588		"""Set the join policy of a group.
589
590		join_policy can be one of:
591		* "invite"
592		* "open"
593		"""
594		return self.db.simple_update_one(
595		table="groups",
596		keyvalues={"group_id": group_id},
597		updatevalues={"join_policy": join_policy},
598		desc="set_group_join_policy",
599		)
600
601		def add_room_to_summary(self, group_id, room_id, category_id, order, is_public):
602		return self.db.runInteraction(
603		"add_room_to_summary",
604		self._add_room_to_summary_txn,
605		group_id,
606		room_id,
607		category_id,
608		order,
609		is_public,
610		)
611
612		def _add_room_to_summary_txn(
613		self, txn, group_id, room_id, category_id, order, is_public
614		):
615		"""Add (or update) room's entry in summary.
616
617		Args:
618		group_id (str)
619		room_id (str)
620		category_id (str): If not None then adds the category to the end of
621		the summary if its not already there. [Optional]
622		order (int): If not None inserts the room at that position, e.g.
623		an order of 1 will put the room first. Otherwise, the room gets
624		added to the end.
625		"""
626		room_in_group = self.db.simple_select_one_onecol_txn(
627		txn,
628		table="group_rooms",
629		keyvalues={"group_id": group_id, "room_id": room_id},
630		retcol="room_id",
631		allow_none=True,
632		)
633		if not room_in_group:
634		raise SynapseError(400, "room not in group")
635
636		if category_id is None:
637		category_id = _DEFAULT_CATEGORY_ID
638		else:
639		cat_exists = self.db.simple_select_one_onecol_txn(
640		txn,
641		table="group_room_categories",
642		keyvalues={"group_id": group_id, "category_id": category_id},
643		retcol="group_id",
644		allow_none=True,
645		)
646		if not cat_exists:
647		raise SynapseError(400, "Category doesn't exist")
648
649		# TODO: Check category is part of summary already
650		cat_exists = self.db.simple_select_one_onecol_txn(
651		txn,
652		table="group_summary_room_categories",
653		keyvalues={"group_id": group_id, "category_id": category_id},
654		retcol="group_id",
655		allow_none=True,
656		)
657		if not cat_exists:
658		# If not, add it with an order larger than all others
659		txn.execute(
660		"""
661		INSERT INTO group_summary_room_categories
662		(group_id, category_id, cat_order)
663		SELECT ?, ?, COALESCE(MAX(cat_order), 0) + 1
664		FROM group_summary_room_categories
665		WHERE group_id = ? AND category_id = ?
666		""",
667		(group_id, category_id, group_id, category_id),
668		)
669
670		existing = self.db.simple_select_one_txn(
671		txn,
672		table="group_summary_rooms",
673		keyvalues={
674		"group_id": group_id,
675		"room_id": room_id,
676		"category_id": category_id,
677		},
678		retcols=("room_order", "is_public"),
679		allow_none=True,
680		)
681
682		if order is not None:
683		# Shuffle other room orders that come after the given order
684		sql = """
685		UPDATE group_summary_rooms SET room_order = room_order + 1
686		WHERE group_id = ? AND category_id = ? AND room_order >= ?
687		"""
688		txn.execute(sql, (group_id, category_id, order))
689		elif not existing:
690		sql = """
691		SELECT COALESCE(MAX(room_order), 0) + 1 FROM group_summary_rooms
692		WHERE group_id = ? AND category_id = ?
693		"""
694		txn.execute(sql, (group_id, category_id))
695		(order,) = txn.fetchone()
696
697		if existing:
698		to_update = {}
699		if order is not None:
700		to_update["room_order"] = order
701		if is_public is not None:
702		to_update["is_public"] = is_public
703		self.db.simple_update_txn(
704		txn,
705		table="group_summary_rooms",
706		keyvalues={
707		"group_id": group_id,
708		"category_id": category_id,
709		"room_id": room_id,
710		},
711		values=to_update,
712		)
713		else:
714		if is_public is None:
715		is_public = True
716
717		self.db.simple_insert_txn(
718		txn,
719		table="group_summary_rooms",
720		values={
721		"group_id": group_id,
722		"category_id": category_id,
723		"room_id": room_id,
724		"room_order": order,
725		"is_public": is_public,
726		},
727		)
728
729		def remove_room_from_summary(self, group_id, room_id, category_id):
730		if category_id is None:
731		category_id = _DEFAULT_CATEGORY_ID
732
733		return self.db.simple_delete(
734		table="group_summary_rooms",
735		keyvalues={
736		"group_id": group_id,
737		"category_id": category_id,
738		"room_id": room_id,
739		},
740		desc="remove_room_from_summary",
741		)
742
743		def upsert_group_category(self, group_id, category_id, profile, is_public):
744		"""Add/update room category for group
745		"""
746		insertion_values = {}
747		update_values = {"category_id": category_id} # This cannot be empty
748
749		if profile is None:
750		insertion_values["profile"] = "{}"
751		else:
752		update_values["profile"] = json.dumps(profile)
753
754		if is_public is None:
755		insertion_values["is_public"] = True
756		else:
757		update_values["is_public"] = is_public
758
759		return self.db.simple_upsert(
760		table="group_room_categories",
761		keyvalues={"group_id": group_id, "category_id": category_id},
762		values=update_values,
763		insertion_values=insertion_values,
764		desc="upsert_group_category",
765		)
766
767		def remove_group_category(self, group_id, category_id):
768		return self.db.simple_delete(
769		table="group_room_categories",
770		keyvalues={"group_id": group_id, "category_id": category_id},
771		desc="remove_group_category",
772		)
773
774		def upsert_group_role(self, group_id, role_id, profile, is_public):
775		"""Add/remove user role
776		"""
777		insertion_values = {}
778		update_values = {"role_id": role_id} # This cannot be empty
779
780		if profile is None:
781		insertion_values["profile"] = "{}"
782		else:
783		update_values["profile"] = json.dumps(profile)
784
785		if is_public is None:
786		insertion_values["is_public"] = True
787		else:
788		update_values["is_public"] = is_public
789
790		return self.db.simple_upsert(
791		table="group_roles",
792		keyvalues={"group_id": group_id, "role_id": role_id},
793		values=update_values,
794		insertion_values=insertion_values,
795		desc="upsert_group_role",
796		)
797
798		def remove_group_role(self, group_id, role_id):
799		return self.db.simple_delete(
800		table="group_roles",
801		keyvalues={"group_id": group_id, "role_id": role_id},
802		desc="remove_group_role",
803		)
804
805		def add_user_to_summary(self, group_id, user_id, role_id, order, is_public):
806		return self.db.runInteraction(
807		"add_user_to_summary",
808		self._add_user_to_summary_txn,
809		group_id,
810		user_id,
811		role_id,
812		order,
813		is_public,
814		)
815
816		def _add_user_to_summary_txn(
817		self, txn, group_id, user_id, role_id, order, is_public
818		):
819		"""Add (or update) user's entry in summary.
820
821		Args:
822		group_id (str)
823		user_id (str)
824		role_id (str): If not None then adds the role to the end of
825		the summary if its not already there. [Optional]
826		order (int): If not None inserts the user at that position, e.g.
827		an order of 1 will put the user first. Otherwise, the user gets
828		added to the end.
829		"""
830		user_in_group = self.db.simple_select_one_onecol_txn(
831		txn,
832		table="group_users",
833		keyvalues={"group_id": group_id, "user_id": user_id},
834		retcol="user_id",
835		allow_none=True,
836		)
837		if not user_in_group:
838		raise SynapseError(400, "user not in group")
839
840		if role_id is None:
841		role_id = _DEFAULT_ROLE_ID
842		else:
843		role_exists = self.db.simple_select_one_onecol_txn(
844		txn,
845		table="group_roles",
846		keyvalues={"group_id": group_id, "role_id": role_id},
847		retcol="group_id",
848		allow_none=True,
849		)
850		if not role_exists:
851		raise SynapseError(400, "Role doesn't exist")
852
853		# TODO: Check role is part of the summary already
854		role_exists = self.db.simple_select_one_onecol_txn(
855		txn,
856		table="group_summary_roles",
857		keyvalues={"group_id": group_id, "role_id": role_id},
858		retcol="group_id",
859		allow_none=True,
860		)
861		if not role_exists:
862		# If not, add it with an order larger than all others
863		txn.execute(
864		"""
865		INSERT INTO group_summary_roles
866		(group_id, role_id, role_order)
867		SELECT ?, ?, COALESCE(MAX(role_order), 0) + 1
868		FROM group_summary_roles
869		WHERE group_id = ? AND role_id = ?
870		""",
871		(group_id, role_id, group_id, role_id),
872		)
873
874		existing = self.db.simple_select_one_txn(
875		txn,
876		table="group_summary_users",
877		keyvalues={"group_id": group_id, "user_id": user_id, "role_id": role_id},
878		retcols=("user_order", "is_public"),
879		allow_none=True,
880		)
881
882		if order is not None:
883		# Shuffle other users orders that come after the given order
884		sql = """
885		UPDATE group_summary_users SET user_order = user_order + 1
886		WHERE group_id = ? AND role_id = ? AND user_order >= ?
887		"""
888		txn.execute(sql, (group_id, role_id, order))
889		elif not existing:
890		sql = """
891		SELECT COALESCE(MAX(user_order), 0) + 1 FROM group_summary_users
892		WHERE group_id = ? AND role_id = ?
893		"""
894		txn.execute(sql, (group_id, role_id))
895		(order,) = txn.fetchone()
896
897		if existing:
898		to_update = {}
899		if order is not None:
900		to_update["user_order"] = order
901		if is_public is not None:
902		to_update["is_public"] = is_public
903		self.db.simple_update_txn(
904		txn,
905		table="group_summary_users",
906		keyvalues={
907		"group_id": group_id,
908		"role_id": role_id,
909		"user_id": user_id,
910		},
911		values=to_update,
912		)
913		else:
914		if is_public is None:
915		is_public = True
916
917		self.db.simple_insert_txn(
918		txn,
919		table="group_summary_users",
920		values={
921		"group_id": group_id,
922		"role_id": role_id,
923		"user_id": user_id,
924		"user_order": order,
925		"is_public": is_public,
926		},
927		)
928
929		def remove_user_from_summary(self, group_id, user_id, role_id):
930		if role_id is None:
931		role_id = _DEFAULT_ROLE_ID
932
933		return self.db.simple_delete(
934		table="group_summary_users",
935		keyvalues={"group_id": group_id, "role_id": role_id, "user_id": user_id},
936		desc="remove_user_from_summary",
937		)
938
939		def add_group_invite(self, group_id, user_id):
940		"""Record that the group server has invited a user
941		"""
942		return self.db.simple_insert(
943		table="group_invites",
944		values={"group_id": group_id, "user_id": user_id},
945		desc="add_group_invite",
946		)
947
948		def add_user_to_group(
949		self,
950		group_id,
951		user_id,
952		is_admin=False,
953		is_public=True,
954		local_attestation=None,
955		remote_attestation=None,
956		):
957		"""Add a user to the group server.
958
959		Args:
960		group_id (str)
961		user_id (str)
962		is_admin (bool)
963		is_public (bool)
964		local_attestation (dict): The attestation the GS created to give
965		to the remote server. Optional if the user and group are on the
966		same server
967		remote_attestation (dict): The attestation given to GS by remote
968		server. Optional if the user and group are on the same server
969		"""
970
971		def _add_user_to_group_txn(txn):
972		self.db.simple_insert_txn(
973		txn,
974		table="group_users",
975		values={
976		"group_id": group_id,
977		"user_id": user_id,
978		"is_admin": is_admin,
979		"is_public": is_public,
980		},
981		)
982
983		self.db.simple_delete_txn(
984		txn,
985		table="group_invites",
986		keyvalues={"group_id": group_id, "user_id": user_id},
987		)
988
989		if local_attestation:
990		self.db.simple_insert_txn(
991		txn,
992		table="group_attestations_renewals",
993		values={
994		"group_id": group_id,
995		"user_id": user_id,
996		"valid_until_ms": local_attestation["valid_until_ms"],
997		},
998		)
999		if remote_attestation:
1000		self.db.simple_insert_txn(
1001		txn,
1002		table="group_attestations_remote",
1003		values={
1004		"group_id": group_id,
1005		"user_id": user_id,
1006		"valid_until_ms": remote_attestation["valid_until_ms"],
1007		"attestation_json": json.dumps(remote_attestation),
1008		},
1009		)
1010
1011		return self.db.runInteraction("add_user_to_group", _add_user_to_group_txn)
1012
1013		def remove_user_from_group(self, group_id, user_id):
1014		def _remove_user_from_group_txn(txn):
1015		self.db.simple_delete_txn(
1016		txn,
1017		table="group_users",
1018		keyvalues={"group_id": group_id, "user_id": user_id},
1019		)
1020		self.db.simple_delete_txn(
1021		txn,
1022		table="group_invites",
1023		keyvalues={"group_id": group_id, "user_id": user_id},
1024		)
1025		self.db.simple_delete_txn(
1026		txn,
1027		table="group_attestations_renewals",
1028		keyvalues={"group_id": group_id, "user_id": user_id},
1029		)
1030		self.db.simple_delete_txn(
1031		txn,
1032		table="group_attestations_remote",
1033		keyvalues={"group_id": group_id, "user_id": user_id},
1034		)
1035		self.db.simple_delete_txn(
1036		txn,
1037		table="group_summary_users",
1038		keyvalues={"group_id": group_id, "user_id": user_id},
1039		)
1040
1041		return self.db.runInteraction(
1042		"remove_user_from_group", _remove_user_from_group_txn
1043		)
1044
1045		def add_room_to_group(self, group_id, room_id, is_public):
1046		return self.db.simple_insert(
1047		table="group_rooms",
1048		values={"group_id": group_id, "room_id": room_id, "is_public": is_public},
1049		desc="add_room_to_group",
1050		)
1051
1052		def update_room_in_group_visibility(self, group_id, room_id, is_public):
1053		return self.db.simple_update(
1054		table="group_rooms",
1055		keyvalues={"group_id": group_id, "room_id": room_id},
1056		updatevalues={"is_public": is_public},
1057		desc="update_room_in_group_visibility",
1058		)
1059
1060		def remove_room_from_group(self, group_id, room_id):
1061		def _remove_room_from_group_txn(txn):
1062		self.db.simple_delete_txn(
1063		txn,
1064		table="group_rooms",
1065		keyvalues={"group_id": group_id, "room_id": room_id},
1066		)
1067
1068		self.db.simple_delete_txn(
1069		txn,
1070		table="group_summary_rooms",
1071		keyvalues={"group_id": group_id, "room_id": room_id},
1072		)
1073
1074		return self.db.runInteraction(
1075		"remove_room_from_group", _remove_room_from_group_txn
1076		)
1077
1078		def update_group_publicity(self, group_id, user_id, publicise):
1079		"""Update whether the user is publicising their membership of the group
1080		"""
1081		return self.db.simple_update_one(
1082		table="local_group_membership",
1083		keyvalues={"group_id": group_id, "user_id": user_id},
1084		updatevalues={"is_publicised": publicise},
1085		desc="update_group_publicity",
1086		)
1087
1088		@defer.inlineCallbacks
1089		def register_user_group_membership(
1090		self,
1091		group_id,
1092		user_id,
1093		membership,
1094		is_admin=False,
1095		content={},
1096		local_attestation=None,
1097		remote_attestation=None,
1098		is_publicised=False,
1099		):
1100		"""Registers that a local user is a member of a (local or remote) group.
1101
1102		Args:
1103		group_id (str)
1104		user_id (str)
1105		membership (str)
1106		is_admin (bool)
1107		content (dict): Content of the membership, e.g. includes the inviter
1108		if the user has been invited.
1109		local_attestation (dict): If remote group then store the fact that we
1110		have given out an attestation, else None.
1111		remote_attestation (dict): If remote group then store the remote
1112		attestation from the group, else None.
1113		"""
1114
1115		def _register_user_group_membership_txn(txn, next_id):
1116		# TODO: Upsert?
1117		self.db.simple_delete_txn(
1118		txn,
1119		table="local_group_membership",
1120		keyvalues={"group_id": group_id, "user_id": user_id},
1121		)
1122		self.db.simple_insert_txn(
1123		txn,
1124		table="local_group_membership",
1125		values={
1126		"group_id": group_id,
1127		"user_id": user_id,
1128		"is_admin": is_admin,
1129		"membership": membership,
1130		"is_publicised": is_publicised,
1131		"content": json.dumps(content),
1132		},
1133		)
1134
1135		self.db.simple_insert_txn(
1136		txn,
1137		table="local_group_updates",
1138		values={
1139		"stream_id": next_id,
1140		"group_id": group_id,
1141		"user_id": user_id,
1142		"type": "membership",
1143		"content": json.dumps(
1144		{"membership": membership, "content": content}
1145		),
1146		},
1147		)
1148		self._group_updates_stream_cache.entity_has_changed(user_id, next_id)
1149
1150		# TODO: Insert profile to ensure it comes down stream if its a join.
1151
1152		if membership == "join":
1153		if local_attestation:
1154		self.db.simple_insert_txn(
1155		txn,
1156		table="group_attestations_renewals",
1157		values={
1158		"group_id": group_id,
1159		"user_id": user_id,
1160		"valid_until_ms": local_attestation["valid_until_ms"],
1161		},
1162		)
1163		if remote_attestation:
1164		self.db.simple_insert_txn(
1165		txn,
1166		table="group_attestations_remote",
1167		values={
1168		"group_id": group_id,
1169		"user_id": user_id,
1170		"valid_until_ms": remote_attestation["valid_until_ms"],
1171		"attestation_json": json.dumps(remote_attestation),
1172		},
1173		)
1174		else:
1175		self.db.simple_delete_txn(
1176		txn,
1177		table="group_attestations_renewals",
1178		keyvalues={"group_id": group_id, "user_id": user_id},
1179		)
1180		self.db.simple_delete_txn(
1181		txn,
1182		table="group_attestations_remote",
1183		keyvalues={"group_id": group_id, "user_id": user_id},
1184		)
1185
1186		return next_id
1187
1188		with self._group_updates_id_gen.get_next() as next_id:
1189		res = yield self.db.runInteraction(
1190		"register_user_group_membership",
1191		_register_user_group_membership_txn,
1192		next_id,
1193		)
1194		return res
1195
1196		@defer.inlineCallbacks
1197		def create_group(
1198		self, group_id, user_id, name, avatar_url, short_description, long_description
1199		):
1200		yield self.db.simple_insert(
1201		table="groups",
1202		values={
1203		"group_id": group_id,
1204		"name": name,
1205		"avatar_url": avatar_url,
1206		"short_description": short_description,
1207		"long_description": long_description,
1208		"is_public": True,
1209		},
1210		desc="create_group",
1211		)
1212
1213		@defer.inlineCallbacks
1214		def update_group_profile(self, group_id, profile):
1215		yield self.db.simple_update_one(
1216		table="groups",
1217		keyvalues={"group_id": group_id},
1218		updatevalues=profile,
1219		desc="update_group_profile",
1220		)
1221
1222		def update_attestation_renewal(self, group_id, user_id, attestation):
1223		"""Update an attestation that we have renewed
1224		"""
1225		return self.db.simple_update_one(
1226		table="group_attestations_renewals",
1227		keyvalues={"group_id": group_id, "user_id": user_id},
1228		updatevalues={"valid_until_ms": attestation["valid_until_ms"]},
1229		desc="update_attestation_renewal",
1230		)
1231
1232		def update_remote_attestion(self, group_id, user_id, attestation):
1233		"""Update an attestation that a remote has renewed
1234		"""
1235		return self.db.simple_update_one(
1236		table="group_attestations_remote",
1237		keyvalues={"group_id": group_id, "user_id": user_id},
1238		updatevalues={
1239		"valid_until_ms": attestation["valid_until_ms"],
1240		"attestation_json": json.dumps(attestation),
1241		},
1242		desc="update_remote_attestion",
1243		)
1244
1245		def remove_attestation_renewal(self, group_id, user_id):
1246		"""Remove an attestation that we thought we should renew, but actually
1247		shouldn't. Ideally this would never get called as we would never
1248		incorrectly try and do attestations for local users on local groups.
1249
1250		Args:
1251		group_id (str)
1252		user_id (str)
1253		"""
1254		return self.db.simple_delete(
1255		table="group_attestations_renewals",
1256		keyvalues={"group_id": group_id, "user_id": user_id},
1257		desc="remove_attestation_renewal",
1258		)
1259
1260		def get_group_stream_token(self):
1261		return self._group_updates_id_gen.get_current_token()
1262
1263		def delete_group(self, group_id):
1264		"""Deletes a group fully from the database.
1265
1266		Args:
1267		group_id (str)
1268
1269		Returns:
1270		Deferred
1271		"""
1272
1273		def _delete_group_txn(txn):
1274		tables = [
1275		"groups",
1276		"group_users",
1277		"group_invites",
1278		"group_rooms",
1279		"group_summary_rooms",
1280		"group_summary_room_categories",
1281		"group_room_categories",
1282		"group_summary_users",
1283		"group_summary_roles",
1284		"group_roles",
1285		"group_attestations_renewals",
1286		"group_attestations_remote",
1287		]
1288
1289		for table in tables:
1290		self.db.simple_delete_txn(
1291		txn, table=table, keyvalues={"group_id": group_id}
1292		)
1293
1294		return self.db.runInteraction("delete_group", _delete_group_txn)

+0

-208

~~synapse/storage/data_stores/main/keys.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2019 New Vector Ltd.
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		import itertools
17		import logging
18
19		from signedjson.key import decode_verify_key_bytes
20
21		from synapse.storage._base import SQLBaseStore
22		from synapse.storage.keys import FetchKeyResult
23		from synapse.util.caches.descriptors import cached, cachedList
24		from synapse.util.iterutils import batch_iter
25
26		logger = logging.getLogger(__name__)
27
28
29		db_binary_type = memoryview
30
31
32		class KeyStore(SQLBaseStore):
33		"""Persistence for signature verification keys
34		"""
35
36		@cached()
37		def _get_server_verify_key(self, server_name_and_key_id):
38		raise NotImplementedError()
39
40		@cachedList(
41		cached_method_name="_get_server_verify_key", list_name="server_name_and_key_ids"
42		)
43		def get_server_verify_keys(self, server_name_and_key_ids):
44		"""
45		Args:
46		server_name_and_key_ids (iterable[Tuple[str, str]]):
47		iterable of (server_name, key-id) tuples to fetch keys for
48
49		Returns:
50		Deferred: resolves to dict[Tuple[str, str], FetchKeyResult\|None]:
51		map from (server_name, key_id) -> FetchKeyResult, or None if the key is
52		unknown
53		"""
54		keys = {}
55
56		def _get_keys(txn, batch):
57		"""Processes a batch of keys to fetch, and adds the result to `keys`."""
58
59		# batch_iter always returns tuples so it's safe to do len(batch)
60		sql = (
61		"SELECT server_name, key_id, verify_key, ts_valid_until_ms "
62		"FROM server_signature_keys WHERE 1=0"
63		) + " OR (server_name=? AND key_id=?)" * len(batch)
64
65		txn.execute(sql, tuple(itertools.chain.from_iterable(batch)))
66
67		for row in txn:
68		server_name, key_id, key_bytes, ts_valid_until_ms = row
69
70		if ts_valid_until_ms is None:
71		# Old keys may be stored with a ts_valid_until_ms of null,
72		# in which case we treat this as if it was set to `0`, i.e.
73		# it won't match key requests that define a minimum
74		# `ts_valid_until_ms`.
75		ts_valid_until_ms = 0
76
77		res = FetchKeyResult(
78		verify_key=decode_verify_key_bytes(key_id, bytes(key_bytes)),
79		valid_until_ts=ts_valid_until_ms,
80		)
81		keys[(server_name, key_id)] = res
82
83		def _txn(txn):
84		for batch in batch_iter(server_name_and_key_ids, 50):
85		_get_keys(txn, batch)
86		return keys
87
88		return self.db.runInteraction("get_server_verify_keys", _txn)
89
90		def store_server_verify_keys(self, from_server, ts_added_ms, verify_keys):
91		"""Stores NACL verification keys for remote servers.
92		Args:
93		from_server (str): Where the verification keys were looked up
94		ts_added_ms (int): The time to record that the key was added
95		verify_keys (iterable[tuple[str, str, FetchKeyResult]]):
96		keys to be stored. Each entry is a triplet of
97		(server_name, key_id, key).
98		"""
99		key_values = []
100		value_values = []
101		invalidations = []
102		for server_name, key_id, fetch_result in verify_keys:
103		key_values.append((server_name, key_id))
104		value_values.append(
105		(
106		from_server,
107		ts_added_ms,
108		fetch_result.valid_until_ts,
109		db_binary_type(fetch_result.verify_key.encode()),
110		)
111		)
112		# invalidate takes a tuple corresponding to the params of
113		# _get_server_verify_key. _get_server_verify_key only takes one
114		# param, which is itself the 2-tuple (server_name, key_id).
115		invalidations.append((server_name, key_id))
116
117		def _invalidate(res):
118		f = self._get_server_verify_key.invalidate
119		for i in invalidations:
120		f((i,))
121		return res
122
123		return self.db.runInteraction(
124		"store_server_verify_keys",
125		self.db.simple_upsert_many_txn,
126		table="server_signature_keys",
127		key_names=("server_name", "key_id"),
128		key_values=key_values,
129		value_names=(
130		"from_server",
131		"ts_added_ms",
132		"ts_valid_until_ms",
133		"verify_key",
134		),
135		value_values=value_values,
136		).addCallback(_invalidate)
137
138		def store_server_keys_json(
139		self, server_name, key_id, from_server, ts_now_ms, ts_expires_ms, key_json_bytes
140		):
141		"""Stores the JSON bytes for a set of keys from a server
142		The JSON should be signed by the originating server, the intermediate
143		server, and by this server. Updates the value for the
144		(server_name, key_id, from_server) triplet if one already existed.
145		Args:
146		server_name (str): The name of the server.
147		key_id (str): The identifer of the key this JSON is for.
148		from_server (str): The server this JSON was fetched from.
149		ts_now_ms (int): The time now in milliseconds.
150		ts_valid_until_ms (int): The time when this json stops being valid.
151		key_json (bytes): The encoded JSON.
152		"""
153		return self.db.simple_upsert(
154		table="server_keys_json",
155		keyvalues={
156		"server_name": server_name,
157		"key_id": key_id,
158		"from_server": from_server,
159		},
160		values={
161		"server_name": server_name,
162		"key_id": key_id,
163		"from_server": from_server,
164		"ts_added_ms": ts_now_ms,
165		"ts_valid_until_ms": ts_expires_ms,
166		"key_json": db_binary_type(key_json_bytes),
167		},
168		desc="store_server_keys_json",
169		)
170
171		def get_server_keys_json(self, server_keys):
172		"""Retrive the key json for a list of server_keys and key ids.
173		If no keys are found for a given server, key_id and source then
174		that server, key_id, and source triplet entry will be an empty list.
175		The JSON is returned as a byte array so that it can be efficiently
176		used in an HTTP response.
177		Args:
178		server_keys (list): List of (server_name, key_id, source) triplets.
179		Returns:
180		Deferred[dict[Tuple[str, str, str\|None], list[dict]]]:
181		Dict mapping (server_name, key_id, source) triplets to lists of dicts
182		"""
183
184		def _get_server_keys_json_txn(txn):
185		results = {}
186		for server_name, key_id, from_server in server_keys:
187		keyvalues = {"server_name": server_name}
188		if key_id is not None:
189		keyvalues["key_id"] = key_id
190		if from_server is not None:
191		keyvalues["from_server"] = from_server
192		rows = self.db.simple_select_list_txn(
193		txn,
194		"server_keys_json",
195		keyvalues=keyvalues,
196		retcols=(
197		"key_id",
198		"from_server",
199		"ts_added_ms",
200		"ts_valid_until_ms",
201		"key_json",
202		),
203		)
204		results[(server_name, key_id, from_server)] = rows
205		return results
206
207		return self.db.runInteraction("get_server_keys_json", _get_server_keys_json_txn)

+0

-394

~~synapse/storage/data_stores/main/media_repository.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14		from synapse.storage._base import SQLBaseStore
15		from synapse.storage.database import Database
16
17
18		class MediaRepositoryBackgroundUpdateStore(SQLBaseStore):
19		def __init__(self, database: Database, db_conn, hs):
20		super(MediaRepositoryBackgroundUpdateStore, self).__init__(
21		database, db_conn, hs
22		)
23
24		self.db.updates.register_background_index_update(
25		update_name="local_media_repository_url_idx",
26		index_name="local_media_repository_url_idx",
27		table="local_media_repository",
28		columns=["created_ts"],
29		where_clause="url_cache IS NOT NULL",
30		)
31
32
33		class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
34		"""Persistence for attachments and avatars"""
35
36		def __init__(self, database: Database, db_conn, hs):
37		super(MediaRepositoryStore, self).__init__(database, db_conn, hs)
38
39		def get_local_media(self, media_id):
40		"""Get the metadata for a local piece of media
41		Returns:
42		None if the media_id doesn't exist.
43		"""
44		return self.db.simple_select_one(
45		"local_media_repository",
46		{"media_id": media_id},
47		(
48		"media_type",
49		"media_length",
50		"upload_name",
51		"created_ts",
52		"quarantined_by",
53		"url_cache",
54		),
55		allow_none=True,
56		desc="get_local_media",
57		)
58
59		def store_local_media(
60		self,
61		media_id,
62		media_type,
63		time_now_ms,
64		upload_name,
65		media_length,
66		user_id,
67		url_cache=None,
68		):
69		return self.db.simple_insert(
70		"local_media_repository",
71		{
72		"media_id": media_id,
73		"media_type": media_type,
74		"created_ts": time_now_ms,
75		"upload_name": upload_name,
76		"media_length": media_length,
77		"user_id": user_id.to_string(),
78		"url_cache": url_cache,
79		},
80		desc="store_local_media",
81		)
82
83		def mark_local_media_as_safe(self, media_id: str):
84		"""Mark a local media as safe from quarantining."""
85		return self.db.simple_update_one(
86		table="local_media_repository",
87		keyvalues={"media_id": media_id},
88		updatevalues={"safe_from_quarantine": True},
89		desc="mark_local_media_as_safe",
90		)
91
92		def get_url_cache(self, url, ts):
93		"""Get the media_id and ts for a cached URL as of the given timestamp
94		Returns:
95		None if the URL isn't cached.
96		"""
97
98		def get_url_cache_txn(txn):
99		# get the most recently cached result (relative to the given ts)
100		sql = (
101		"SELECT response_code, etag, expires_ts, og, media_id, download_ts"
102		" FROM local_media_repository_url_cache"
103		" WHERE url = ? AND download_ts <= ?"
104		" ORDER BY download_ts DESC LIMIT 1"
105		)
106		txn.execute(sql, (url, ts))
107		row = txn.fetchone()
108
109		if not row:
110		# ...or if we've requested a timestamp older than the oldest
111		# copy in the cache, return the oldest copy (if any)
112		sql = (
113		"SELECT response_code, etag, expires_ts, og, media_id, download_ts"
114		" FROM local_media_repository_url_cache"
115		" WHERE url = ? AND download_ts > ?"
116		" ORDER BY download_ts ASC LIMIT 1"
117		)
118		txn.execute(sql, (url, ts))
119		row = txn.fetchone()
120
121		if not row:
122		return None
123
124		return dict(
125		zip(
126		(
127		"response_code",
128		"etag",
129		"expires_ts",
130		"og",
131		"media_id",
132		"download_ts",
133		),
134		row,
135		)
136		)
137
138		return self.db.runInteraction("get_url_cache", get_url_cache_txn)
139
140		def store_url_cache(
141		self, url, response_code, etag, expires_ts, og, media_id, download_ts
142		):
143		return self.db.simple_insert(
144		"local_media_repository_url_cache",
145		{
146		"url": url,
147		"response_code": response_code,
148		"etag": etag,
149		"expires_ts": expires_ts,
150		"og": og,
151		"media_id": media_id,
152		"download_ts": download_ts,
153		},
154		desc="store_url_cache",
155		)
156
157		def get_local_media_thumbnails(self, media_id):
158		return self.db.simple_select_list(
159		"local_media_repository_thumbnails",
160		{"media_id": media_id},
161		(
162		"thumbnail_width",
163		"thumbnail_height",
164		"thumbnail_method",
165		"thumbnail_type",
166		"thumbnail_length",
167		),
168		desc="get_local_media_thumbnails",
169		)
170
171		def store_local_thumbnail(
172		self,
173		media_id,
174		thumbnail_width,
175		thumbnail_height,
176		thumbnail_type,
177		thumbnail_method,
178		thumbnail_length,
179		):
180		return self.db.simple_insert(
181		"local_media_repository_thumbnails",
182		{
183		"media_id": media_id,
184		"thumbnail_width": thumbnail_width,
185		"thumbnail_height": thumbnail_height,
186		"thumbnail_method": thumbnail_method,
187		"thumbnail_type": thumbnail_type,
188		"thumbnail_length": thumbnail_length,
189		},
190		desc="store_local_thumbnail",
191		)
192
193		def get_cached_remote_media(self, origin, media_id):
194		return self.db.simple_select_one(
195		"remote_media_cache",
196		{"media_origin": origin, "media_id": media_id},
197		(
198		"media_type",
199		"media_length",
200		"upload_name",
201		"created_ts",
202		"filesystem_id",
203		"quarantined_by",
204		),
205		allow_none=True,
206		desc="get_cached_remote_media",
207		)
208
209		def store_cached_remote_media(
210		self,
211		origin,
212		media_id,
213		media_type,
214		media_length,
215		time_now_ms,
216		upload_name,
217		filesystem_id,
218		):
219		return self.db.simple_insert(
220		"remote_media_cache",
221		{
222		"media_origin": origin,
223		"media_id": media_id,
224		"media_type": media_type,
225		"media_length": media_length,
226		"created_ts": time_now_ms,
227		"upload_name": upload_name,
228		"filesystem_id": filesystem_id,
229		"last_access_ts": time_now_ms,
230		},
231		desc="store_cached_remote_media",
232		)
233
234		def update_cached_last_access_time(self, local_media, remote_media, time_ms):
235		"""Updates the last access time of the given media
236
237		Args:
238		local_media (iterable[str]): Set of media_ids
239		remote_media (iterable[(str, str)]): Set of (server_name, media_id)
240		time_ms: Current time in milliseconds
241		"""
242
243		def update_cache_txn(txn):
244		sql = (
245		"UPDATE remote_media_cache SET last_access_ts = ?"
246		" WHERE media_origin = ? AND media_id = ?"
247		)
248
249		txn.executemany(
250		sql,
251		(
252		(time_ms, media_origin, media_id)
253		for media_origin, media_id in remote_media
254		),
255		)
256
257		sql = (
258		"UPDATE local_media_repository SET last_access_ts = ?"
259		" WHERE media_id = ?"
260		)
261
262		txn.executemany(sql, ((time_ms, media_id) for media_id in local_media))
263
264		return self.db.runInteraction(
265		"update_cached_last_access_time", update_cache_txn
266		)
267
268		def get_remote_media_thumbnails(self, origin, media_id):
269		return self.db.simple_select_list(
270		"remote_media_cache_thumbnails",
271		{"media_origin": origin, "media_id": media_id},
272		(
273		"thumbnail_width",
274		"thumbnail_height",
275		"thumbnail_method",
276		"thumbnail_type",
277		"thumbnail_length",
278		"filesystem_id",
279		),
280		desc="get_remote_media_thumbnails",
281		)
282
283		def store_remote_media_thumbnail(
284		self,
285		origin,
286		media_id,
287		filesystem_id,
288		thumbnail_width,
289		thumbnail_height,
290		thumbnail_type,
291		thumbnail_method,
292		thumbnail_length,
293		):
294		return self.db.simple_insert(
295		"remote_media_cache_thumbnails",
296		{
297		"media_origin": origin,
298		"media_id": media_id,
299		"thumbnail_width": thumbnail_width,
300		"thumbnail_height": thumbnail_height,
301		"thumbnail_method": thumbnail_method,
302		"thumbnail_type": thumbnail_type,
303		"thumbnail_length": thumbnail_length,
304		"filesystem_id": filesystem_id,
305		},
306		desc="store_remote_media_thumbnail",
307		)
308
309		def get_remote_media_before(self, before_ts):
310		sql = (
311		"SELECT media_origin, media_id, filesystem_id"
312		" FROM remote_media_cache"
313		" WHERE last_access_ts < ?"
314		)
315
316		return self.db.execute(
317		"get_remote_media_before", self.db.cursor_to_dict, sql, before_ts
318		)
319
320		def delete_remote_media(self, media_origin, media_id):
321		def delete_remote_media_txn(txn):
322		self.db.simple_delete_txn(
323		txn,
324		"remote_media_cache",
325		keyvalues={"media_origin": media_origin, "media_id": media_id},
326		)
327		self.db.simple_delete_txn(
328		txn,
329		"remote_media_cache_thumbnails",
330		keyvalues={"media_origin": media_origin, "media_id": media_id},
331		)
332
333		return self.db.runInteraction("delete_remote_media", delete_remote_media_txn)
334
335		def get_expired_url_cache(self, now_ts):
336		sql = (
337		"SELECT media_id FROM local_media_repository_url_cache"
338		" WHERE expires_ts < ?"
339		" ORDER BY expires_ts ASC"
340		" LIMIT 500"
341		)
342
343		def _get_expired_url_cache_txn(txn):
344		txn.execute(sql, (now_ts,))
345		return [row[0] for row in txn]
346
347		return self.db.runInteraction(
348		"get_expired_url_cache", _get_expired_url_cache_txn
349		)
350
351		async def delete_url_cache(self, media_ids):
352		if len(media_ids) == 0:
353		return
354
355		sql = "DELETE FROM local_media_repository_url_cache WHERE media_id = ?"
356
357		def _delete_url_cache_txn(txn):
358		txn.executemany(sql, [(media_id,) for media_id in media_ids])
359
360		return await self.db.runInteraction("delete_url_cache", _delete_url_cache_txn)
361
362		def get_url_cache_media_before(self, before_ts):
363		sql = (
364		"SELECT media_id FROM local_media_repository"
365		" WHERE created_ts < ? AND url_cache IS NOT NULL"
366		" ORDER BY created_ts ASC"
367		" LIMIT 500"
368		)
369
370		def _get_url_cache_media_before_txn(txn):
371		txn.execute(sql, (before_ts,))
372		return [row[0] for row in txn]
373
374		return self.db.runInteraction(
375		"get_url_cache_media_before", _get_url_cache_media_before_txn
376		)
377
378		async def delete_url_cache_media(self, media_ids):
379		if len(media_ids) == 0:
380		return
381
382		def _delete_url_cache_media_txn(txn):
383		sql = "DELETE FROM local_media_repository WHERE media_id = ?"
384
385		txn.executemany(sql, [(media_id,) for media_id in media_ids])
386
387		sql = "DELETE FROM local_media_repository_thumbnails WHERE media_id = ?"
388
389		txn.executemany(sql, [(media_id,) for media_id in media_ids])
390
391		return await self.db.runInteraction(
392		"delete_url_cache_media", _delete_url_cache_media_txn
393		)

+0

-128

~~synapse/storage/data_stores/main/metrics.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2020 The Matrix.org Foundation C.I.C.
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14		import typing
15		from collections import Counter
16
17		from twisted.internet import defer
18
19		from synapse.metrics import BucketCollector
20		from synapse.metrics.background_process_metrics import run_as_background_process
21		from synapse.storage._base import SQLBaseStore
22		from synapse.storage.data_stores.main.event_push_actions import (
23		EventPushActionsWorkerStore,
24		)
25		from synapse.storage.database import Database
26
27
28		class ServerMetricsStore(EventPushActionsWorkerStore, SQLBaseStore):
29		"""Functions to pull various metrics from the DB, for e.g. phone home
30		stats and prometheus metrics.
31		"""
32
33		def __init__(self, database: Database, db_conn, hs):
34		super().__init__(database, db_conn, hs)
35
36		# Collect metrics on the number of forward extremities that exist.
37		# Counter of number of extremities to count
38		self._current_forward_extremities_amount = (
39		Counter()
40		) # type: typing.Counter[int]
41
42		BucketCollector(
43		"synapse_forward_extremities",
44		lambda: self._current_forward_extremities_amount,
45		buckets=[1, 2, 3, 5, 7, 10, 15, 20, 50, 100, 200, 500, "+Inf"],
46		)
47
48		# Read the extrems every 60 minutes
49		def read_forward_extremities():
50		# run as a background process to make sure that the database transactions
51		# have a logcontext to report to
52		return run_as_background_process(
53		"read_forward_extremities", self._read_forward_extremities
54		)
55
56		hs.get_clock().looping_call(read_forward_extremities, 60 * 60 * 1000)
57
58		async def _read_forward_extremities(self):
59		def fetch(txn):
60		txn.execute(
61		"""
62		select count(*) c from event_forward_extremities
63		group by room_id
64		"""
65		)
66		return txn.fetchall()
67
68		res = await self.db.runInteraction("read_forward_extremities", fetch)
69		self._current_forward_extremities_amount = Counter([x[0] for x in res])
70
71		@defer.inlineCallbacks
72		def count_daily_messages(self):
73		"""
74		Returns an estimate of the number of messages sent in the last day.
75
76		If it has been significantly less or more than one day since the last
77		call to this function, it will return None.
78		"""
79
80		def _count_messages(txn):
81		sql = """
82		SELECT COALESCE(COUNT(*), 0) FROM events
83		WHERE type = 'm.room.message'
84		AND stream_ordering > ?
85		"""
86		txn.execute(sql, (self.stream_ordering_day_ago,))
87		(count,) = txn.fetchone()
88		return count
89
90		ret = yield self.db.runInteraction("count_messages", _count_messages)
91		return ret
92
93		@defer.inlineCallbacks
94		def count_daily_sent_messages(self):
95		def _count_messages(txn):
96		# This is good enough as if you have silly characters in your own
97		# hostname then thats your own fault.
98		like_clause = "%:" + self.hs.hostname
99
100		sql = """
101		SELECT COALESCE(COUNT(*), 0) FROM events
102		WHERE type = 'm.room.message'
103		AND sender LIKE ?
104		AND stream_ordering > ?
105		"""
106
107		txn.execute(sql, (like_clause, self.stream_ordering_day_ago))
108		(count,) = txn.fetchone()
109		return count
110
111		ret = yield self.db.runInteraction("count_daily_sent_messages", _count_messages)
112		return ret
113
114		@defer.inlineCallbacks
115		def count_daily_active_rooms(self):
116		def _count(txn):
117		sql = """
118		SELECT COALESCE(COUNT(DISTINCT room_id), 0) FROM events
119		WHERE type = 'm.room.message'
120		AND stream_ordering > ?
121		"""
122		txn.execute(sql, (self.stream_ordering_day_ago,))
123		(count,) = txn.fetchone()
124		return count
125
126		ret = yield self.db.runInteraction("count_daily_active_rooms", _count)
127		return ret

+0

-359

~~synapse/storage/data_stores/main/monthly_active_users.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2018 New Vector
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14		import logging
15		from typing import List
16
17		from twisted.internet import defer
18
19		from synapse.storage._base import SQLBaseStore
20		from synapse.storage.database import Database, make_in_list_sql_clause
21		from synapse.util.caches.descriptors import cached
22
23		logger = logging.getLogger(__name__)
24
25		# Number of msec of granularity to store the monthly_active_user timestamp
26		# This means it is not necessary to update the table on every request
27		LAST_SEEN_GRANULARITY = 60 * 60 * 1000
28
29
30		class MonthlyActiveUsersWorkerStore(SQLBaseStore):
31		def __init__(self, database: Database, db_conn, hs):
32		super(MonthlyActiveUsersWorkerStore, self).__init__(database, db_conn, hs)
33		self._clock = hs.get_clock()
34		self.hs = hs
35
36		@cached(num_args=0)
37		def get_monthly_active_count(self):
38		"""Generates current count of monthly active users
39
40		Returns:
41		Defered[int]: Number of current monthly active users
42		"""
43
44		def _count_users(txn):
45		sql = "SELECT COALESCE(count(*), 0) FROM monthly_active_users"
46		txn.execute(sql)
47		(count,) = txn.fetchone()
48		return count
49
50		return self.db.runInteraction("count_users", _count_users)
51
52		@cached(num_args=0)
53		def get_monthly_active_count_by_service(self):
54		"""Generates current count of monthly active users broken down by service.
55		A service is typically an appservice but also includes native matrix users.
56		Since the `monthly_active_users` table is populated from the `user_ips` table
57		`config.track_appservice_user_ips` must be set to `true` for this
58		method to return anything other than native matrix users.
59
60		Returns:
61		Deferred[dict]: dict that includes a mapping between app_service_id
62		and the number of occurrences.
63
64		"""
65
66		def _count_users_by_service(txn):
67		sql = """
68		SELECT COALESCE(appservice_id, 'native'), COALESCE(count(*), 0)
69		FROM monthly_active_users
70		LEFT JOIN users ON monthly_active_users.user_id=users.name
71		GROUP BY appservice_id;
72		"""
73
74		txn.execute(sql)
75		result = txn.fetchall()
76		return dict(result)
77
78		return self.db.runInteraction("count_users_by_service", _count_users_by_service)
79
80		async def get_registered_reserved_users(self) -> List[str]:
81		"""Of the reserved threepids defined in config, retrieve those that are associated
82		with registered users
83
84		Returns:
85		User IDs of actual users that are reserved
86		"""
87		users = []
88
89		for tp in self.hs.config.mau_limits_reserved_threepids[
90		: self.hs.config.max_mau_value
91		]:
92		user_id = await self.hs.get_datastore().get_user_id_by_threepid(
93		tp["medium"], tp["address"]
94		)
95		if user_id:
96		users.append(user_id)
97
98		return users
99
100		@cached(num_args=1)
101		def user_last_seen_monthly_active(self, user_id):
102		"""
103		Checks if a given user is part of the monthly active user group
104		Arguments:
105		user_id (str): user to add/update
106		Return:
107		Deferred[int] : timestamp since last seen, None if never seen
108
109		"""
110
111		return self.db.simple_select_one_onecol(
112		table="monthly_active_users",
113		keyvalues={"user_id": user_id},
114		retcol="timestamp",
115		allow_none=True,
116		desc="user_last_seen_monthly_active",
117		)
118
119
120		class MonthlyActiveUsersStore(MonthlyActiveUsersWorkerStore):
121		def __init__(self, database: Database, db_conn, hs):
122		super(MonthlyActiveUsersStore, self).__init__(database, db_conn, hs)
123
124		self._limit_usage_by_mau = hs.config.limit_usage_by_mau
125		self._mau_stats_only = hs.config.mau_stats_only
126		self._max_mau_value = hs.config.max_mau_value
127
128		# Do not add more reserved users than the total allowable number
129		# cur = LoggingTransaction(
130		self.db.new_transaction(
131		db_conn,
132		"initialise_mau_threepids",
133		[],
134		[],
135		self._initialise_reserved_users,
136		hs.config.mau_limits_reserved_threepids[: self._max_mau_value],
137		)
138
139		def _initialise_reserved_users(self, txn, threepids):
140		"""Ensures that reserved threepids are accounted for in the MAU table, should
141		be called on start up.
142
143		Args:
144		txn (cursor):
145		threepids (list[dict]): List of threepid dicts to reserve
146		"""
147
148		# XXX what is this function trying to achieve? It upserts into
149		# monthly_active_users for each registered reserved mau user, but why?
150		#
151		# - shouldn't there already be an entry for each reserved user (at least
152		# if they have been active recently)?
153		#
154		# - if it's important that the timestamp is kept up to date, why do we only
155		# run this at startup?
156
157		for tp in threepids:
158		user_id = self.get_user_id_by_threepid_txn(txn, tp["medium"], tp["address"])
159
160		if user_id:
161		is_support = self.is_support_user_txn(txn, user_id)
162		if not is_support:
163		# We do this manually here to avoid hitting #6791
164		self.db.simple_upsert_txn(
165		txn,
166		table="monthly_active_users",
167		keyvalues={"user_id": user_id},
168		values={"timestamp": int(self._clock.time_msec())},
169		)
170		else:
171		logger.warning("mau limit reserved threepid %s not found in db" % tp)
172
173		async def reap_monthly_active_users(self):
174		"""Cleans out monthly active user table to ensure that no stale
175		entries exist.
176		"""
177
178		def _reap_users(txn, reserved_users):
179		"""
180		Args:
181		reserved_users (tuple): reserved users to preserve
182		"""
183
184		thirty_days_ago = int(self._clock.time_msec()) - (1000 * 60 * 60 * 24 * 30)
185
186		in_clause, in_clause_args = make_in_list_sql_clause(
187		self.database_engine, "user_id", reserved_users
188		)
189
190		txn.execute(
191		"DELETE FROM monthly_active_users WHERE timestamp < ? AND NOT %s"
192		% (in_clause,),
193		[thirty_days_ago] + in_clause_args,
194		)
195
196		if self._limit_usage_by_mau:
197		# If MAU user count still exceeds the MAU threshold, then delete on
198		# a least recently active basis.
199		# Note it is not possible to write this query using OFFSET due to
200		# incompatibilities in how sqlite and postgres support the feature.
201		# Sqlite requires 'LIMIT -1 OFFSET ?', the LIMIT must be present,
202		# while Postgres does not require 'LIMIT', but also does not support
203		# negative LIMIT values. So there is no way to write it that both can
204		# support
205
206		# Limit must be >= 0 for postgres
207		num_of_non_reserved_users_to_remove = max(
208		self._max_mau_value - len(reserved_users), 0
209		)
210
211		# It is important to filter reserved users twice to guard
212		# against the case where the reserved user is present in the
213		# SELECT, meaning that a legitimate mau is deleted.
214		sql = """
215		DELETE FROM monthly_active_users
216		WHERE user_id NOT IN (
217		SELECT user_id FROM monthly_active_users
218		WHERE NOT %s
219		ORDER BY timestamp DESC
220		LIMIT ?
221		)
222		AND NOT %s
223		""" % (
224		in_clause,
225		in_clause,
226		)
227
228		query_args = (
229		in_clause_args
230		+ [num_of_non_reserved_users_to_remove]
231		+ in_clause_args
232		)
233		txn.execute(sql, query_args)
234
235		# It seems poor to invalidate the whole cache. Postgres supports
236		# 'Returning' which would allow me to invalidate only the
237		# specific users, but sqlite has no way to do this and instead
238		# I would need to SELECT and the DELETE which without locking
239		# is racy.
240		# Have resolved to invalidate the whole cache for now and do
241		# something about it if and when the perf becomes significant
242		self._invalidate_all_cache_and_stream(
243		txn, self.user_last_seen_monthly_active
244		)
245		self._invalidate_cache_and_stream(txn, self.get_monthly_active_count, ())
246
247		reserved_users = await self.get_registered_reserved_users()
248		await self.db.runInteraction(
249		"reap_monthly_active_users", _reap_users, reserved_users
250		)
251
252		@defer.inlineCallbacks
253		def upsert_monthly_active_user(self, user_id):
254		"""Updates or inserts the user into the monthly active user table, which
255		is used to track the current MAU usage of the server
256
257		Args:
258		user_id (str): user to add/update
259
260		Returns:
261		Deferred
262		"""
263		# Support user never to be included in MAU stats. Note I can't easily call this
264		# from upsert_monthly_active_user_txn because then I need a _txn form of
265		# is_support_user which is complicated because I want to cache the result.
266		# Therefore I call it here and ignore the case where
267		# upsert_monthly_active_user_txn is called directly from
268		# _initialise_reserved_users reasoning that it would be very strange to
269		# include a support user in this context.
270
271		is_support = yield self.is_support_user(user_id)
272		if is_support:
273		return
274
275		yield self.db.runInteraction(
276		"upsert_monthly_active_user", self.upsert_monthly_active_user_txn, user_id
277		)
278
279		def upsert_monthly_active_user_txn(self, txn, user_id):
280		"""Updates or inserts monthly active user member
281
282		We consciously do not call is_support_txn from this method because it
283		is not possible to cache the response. is_support_txn will be false in
284		almost all cases, so it seems reasonable to call it only for
285		upsert_monthly_active_user and to call is_support_txn manually
286		for cases where upsert_monthly_active_user_txn is called directly,
287		like _initialise_reserved_users
288
289		In short, don't call this method with support users. (Support users
290		should not appear in the MAU stats).
291
292		Args:
293		txn (cursor):
294		user_id (str): user to add/update
295
296		Returns:
297		bool: True if a new entry was created, False if an
298		existing one was updated.
299		"""
300
301		# Am consciously deciding to lock the table on the basis that is ought
302		# never be a big table and alternative approaches (batching multiple
303		# upserts into a single txn) introduced a lot of extra complexity.
304		# See https://github.com/matrix-org/synapse/issues/3854 for more
305		is_insert = self.db.simple_upsert_txn(
306		txn,
307		table="monthly_active_users",
308		keyvalues={"user_id": user_id},
309		values={"timestamp": int(self._clock.time_msec())},
310		)
311
312		self._invalidate_cache_and_stream(txn, self.get_monthly_active_count, ())
313		self._invalidate_cache_and_stream(
314		txn, self.get_monthly_active_count_by_service, ()
315		)
316		self._invalidate_cache_and_stream(
317		txn, self.user_last_seen_monthly_active, (user_id,)
318		)
319
320		return is_insert
321
322		@defer.inlineCallbacks
323		def populate_monthly_active_users(self, user_id):
324		"""Checks on the state of monthly active user limits and optionally
325		add the user to the monthly active tables
326
327		Args:
328		user_id(str): the user_id to query
329		"""
330		if self._limit_usage_by_mau or self._mau_stats_only:
331		# Trial users and guests should not be included as part of MAU group
332		is_guest = yield self.is_guest(user_id)
333		if is_guest:
334		return
335		is_trial = yield self.is_trial_user(user_id)
336		if is_trial:
337		return
338
339		last_seen_timestamp = yield self.user_last_seen_monthly_active(user_id)
340		now = self.hs.get_clock().time_msec()
341
342		# We want to reduce to the total number of db writes, and are happy
343		# to trade accuracy of timestamp in order to lighten load. This means
344		# We always insert new users (where MAU threshold has not been reached),
345		# but only update if we have not previously seen the user for
346		# LAST_SEEN_GRANULARITY ms
347		if last_seen_timestamp is None:
348		# In the case where mau_stats_only is True and limit_usage_by_mau is
349		# False, there is no point in checking get_monthly_active_count - it
350		# adds no value and will break the logic if max_mau_value is exceeded.
351		if not self._limit_usage_by_mau:
352		yield self.upsert_monthly_active_user(user_id)
353		else:
354		count = yield self.get_monthly_active_count()
355		if count < self._max_mau_value:
356		yield self.upsert_monthly_active_user(user_id)
357		elif now - last_seen_timestamp > LAST_SEEN_GRANULARITY:
358		yield self.upsert_monthly_active_user(user_id)

+0

-33

~~synapse/storage/data_stores/main/openid.py~~ less more

0		from synapse.storage._base import SQLBaseStore
1
2
3		class OpenIdStore(SQLBaseStore):
4		def insert_open_id_token(self, token, ts_valid_until_ms, user_id):
5		return self.db.simple_insert(
6		table="open_id_tokens",
7		values={
8		"token": token,
9		"ts_valid_until_ms": ts_valid_until_ms,
10		"user_id": user_id,
11		},
12		desc="insert_open_id_token",
13		)
14
15		def get_user_id_for_open_id_token(self, token, ts_now_ms):
16		def get_user_id_for_token_txn(txn):
17		sql = (
18		"SELECT user_id FROM open_id_tokens"
19		" WHERE token = ? AND ? <= ts_valid_until_ms"
20		)
21
22		txn.execute(sql, (token, ts_now_ms))
23
24		rows = txn.fetchall()
25		if not rows:
26		return None
27		else:
28		return rows[0][0]
29
30		return self.db.runInteraction(
31		"get_user_id_for_token", get_user_id_for_token_txn
32		)

+0

-186

~~synapse/storage/data_stores/main/presence.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		from typing import List, Tuple
16
17		from twisted.internet import defer
18
19		from synapse.storage._base import SQLBaseStore, make_in_list_sql_clause
20		from synapse.storage.presence import UserPresenceState
21		from synapse.util.caches.descriptors import cached, cachedList
22		from synapse.util.iterutils import batch_iter
23
24
25		class PresenceStore(SQLBaseStore):
26		@defer.inlineCallbacks
27		def update_presence(self, presence_states):
28		stream_ordering_manager = self._presence_id_gen.get_next_mult(
29		len(presence_states)
30		)
31
32		with stream_ordering_manager as stream_orderings:
33		yield self.db.runInteraction(
34		"update_presence",
35		self._update_presence_txn,
36		stream_orderings,
37		presence_states,
38		)
39
40		return stream_orderings[-1], self._presence_id_gen.get_current_token()
41
42		def _update_presence_txn(self, txn, stream_orderings, presence_states):
43		for stream_id, state in zip(stream_orderings, presence_states):
44		txn.call_after(
45		self.presence_stream_cache.entity_has_changed, state.user_id, stream_id
46		)
47		txn.call_after(self._get_presence_for_user.invalidate, (state.user_id,))
48
49		# Actually insert new rows
50		self.db.simple_insert_many_txn(
51		txn,
52		table="presence_stream",
53		values=[
54		{
55		"stream_id": stream_id,
56		"user_id": state.user_id,
57		"state": state.state,
58		"last_active_ts": state.last_active_ts,
59		"last_federation_update_ts": state.last_federation_update_ts,
60		"last_user_sync_ts": state.last_user_sync_ts,
61		"status_msg": state.status_msg,
62		"currently_active": state.currently_active,
63		}
64		for stream_id, state in zip(stream_orderings, presence_states)
65		],
66		)
67
68		# Delete old rows to stop database from getting really big
69		sql = "DELETE FROM presence_stream WHERE stream_id < ? AND "
70
71		for states in batch_iter(presence_states, 50):
72		clause, args = make_in_list_sql_clause(
73		self.database_engine, "user_id", [s.user_id for s in states]
74		)
75		txn.execute(sql + clause, [stream_id] + list(args))
76
77		async def get_all_presence_updates(
78		self, instance_name: str, last_id: int, current_id: int, limit: int
79		) -> Tuple[List[Tuple[int, list]], int, bool]:
80		"""Get updates for presence replication stream.
81
82		Args:
83		instance_name: The writer we want to fetch updates from. Unused
84		here since there is only ever one writer.
85		last_id: The token to fetch updates from. Exclusive.
86		current_id: The token to fetch updates up to. Inclusive.
87		limit: The requested limit for the number of rows to return. The
88		function may return more or fewer rows.
89
90		Returns:
91		A tuple consisting of: the updates, a token to use to fetch
92		subsequent updates, and whether we returned fewer rows than exists
93		between the requested tokens due to the limit.
94
95		The token returned can be used in a subsequent call to this
96		function to get further updatees.
97
98		The updates are a list of 2-tuples of stream ID and the row data
99		"""
100
101		if last_id == current_id:
102		return [], current_id, False
103
104		def get_all_presence_updates_txn(txn):
105		sql = """
106		SELECT stream_id, user_id, state, last_active_ts,
107		last_federation_update_ts, last_user_sync_ts,
108		status_msg,
109		currently_active
110		FROM presence_stream
111		WHERE ? < stream_id AND stream_id <= ?
112		ORDER BY stream_id ASC
113		LIMIT ?
114		"""
115		txn.execute(sql, (last_id, current_id, limit))
116		updates = [(row[0], row[1:]) for row in txn]
117
118		upper_bound = current_id
119		limited = False
120		if len(updates) >= limit:
121		upper_bound = updates[-1][0]
122		limited = True
123
124		return updates, upper_bound, limited
125
126		return await self.db.runInteraction(
127		"get_all_presence_updates", get_all_presence_updates_txn
128		)
129
130		@cached()
131		def _get_presence_for_user(self, user_id):
132		raise NotImplementedError()
133
134		@cachedList(
135		cached_method_name="_get_presence_for_user",
136		list_name="user_ids",
137		num_args=1,
138		inlineCallbacks=True,
139		)
140		def get_presence_for_users(self, user_ids):
141		rows = yield self.db.simple_select_many_batch(
142		table="presence_stream",
143		column="user_id",
144		iterable=user_ids,
145		keyvalues={},
146		retcols=(
147		"user_id",
148		"state",
149		"last_active_ts",
150		"last_federation_update_ts",
151		"last_user_sync_ts",
152		"status_msg",
153		"currently_active",
154		),
155		desc="get_presence_for_users",
156		)
157
158		for row in rows:
159		row["currently_active"] = bool(row["currently_active"])
160
161		return {row["user_id"]: UserPresenceState(**row) for row in rows}
162
163		def get_current_presence_token(self):
164		return self._presence_id_gen.get_current_token()
165
166		def allow_presence_visible(self, observed_localpart, observer_userid):
167		return self.db.simple_insert(
168		table="presence_allow_inbound",
169		values={
170		"observed_user_id": observed_localpart,
171		"observer_user_id": observer_userid,
172		},
173		desc="allow_presence_visible",
174		or_ignore=True,
175		)
176
177		def disallow_presence_visible(self, observed_localpart, observer_userid):
178		return self.db.simple_delete_one(
179		table="presence_allow_inbound",
180		keyvalues={
181		"observed_user_id": observed_localpart,
182		"observer_user_id": observer_userid,
183		},
184		desc="disallow_presence_visible",
185		)

+0

-178

~~synapse/storage/data_stores/main/profile.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		from twisted.internet import defer
16
17		from synapse.api.errors import StoreError
18		from synapse.storage._base import SQLBaseStore
19		from synapse.storage.data_stores.main.roommember import ProfileInfo
20
21
22		class ProfileWorkerStore(SQLBaseStore):
23		@defer.inlineCallbacks
24		def get_profileinfo(self, user_localpart):
25		try:
26		profile = yield self.db.simple_select_one(
27		table="profiles",
28		keyvalues={"user_id": user_localpart},
29		retcols=("displayname", "avatar_url"),
30		desc="get_profileinfo",
31		)
32		except StoreError as e:
33		if e.code == 404:
34		# no match
35		return ProfileInfo(None, None)
36		else:
37		raise
38
39		return ProfileInfo(
40		avatar_url=profile["avatar_url"], display_name=profile["displayname"]
41		)
42
43		def get_profile_displayname(self, user_localpart):
44		return self.db.simple_select_one_onecol(
45		table="profiles",
46		keyvalues={"user_id": user_localpart},
47		retcol="displayname",
48		desc="get_profile_displayname",
49		)
50
51		def get_profile_avatar_url(self, user_localpart):
52		return self.db.simple_select_one_onecol(
53		table="profiles",
54		keyvalues={"user_id": user_localpart},
55		retcol="avatar_url",
56		desc="get_profile_avatar_url",
57		)
58
59		def get_from_remote_profile_cache(self, user_id):
60		return self.db.simple_select_one(
61		table="remote_profile_cache",
62		keyvalues={"user_id": user_id},
63		retcols=("displayname", "avatar_url"),
64		allow_none=True,
65		desc="get_from_remote_profile_cache",
66		)
67
68		def create_profile(self, user_localpart):
69		return self.db.simple_insert(
70		table="profiles", values={"user_id": user_localpart}, desc="create_profile"
71		)
72
73		def set_profile_displayname(self, user_localpart, new_displayname):
74		return self.db.simple_update_one(
75		table="profiles",
76		keyvalues={"user_id": user_localpart},
77		updatevalues={"displayname": new_displayname},
78		desc="set_profile_displayname",
79		)
80
81		def set_profile_avatar_url(self, user_localpart, new_avatar_url):
82		return self.db.simple_update_one(
83		table="profiles",
84		keyvalues={"user_id": user_localpart},
85		updatevalues={"avatar_url": new_avatar_url},
86		desc="set_profile_avatar_url",
87		)
88
89
90		class ProfileStore(ProfileWorkerStore):
91		def add_remote_profile_cache(self, user_id, displayname, avatar_url):
92		"""Ensure we are caching the remote user's profiles.
93
94		This should only be called when `is_subscribed_remote_profile_for_user`
95		would return true for the user.
96		"""
97		return self.db.simple_upsert(
98		table="remote_profile_cache",
99		keyvalues={"user_id": user_id},
100		values={
101		"displayname": displayname,
102		"avatar_url": avatar_url,
103		"last_check": self._clock.time_msec(),
104		},
105		desc="add_remote_profile_cache",
106		)
107
108		def update_remote_profile_cache(self, user_id, displayname, avatar_url):
109		return self.db.simple_update(
110		table="remote_profile_cache",
111		keyvalues={"user_id": user_id},
112		updatevalues={
113		"displayname": displayname,
114		"avatar_url": avatar_url,
115		"last_check": self._clock.time_msec(),
116		},
117		desc="update_remote_profile_cache",
118		)
119
120		@defer.inlineCallbacks
121		def maybe_delete_remote_profile_cache(self, user_id):
122		"""Check if we still care about the remote user's profile, and if we
123		don't then remove their profile from the cache
124		"""
125		subscribed = yield self.is_subscribed_remote_profile_for_user(user_id)
126		if not subscribed:
127		yield self.db.simple_delete(
128		table="remote_profile_cache",
129		keyvalues={"user_id": user_id},
130		desc="delete_remote_profile_cache",
131		)
132
133		def get_remote_profile_cache_entries_that_expire(self, last_checked):
134		"""Get all users who haven't been checked since `last_checked`
135		"""
136
137		def _get_remote_profile_cache_entries_that_expire_txn(txn):
138		sql = """
139		SELECT user_id, displayname, avatar_url
140		FROM remote_profile_cache
141		WHERE last_check < ?
142		"""
143
144		txn.execute(sql, (last_checked,))
145
146		return self.db.cursor_to_dict(txn)
147
148		return self.db.runInteraction(
149		"get_remote_profile_cache_entries_that_expire",
150		_get_remote_profile_cache_entries_that_expire_txn,
151		)
152
153		@defer.inlineCallbacks
154		def is_subscribed_remote_profile_for_user(self, user_id):
155		"""Check whether we are interested in a remote user's profile.
156		"""
157		res = yield self.db.simple_select_one_onecol(
158		table="group_users",
159		keyvalues={"user_id": user_id},
160		retcol="user_id",
161		allow_none=True,
162		desc="should_update_remote_profile_cache_for_user",
163		)
164
165		if res:
166		return True
167
168		res = yield self.db.simple_select_one_onecol(
169		table="group_invites",
170		keyvalues={"user_id": user_id},
171		retcol="user_id",
172		allow_none=True,
173		desc="should_update_remote_profile_cache_for_user",
174		)
175
176		if res:
177		return True

+0

-398

~~synapse/storage/data_stores/main/purge_events.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2020 The Matrix.org Foundation C.I.C.
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16		from typing import Any, Tuple
17
18		from synapse.api.errors import SynapseError
19		from synapse.storage._base import SQLBaseStore
20		from synapse.storage.data_stores.main.state import StateGroupWorkerStore
21		from synapse.types import RoomStreamToken
22
23		logger = logging.getLogger(__name__)
24
25
26		class PurgeEventsStore(StateGroupWorkerStore, SQLBaseStore):
27		def purge_history(self, room_id, token, delete_local_events):
28		"""Deletes room history before a certain point
29
30		Args:
31		room_id (str):
32
33		token (str): A topological token to delete events before
34
35		delete_local_events (bool):
36		if True, we will delete local events as well as remote ones
37		(instead of just marking them as outliers and deleting their
38		state groups).
39
40		Returns:
41		Deferred[set[int]]: The set of state groups that are referenced by
42		deleted events.
43		"""
44
45		return self.db.runInteraction(
46		"purge_history",
47		self._purge_history_txn,
48		room_id,
49		token,
50		delete_local_events,
51		)
52
53		def _purge_history_txn(self, txn, room_id, token_str, delete_local_events):
54		token = RoomStreamToken.parse(token_str)
55
56		# Tables that should be pruned:
57		# event_auth
58		# event_backward_extremities
59		# event_edges
60		# event_forward_extremities
61		# event_json
62		# event_push_actions
63		# event_reference_hashes
64		# event_search
65		# event_to_state_groups
66		# events
67		# rejections
68		# room_depth
69		# state_groups
70		# state_groups_state
71
72		# we will build a temporary table listing the events so that we don't
73		# have to keep shovelling the list back and forth across the
74		# connection. Annoyingly the python sqlite driver commits the
75		# transaction on CREATE, so let's do this first.
76		#
77		# furthermore, we might already have the table from a previous (failed)
78		# purge attempt, so let's drop the table first.
79
80		txn.execute("DROP TABLE IF EXISTS events_to_purge")
81
82		txn.execute(
83		"CREATE TEMPORARY TABLE events_to_purge ("
84		" event_id TEXT NOT NULL,"
85		" should_delete BOOLEAN NOT NULL"
86		")"
87		)
88
89		# First ensure that we're not about to delete all the forward extremeties
90		txn.execute(
91		"SELECT e.event_id, e.depth FROM events as e "
92		"INNER JOIN event_forward_extremities as f "
93		"ON e.event_id = f.event_id "
94		"AND e.room_id = f.room_id "
95		"WHERE f.room_id = ?",
96		(room_id,),
97		)
98		rows = txn.fetchall()
99		max_depth = max(row[1] for row in rows)
100
101		if max_depth < token.topological:
102		# We need to ensure we don't delete all the events from the database
103		# otherwise we wouldn't be able to send any events (due to not
104		# having any backwards extremeties)
105		raise SynapseError(
106		400, "topological_ordering is greater than forward extremeties"
107		)
108
109		logger.info("[purge] looking for events to delete")
110
111		should_delete_expr = "state_key IS NULL"
112		should_delete_params = () # type: Tuple[Any, ...]
113		if not delete_local_events:
114		should_delete_expr += " AND event_id NOT LIKE ?"
115
116		# We include the parameter twice since we use the expression twice
117		should_delete_params += ("%:" + self.hs.hostname, "%:" + self.hs.hostname)
118
119		should_delete_params += (room_id, token.topological)
120
121		# Note that we insert events that are outliers and aren't going to be
122		# deleted, as nothing will happen to them.
123		txn.execute(
124		"INSERT INTO events_to_purge"
125		" SELECT event_id, %s"
126		" FROM events AS e LEFT JOIN state_events USING (event_id)"
127		" WHERE (NOT outlier OR (%s)) AND e.room_id = ? AND topological_ordering < ?"
128		% (should_delete_expr, should_delete_expr),
129		should_delete_params,
130		)
131
132		# We create the indices after insertion as that's a lot faster.
133
134		# create an index on should_delete because later we'll be looking for
135		# the should_delete / shouldn't_delete subsets
136		txn.execute(
137		"CREATE INDEX events_to_purge_should_delete"
138		" ON events_to_purge(should_delete)"
139		)
140
141		# We do joins against events_to_purge for e.g. calculating state
142		# groups to purge, etc., so lets make an index.
143		txn.execute("CREATE INDEX events_to_purge_id ON events_to_purge(event_id)")
144
145		txn.execute("SELECT event_id, should_delete FROM events_to_purge")
146		event_rows = txn.fetchall()
147		logger.info(
148		"[purge] found %i events before cutoff, of which %i can be deleted",
149		len(event_rows),
150		sum(1 for e in event_rows if e[1]),
151		)
152
153		logger.info("[purge] Finding new backward extremities")
154
155		# We calculate the new entries for the backward extremeties by finding
156		# events to be purged that are pointed to by events we're not going to
157		# purge.
158		txn.execute(
159		"SELECT DISTINCT e.event_id FROM events_to_purge AS e"
160		" INNER JOIN event_edges AS ed ON e.event_id = ed.prev_event_id"
161		" LEFT JOIN events_to_purge AS ep2 ON ed.event_id = ep2.event_id"
162		" WHERE ep2.event_id IS NULL"
163		)
164		new_backwards_extrems = txn.fetchall()
165
166		logger.info("[purge] replacing backward extremities: %r", new_backwards_extrems)
167
168		txn.execute(
169		"DELETE FROM event_backward_extremities WHERE room_id = ?", (room_id,)
170		)
171
172		# Update backward extremeties
173		txn.executemany(
174		"INSERT INTO event_backward_extremities (room_id, event_id)"
175		" VALUES (?, ?)",
176		[(room_id, event_id) for event_id, in new_backwards_extrems],
177		)
178
179		logger.info("[purge] finding state groups referenced by deleted events")
180
181		# Get all state groups that are referenced by events that are to be
182		# deleted.
183		txn.execute(
184		"""
185		SELECT DISTINCT state_group FROM events_to_purge
186		INNER JOIN event_to_state_groups USING (event_id)
187		"""
188		)
189
190		referenced_state_groups = {sg for sg, in txn}
191		logger.info(
192		"[purge] found %i referenced state groups", len(referenced_state_groups)
193		)
194
195		logger.info("[purge] removing events from event_to_state_groups")
196		txn.execute(
197		"DELETE FROM event_to_state_groups "
198		"WHERE event_id IN (SELECT event_id from events_to_purge)"
199		)
200		for event_id, _ in event_rows:
201		txn.call_after(self._get_state_group_for_event.invalidate, (event_id,))
202
203		# Delete all remote non-state events
204		for table in (
205		"events",
206		"event_json",
207		"event_auth",
208		"event_edges",
209		"event_forward_extremities",
210		"event_reference_hashes",
211		"event_search",
212		"rejections",
213		):
214		logger.info("[purge] removing events from %s", table)
215
216		txn.execute(
217		"DELETE FROM %s WHERE event_id IN ("
218		" SELECT event_id FROM events_to_purge WHERE should_delete"
219		")" % (table,)
220		)
221
222		# event_push_actions lacks an index on event_id, and has one on
223		# (room_id, event_id) instead.
224		for table in ("event_push_actions",):
225		logger.info("[purge] removing events from %s", table)
226
227		txn.execute(
228		"DELETE FROM %s WHERE room_id = ? AND event_id IN ("
229		" SELECT event_id FROM events_to_purge WHERE should_delete"
230		")" % (table,),
231		(room_id,),
232		)
233
234		# Mark all state and own events as outliers
235		logger.info("[purge] marking remaining events as outliers")
236		txn.execute(
237		"UPDATE events SET outlier = ?"
238		" WHERE event_id IN ("
239		" SELECT event_id FROM events_to_purge "
240		" WHERE NOT should_delete"
241		")",
242		(True,),
243		)
244
245		# synapse tries to take out an exclusive lock on room_depth whenever it
246		# persists events (because upsert), and once we run this update, we
247		# will block that for the rest of our transaction.
248		#
249		# So, let's stick it at the end so that we don't block event
250		# persistence.
251		#
252		# We do this by calculating the minimum depth of the backwards
253		# extremities. However, the events in event_backward_extremities
254		# are ones we don't have yet so we need to look at the events that
255		# point to it via event_edges table.
256		txn.execute(
257		"""
258		SELECT COALESCE(MIN(depth), 0)
259		FROM event_backward_extremities AS eb
260		INNER JOIN event_edges AS eg ON eg.prev_event_id = eb.event_id
261		INNER JOIN events AS e ON e.event_id = eg.event_id
262		WHERE eb.room_id = ?
263		""",
264		(room_id,),
265		)
266		(min_depth,) = txn.fetchone()
267
268		logger.info("[purge] updating room_depth to %d", min_depth)
269
270		txn.execute(
271		"UPDATE room_depth SET min_depth = ? WHERE room_id = ?",
272		(min_depth, room_id),
273		)
274
275		# finally, drop the temp table. this will commit the txn in sqlite,
276		# so make sure to keep this actually last.
277		txn.execute("DROP TABLE events_to_purge")
278
279		logger.info("[purge] done")
280
281		return referenced_state_groups
282
283		def purge_room(self, room_id):
284		"""Deletes all record of a room
285
286		Args:
287		room_id (str)
288
289		Returns:
290		Deferred[List[int]]: The list of state groups to delete.
291		"""
292
293		return self.db.runInteraction("purge_room", self._purge_room_txn, room_id)
294
295		def _purge_room_txn(self, txn, room_id):
296		# First we fetch all the state groups that should be deleted, before
297		# we delete that information.
298		txn.execute(
299		"""
300		SELECT DISTINCT state_group FROM events
301		INNER JOIN event_to_state_groups USING(event_id)
302		WHERE events.room_id = ?
303		""",
304		(room_id,),
305		)
306
307		state_groups = [row[0] for row in txn]
308
309		# Now we delete tables which lack an index on room_id but have one on event_id
310		for table in (
311		"event_auth",
312		"event_edges",
313		"event_push_actions_staging",
314		"event_reference_hashes",
315		"event_relations",
316		"event_to_state_groups",
317		"redactions",
318		"rejections",
319		"state_events",
320		):
321		logger.info("[purge] removing %s from %s", room_id, table)
322
323		txn.execute(
324		"""
325		DELETE FROM %s WHERE event_id IN (
326		SELECT event_id FROM events WHERE room_id=?
327		)
328		"""
329		% (table,),
330		(room_id,),
331		)
332
333		# and finally, the tables with an index on room_id (or no useful index)
334		for table in (
335		"current_state_events",
336		"event_backward_extremities",
337		"event_forward_extremities",
338		"event_json",
339		"event_push_actions",
340		"event_search",
341		"events",
342		"group_rooms",
343		"public_room_list_stream",
344		"receipts_graph",
345		"receipts_linearized",
346		"room_aliases",
347		"room_depth",
348		"room_memberships",
349		"room_stats_state",
350		"room_stats_current",
351		"room_stats_historical",
352		"room_stats_earliest_token",
353		"rooms",
354		"stream_ordering_to_exterm",
355		"users_in_public_rooms",
356		"users_who_share_private_rooms",
357		# no useful index, but let's clear them anyway
358		"appservice_room_list",
359		"e2e_room_keys",
360		"event_push_summary",
361		"pusher_throttle",
362		"group_summary_rooms",
363		"room_account_data",
364		"room_tags",
365		"local_current_membership",
366		):
367		logger.info("[purge] removing %s from %s", room_id, table)
368		txn.execute("DELETE FROM %s WHERE room_id=?" % (table,), (room_id,))
369
370		# Other tables we do NOT need to clear out:
371		#
372		# - blocked_rooms
373		# This is important, to make sure that we don't accidentally rejoin a blocked
374		# room after it was purged
375		#
376		# - user_directory
377		# This has a room_id column, but it is unused
378		#
379
380		# Other tables that we might want to consider clearing out include:
381		#
382		# - event_reports
383		# Given that these are intended for abuse management my initial
384		# inclination is to leave them in place.
385		#
386		# - current_state_delta_stream
387		# - ex_outlier_stream
388		# - room_tags_revisions
389		# The problem with these is that they are largeish and there is no room_id
390		# index on them. In any case we should be clearing out 'stream' tables
391		# periodically anyway (#5888)
392
393		# TODO: we could probably usefully do a bunch of cache invalidation here
394
395		logger.info("[purge] done")
396
397		return state_groups

+0

-759

~~synapse/storage/data_stores/main/push_rule.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2018 New Vector Ltd
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		import abc
17		import logging
18		from typing import List, Tuple, Union
19
20		from canonicaljson import json
21
22		from twisted.internet import defer
23
24		from synapse.push.baserules import list_with_base_rules
25		from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
26		from synapse.storage._base import SQLBaseStore, db_to_json
27		from synapse.storage.data_stores.main.appservice import ApplicationServiceWorkerStore
28		from synapse.storage.data_stores.main.events_worker import EventsWorkerStore
29		from synapse.storage.data_stores.main.pusher import PusherWorkerStore
30		from synapse.storage.data_stores.main.receipts import ReceiptsWorkerStore
31		from synapse.storage.data_stores.main.roommember import RoomMemberWorkerStore
32		from synapse.storage.database import Database
33		from synapse.storage.push_rule import InconsistentRuleException, RuleNotFoundException
34		from synapse.storage.util.id_generators import ChainedIdGenerator
35		from synapse.util.caches.descriptors import cachedInlineCallbacks, cachedList
36		from synapse.util.caches.stream_change_cache import StreamChangeCache
37
38		logger = logging.getLogger(__name__)
39
40
41		def _load_rules(rawrules, enabled_map):
42		ruleslist = []
43		for rawrule in rawrules:
44		rule = dict(rawrule)
45		rule["conditions"] = db_to_json(rawrule["conditions"])
46		rule["actions"] = db_to_json(rawrule["actions"])
47		rule["default"] = False
48		ruleslist.append(rule)
49
50		# We're going to be mutating this a lot, so do a deep copy
51		rules = list(list_with_base_rules(ruleslist))
52
53		for i, rule in enumerate(rules):
54		rule_id = rule["rule_id"]
55		if rule_id in enabled_map:
56		if rule.get("enabled", True) != bool(enabled_map[rule_id]):
57		# Rules are cached across users.
58		rule = dict(rule)
59		rule["enabled"] = bool(enabled_map[rule_id])
60		rules[i] = rule
61
62		return rules
63
64
65		class PushRulesWorkerStore(
66		ApplicationServiceWorkerStore,
67		ReceiptsWorkerStore,
68		PusherWorkerStore,
69		RoomMemberWorkerStore,
70		EventsWorkerStore,
71		SQLBaseStore,
72		):
73		"""This is an abstract base class where subclasses must implement
74		`get_max_push_rules_stream_id` which can be called in the initializer.
75		"""
76
77		# This ABCMeta metaclass ensures that we cannot be instantiated without
78		# the abstract methods being implemented.
79		__metaclass__ = abc.ABCMeta
80
81		def __init__(self, database: Database, db_conn, hs):
82		super(PushRulesWorkerStore, self).__init__(database, db_conn, hs)
83
84		if hs.config.worker.worker_app is None:
85		self._push_rules_stream_id_gen = ChainedIdGenerator(
86		self._stream_id_gen, db_conn, "push_rules_stream", "stream_id"
87		) # type: Union[ChainedIdGenerator, SlavedIdTracker]
88		else:
89		self._push_rules_stream_id_gen = SlavedIdTracker(
90		db_conn, "push_rules_stream", "stream_id"
91		)
92
93		push_rules_prefill, push_rules_id = self.db.get_cache_dict(
94		db_conn,
95		"push_rules_stream",
96		entity_column="user_id",
97		stream_column="stream_id",
98		max_value=self.get_max_push_rules_stream_id(),
99		)
100
101		self.push_rules_stream_cache = StreamChangeCache(
102		"PushRulesStreamChangeCache",
103		push_rules_id,
104		prefilled_cache=push_rules_prefill,
105		)
106
107		@abc.abstractmethod
108		def get_max_push_rules_stream_id(self):
109		"""Get the position of the push rules stream.
110
111		Returns:
112		int
113		"""
114		raise NotImplementedError()
115
116		@cachedInlineCallbacks(max_entries=5000)
117		def get_push_rules_for_user(self, user_id):
118		rows = yield self.db.simple_select_list(
119		table="push_rules",
120		keyvalues={"user_name": user_id},
121		retcols=(
122		"user_name",
123		"rule_id",
124		"priority_class",
125		"priority",
126		"conditions",
127		"actions",
128		),
129		desc="get_push_rules_enabled_for_user",
130		)
131
132		rows.sort(key=lambda row: (-int(row["priority_class"]), -int(row["priority"])))
133
134		enabled_map = yield self.get_push_rules_enabled_for_user(user_id)
135
136		rules = _load_rules(rows, enabled_map)
137
138		return rules
139
140		@cachedInlineCallbacks(max_entries=5000)
141		def get_push_rules_enabled_for_user(self, user_id):
142		results = yield self.db.simple_select_list(
143		table="push_rules_enable",
144		keyvalues={"user_name": user_id},
145		retcols=("user_name", "rule_id", "enabled"),
146		desc="get_push_rules_enabled_for_user",
147		)
148		return {r["rule_id"]: False if r["enabled"] == 0 else True for r in results}
149
150		def have_push_rules_changed_for_user(self, user_id, last_id):
151		if not self.push_rules_stream_cache.has_entity_changed(user_id, last_id):
152		return defer.succeed(False)
153		else:
154
155		def have_push_rules_changed_txn(txn):
156		sql = (
157		"SELECT COUNT(stream_id) FROM push_rules_stream"
158		" WHERE user_id = ? AND ? < stream_id"
159		)
160		txn.execute(sql, (user_id, last_id))
161		(count,) = txn.fetchone()
162		return bool(count)
163
164		return self.db.runInteraction(
165		"have_push_rules_changed", have_push_rules_changed_txn
166		)
167
168		@cachedList(
169		cached_method_name="get_push_rules_for_user",
170		list_name="user_ids",
171		num_args=1,
172		inlineCallbacks=True,
173		)
174		def bulk_get_push_rules(self, user_ids):
175		if not user_ids:
176		return {}
177
178		results = {user_id: [] for user_id in user_ids}
179
180		rows = yield self.db.simple_select_many_batch(
181		table="push_rules",
182		column="user_name",
183		iterable=user_ids,
184		retcols=("*",),
185		desc="bulk_get_push_rules",
186		)
187
188		rows.sort(key=lambda row: (-int(row["priority_class"]), -int(row["priority"])))
189
190		for row in rows:
191		results.setdefault(row["user_name"], []).append(row)
192
193		enabled_map_by_user = yield self.bulk_get_push_rules_enabled(user_ids)
194
195		for user_id, rules in results.items():
196		results[user_id] = _load_rules(rules, enabled_map_by_user.get(user_id, {}))
197
198		return results
199
200		@defer.inlineCallbacks
201		def copy_push_rule_from_room_to_room(self, new_room_id, user_id, rule):
202		"""Copy a single push rule from one room to another for a specific user.
203
204		Args:
205		new_room_id (str): ID of the new room.
206		user_id (str): ID of user the push rule belongs to.
207		rule (Dict): A push rule.
208		"""
209		# Create new rule id
210		rule_id_scope = "/".join(rule["rule_id"].split("/")[:-1])
211		new_rule_id = rule_id_scope + "/" + new_room_id
212
213		# Change room id in each condition
214		for condition in rule.get("conditions", []):
215		if condition.get("key") == "room_id":
216		condition["pattern"] = new_room_id
217
218		# Add the rule for the new room
219		yield self.add_push_rule(
220		user_id=user_id,
221		rule_id=new_rule_id,
222		priority_class=rule["priority_class"],
223		conditions=rule["conditions"],
224		actions=rule["actions"],
225		)
226
227		@defer.inlineCallbacks
228		def copy_push_rules_from_room_to_room_for_user(
229		self, old_room_id, new_room_id, user_id
230		):
231		"""Copy all of the push rules from one room to another for a specific
232		user.
233
234		Args:
235		old_room_id (str): ID of the old room.
236		new_room_id (str): ID of the new room.
237		user_id (str): ID of user to copy push rules for.
238		"""
239		# Retrieve push rules for this user
240		user_push_rules = yield self.get_push_rules_for_user(user_id)
241
242		# Get rules relating to the old room and copy them to the new room
243		for rule in user_push_rules:
244		conditions = rule.get("conditions", [])
245		if any(
246		(c.get("key") == "room_id" and c.get("pattern") == old_room_id)
247		for c in conditions
248		):
249		yield self.copy_push_rule_from_room_to_room(new_room_id, user_id, rule)
250
251		@defer.inlineCallbacks
252		def bulk_get_push_rules_for_room(self, event, context):
253		state_group = context.state_group
254		if not state_group:
255		# If state_group is None it means it has yet to be assigned a
256		# state group, i.e. we need to make sure that calls with a state_group
257		# of None don't hit previous cached calls with a None state_group.
258		# To do this we set the state_group to a new object as object() != object()
259		state_group = object()
260
261		current_state_ids = yield defer.ensureDeferred(context.get_current_state_ids())
262		result = yield self._bulk_get_push_rules_for_room(
263		event.room_id, state_group, current_state_ids, event=event
264		)
265		return result
266
267		@cachedInlineCallbacks(num_args=2, cache_context=True)
268		def _bulk_get_push_rules_for_room(
269		self, room_id, state_group, current_state_ids, cache_context, event=None
270		):
271		# We don't use `state_group`, its there so that we can cache based
272		# on it. However, its important that its never None, since two current_state's
273		# with a state_group of None are likely to be different.
274		# See bulk_get_push_rules_for_room for how we work around this.
275		assert state_group is not None
276
277		# We also will want to generate notifs for other people in the room so
278		# their unread countss are correct in the event stream, but to avoid
279		# generating them for bot / AS users etc, we only do so for people who've
280		# sent a read receipt into the room.
281
282		users_in_room = yield self._get_joined_users_from_context(
283		room_id,
284		state_group,
285		current_state_ids,
286		on_invalidate=cache_context.invalidate,
287		event=event,
288		)
289
290		# We ignore app service users for now. This is so that we don't fill
291		# up the `get_if_users_have_pushers` cache with AS entries that we
292		# know don't have pushers, nor even read receipts.
293		local_users_in_room = {
294		u
295		for u in users_in_room
296		if self.hs.is_mine_id(u)
297		and not self.get_if_app_services_interested_in_user(u)
298		}
299
300		# users in the room who have pushers need to get push rules run because
301		# that's how their pushers work
302		if_users_with_pushers = yield self.get_if_users_have_pushers(
303		local_users_in_room, on_invalidate=cache_context.invalidate
304		)
305		user_ids = {
306		uid for uid, have_pusher in if_users_with_pushers.items() if have_pusher
307		}
308
309		users_with_receipts = yield self.get_users_with_read_receipts_in_room(
310		room_id, on_invalidate=cache_context.invalidate
311		)
312
313		# any users with pushers must be ours: they have pushers
314		for uid in users_with_receipts:
315		if uid in local_users_in_room:
316		user_ids.add(uid)
317
318		rules_by_user = yield self.bulk_get_push_rules(
319		user_ids, on_invalidate=cache_context.invalidate
320		)
321
322		rules_by_user = {k: v for k, v in rules_by_user.items() if v is not None}
323
324		return rules_by_user
325
326		@cachedList(
327		cached_method_name="get_push_rules_enabled_for_user",
328		list_name="user_ids",
329		num_args=1,
330		inlineCallbacks=True,
331		)
332		def bulk_get_push_rules_enabled(self, user_ids):
333		if not user_ids:
334		return {}
335
336		results = {user_id: {} for user_id in user_ids}
337
338		rows = yield self.db.simple_select_many_batch(
339		table="push_rules_enable",
340		column="user_name",
341		iterable=user_ids,
342		retcols=("user_name", "rule_id", "enabled"),
343		desc="bulk_get_push_rules_enabled",
344		)
345		for row in rows:
346		enabled = bool(row["enabled"])
347		results.setdefault(row["user_name"], {})[row["rule_id"]] = enabled
348		return results
349
350		async def get_all_push_rule_updates(
351		self, instance_name: str, last_id: int, current_id: int, limit: int
352		) -> Tuple[List[Tuple[int, tuple]], int, bool]:
353		"""Get updates for push_rules replication stream.
354
355		Args:
356		instance_name: The writer we want to fetch updates from. Unused
357		here since there is only ever one writer.
358		last_id: The token to fetch updates from. Exclusive.
359		current_id: The token to fetch updates up to. Inclusive.
360		limit: The requested limit for the number of rows to return. The
361		function may return more or fewer rows.
362
363		Returns:
364		A tuple consisting of: the updates, a token to use to fetch
365		subsequent updates, and whether we returned fewer rows than exists
366		between the requested tokens due to the limit.
367
368		The token returned can be used in a subsequent call to this
369		function to get further updatees.
370
371		The updates are a list of 2-tuples of stream ID and the row data
372		"""
373
374		if last_id == current_id:
375		return [], current_id, False
376
377		def get_all_push_rule_updates_txn(txn):
378		sql = """
379		SELECT stream_id, user_id
380		FROM push_rules_stream
381		WHERE ? < stream_id AND stream_id <= ?
382		ORDER BY stream_id ASC
383		LIMIT ?
384		"""
385		txn.execute(sql, (last_id, current_id, limit))
386		updates = [(stream_id, (user_id,)) for stream_id, user_id in txn]
387
388		limited = False
389		upper_bound = current_id
390		if len(updates) == limit:
391		limited = True
392		upper_bound = updates[-1][0]
393
394		return updates, upper_bound, limited
395
396		return await self.db.runInteraction(
397		"get_all_push_rule_updates", get_all_push_rule_updates_txn
398		)
399
400
401		class PushRuleStore(PushRulesWorkerStore):
402		@defer.inlineCallbacks
403		def add_push_rule(
404		self,
405		user_id,
406		rule_id,
407		priority_class,
408		conditions,
409		actions,
410		before=None,
411		after=None,
412		):
413		conditions_json = json.dumps(conditions)
414		actions_json = json.dumps(actions)
415		with self._push_rules_stream_id_gen.get_next() as ids:
416		stream_id, event_stream_ordering = ids
417		if before or after:
418		yield self.db.runInteraction(
419		"_add_push_rule_relative_txn",
420		self._add_push_rule_relative_txn,
421		stream_id,
422		event_stream_ordering,
423		user_id,
424		rule_id,
425		priority_class,
426		conditions_json,
427		actions_json,
428		before,
429		after,
430		)
431		else:
432		yield self.db.runInteraction(
433		"_add_push_rule_highest_priority_txn",
434		self._add_push_rule_highest_priority_txn,
435		stream_id,
436		event_stream_ordering,
437		user_id,
438		rule_id,
439		priority_class,
440		conditions_json,
441		actions_json,
442		)
443
444		def _add_push_rule_relative_txn(
445		self,
446		txn,
447		stream_id,
448		event_stream_ordering,
449		user_id,
450		rule_id,
451		priority_class,
452		conditions_json,
453		actions_json,
454		before,
455		after,
456		):
457		# Lock the table since otherwise we'll have annoying races between the
458		# SELECT here and the UPSERT below.
459		self.database_engine.lock_table(txn, "push_rules")
460
461		relative_to_rule = before or after
462
463		res = self.db.simple_select_one_txn(
464		txn,
465		table="push_rules",
466		keyvalues={"user_name": user_id, "rule_id": relative_to_rule},
467		retcols=["priority_class", "priority"],
468		allow_none=True,
469		)
470
471		if not res:
472		raise RuleNotFoundException(
473		"before/after rule not found: %s" % (relative_to_rule,)
474		)
475
476		base_priority_class = res["priority_class"]
477		base_rule_priority = res["priority"]
478
479		if base_priority_class != priority_class:
480		raise InconsistentRuleException(
481		"Given priority class does not match class of relative rule"
482		)
483
484		if before:
485		# Higher priority rules are executed first, So adding a rule before
486		# a rule means giving it a higher priority than that rule.
487		new_rule_priority = base_rule_priority + 1
488		else:
489		# We increment the priority of the existing rules to make space for
490		# the new rule. Therefore if we want this rule to appear after
491		# an existing rule we give it the priority of the existing rule,
492		# and then increment the priority of the existing rule.
493		new_rule_priority = base_rule_priority
494
495		sql = (
496		"UPDATE push_rules SET priority = priority + 1"
497		" WHERE user_name = ? AND priority_class = ? AND priority >= ?"
498		)
499
500		txn.execute(sql, (user_id, priority_class, new_rule_priority))
501
502		self._upsert_push_rule_txn(
503		txn,
504		stream_id,
505		event_stream_ordering,
506		user_id,
507		rule_id,
508		priority_class,
509		new_rule_priority,
510		conditions_json,
511		actions_json,
512		)
513
514		def _add_push_rule_highest_priority_txn(
515		self,
516		txn,
517		stream_id,
518		event_stream_ordering,
519		user_id,
520		rule_id,
521		priority_class,
522		conditions_json,
523		actions_json,
524		):
525		# Lock the table since otherwise we'll have annoying races between the
526		# SELECT here and the UPSERT below.
527		self.database_engine.lock_table(txn, "push_rules")
528
529		# find the highest priority rule in that class
530		sql = (
531		"SELECT COUNT(*), MAX(priority) FROM push_rules"
532		" WHERE user_name = ? and priority_class = ?"
533		)
534		txn.execute(sql, (user_id, priority_class))
535		res = txn.fetchall()
536		(how_many, highest_prio) = res[0]
537
538		new_prio = 0
539		if how_many > 0:
540		new_prio = highest_prio + 1
541
542		self._upsert_push_rule_txn(
543		txn,
544		stream_id,
545		event_stream_ordering,
546		user_id,
547		rule_id,
548		priority_class,
549		new_prio,
550		conditions_json,
551		actions_json,
552		)
553
554		def _upsert_push_rule_txn(
555		self,
556		txn,
557		stream_id,
558		event_stream_ordering,
559		user_id,
560		rule_id,
561		priority_class,
562		priority,
563		conditions_json,
564		actions_json,
565		update_stream=True,
566		):
567		"""Specialised version of simple_upsert_txn that picks a push_rule_id
568		using the _push_rule_id_gen if it needs to insert the rule. It assumes
569		that the "push_rules" table is locked"""
570
571		sql = (
572		"UPDATE push_rules"
573		" SET priority_class = ?, priority = ?, conditions = ?, actions = ?"
574		" WHERE user_name = ? AND rule_id = ?"
575		)
576
577		txn.execute(
578		sql,
579		(priority_class, priority, conditions_json, actions_json, user_id, rule_id),
580		)
581
582		if txn.rowcount == 0:
583		# We didn't update a row with the given rule_id so insert one
584		push_rule_id = self._push_rule_id_gen.get_next()
585
586		self.db.simple_insert_txn(
587		txn,
588		table="push_rules",
589		values={
590		"id": push_rule_id,
591		"user_name": user_id,
592		"rule_id": rule_id,
593		"priority_class": priority_class,
594		"priority": priority,
595		"conditions": conditions_json,
596		"actions": actions_json,
597		},
598		)
599
600		if update_stream:
601		self._insert_push_rules_update_txn(
602		txn,
603		stream_id,
604		event_stream_ordering,
605		user_id,
606		rule_id,
607		op="ADD",
608		data={
609		"priority_class": priority_class,
610		"priority": priority,
611		"conditions": conditions_json,
612		"actions": actions_json,
613		},
614		)
615
616		@defer.inlineCallbacks
617		def delete_push_rule(self, user_id, rule_id):
618		"""
619		Delete a push rule. Args specify the row to be deleted and can be
620		any of the columns in the push_rule table, but below are the
621		standard ones
622
623		Args:
624		user_id (str): The matrix ID of the push rule owner
625		rule_id (str): The rule_id of the rule to be deleted
626		"""
627
628		def delete_push_rule_txn(txn, stream_id, event_stream_ordering):
629		self.db.simple_delete_one_txn(
630		txn, "push_rules", {"user_name": user_id, "rule_id": rule_id}
631		)
632
633		self._insert_push_rules_update_txn(
634		txn, stream_id, event_stream_ordering, user_id, rule_id, op="DELETE"
635		)
636
637		with self._push_rules_stream_id_gen.get_next() as ids:
638		stream_id, event_stream_ordering = ids
639		yield self.db.runInteraction(
640		"delete_push_rule",
641		delete_push_rule_txn,
642		stream_id,
643		event_stream_ordering,
644		)
645
646		@defer.inlineCallbacks
647		def set_push_rule_enabled(self, user_id, rule_id, enabled):
648		with self._push_rules_stream_id_gen.get_next() as ids:
649		stream_id, event_stream_ordering = ids
650		yield self.db.runInteraction(
651		"_set_push_rule_enabled_txn",
652		self._set_push_rule_enabled_txn,
653		stream_id,
654		event_stream_ordering,
655		user_id,
656		rule_id,
657		enabled,
658		)
659
660		def _set_push_rule_enabled_txn(
661		self, txn, stream_id, event_stream_ordering, user_id, rule_id, enabled
662		):
663		new_id = self._push_rules_enable_id_gen.get_next()
664		self.db.simple_upsert_txn(
665		txn,
666		"push_rules_enable",
667		{"user_name": user_id, "rule_id": rule_id},
668		{"enabled": 1 if enabled else 0},
669		{"id": new_id},
670		)
671
672		self._insert_push_rules_update_txn(
673		txn,
674		stream_id,
675		event_stream_ordering,
676		user_id,
677		rule_id,
678		op="ENABLE" if enabled else "DISABLE",
679		)
680
681		@defer.inlineCallbacks
682		def set_push_rule_actions(self, user_id, rule_id, actions, is_default_rule):
683		actions_json = json.dumps(actions)
684
685		def set_push_rule_actions_txn(txn, stream_id, event_stream_ordering):
686		if is_default_rule:
687		# Add a dummy rule to the rules table with the user specified
688		# actions.
689		priority_class = -1
690		priority = 1
691		self._upsert_push_rule_txn(
692		txn,
693		stream_id,
694		event_stream_ordering,
695		user_id,
696		rule_id,
697		priority_class,
698		priority,
699		"[]",
700		actions_json,
701		update_stream=False,
702		)
703		else:
704		self.db.simple_update_one_txn(
705		txn,
706		"push_rules",
707		{"user_name": user_id, "rule_id": rule_id},
708		{"actions": actions_json},
709		)
710
711		self._insert_push_rules_update_txn(
712		txn,
713		stream_id,
714		event_stream_ordering,
715		user_id,
716		rule_id,
717		op="ACTIONS",
718		data={"actions": actions_json},
719		)
720
721		with self._push_rules_stream_id_gen.get_next() as ids:
722		stream_id, event_stream_ordering = ids
723		yield self.db.runInteraction(
724		"set_push_rule_actions",
725		set_push_rule_actions_txn,
726		stream_id,
727		event_stream_ordering,
728		)
729
730		def _insert_push_rules_update_txn(
731		self, txn, stream_id, event_stream_ordering, user_id, rule_id, op, data=None
732		):
733		values = {
734		"stream_id": stream_id,
735		"event_stream_ordering": event_stream_ordering,
736		"user_id": user_id,
737		"rule_id": rule_id,
738		"op": op,
739		}
740		if data is not None:
741		values.update(data)
742
743		self.db.simple_insert_txn(txn, "push_rules_stream", values=values)
744
745		txn.call_after(self.get_push_rules_for_user.invalidate, (user_id,))
746		txn.call_after(self.get_push_rules_enabled_for_user.invalidate, (user_id,))
747		txn.call_after(
748		self.push_rules_stream_cache.entity_has_changed, user_id, stream_id
749		)
750
751		def get_push_rules_stream_token(self):
752		"""Get the position of the push rules stream.
753		Returns a pair of a stream id for the push_rules stream and the
754		room stream ordering it corresponds to."""
755		return self._push_rules_stream_id_gen.get_current_token()
756
757		def get_max_push_rules_stream_id(self):
758		return self.get_push_rules_stream_token()[0]

+0

-354

~~synapse/storage/data_stores/main/pusher.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2018 New Vector Ltd
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		import logging
17		from typing import Iterable, Iterator, List, Tuple
18
19		from canonicaljson import encode_canonical_json
20
21		from twisted.internet import defer
22
23		from synapse.storage._base import SQLBaseStore, db_to_json
24		from synapse.util.caches.descriptors import cachedInlineCallbacks, cachedList
25
26		logger = logging.getLogger(__name__)
27
28
29		class PusherWorkerStore(SQLBaseStore):
30		def _decode_pushers_rows(self, rows: Iterable[dict]) -> Iterator[dict]:
31		"""JSON-decode the data in the rows returned from the `pushers` table
32
33		Drops any rows whose data cannot be decoded
34		"""
35		for r in rows:
36		dataJson = r["data"]
37		try:
38		r["data"] = db_to_json(dataJson)
39		except Exception as e:
40		logger.warning(
41		"Invalid JSON in data for pusher %d: %s, %s",
42		r["id"],
43		dataJson,
44		e.args[0],
45		)
46		continue
47
48		yield r
49
50		@defer.inlineCallbacks
51		def user_has_pusher(self, user_id):
52		ret = yield self.db.simple_select_one_onecol(
53		"pushers", {"user_name": user_id}, "id", allow_none=True
54		)
55		return ret is not None
56
57		def get_pushers_by_app_id_and_pushkey(self, app_id, pushkey):
58		return self.get_pushers_by({"app_id": app_id, "pushkey": pushkey})
59
60		def get_pushers_by_user_id(self, user_id):
61		return self.get_pushers_by({"user_name": user_id})
62
63		@defer.inlineCallbacks
64		def get_pushers_by(self, keyvalues):
65		ret = yield self.db.simple_select_list(
66		"pushers",
67		keyvalues,
68		[
69		"id",
70		"user_name",
71		"access_token",
72		"profile_tag",
73		"kind",
74		"app_id",
75		"app_display_name",
76		"device_display_name",
77		"pushkey",
78		"ts",
79		"lang",
80		"data",
81		"last_stream_ordering",
82		"last_success",
83		"failing_since",
84		],
85		desc="get_pushers_by",
86		)
87		return self._decode_pushers_rows(ret)
88
89		@defer.inlineCallbacks
90		def get_all_pushers(self):
91		def get_pushers(txn):
92		txn.execute("SELECT * FROM pushers")
93		rows = self.db.cursor_to_dict(txn)
94
95		return self._decode_pushers_rows(rows)
96
97		rows = yield self.db.runInteraction("get_all_pushers", get_pushers)
98		return rows
99
100		async def get_all_updated_pushers_rows(
101		self, instance_name: str, last_id: int, current_id: int, limit: int
102		) -> Tuple[List[Tuple[int, tuple]], int, bool]:
103		"""Get updates for pushers replication stream.
104
105		Args:
106		instance_name: The writer we want to fetch updates from. Unused
107		here since there is only ever one writer.
108		last_id: The token to fetch updates from. Exclusive.
109		current_id: The token to fetch updates up to. Inclusive.
110		limit: The requested limit for the number of rows to return. The
111		function may return more or fewer rows.
112
113		Returns:
114		A tuple consisting of: the updates, a token to use to fetch
115		subsequent updates, and whether we returned fewer rows than exists
116		between the requested tokens due to the limit.
117
118		The token returned can be used in a subsequent call to this
119		function to get further updatees.
120
121		The updates are a list of 2-tuples of stream ID and the row data
122		"""
123
124		if last_id == current_id:
125		return [], current_id, False
126
127		def get_all_updated_pushers_rows_txn(txn):
128		sql = """
129		SELECT id, user_name, app_id, pushkey
130		FROM pushers
131		WHERE ? < id AND id <= ?
132		ORDER BY id ASC LIMIT ?
133		"""
134		txn.execute(sql, (last_id, current_id, limit))
135		updates = [
136		(stream_id, (user_name, app_id, pushkey, False))
137		for stream_id, user_name, app_id, pushkey in txn
138		]
139
140		sql = """
141		SELECT stream_id, user_id, app_id, pushkey
142		FROM deleted_pushers
143		WHERE ? < stream_id AND stream_id <= ?
144		ORDER BY stream_id ASC LIMIT ?
145		"""
146		txn.execute(sql, (last_id, current_id, limit))
147		updates.extend(
148		(stream_id, (user_name, app_id, pushkey, True))
149		for stream_id, user_name, app_id, pushkey in txn
150		)
151
152		updates.sort() # Sort so that they're ordered by stream id
153
154		limited = False
155		upper_bound = current_id
156		if len(updates) >= limit:
157		limited = True
158		upper_bound = updates[-1][0]
159
160		return updates, upper_bound, limited
161
162		return await self.db.runInteraction(
163		"get_all_updated_pushers_rows", get_all_updated_pushers_rows_txn
164		)
165
166		@cachedInlineCallbacks(num_args=1, max_entries=15000)
167		def get_if_user_has_pusher(self, user_id):
168		# This only exists for the cachedList decorator
169		raise NotImplementedError()
170
171		@cachedList(
172		cached_method_name="get_if_user_has_pusher",
173		list_name="user_ids",
174		num_args=1,
175		inlineCallbacks=True,
176		)
177		def get_if_users_have_pushers(self, user_ids):
178		rows = yield self.db.simple_select_many_batch(
179		table="pushers",
180		column="user_name",
181		iterable=user_ids,
182		retcols=["user_name"],
183		desc="get_if_users_have_pushers",
184		)
185
186		result = {user_id: False for user_id in user_ids}
187		result.update({r["user_name"]: True for r in rows})
188
189		return result
190
191		@defer.inlineCallbacks
192		def update_pusher_last_stream_ordering(
193		self, app_id, pushkey, user_id, last_stream_ordering
194		):
195		yield self.db.simple_update_one(
196		"pushers",
197		{"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
198		{"last_stream_ordering": last_stream_ordering},
199		desc="update_pusher_last_stream_ordering",
200		)
201
202		@defer.inlineCallbacks
203		def update_pusher_last_stream_ordering_and_success(
204		self, app_id, pushkey, user_id, last_stream_ordering, last_success
205		):
206		"""Update the last stream ordering position we've processed up to for
207		the given pusher.
208
209		Args:
210		app_id (str)
211		pushkey (str)
212		last_stream_ordering (int)
213		last_success (int)
214
215		Returns:
216		Deferred[bool]: True if the pusher still exists; False if it has been deleted.
217		"""
218		updated = yield self.db.simple_update(
219		table="pushers",
220		keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
221		updatevalues={
222		"last_stream_ordering": last_stream_ordering,
223		"last_success": last_success,
224		},
225		desc="update_pusher_last_stream_ordering_and_success",
226		)
227
228		return bool(updated)
229
230		@defer.inlineCallbacks
231		def update_pusher_failing_since(self, app_id, pushkey, user_id, failing_since):
232		yield self.db.simple_update(
233		table="pushers",
234		keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
235		updatevalues={"failing_since": failing_since},
236		desc="update_pusher_failing_since",
237		)
238
239		@defer.inlineCallbacks
240		def get_throttle_params_by_room(self, pusher_id):
241		res = yield self.db.simple_select_list(
242		"pusher_throttle",
243		{"pusher": pusher_id},
244		["room_id", "last_sent_ts", "throttle_ms"],
245		desc="get_throttle_params_by_room",
246		)
247
248		params_by_room = {}
249		for row in res:
250		params_by_room[row["room_id"]] = {
251		"last_sent_ts": row["last_sent_ts"],
252		"throttle_ms": row["throttle_ms"],
253		}
254
255		return params_by_room
256
257		@defer.inlineCallbacks
258		def set_throttle_params(self, pusher_id, room_id, params):
259		# no need to lock because `pusher_throttle` has a primary key on
260		# (pusher, room_id) so simple_upsert will retry
261		yield self.db.simple_upsert(
262		"pusher_throttle",
263		{"pusher": pusher_id, "room_id": room_id},
264		params,
265		desc="set_throttle_params",
266		lock=False,
267		)
268
269
270		class PusherStore(PusherWorkerStore):
271		def get_pushers_stream_token(self):
272		return self._pushers_id_gen.get_current_token()
273
274		@defer.inlineCallbacks
275		def add_pusher(
276		self,
277		user_id,
278		access_token,
279		kind,
280		app_id,
281		app_display_name,
282		device_display_name,
283		pushkey,
284		pushkey_ts,
285		lang,
286		data,
287		last_stream_ordering,
288		profile_tag="",
289		):
290		with self._pushers_id_gen.get_next() as stream_id:
291		# no need to lock because `pushers` has a unique key on
292		# (app_id, pushkey, user_name) so simple_upsert will retry
293		yield self.db.simple_upsert(
294		table="pushers",
295		keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
296		values={
297		"access_token": access_token,
298		"kind": kind,
299		"app_display_name": app_display_name,
300		"device_display_name": device_display_name,
301		"ts": pushkey_ts,
302		"lang": lang,
303		"data": bytearray(encode_canonical_json(data)),
304		"last_stream_ordering": last_stream_ordering,
305		"profile_tag": profile_tag,
306		"id": stream_id,
307		},
308		desc="add_pusher",
309		lock=False,
310		)
311
312		user_has_pusher = self.get_if_user_has_pusher.cache.get(
313		(user_id,), None, update_metrics=False
314		)
315
316		if user_has_pusher is not True:
317		# invalidate, since we the user might not have had a pusher before
318		yield self.db.runInteraction(
319		"add_pusher",
320		self._invalidate_cache_and_stream,
321		self.get_if_user_has_pusher,
322		(user_id,),
323		)
324
325		@defer.inlineCallbacks
326		def delete_pusher_by_app_id_pushkey_user_id(self, app_id, pushkey, user_id):
327		def delete_pusher_txn(txn, stream_id):
328		self._invalidate_cache_and_stream(
329		txn, self.get_if_user_has_pusher, (user_id,)
330		)
331
332		self.db.simple_delete_one_txn(
333		txn,
334		"pushers",
335		{"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
336		)
337
338		# it's possible for us to end up with duplicate rows for
339		# (app_id, pushkey, user_id) at different stream_ids, but that
340		# doesn't really matter.
341		self.db.simple_insert_txn(
342		txn,
343		table="deleted_pushers",
344		values={
345		"stream_id": stream_id,
346		"app_id": app_id,
347		"pushkey": pushkey,
348		"user_id": user_id,
349		},
350		)
351
352		with self._pushers_id_gen.get_next() as stream_id:
353		yield self.db.runInteraction("delete_pusher", delete_pusher_txn, stream_id)

+0

-589

~~synapse/storage/data_stores/main/receipts.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2018 New Vector Ltd
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		import abc
17		import logging
18		from typing import List, Tuple
19
20		from canonicaljson import json
21
22		from twisted.internet import defer
23
24		from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
25		from synapse.storage.database import Database
26		from synapse.storage.util.id_generators import StreamIdGenerator
27		from synapse.util.async_helpers import ObservableDeferred
28		from synapse.util.caches.descriptors import cached, cachedInlineCallbacks, cachedList
29		from synapse.util.caches.stream_change_cache import StreamChangeCache
30
31		logger = logging.getLogger(__name__)
32
33
34		class ReceiptsWorkerStore(SQLBaseStore):
35		"""This is an abstract base class where subclasses must implement
36		`get_max_receipt_stream_id` which can be called in the initializer.
37		"""
38
39		# This ABCMeta metaclass ensures that we cannot be instantiated without
40		# the abstract methods being implemented.
41		__metaclass__ = abc.ABCMeta
42
43		def __init__(self, database: Database, db_conn, hs):
44		super(ReceiptsWorkerStore, self).__init__(database, db_conn, hs)
45
46		self._receipts_stream_cache = StreamChangeCache(
47		"ReceiptsRoomChangeCache", self.get_max_receipt_stream_id()
48		)
49
50		@abc.abstractmethod
51		def get_max_receipt_stream_id(self):
52		"""Get the current max stream ID for receipts stream
53
54		Returns:
55		int
56		"""
57		raise NotImplementedError()
58
59		@cachedInlineCallbacks()
60		def get_users_with_read_receipts_in_room(self, room_id):
61		receipts = yield self.get_receipts_for_room(room_id, "m.read")
62		return {r["user_id"] for r in receipts}
63
64		@cached(num_args=2)
65		def get_receipts_for_room(self, room_id, receipt_type):
66		return self.db.simple_select_list(
67		table="receipts_linearized",
68		keyvalues={"room_id": room_id, "receipt_type": receipt_type},
69		retcols=("user_id", "event_id"),
70		desc="get_receipts_for_room",
71		)
72
73		@cached(num_args=3)
74		def get_last_receipt_event_id_for_user(self, user_id, room_id, receipt_type):
75		return self.db.simple_select_one_onecol(
76		table="receipts_linearized",
77		keyvalues={
78		"room_id": room_id,
79		"receipt_type": receipt_type,
80		"user_id": user_id,
81		},
82		retcol="event_id",
83		desc="get_own_receipt_for_user",
84		allow_none=True,
85		)
86
87		@cachedInlineCallbacks(num_args=2)
88		def get_receipts_for_user(self, user_id, receipt_type):
89		rows = yield self.db.simple_select_list(
90		table="receipts_linearized",
91		keyvalues={"user_id": user_id, "receipt_type": receipt_type},
92		retcols=("room_id", "event_id"),
93		desc="get_receipts_for_user",
94		)
95
96		return {row["room_id"]: row["event_id"] for row in rows}
97
98		@defer.inlineCallbacks
99		def get_receipts_for_user_with_orderings(self, user_id, receipt_type):
100		def f(txn):
101		sql = (
102		"SELECT rl.room_id, rl.event_id,"
103		" e.topological_ordering, e.stream_ordering"
104		" FROM receipts_linearized AS rl"
105		" INNER JOIN events AS e USING (room_id, event_id)"
106		" WHERE rl.room_id = e.room_id"
107		" AND rl.event_id = e.event_id"
108		" AND user_id = ?"
109		)
110		txn.execute(sql, (user_id,))
111		return txn.fetchall()
112
113		rows = yield self.db.runInteraction("get_receipts_for_user_with_orderings", f)
114		return {
115		row[0]: {
116		"event_id": row[1],
117		"topological_ordering": row[2],
118		"stream_ordering": row[3],
119		}
120		for row in rows
121		}
122
123		@defer.inlineCallbacks
124		def get_linearized_receipts_for_rooms(self, room_ids, to_key, from_key=None):
125		"""Get receipts for multiple rooms for sending to clients.
126
127		Args:
128		room_ids (list): List of room_ids.
129		to_key (int): Max stream id to fetch receipts upto.
130		from_key (int): Min stream id to fetch receipts from. None fetches
131		from the start.
132
133		Returns:
134		list: A list of receipts.
135		"""
136		room_ids = set(room_ids)
137
138		if from_key is not None:
139		# Only ask the database about rooms where there have been new
140		# receipts added since `from_key`
141		room_ids = yield self._receipts_stream_cache.get_entities_changed(
142		room_ids, from_key
143		)
144
145		results = yield self._get_linearized_receipts_for_rooms(
146		room_ids, to_key, from_key=from_key
147		)
148
149		return [ev for res in results.values() for ev in res]
150
151		def get_linearized_receipts_for_room(self, room_id, to_key, from_key=None):
152		"""Get receipts for a single room for sending to clients.
153
154		Args:
155		room_ids (str): The room id.
156		to_key (int): Max stream id to fetch receipts upto.
157		from_key (int): Min stream id to fetch receipts from. None fetches
158		from the start.
159
160		Returns:
161		Deferred[list]: A list of receipts.
162		"""
163		if from_key is not None:
164		# Check the cache first to see if any new receipts have been added
165		# since`from_key`. If not we can no-op.
166		if not self._receipts_stream_cache.has_entity_changed(room_id, from_key):
167		defer.succeed([])
168
169		return self._get_linearized_receipts_for_room(room_id, to_key, from_key)
170
171		@cachedInlineCallbacks(num_args=3, tree=True)
172		def _get_linearized_receipts_for_room(self, room_id, to_key, from_key=None):
173		"""See get_linearized_receipts_for_room
174		"""
175
176		def f(txn):
177		if from_key:
178		sql = (
179		"SELECT * FROM receipts_linearized WHERE"
180		" room_id = ? AND stream_id > ? AND stream_id <= ?"
181		)
182
183		txn.execute(sql, (room_id, from_key, to_key))
184		else:
185		sql = (
186		"SELECT * FROM receipts_linearized WHERE"
187		" room_id = ? AND stream_id <= ?"
188		)
189
190		txn.execute(sql, (room_id, to_key))
191
192		rows = self.db.cursor_to_dict(txn)
193
194		return rows
195
196		rows = yield self.db.runInteraction("get_linearized_receipts_for_room", f)
197
198		if not rows:
199		return []
200
201		content = {}
202		for row in rows:
203		content.setdefault(row["event_id"], {}).setdefault(row["receipt_type"], {})[
204		row["user_id"]
205		] = db_to_json(row["data"])
206
207		return [{"type": "m.receipt", "room_id": room_id, "content": content}]
208
209		@cachedList(
210		cached_method_name="_get_linearized_receipts_for_room",
211		list_name="room_ids",
212		num_args=3,
213		inlineCallbacks=True,
214		)
215		def _get_linearized_receipts_for_rooms(self, room_ids, to_key, from_key=None):
216		if not room_ids:
217		return {}
218
219		def f(txn):
220		if from_key:
221		sql = """
222		SELECT * FROM receipts_linearized WHERE
223		stream_id > ? AND stream_id <= ? AND
224		"""
225		clause, args = make_in_list_sql_clause(
226		self.database_engine, "room_id", room_ids
227		)
228
229		txn.execute(sql + clause, [from_key, to_key] + list(args))
230		else:
231		sql = """
232		SELECT * FROM receipts_linearized WHERE
233		stream_id <= ? AND
234		"""
235
236		clause, args = make_in_list_sql_clause(
237		self.database_engine, "room_id", room_ids
238		)
239
240		txn.execute(sql + clause, [to_key] + list(args))
241
242		return self.db.cursor_to_dict(txn)
243
244		txn_results = yield self.db.runInteraction(
245		"_get_linearized_receipts_for_rooms", f
246		)
247
248		results = {}
249		for row in txn_results:
250		# We want a single event per room, since we want to batch the
251		# receipts by room, event and type.
252		room_event = results.setdefault(
253		row["room_id"],
254		{"type": "m.receipt", "room_id": row["room_id"], "content": {}},
255		)
256
257		# The content is of the form:
258		# {"$foo:bar": { "read": { "@user:host": <receipt> }, .. }, .. }
259		event_entry = room_event["content"].setdefault(row["event_id"], {})
260		receipt_type = event_entry.setdefault(row["receipt_type"], {})
261
262		receipt_type[row["user_id"]] = db_to_json(row["data"])
263
264		results = {
265		room_id: [results[room_id]] if room_id in results else []
266		for room_id in room_ids
267		}
268		return results
269
270		def get_users_sent_receipts_between(self, last_id: int, current_id: int):
271		"""Get all users who sent receipts between `last_id` exclusive and
272		`current_id` inclusive.
273
274		Returns:
275		Deferred[List[str]]
276		"""
277
278		if last_id == current_id:
279		return defer.succeed([])
280
281		def _get_users_sent_receipts_between_txn(txn):
282		sql = """
283		SELECT DISTINCT user_id FROM receipts_linearized
284		WHERE ? < stream_id AND stream_id <= ?
285		"""
286		txn.execute(sql, (last_id, current_id))
287
288		return [r[0] for r in txn]
289
290		return self.db.runInteraction(
291		"get_users_sent_receipts_between", _get_users_sent_receipts_between_txn
292		)
293
294		async def get_all_updated_receipts(
295		self, instance_name: str, last_id: int, current_id: int, limit: int
296		) -> Tuple[List[Tuple[int, list]], int, bool]:
297		"""Get updates for receipts replication stream.
298
299		Args:
300		instance_name: The writer we want to fetch updates from. Unused
301		here since there is only ever one writer.
302		last_id: The token to fetch updates from. Exclusive.
303		current_id: The token to fetch updates up to. Inclusive.
304		limit: The requested limit for the number of rows to return. The
305		function may return more or fewer rows.
306
307		Returns:
308		A tuple consisting of: the updates, a token to use to fetch
309		subsequent updates, and whether we returned fewer rows than exists
310		between the requested tokens due to the limit.
311
312		The token returned can be used in a subsequent call to this
313		function to get further updatees.
314
315		The updates are a list of 2-tuples of stream ID and the row data
316		"""
317
318		if last_id == current_id:
319		return [], current_id, False
320
321		def get_all_updated_receipts_txn(txn):
322		sql = """
323		SELECT stream_id, room_id, receipt_type, user_id, event_id, data
324		FROM receipts_linearized
325		WHERE ? < stream_id AND stream_id <= ?
326		ORDER BY stream_id ASC
327		LIMIT ?
328		"""
329		txn.execute(sql, (last_id, current_id, limit))
330
331		updates = [(r[0], r[1:5] + (db_to_json(r[5]),)) for r in txn]
332
333		limited = False
334		upper_bound = current_id
335
336		if len(updates) == limit:
337		limited = True
338		upper_bound = updates[-1][0]
339
340		return updates, upper_bound, limited
341
342		return await self.db.runInteraction(
343		"get_all_updated_receipts", get_all_updated_receipts_txn
344		)
345
346		def _invalidate_get_users_with_receipts_in_room(
347		self, room_id, receipt_type, user_id
348		):
349		if receipt_type != "m.read":
350		return
351
352		# Returns either an ObservableDeferred or the raw result
353		res = self.get_users_with_read_receipts_in_room.cache.get(
354		room_id, None, update_metrics=False
355		)
356
357		# first handle the ObservableDeferred case
358		if isinstance(res, ObservableDeferred):
359		if res.has_called():
360		res = res.get_result()
361		else:
362		res = None
363
364		if res and user_id in res:
365		# We'd only be adding to the set, so no point invalidating if the
366		# user is already there
367		return
368
369		self.get_users_with_read_receipts_in_room.invalidate((room_id,))
370
371
372		class ReceiptsStore(ReceiptsWorkerStore):
373		def __init__(self, database: Database, db_conn, hs):
374		# We instantiate this first as the ReceiptsWorkerStore constructor
375		# needs to be able to call get_max_receipt_stream_id
376		self._receipts_id_gen = StreamIdGenerator(
377		db_conn, "receipts_linearized", "stream_id"
378		)
379
380		super(ReceiptsStore, self).__init__(database, db_conn, hs)
381
382		def get_max_receipt_stream_id(self):
383		return self._receipts_id_gen.get_current_token()
384
385		def insert_linearized_receipt_txn(
386		self, txn, room_id, receipt_type, user_id, event_id, data, stream_id
387		):
388		"""Inserts a read-receipt into the database if it's newer than the current RR
389
390		Returns: int\|None
391		None if the RR is older than the current RR
392		otherwise, the rx timestamp of the event that the RR corresponds to
393		(or 0 if the event is unknown)
394		"""
395		res = self.db.simple_select_one_txn(
396		txn,
397		table="events",
398		retcols=["stream_ordering", "received_ts"],
399		keyvalues={"event_id": event_id},
400		allow_none=True,
401		)
402
403		stream_ordering = int(res["stream_ordering"]) if res else None
404		rx_ts = res["received_ts"] if res else 0
405
406		# We don't want to clobber receipts for more recent events, so we
407		# have to compare orderings of existing receipts
408		if stream_ordering is not None:
409		sql = (
410		"SELECT stream_ordering, event_id FROM events"
411		" INNER JOIN receipts_linearized as r USING (event_id, room_id)"
412		" WHERE r.room_id = ? AND r.receipt_type = ? AND r.user_id = ?"
413		)
414		txn.execute(sql, (room_id, receipt_type, user_id))
415
416		for so, eid in txn:
417		if int(so) >= stream_ordering:
418		logger.debug(
419		"Ignoring new receipt for %s in favour of existing "
420		"one for later event %s",
421		event_id,
422		eid,
423		)
424		return None
425
426		txn.call_after(self.get_receipts_for_room.invalidate, (room_id, receipt_type))
427		txn.call_after(
428		self._invalidate_get_users_with_receipts_in_room,
429		room_id,
430		receipt_type,
431		user_id,
432		)
433		txn.call_after(self.get_receipts_for_user.invalidate, (user_id, receipt_type))
434		# FIXME: This shouldn't invalidate the whole cache
435		txn.call_after(
436		self._get_linearized_receipts_for_room.invalidate_many, (room_id,)
437		)
438
439		txn.call_after(
440		self._receipts_stream_cache.entity_has_changed, room_id, stream_id
441		)
442
443		txn.call_after(
444		self.get_last_receipt_event_id_for_user.invalidate,
445		(user_id, room_id, receipt_type),
446		)
447
448		self.db.simple_upsert_txn(
449		txn,
450		table="receipts_linearized",
451		keyvalues={
452		"room_id": room_id,
453		"receipt_type": receipt_type,
454		"user_id": user_id,
455		},
456		values={
457		"stream_id": stream_id,
458		"event_id": event_id,
459		"data": json.dumps(data),
460		},
461		# receipts_linearized has a unique constraint on
462		# (user_id, room_id, receipt_type), so no need to lock
463		lock=False,
464		)
465
466		if receipt_type == "m.read" and stream_ordering is not None:
467		self._remove_old_push_actions_before_txn(
468		txn, room_id=room_id, user_id=user_id, stream_ordering=stream_ordering
469		)
470
471		return rx_ts
472
473		@defer.inlineCallbacks
474		def insert_receipt(self, room_id, receipt_type, user_id, event_ids, data):
475		"""Insert a receipt, either from local client or remote server.
476
477		Automatically does conversion between linearized and graph
478		representations.
479		"""
480		if not event_ids:
481		return
482
483		if len(event_ids) == 1:
484		linearized_event_id = event_ids[0]
485		else:
486		# we need to points in graph -> linearized form.
487		# TODO: Make this better.
488		def graph_to_linear(txn):
489		clause, args = make_in_list_sql_clause(
490		self.database_engine, "event_id", event_ids
491		)
492
493		sql = """
494		SELECT event_id WHERE room_id = ? AND stream_ordering IN (
495		SELECT max(stream_ordering) WHERE %s
496		)
497		""" % (
498		clause,
499		)
500
501		txn.execute(sql, [room_id] + list(args))
502		rows = txn.fetchall()
503		if rows:
504		return rows[0][0]
505		else:
506		raise RuntimeError("Unrecognized event_ids: %r" % (event_ids,))
507
508		linearized_event_id = yield self.db.runInteraction(
509		"insert_receipt_conv", graph_to_linear
510		)
511
512		stream_id_manager = self._receipts_id_gen.get_next()
513		with stream_id_manager as stream_id:
514		event_ts = yield self.db.runInteraction(
515		"insert_linearized_receipt",
516		self.insert_linearized_receipt_txn,
517		room_id,
518		receipt_type,
519		user_id,
520		linearized_event_id,
521		data,
522		stream_id=stream_id,
523		)
524
525		if event_ts is None:
526		return None
527
528		now = self._clock.time_msec()
529		logger.debug(
530		"RR for event %s in %s (%i ms old)",
531		linearized_event_id,
532		room_id,
533		now - event_ts,
534		)
535
536		yield self.insert_graph_receipt(room_id, receipt_type, user_id, event_ids, data)
537
538		max_persisted_id = self._receipts_id_gen.get_current_token()
539
540		return stream_id, max_persisted_id
541
542		def insert_graph_receipt(self, room_id, receipt_type, user_id, event_ids, data):
543		return self.db.runInteraction(
544		"insert_graph_receipt",
545		self.insert_graph_receipt_txn,
546		room_id,
547		receipt_type,
548		user_id,
549		event_ids,
550		data,
551		)
552
553		def insert_graph_receipt_txn(
554		self, txn, room_id, receipt_type, user_id, event_ids, data
555		):
556		txn.call_after(self.get_receipts_for_room.invalidate, (room_id, receipt_type))
557		txn.call_after(
558		self._invalidate_get_users_with_receipts_in_room,
559		room_id,
560		receipt_type,
561		user_id,
562		)
563		txn.call_after(self.get_receipts_for_user.invalidate, (user_id, receipt_type))
564		# FIXME: This shouldn't invalidate the whole cache
565		txn.call_after(
566		self._get_linearized_receipts_for_room.invalidate_many, (room_id,)
567		)
568
569		self.db.simple_delete_txn(
570		txn,
571		table="receipts_graph",
572		keyvalues={
573		"room_id": room_id,
574		"receipt_type": receipt_type,
575		"user_id": user_id,
576		},
577		)
578		self.db.simple_insert_txn(
579		txn,
580		table="receipts_graph",
581		values={
582		"room_id": room_id,
583		"receipt_type": receipt_type,
584		"user_id": user_id,
585		"event_ids": json.dumps(event_ids),
586		"data": json.dumps(data),
587		},
588		)

+0

-1582

~~synapse/storage/data_stores/main/registration.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2017-2018 New Vector Ltd
3		# Copyright 2019 The Matrix.org Foundation C.I.C.
4		#
5		# Licensed under the Apache License, Version 2.0 (the "License");
6		# you may not use this file except in compliance with the License.
7		# You may obtain a copy of the License at
8		#
9		# http://www.apache.org/licenses/LICENSE-2.0
10		#
11		# Unless required by applicable law or agreed to in writing, software
12		# distributed under the License is distributed on an "AS IS" BASIS,
13		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14		# See the License for the specific language governing permissions and
15		# limitations under the License.
16
17		import logging
18		import re
19		from typing import Optional
20
21		from twisted.internet import defer
22		from twisted.internet.defer import Deferred
23
24		from synapse.api.constants import UserTypes
25		from synapse.api.errors import Codes, StoreError, SynapseError, ThreepidValidationError
26		from synapse.metrics.background_process_metrics import run_as_background_process
27		from synapse.storage._base import SQLBaseStore
28		from synapse.storage.database import Database
29		from synapse.storage.types import Cursor
30		from synapse.storage.util.sequence import build_sequence_generator
31		from synapse.types import UserID
32		from synapse.util.caches.descriptors import cached, cachedInlineCallbacks
33
34		THIRTY_MINUTES_IN_MS = 30 * 60 * 1000
35
36		logger = logging.getLogger(__name__)
37
38
39		class RegistrationWorkerStore(SQLBaseStore):
40		def __init__(self, database: Database, db_conn, hs):
41		super(RegistrationWorkerStore, self).__init__(database, db_conn, hs)
42
43		self.config = hs.config
44		self.clock = hs.get_clock()
45
46		self._user_id_seq = build_sequence_generator(
47		database.engine, find_max_generated_user_id_localpart, "user_id_seq",
48		)
49
50		@cached()
51		def get_user_by_id(self, user_id):
52		return self.db.simple_select_one(
53		table="users",
54		keyvalues={"name": user_id},
55		retcols=[
56		"name",
57		"password_hash",
58		"is_guest",
59		"admin",
60		"consent_version",
61		"consent_server_notice_sent",
62		"appservice_id",
63		"creation_ts",
64		"user_type",
65		"deactivated",
66		],
67		allow_none=True,
68		desc="get_user_by_id",
69		)
70
71		@defer.inlineCallbacks
72		def is_trial_user(self, user_id):
73		"""Checks if user is in the "trial" period, i.e. within the first
74		N days of registration defined by `mau_trial_days` config
75
76		Args:
77		user_id (str)
78
79		Returns:
80		Deferred[bool]
81		"""
82
83		info = yield self.get_user_by_id(user_id)
84		if not info:
85		return False
86
87		now = self.clock.time_msec()
88		trial_duration_ms = self.config.mau_trial_days * 24 * 60 * 60 * 1000
89		is_trial = (now - info["creation_ts"] * 1000) < trial_duration_ms
90		return is_trial
91
92		@cached()
93		def get_user_by_access_token(self, token):
94		"""Get a user from the given access token.
95
96		Args:
97		token (str): The access token of a user.
98		Returns:
99		defer.Deferred: None, if the token did not match, otherwise dict
100		including the keys `name`, `is_guest`, `device_id`, `token_id`,
101		`valid_until_ms`.
102		"""
103		return self.db.runInteraction(
104		"get_user_by_access_token", self._query_for_auth, token
105		)
106
107		@cachedInlineCallbacks()
108		def get_expiration_ts_for_user(self, user_id):
109		"""Get the expiration timestamp for the account bearing a given user ID.
110
111		Args:
112		user_id (str): The ID of the user.
113		Returns:
114		defer.Deferred: None, if the account has no expiration timestamp,
115		otherwise int representation of the timestamp (as a number of
116		milliseconds since epoch).
117		"""
118		res = yield self.db.simple_select_one_onecol(
119		table="account_validity",
120		keyvalues={"user_id": user_id},
121		retcol="expiration_ts_ms",
122		allow_none=True,
123		desc="get_expiration_ts_for_user",
124		)
125		return res
126
127		@defer.inlineCallbacks
128		def set_account_validity_for_user(
129		self, user_id, expiration_ts, email_sent, renewal_token=None
130		):
131		"""Updates the account validity properties of the given account, with the
132		given values.
133
134		Args:
135		user_id (str): ID of the account to update properties for.
136		expiration_ts (int): New expiration date, as a timestamp in milliseconds
137		since epoch.
138		email_sent (bool): True means a renewal email has been sent for this
139		account and there's no need to send another one for the current validity
140		period.
141		renewal_token (str): Renewal token the user can use to extend the validity
142		of their account. Defaults to no token.
143		"""
144
145		def set_account_validity_for_user_txn(txn):
146		self.db.simple_update_txn(
147		txn=txn,
148		table="account_validity",
149		keyvalues={"user_id": user_id},
150		updatevalues={
151		"expiration_ts_ms": expiration_ts,
152		"email_sent": email_sent,
153		"renewal_token": renewal_token,
154		},
155		)
156		self._invalidate_cache_and_stream(
157		txn, self.get_expiration_ts_for_user, (user_id,)
158		)
159
160		yield self.db.runInteraction(
161		"set_account_validity_for_user", set_account_validity_for_user_txn
162		)
163
164		@defer.inlineCallbacks
165		def set_renewal_token_for_user(self, user_id, renewal_token):
166		"""Defines a renewal token for a given user.
167
168		Args:
169		user_id (str): ID of the user to set the renewal token for.
170		renewal_token (str): Random unique string that will be used to renew the
171		user's account.
172
173		Raises:
174		StoreError: The provided token is already set for another user.
175		"""
176		yield self.db.simple_update_one(
177		table="account_validity",
178		keyvalues={"user_id": user_id},
179		updatevalues={"renewal_token": renewal_token},
180		desc="set_renewal_token_for_user",
181		)
182
183		@defer.inlineCallbacks
184		def get_user_from_renewal_token(self, renewal_token):
185		"""Get a user ID from a renewal token.
186
187		Args:
188		renewal_token (str): The renewal token to perform the lookup with.
189
190		Returns:
191		defer.Deferred[str]: The ID of the user to which the token belongs.
192		"""
193		res = yield self.db.simple_select_one_onecol(
194		table="account_validity",
195		keyvalues={"renewal_token": renewal_token},
196		retcol="user_id",
197		desc="get_user_from_renewal_token",
198		)
199
200		return res
201
202		@defer.inlineCallbacks
203		def get_renewal_token_for_user(self, user_id):
204		"""Get the renewal token associated with a given user ID.
205
206		Args:
207		user_id (str): The user ID to lookup a token for.
208
209		Returns:
210		defer.Deferred[str]: The renewal token associated with this user ID.
211		"""
212		res = yield self.db.simple_select_one_onecol(
213		table="account_validity",
214		keyvalues={"user_id": user_id},
215		retcol="renewal_token",
216		desc="get_renewal_token_for_user",
217		)
218
219		return res
220
221		@defer.inlineCallbacks
222		def get_users_expiring_soon(self):
223		"""Selects users whose account will expire in the [now, now + renew_at] time
224		window (see configuration for account_validity for information on what renew_at
225		refers to).
226
227		Returns:
228		Deferred: Resolves to a list[dict[user_id (str), expiration_ts_ms (int)]]
229		"""
230
231		def select_users_txn(txn, now_ms, renew_at):
232		sql = (
233		"SELECT user_id, expiration_ts_ms FROM account_validity"
234		" WHERE email_sent = ? AND (expiration_ts_ms - ?) <= ?"
235		)
236		values = [False, now_ms, renew_at]
237		txn.execute(sql, values)
238		return self.db.cursor_to_dict(txn)
239
240		res = yield self.db.runInteraction(
241		"get_users_expiring_soon",
242		select_users_txn,
243		self.clock.time_msec(),
244		self.config.account_validity.renew_at,
245		)
246
247		return res
248
249		@defer.inlineCallbacks
250		def set_renewal_mail_status(self, user_id, email_sent):
251		"""Sets or unsets the flag that indicates whether a renewal email has been sent
252		to the user (and the user hasn't renewed their account yet).
253
254		Args:
255		user_id (str): ID of the user to set/unset the flag for.
256		email_sent (bool): Flag which indicates whether a renewal email has been sent
257		to this user.
258		"""
259		yield self.db.simple_update_one(
260		table="account_validity",
261		keyvalues={"user_id": user_id},
262		updatevalues={"email_sent": email_sent},
263		desc="set_renewal_mail_status",
264		)
265
266		@defer.inlineCallbacks
267		def delete_account_validity_for_user(self, user_id):
268		"""Deletes the entry for the given user in the account validity table, removing
269		their expiration date and renewal token.
270
271		Args:
272		user_id (str): ID of the user to remove from the account validity table.
273		"""
274		yield self.db.simple_delete_one(
275		table="account_validity",
276		keyvalues={"user_id": user_id},
277		desc="delete_account_validity_for_user",
278		)
279
280		async def is_server_admin(self, user):
281		"""Determines if a user is an admin of this homeserver.
282
283		Args:
284		user (UserID): user ID of the user to test
285
286		Returns (bool):
287		true iff the user is a server admin, false otherwise.
288		"""
289		res = await self.db.simple_select_one_onecol(
290		table="users",
291		keyvalues={"name": user.to_string()},
292		retcol="admin",
293		allow_none=True,
294		desc="is_server_admin",
295		)
296
297		return bool(res) if res else False
298
299		def set_server_admin(self, user, admin):
300		"""Sets whether a user is an admin of this homeserver.
301
302		Args:
303		user (UserID): user ID of the user to test
304		admin (bool): true iff the user is to be a server admin,
305		false otherwise.
306		"""
307
308		def set_server_admin_txn(txn):
309		self.db.simple_update_one_txn(
310		txn, "users", {"name": user.to_string()}, {"admin": 1 if admin else 0}
311		)
312		self._invalidate_cache_and_stream(
313		txn, self.get_user_by_id, (user.to_string(),)
314		)
315
316		return self.db.runInteraction("set_server_admin", set_server_admin_txn)
317
318		def _query_for_auth(self, txn, token):
319		sql = (
320		"SELECT users.name, users.is_guest, access_tokens.id as token_id,"
321		" access_tokens.device_id, access_tokens.valid_until_ms"
322		" FROM users"
323		" INNER JOIN access_tokens on users.name = access_tokens.user_id"
324		" WHERE token = ?"
325		)
326
327		txn.execute(sql, (token,))
328		rows = self.db.cursor_to_dict(txn)
329		if rows:
330		return rows[0]
331
332		return None
333
334		@cachedInlineCallbacks()
335		def is_real_user(self, user_id):
336		"""Determines if the user is a real user, ie does not have a 'user_type'.
337
338		Args:
339		user_id (str): user id to test
340
341		Returns:
342		Deferred[bool]: True if user 'user_type' is null or empty string
343		"""
344		res = yield self.db.runInteraction(
345		"is_real_user", self.is_real_user_txn, user_id
346		)
347		return res
348
349		@cached()
350		def is_support_user(self, user_id):
351		"""Determines if the user is of type UserTypes.SUPPORT
352
353		Args:
354		user_id (str): user id to test
355
356		Returns:
357		Deferred[bool]: True if user is of type UserTypes.SUPPORT
358		"""
359		return self.db.runInteraction(
360		"is_support_user", self.is_support_user_txn, user_id
361		)
362
363		def is_real_user_txn(self, txn, user_id):
364		res = self.db.simple_select_one_onecol_txn(
365		txn=txn,
366		table="users",
367		keyvalues={"name": user_id},
368		retcol="user_type",
369		allow_none=True,
370		)
371		return res is None
372
373		def is_support_user_txn(self, txn, user_id):
374		res = self.db.simple_select_one_onecol_txn(
375		txn=txn,
376		table="users",
377		keyvalues={"name": user_id},
378		retcol="user_type",
379		allow_none=True,
380		)
381		return True if res == UserTypes.SUPPORT else False
382
383		def get_users_by_id_case_insensitive(self, user_id):
384		"""Gets users that match user_id case insensitively.
385		Returns a mapping of user_id -> password_hash.
386		"""
387
388		def f(txn):
389		sql = "SELECT name, password_hash FROM users WHERE lower(name) = lower(?)"
390		txn.execute(sql, (user_id,))
391		return dict(txn)
392
393		return self.db.runInteraction("get_users_by_id_case_insensitive", f)
394
395		async def get_user_by_external_id(
396		self, auth_provider: str, external_id: str
397		) -> str:
398		"""Look up a user by their external auth id
399
400		Args:
401		auth_provider: identifier for the remote auth provider
402		external_id: id on that system
403
404		Returns:
405		str\|None: the mxid of the user, or None if they are not known
406		"""
407		return await self.db.simple_select_one_onecol(
408		table="user_external_ids",
409		keyvalues={"auth_provider": auth_provider, "external_id": external_id},
410		retcol="user_id",
411		allow_none=True,
412		desc="get_user_by_external_id",
413		)
414
415		@defer.inlineCallbacks
416		def count_all_users(self):
417		"""Counts all users registered on the homeserver."""
418
419		def _count_users(txn):
420		txn.execute("SELECT COUNT(*) AS users FROM users")
421		rows = self.db.cursor_to_dict(txn)
422		if rows:
423		return rows[0]["users"]
424		return 0
425
426		ret = yield self.db.runInteraction("count_users", _count_users)
427		return ret
428
429		def count_daily_user_type(self):
430		"""
431		Counts 1) native non guest users
432		2) native guests users
433		3) bridged users
434		who registered on the homeserver in the past 24 hours
435		"""
436
437		def _count_daily_user_type(txn):
438		yesterday = int(self._clock.time()) - (60 * 60 * 24)
439
440		sql = """
441		SELECT user_type, COALESCE(count(*), 0) AS count FROM (
442		SELECT
443		CASE
444		WHEN is_guest=0 AND appservice_id IS NULL THEN 'native'
445		WHEN is_guest=1 AND appservice_id IS NULL THEN 'guest'
446		WHEN is_guest=0 AND appservice_id IS NOT NULL THEN 'bridged'
447		END AS user_type
448		FROM users
449		WHERE creation_ts > ?
450		) AS t GROUP BY user_type
451		"""
452		results = {"native": 0, "guest": 0, "bridged": 0}
453		txn.execute(sql, (yesterday,))
454		for row in txn:
455		results[row[0]] = row[1]
456		return results
457
458		return self.db.runInteraction("count_daily_user_type", _count_daily_user_type)
459
460		@defer.inlineCallbacks
461		def count_nonbridged_users(self):
462		def _count_users(txn):
463		txn.execute(
464		"""
465		SELECT COALESCE(COUNT(*), 0) FROM users
466		WHERE appservice_id IS NULL
467		"""
468		)
469		(count,) = txn.fetchone()
470		return count
471
472		ret = yield self.db.runInteraction("count_users", _count_users)
473		return ret
474
475		@defer.inlineCallbacks
476		def count_real_users(self):
477		"""Counts all users without a special user_type registered on the homeserver."""
478
479		def _count_users(txn):
480		txn.execute("SELECT COUNT(*) AS users FROM users where user_type is null")
481		rows = self.db.cursor_to_dict(txn)
482		if rows:
483		return rows[0]["users"]
484		return 0
485
486		ret = yield self.db.runInteraction("count_real_users", _count_users)
487		return ret
488
489		async def generate_user_id(self) -> str:
490		"""Generate a suitable localpart for a guest user
491
492		Returns: a (hopefully) free localpart
493		"""
494		next_id = await self.db.runInteraction(
495		"generate_user_id", self._user_id_seq.get_next_id_txn
496		)
497
498		return str(next_id)
499
500		async def get_user_id_by_threepid(self, medium: str, address: str) -> Optional[str]:
501		"""Returns user id from threepid
502
503		Args:
504		medium: threepid medium e.g. email
505		address: threepid address e.g. me@example.com
506
507		Returns:
508		The user ID or None if no user id/threepid mapping exists
509		"""
510		user_id = await self.db.runInteraction(
511		"get_user_id_by_threepid", self.get_user_id_by_threepid_txn, medium, address
512		)
513		return user_id
514
515		def get_user_id_by_threepid_txn(self, txn, medium, address):
516		"""Returns user id from threepid
517
518		Args:
519		txn (cursor):
520		medium (str): threepid medium e.g. email
521		address (str): threepid address e.g. me@example.com
522
523		Returns:
524		str\|None: user id or None if no user id/threepid mapping exists
525		"""
526		ret = self.db.simple_select_one_txn(
527		txn,
528		"user_threepids",
529		{"medium": medium, "address": address},
530		["user_id"],
531		True,
532		)
533		if ret:
534		return ret["user_id"]
535		return None
536
537		@defer.inlineCallbacks
538		def user_add_threepid(self, user_id, medium, address, validated_at, added_at):
539		yield self.db.simple_upsert(
540		"user_threepids",
541		{"medium": medium, "address": address},
542		{"user_id": user_id, "validated_at": validated_at, "added_at": added_at},
543		)
544
545		@defer.inlineCallbacks
546		def user_get_threepids(self, user_id):
547		ret = yield self.db.simple_select_list(
548		"user_threepids",
549		{"user_id": user_id},
550		["medium", "address", "validated_at", "added_at"],
551		"user_get_threepids",
552		)
553		return ret
554
555		def user_delete_threepid(self, user_id, medium, address):
556		return self.db.simple_delete(
557		"user_threepids",
558		keyvalues={"user_id": user_id, "medium": medium, "address": address},
559		desc="user_delete_threepid",
560		)
561
562		def user_delete_threepids(self, user_id: str):
563		"""Delete all threepid this user has bound
564
565		Args:
566		user_id: The user id to delete all threepids of
567
568		"""
569		return self.db.simple_delete(
570		"user_threepids",
571		keyvalues={"user_id": user_id},
572		desc="user_delete_threepids",
573		)
574
575		def add_user_bound_threepid(self, user_id, medium, address, id_server):
576		"""The server proxied a bind request to the given identity server on
577		behalf of the given user. We need to remember this in case the user
578		asks us to unbind the threepid.
579
580		Args:
581		user_id (str)
582		medium (str)
583		address (str)
584		id_server (str)
585
586		Returns:
587		Deferred
588		"""
589		# We need to use an upsert, in case they user had already bound the
590		# threepid
591		return self.db.simple_upsert(
592		table="user_threepid_id_server",
593		keyvalues={
594		"user_id": user_id,
595		"medium": medium,
596		"address": address,
597		"id_server": id_server,
598		},
599		values={},
600		insertion_values={},
601		desc="add_user_bound_threepid",
602		)
603
604		def user_get_bound_threepids(self, user_id):
605		"""Get the threepids that a user has bound to an identity server through the homeserver
606		The homeserver remembers where binds to an identity server occurred. Using this
607		method can retrieve those threepids.
608
609		Args:
610		user_id (str): The ID of the user to retrieve threepids for
611
612		Returns:
613		Deferred[list[dict]]: List of dictionaries containing the following:
614		medium (str): The medium of the threepid (e.g "email")
615		address (str): The address of the threepid (e.g "bob@example.com")
616		"""
617		return self.db.simple_select_list(
618		table="user_threepid_id_server",
619		keyvalues={"user_id": user_id},
620		retcols=["medium", "address"],
621		desc="user_get_bound_threepids",
622		)
623
624		def remove_user_bound_threepid(self, user_id, medium, address, id_server):
625		"""The server proxied an unbind request to the given identity server on
626		behalf of the given user, so we remove the mapping of threepid to
627		identity server.
628
629		Args:
630		user_id (str)
631		medium (str)
632		address (str)
633		id_server (str)
634
635		Returns:
636		Deferred
637		"""
638		return self.db.simple_delete(
639		table="user_threepid_id_server",
640		keyvalues={
641		"user_id": user_id,
642		"medium": medium,
643		"address": address,
644		"id_server": id_server,
645		},
646		desc="remove_user_bound_threepid",
647		)
648
649		def get_id_servers_user_bound(self, user_id, medium, address):
650		"""Get the list of identity servers that the server proxied bind
651		requests to for given user and threepid
652
653		Args:
654		user_id (str)
655		medium (str)
656		address (str)
657
658		Returns:
659		Deferred[list[str]]: Resolves to a list of identity servers
660		"""
661		return self.db.simple_select_onecol(
662		table="user_threepid_id_server",
663		keyvalues={"user_id": user_id, "medium": medium, "address": address},
664		retcol="id_server",
665		desc="get_id_servers_user_bound",
666		)
667
668		@cachedInlineCallbacks()
669		def get_user_deactivated_status(self, user_id):
670		"""Retrieve the value for the `deactivated` property for the provided user.
671
672		Args:
673		user_id (str): The ID of the user to retrieve the status for.
674
675		Returns:
676		defer.Deferred(bool): The requested value.
677		"""
678
679		res = yield self.db.simple_select_one_onecol(
680		table="users",
681		keyvalues={"name": user_id},
682		retcol="deactivated",
683		desc="get_user_deactivated_status",
684		)
685
686		# Convert the integer into a boolean.
687		return res == 1
688
689		def get_threepid_validation_session(
690		self, medium, client_secret, address=None, sid=None, validated=True
691		):
692		"""Gets a session_id and last_send_attempt (if available) for a
693		combination of validation metadata
694
695		Args:
696		medium (str\|None): The medium of the 3PID
697		address (str\|None): The address of the 3PID
698		sid (str\|None): The ID of the validation session
699		client_secret (str): A unique string provided by the client to help identify this
700		validation attempt
701		validated (bool\|None): Whether sessions should be filtered by
702		whether they have been validated already or not. None to
703		perform no filtering
704
705		Returns:
706		Deferred[dict\|None]: A dict containing the following:
707		* address - address of the 3pid
708		* medium - medium of the 3pid
709		* client_secret - a secret provided by the client for this validation session
710		* session_id - ID of the validation session
711		* send_attempt - a number serving to dedupe send attempts for this session
712		* validated_at - timestamp of when this session was validated if so
713
714		Otherwise None if a validation session is not found
715		"""
716		if not client_secret:
717		raise SynapseError(
718		400, "Missing parameter: client_secret", errcode=Codes.MISSING_PARAM
719		)
720
721		keyvalues = {"client_secret": client_secret}
722		if medium:
723		keyvalues["medium"] = medium
724		if address:
725		keyvalues["address"] = address
726		if sid:
727		keyvalues["session_id"] = sid
728
729		assert address or sid
730
731		def get_threepid_validation_session_txn(txn):
732		sql = """
733		SELECT address, session_id, medium, client_secret,
734		last_send_attempt, validated_at
735		FROM threepid_validation_session WHERE %s
736		""" % (
737		" AND ".join("%s = ?" % k for k in keyvalues.keys()),
738		)
739
740		if validated is not None:
741		sql += " AND validated_at IS " + ("NOT NULL" if validated else "NULL")
742
743		sql += " LIMIT 1"
744
745		txn.execute(sql, list(keyvalues.values()))
746		rows = self.db.cursor_to_dict(txn)
747		if not rows:
748		return None
749
750		return rows[0]
751
752		return self.db.runInteraction(
753		"get_threepid_validation_session", get_threepid_validation_session_txn
754		)
755
756		def delete_threepid_session(self, session_id):
757		"""Removes a threepid validation session from the database. This can
758		be done after validation has been performed and whatever action was
759		waiting on it has been carried out
760
761		Args:
762		session_id (str): The ID of the session to delete
763		"""
764
765		def delete_threepid_session_txn(txn):
766		self.db.simple_delete_txn(
767		txn,
768		table="threepid_validation_token",
769		keyvalues={"session_id": session_id},
770		)
771		self.db.simple_delete_txn(
772		txn,
773		table="threepid_validation_session",
774		keyvalues={"session_id": session_id},
775		)
776
777		return self.db.runInteraction(
778		"delete_threepid_session", delete_threepid_session_txn
779		)
780
781
782		class RegistrationBackgroundUpdateStore(RegistrationWorkerStore):
783		def __init__(self, database: Database, db_conn, hs):
784		super(RegistrationBackgroundUpdateStore, self).__init__(database, db_conn, hs)
785
786		self.clock = hs.get_clock()
787		self.config = hs.config
788
789		self.db.updates.register_background_index_update(
790		"access_tokens_device_index",
791		index_name="access_tokens_device_id",
792		table="access_tokens",
793		columns=["user_id", "device_id"],
794		)
795
796		self.db.updates.register_background_index_update(
797		"users_creation_ts",
798		index_name="users_creation_ts",
799		table="users",
800		columns=["creation_ts"],
801		)
802
803		# we no longer use refresh tokens, but it's possible that some people
804		# might have a background update queued to build this index. Just
805		# clear the background update.
806		self.db.updates.register_noop_background_update("refresh_tokens_device_index")
807
808		self.db.updates.register_background_update_handler(
809		"user_threepids_grandfather", self._bg_user_threepids_grandfather
810		)
811
812		self.db.updates.register_background_update_handler(
813		"users_set_deactivated_flag", self._background_update_set_deactivated_flag
814		)
815
816		@defer.inlineCallbacks
817		def _background_update_set_deactivated_flag(self, progress, batch_size):
818		"""Retrieves a list of all deactivated users and sets the 'deactivated' flag to 1
819		for each of them.
820		"""
821
822		last_user = progress.get("user_id", "")
823
824		def _background_update_set_deactivated_flag_txn(txn):
825		txn.execute(
826		"""
827		SELECT
828		users.name,
829		COUNT(access_tokens.token) AS count_tokens,
830		COUNT(user_threepids.address) AS count_threepids
831		FROM users
832		LEFT JOIN access_tokens ON (access_tokens.user_id = users.name)
833		LEFT JOIN user_threepids ON (user_threepids.user_id = users.name)
834		WHERE (users.password_hash IS NULL OR users.password_hash = '')
835		AND (users.appservice_id IS NULL OR users.appservice_id = '')
836		AND users.is_guest = 0
837		AND users.name > ?
838		GROUP BY users.name
839		ORDER BY users.name ASC
840		LIMIT ?;
841		""",
842		(last_user, batch_size),
843		)
844
845		rows = self.db.cursor_to_dict(txn)
846
847		if not rows:
848		return True, 0
849
850		rows_processed_nb = 0
851
852		for user in rows:
853		if not user["count_tokens"] and not user["count_threepids"]:
854		self.set_user_deactivated_status_txn(txn, user["name"], True)
855		rows_processed_nb += 1
856
857		logger.info("Marked %d rows as deactivated", rows_processed_nb)
858
859		self.db.updates._background_update_progress_txn(
860		txn, "users_set_deactivated_flag", {"user_id": rows[-1]["name"]}
861		)
862
863		if batch_size > len(rows):
864		return True, len(rows)
865		else:
866		return False, len(rows)
867
868		end, nb_processed = yield self.db.runInteraction(
869		"users_set_deactivated_flag", _background_update_set_deactivated_flag_txn
870		)
871
872		if end:
873		yield self.db.updates._end_background_update("users_set_deactivated_flag")
874
875		return nb_processed
876
877		@defer.inlineCallbacks
878		def _bg_user_threepids_grandfather(self, progress, batch_size):
879		"""We now track which identity servers a user binds their 3PID to, so
880		we need to handle the case of existing bindings where we didn't track
881		this.
882
883		We do this by grandfathering in existing user threepids assuming that
884		they used one of the server configured trusted identity servers.
885		"""
886		id_servers = set(self.config.trusted_third_party_id_servers)
887
888		def _bg_user_threepids_grandfather_txn(txn):
889		sql = """
890		INSERT INTO user_threepid_id_server
891		(user_id, medium, address, id_server)
892		SELECT user_id, medium, address, ?
893		FROM user_threepids
894		"""
895
896		txn.executemany(sql, [(id_server,) for id_server in id_servers])
897
898		if id_servers:
899		yield self.db.runInteraction(
900		"_bg_user_threepids_grandfather", _bg_user_threepids_grandfather_txn
901		)
902
903		yield self.db.updates._end_background_update("user_threepids_grandfather")
904
905		return 1
906
907
908		class RegistrationStore(RegistrationBackgroundUpdateStore):
909		def __init__(self, database: Database, db_conn, hs):
910		super(RegistrationStore, self).__init__(database, db_conn, hs)
911
912		self._account_validity = hs.config.account_validity
913
914		if self._account_validity.enabled:
915		self._clock.call_later(
916		0.0,
917		run_as_background_process,
918		"account_validity_set_expiration_dates",
919		self._set_expiration_date_when_missing,
920		)
921
922		# Create a background job for culling expired 3PID validity tokens
923		def start_cull():
924		# run as a background process to make sure that the database transactions
925		# have a logcontext to report to
926		return run_as_background_process(
927		"cull_expired_threepid_validation_tokens",
928		self.cull_expired_threepid_validation_tokens,
929		)
930
931		hs.get_clock().looping_call(start_cull, THIRTY_MINUTES_IN_MS)
932
933		@defer.inlineCallbacks
934		def add_access_token_to_user(self, user_id, token, device_id, valid_until_ms):
935		"""Adds an access token for the given user.
936
937		Args:
938		user_id (str): The user ID.
939		token (str): The new access token to add.
940		device_id (str): ID of the device to associate with the access
941		token
942		valid_until_ms (int\|None): when the token is valid until. None for
943		no expiry.
944		Raises:
945		StoreError if there was a problem adding this.
946		"""
947		next_id = self._access_tokens_id_gen.get_next()
948
949		yield self.db.simple_insert(
950		"access_tokens",
951		{
952		"id": next_id,
953		"user_id": user_id,
954		"token": token,
955		"device_id": device_id,
956		"valid_until_ms": valid_until_ms,
957		},
958		desc="add_access_token_to_user",
959		)
960
961		def register_user(
962		self,
963		user_id,
964		password_hash=None,
965		was_guest=False,
966		make_guest=False,
967		appservice_id=None,
968		create_profile_with_displayname=None,
969		admin=False,
970		user_type=None,
971		):
972		"""Attempts to register an account.
973
974		Args:
975		user_id (str): The desired user ID to register.
976		password_hash (str\|None): Optional. The password hash for this user.
977		was_guest (bool): Optional. Whether this is a guest account being
978		upgraded to a non-guest account.
979		make_guest (boolean): True if the the new user should be guest,
980		false to add a regular user account.
981		appservice_id (str): The ID of the appservice registering the user.
982		create_profile_with_displayname (unicode): Optionally create a profile for
983		the user, setting their displayname to the given value
984		admin (boolean): is an admin user?
985		user_type (str\|None): type of user. One of the values from
986		api.constants.UserTypes, or None for a normal user.
987
988		Raises:
989		StoreError if the user_id could not be registered.
990
991		Returns:
992		Deferred
993		"""
994		return self.db.runInteraction(
995		"register_user",
996		self._register_user,
997		user_id,
998		password_hash,
999		was_guest,
1000		make_guest,
1001		appservice_id,
1002		create_profile_with_displayname,
1003		admin,
1004		user_type,
1005		)
1006
1007		def _register_user(
1008		self,
1009		txn,
1010		user_id,
1011		password_hash,
1012		was_guest,
1013		make_guest,
1014		appservice_id,
1015		create_profile_with_displayname,
1016		admin,
1017		user_type,
1018		):
1019		user_id_obj = UserID.from_string(user_id)
1020
1021		now = int(self.clock.time())
1022
1023		try:
1024		if was_guest:
1025		# Ensure that the guest user actually exists
1026		# ``allow_none=False`` makes this raise an exception
1027		# if the row isn't in the database.
1028		self.db.simple_select_one_txn(
1029		txn,
1030		"users",
1031		keyvalues={"name": user_id, "is_guest": 1},
1032		retcols=("name",),
1033		allow_none=False,
1034		)
1035
1036		self.db.simple_update_one_txn(
1037		txn,
1038		"users",
1039		keyvalues={"name": user_id, "is_guest": 1},
1040		updatevalues={
1041		"password_hash": password_hash,
1042		"upgrade_ts": now,
1043		"is_guest": 1 if make_guest else 0,
1044		"appservice_id": appservice_id,
1045		"admin": 1 if admin else 0,
1046		"user_type": user_type,
1047		},
1048		)
1049		else:
1050		self.db.simple_insert_txn(
1051		txn,
1052		"users",
1053		values={
1054		"name": user_id,
1055		"password_hash": password_hash,
1056		"creation_ts": now,
1057		"is_guest": 1 if make_guest else 0,
1058		"appservice_id": appservice_id,
1059		"admin": 1 if admin else 0,
1060		"user_type": user_type,
1061		},
1062		)
1063
1064		except self.database_engine.module.IntegrityError:
1065		raise StoreError(400, "User ID already taken.", errcode=Codes.USER_IN_USE)
1066
1067		if self._account_validity.enabled:
1068		self.set_expiration_date_for_user_txn(txn, user_id)
1069
1070		if create_profile_with_displayname:
1071		# set a default displayname serverside to avoid ugly race
1072		# between auto-joins and clients trying to set displaynames
1073		#
1074		# obviously the 'profiles' table uses localpart for user_id
1075		# while everything else uses the full mxid.
1076		txn.execute(
1077		"INSERT INTO profiles(user_id, displayname) VALUES (?,?)",
1078		(user_id_obj.localpart, create_profile_with_displayname),
1079		)
1080
1081		if self.hs.config.stats_enabled:
1082		# we create a new completed user statistics row
1083
1084		# we don't strictly need current_token since this user really can't
1085		# have any state deltas before now (as it is a new user), but still,
1086		# we include it for completeness.
1087		current_token = self._get_max_stream_id_in_current_state_deltas_txn(txn)
1088		self._update_stats_delta_txn(
1089		txn, now, "user", user_id, {}, complete_with_stream_id=current_token
1090		)
1091
1092		self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
1093		txn.call_after(self.is_guest.invalidate, (user_id,))
1094
1095		def record_user_external_id(
1096		self, auth_provider: str, external_id: str, user_id: str
1097		) -> Deferred:
1098		"""Record a mapping from an external user id to a mxid
1099
1100		Args:
1101		auth_provider: identifier for the remote auth provider
1102		external_id: id on that system
1103		user_id: complete mxid that it is mapped to
1104		"""
1105		return self.db.simple_insert(
1106		table="user_external_ids",
1107		values={
1108		"auth_provider": auth_provider,
1109		"external_id": external_id,
1110		"user_id": user_id,
1111		},
1112		desc="record_user_external_id",
1113		)
1114
1115		def user_set_password_hash(self, user_id, password_hash):
1116		"""
1117		NB. This does not evict any cache because the one use for this
1118		removes most of the entries subsequently anyway so it would be
1119		pointless. Use flush_user separately.
1120		"""
1121
1122		def user_set_password_hash_txn(txn):
1123		self.db.simple_update_one_txn(
1124		txn, "users", {"name": user_id}, {"password_hash": password_hash}
1125		)
1126		self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
1127
1128		return self.db.runInteraction(
1129		"user_set_password_hash", user_set_password_hash_txn
1130		)
1131
1132		def user_set_consent_version(self, user_id, consent_version):
1133		"""Updates the user table to record privacy policy consent
1134
1135		Args:
1136		user_id (str): full mxid of the user to update
1137		consent_version (str): version of the policy the user has consented
1138		to
1139
1140		Raises:
1141		StoreError(404) if user not found
1142		"""
1143
1144		def f(txn):
1145		self.db.simple_update_one_txn(
1146		txn,
1147		table="users",
1148		keyvalues={"name": user_id},
1149		updatevalues={"consent_version": consent_version},
1150		)
1151		self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
1152
1153		return self.db.runInteraction("user_set_consent_version", f)
1154
1155		def user_set_consent_server_notice_sent(self, user_id, consent_version):
1156		"""Updates the user table to record that we have sent the user a server
1157		notice about privacy policy consent
1158
1159		Args:
1160		user_id (str): full mxid of the user to update
1161		consent_version (str): version of the policy we have notified the
1162		user about
1163
1164		Raises:
1165		StoreError(404) if user not found
1166		"""
1167
1168		def f(txn):
1169		self.db.simple_update_one_txn(
1170		txn,
1171		table="users",
1172		keyvalues={"name": user_id},
1173		updatevalues={"consent_server_notice_sent": consent_version},
1174		)
1175		self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
1176
1177		return self.db.runInteraction("user_set_consent_server_notice_sent", f)
1178
1179		def user_delete_access_tokens(self, user_id, except_token_id=None, device_id=None):
1180		"""
1181		Invalidate access tokens belonging to a user
1182
1183		Args:
1184		user_id (str): ID of user the tokens belong to
1185		except_token_id (str): list of access_tokens IDs which should
1186		not be deleted
1187		device_id (str\|None): ID of device the tokens are associated with.
1188		If None, tokens associated with any device (or no device) will
1189		be deleted
1190		Returns:
1191		defer.Deferred[list[str, int, str\|None, int]]: a list of
1192		(token, token id, device id) for each of the deleted tokens
1193		"""
1194
1195		def f(txn):
1196		keyvalues = {"user_id": user_id}
1197		if device_id is not None:
1198		keyvalues["device_id"] = device_id
1199
1200		items = keyvalues.items()
1201		where_clause = " AND ".join(k + " = ?" for k, _ in items)
1202		values = [v for _, v in items]
1203		if except_token_id:
1204		where_clause += " AND id != ?"
1205		values.append(except_token_id)
1206
1207		txn.execute(
1208		"SELECT token, id, device_id FROM access_tokens WHERE %s"
1209		% where_clause,
1210		values,
1211		)
1212		tokens_and_devices = [(r[0], r[1], r[2]) for r in txn]
1213
1214		for token, _, _ in tokens_and_devices:
1215		self._invalidate_cache_and_stream(
1216		txn, self.get_user_by_access_token, (token,)
1217		)
1218
1219		txn.execute("DELETE FROM access_tokens WHERE %s" % where_clause, values)
1220
1221		return tokens_and_devices
1222
1223		return self.db.runInteraction("user_delete_access_tokens", f)
1224
1225		def delete_access_token(self, access_token):
1226		def f(txn):
1227		self.db.simple_delete_one_txn(
1228		txn, table="access_tokens", keyvalues={"token": access_token}
1229		)
1230
1231		self._invalidate_cache_and_stream(
1232		txn, self.get_user_by_access_token, (access_token,)
1233		)
1234
1235		return self.db.runInteraction("delete_access_token", f)
1236
1237		@cachedInlineCallbacks()
1238		def is_guest(self, user_id):
1239		res = yield self.db.simple_select_one_onecol(
1240		table="users",
1241		keyvalues={"name": user_id},
1242		retcol="is_guest",
1243		allow_none=True,
1244		desc="is_guest",
1245		)
1246
1247		return res if res else False
1248
1249		def add_user_pending_deactivation(self, user_id):
1250		"""
1251		Adds a user to the table of users who need to be parted from all the rooms they're
1252		in
1253		"""
1254		return self.db.simple_insert(
1255		"users_pending_deactivation",
1256		values={"user_id": user_id},
1257		desc="add_user_pending_deactivation",
1258		)
1259
1260		def del_user_pending_deactivation(self, user_id):
1261		"""
1262		Removes the given user to the table of users who need to be parted from all the
1263		rooms they're in, effectively marking that user as fully deactivated.
1264		"""
1265		# XXX: This should be simple_delete_one but we failed to put a unique index on
1266		# the table, so somehow duplicate entries have ended up in it.
1267		return self.db.simple_delete(
1268		"users_pending_deactivation",
1269		keyvalues={"user_id": user_id},
1270		desc="del_user_pending_deactivation",
1271		)
1272
1273		def get_user_pending_deactivation(self):
1274		"""
1275		Gets one user from the table of users waiting to be parted from all the rooms
1276		they're in.
1277		"""
1278		return self.db.simple_select_one_onecol(
1279		"users_pending_deactivation",
1280		keyvalues={},
1281		retcol="user_id",
1282		allow_none=True,
1283		desc="get_users_pending_deactivation",
1284		)
1285
1286		def validate_threepid_session(self, session_id, client_secret, token, current_ts):
1287		"""Attempt to validate a threepid session using a token
1288
1289		Args:
1290		session_id (str): The id of a validation session
1291		client_secret (str): A unique string provided by the client to
1292		help identify this validation attempt
1293		token (str): A validation token
1294		current_ts (int): The current unix time in milliseconds. Used for
1295		checking token expiry status
1296
1297		Raises:
1298		ThreepidValidationError: if a matching validation token was not found or has
1299		expired
1300
1301		Returns:
1302		deferred str\|None: A str representing a link to redirect the user
1303		to if there is one.
1304		"""
1305
1306		# Insert everything into a transaction in order to run atomically
1307		def validate_threepid_session_txn(txn):
1308		row = self.db.simple_select_one_txn(
1309		txn,
1310		table="threepid_validation_session",
1311		keyvalues={"session_id": session_id},
1312		retcols=["client_secret", "validated_at"],
1313		allow_none=True,
1314		)
1315
1316		if not row:
1317		raise ThreepidValidationError(400, "Unknown session_id")
1318		retrieved_client_secret = row["client_secret"]
1319		validated_at = row["validated_at"]
1320
1321		if retrieved_client_secret != client_secret:
1322		raise ThreepidValidationError(
1323		400, "This client_secret does not match the provided session_id"
1324		)
1325
1326		row = self.db.simple_select_one_txn(
1327		txn,
1328		table="threepid_validation_token",
1329		keyvalues={"session_id": session_id, "token": token},
1330		retcols=["expires", "next_link"],
1331		allow_none=True,
1332		)
1333
1334		if not row:
1335		raise ThreepidValidationError(
1336		400, "Validation token not found or has expired"
1337		)
1338		expires = row["expires"]
1339		next_link = row["next_link"]
1340
1341		# If the session is already validated, no need to revalidate
1342		if validated_at:
1343		return next_link
1344
1345		if expires <= current_ts:
1346		raise ThreepidValidationError(
1347		400, "This token has expired. Please request a new one"
1348		)
1349
1350		# Looks good. Validate the session
1351		self.db.simple_update_txn(
1352		txn,
1353		table="threepid_validation_session",
1354		keyvalues={"session_id": session_id},
1355		updatevalues={"validated_at": self.clock.time_msec()},
1356		)
1357
1358		return next_link
1359
1360		# Return next_link if it exists
1361		return self.db.runInteraction(
1362		"validate_threepid_session_txn", validate_threepid_session_txn
1363		)
1364
1365		def upsert_threepid_validation_session(
1366		self,
1367		medium,
1368		address,
1369		client_secret,
1370		send_attempt,
1371		session_id,
1372		validated_at=None,
1373		):
1374		"""Upsert a threepid validation session
1375		Args:
1376		medium (str): The medium of the 3PID
1377		address (str): The address of the 3PID
1378		client_secret (str): A unique string provided by the client to
1379		help identify this validation attempt
1380		send_attempt (int): The latest send_attempt on this session
1381		session_id (str): The id of this validation session
1382		validated_at (int\|None): The unix timestamp in milliseconds of
1383		when the session was marked as valid
1384		"""
1385		insertion_values = {
1386		"medium": medium,
1387		"address": address,
1388		"client_secret": client_secret,
1389		}
1390
1391		if validated_at:
1392		insertion_values["validated_at"] = validated_at
1393
1394		return self.db.simple_upsert(
1395		table="threepid_validation_session",
1396		keyvalues={"session_id": session_id},
1397		values={"last_send_attempt": send_attempt},
1398		insertion_values=insertion_values,
1399		desc="upsert_threepid_validation_session",
1400		)
1401
1402		def start_or_continue_validation_session(
1403		self,
1404		medium,
1405		address,
1406		session_id,
1407		client_secret,
1408		send_attempt,
1409		next_link,
1410		token,
1411		token_expires,
1412		):
1413		"""Creates a new threepid validation session if it does not already
1414		exist and associates a new validation token with it
1415
1416		Args:
1417		medium (str): The medium of the 3PID
1418		address (str): The address of the 3PID
1419		session_id (str): The id of this validation session
1420		client_secret (str): A unique string provided by the client to
1421		help identify this validation attempt
1422		send_attempt (int): The latest send_attempt on this session
1423		next_link (str\|None): The link to redirect the user to upon
1424		successful validation
1425		token (str): The validation token
1426		token_expires (int): The timestamp for which after the token
1427		will no longer be valid
1428		"""
1429
1430		def start_or_continue_validation_session_txn(txn):
1431		# Create or update a validation session
1432		self.db.simple_upsert_txn(
1433		txn,
1434		table="threepid_validation_session",
1435		keyvalues={"session_id": session_id},
1436		values={"last_send_attempt": send_attempt},
1437		insertion_values={
1438		"medium": medium,
1439		"address": address,
1440		"client_secret": client_secret,
1441		},
1442		)
1443
1444		# Create a new validation token with this session ID
1445		self.db.simple_insert_txn(
1446		txn,
1447		table="threepid_validation_token",
1448		values={
1449		"session_id": session_id,
1450		"token": token,
1451		"next_link": next_link,
1452		"expires": token_expires,
1453		},
1454		)
1455
1456		return self.db.runInteraction(
1457		"start_or_continue_validation_session",
1458		start_or_continue_validation_session_txn,
1459		)
1460
1461		def cull_expired_threepid_validation_tokens(self):
1462		"""Remove threepid validation tokens with expiry dates that have passed"""
1463
1464		def cull_expired_threepid_validation_tokens_txn(txn, ts):
1465		sql = """
1466		DELETE FROM threepid_validation_token WHERE
1467		expires < ?
1468		"""
1469		return txn.execute(sql, (ts,))
1470
1471		return self.db.runInteraction(
1472		"cull_expired_threepid_validation_tokens",
1473		cull_expired_threepid_validation_tokens_txn,
1474		self.clock.time_msec(),
1475		)
1476
1477		@defer.inlineCallbacks
1478		def set_user_deactivated_status(self, user_id, deactivated):
1479		"""Set the `deactivated` property for the provided user to the provided value.
1480
1481		Args:
1482		user_id (str): The ID of the user to set the status for.
1483		deactivated (bool): The value to set for `deactivated`.
1484		"""
1485
1486		yield self.db.runInteraction(
1487		"set_user_deactivated_status",
1488		self.set_user_deactivated_status_txn,
1489		user_id,
1490		deactivated,
1491		)
1492
1493		def set_user_deactivated_status_txn(self, txn, user_id, deactivated):
1494		self.db.simple_update_one_txn(
1495		txn=txn,
1496		table="users",
1497		keyvalues={"name": user_id},
1498		updatevalues={"deactivated": 1 if deactivated else 0},
1499		)
1500		self._invalidate_cache_and_stream(
1501		txn, self.get_user_deactivated_status, (user_id,)
1502		)
1503
1504		@defer.inlineCallbacks
1505		def _set_expiration_date_when_missing(self):
1506		"""
1507		Retrieves the list of registered users that don't have an expiration date, and
1508		adds an expiration date for each of them.
1509		"""
1510
1511		def select_users_with_no_expiration_date_txn(txn):
1512		"""Retrieves the list of registered users with no expiration date from the
1513		database, filtering out deactivated users.
1514		"""
1515		sql = (
1516		"SELECT users.name FROM users"
1517		" LEFT JOIN account_validity ON (users.name = account_validity.user_id)"
1518		" WHERE account_validity.user_id is NULL AND users.deactivated = 0;"
1519		)
1520		txn.execute(sql, [])
1521
1522		res = self.db.cursor_to_dict(txn)
1523		if res:
1524		for user in res:
1525		self.set_expiration_date_for_user_txn(
1526		txn, user["name"], use_delta=True
1527		)
1528
1529		yield self.db.runInteraction(
1530		"get_users_with_no_expiration_date",
1531		select_users_with_no_expiration_date_txn,
1532		)
1533
1534		def set_expiration_date_for_user_txn(self, txn, user_id, use_delta=False):
1535		"""Sets an expiration date to the account with the given user ID.
1536
1537		Args:
1538		user_id (str): User ID to set an expiration date for.
1539		use_delta (bool): If set to False, the expiration date for the user will be
1540		now + validity period. If set to True, this expiration date will be a
1541		random value in the [now + period - d ; now + period] range, d being a
1542		delta equal to 10% of the validity period.
1543		"""
1544		now_ms = self._clock.time_msec()
1545		expiration_ts = now_ms + self._account_validity.period
1546
1547		if use_delta:
1548		expiration_ts = self.rand.randrange(
1549		expiration_ts - self._account_validity.startup_job_max_delta,
1550		expiration_ts,
1551		)
1552
1553		self.db.simple_upsert_txn(
1554		txn,
1555		"account_validity",
1556		keyvalues={"user_id": user_id},
1557		values={"expiration_ts_ms": expiration_ts, "email_sent": False},
1558		)
1559
1560
1561		def find_max_generated_user_id_localpart(cur: Cursor) -> int:
1562		"""
1563		Gets the localpart of the max current generated user ID.
1564
1565		Generated user IDs are integers, so we find the largest integer user ID
1566		already taken and return that.
1567		"""
1568
1569		# We bound between '@0' and '@a' to avoid pulling the entire table
1570		# out.
1571		cur.execute("SELECT name FROM users WHERE '@0' <= name AND name < '@a'")
1572
1573		regex = re.compile(r"^@(\d+):")
1574
1575		max_found = 0
1576
1577		for (user_id,) in cur:
1578		match = regex.search(user_id)
1579		if match:
1580		max_found = max(int(match.group(1)), max_found)
1581		return max_found

+0

-31

~~synapse/storage/data_stores/main/rejections.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16
17		from synapse.storage._base import SQLBaseStore
18
19		logger = logging.getLogger(__name__)
20
21
22		class RejectionsStore(SQLBaseStore):
23		def get_rejection_reason(self, event_id):
24		return self.db.simple_select_one_onecol(
25		table="rejections",
26		retcol="reason",
27		keyvalues={"event_id": event_id},
28		allow_none=True,
29		desc="get_rejection_reason",
30		)

+0

-327

~~synapse/storage/data_stores/main/relations.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2019 New Vector Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16
17		import attr
18
19		from synapse.api.constants import RelationTypes
20		from synapse.storage._base import SQLBaseStore
21		from synapse.storage.data_stores.main.stream import generate_pagination_where_clause
22		from synapse.storage.relations import (
23		AggregationPaginationToken,
24		PaginationChunk,
25		RelationPaginationToken,
26		)
27		from synapse.util.caches.descriptors import cached, cachedInlineCallbacks
28
29		logger = logging.getLogger(__name__)
30
31
32		class RelationsWorkerStore(SQLBaseStore):
33		@cached(tree=True)
34		def get_relations_for_event(
35		self,
36		event_id,
37		relation_type=None,
38		event_type=None,
39		aggregation_key=None,
40		limit=5,
41		direction="b",
42		from_token=None,
43		to_token=None,
44		):
45		"""Get a list of relations for an event, ordered by topological ordering.
46
47		Args:
48		event_id (str): Fetch events that relate to this event ID.
49		relation_type (str\|None): Only fetch events with this relation
50		type, if given.
51		event_type (str\|None): Only fetch events with this event type, if
52		given.
53		aggregation_key (str\|None): Only fetch events with this aggregation
54		key, if given.
55		limit (int): Only fetch the most recent `limit` events.
56		direction (str): Whether to fetch the most recent first (`"b"`) or
57		the oldest first (`"f"`).
58		from_token (RelationPaginationToken\|None): Fetch rows from the given
59		token, or from the start if None.
60		to_token (RelationPaginationToken\|None): Fetch rows up to the given
61		token, or up to the end if None.
62
63		Returns:
64		Deferred[PaginationChunk]: List of event IDs that match relations
65		requested. The rows are of the form `{"event_id": "..."}`.
66		"""
67
68		where_clause = ["relates_to_id = ?"]
69		where_args = [event_id]
70
71		if relation_type is not None:
72		where_clause.append("relation_type = ?")
73		where_args.append(relation_type)
74
75		if event_type is not None:
76		where_clause.append("type = ?")
77		where_args.append(event_type)
78
79		if aggregation_key:
80		where_clause.append("aggregation_key = ?")
81		where_args.append(aggregation_key)
82
83		pagination_clause = generate_pagination_where_clause(
84		direction=direction,
85		column_names=("topological_ordering", "stream_ordering"),
86		from_token=attr.astuple(from_token) if from_token else None,
87		to_token=attr.astuple(to_token) if to_token else None,
88		engine=self.database_engine,
89		)
90
91		if pagination_clause:
92		where_clause.append(pagination_clause)
93
94		if direction == "b":
95		order = "DESC"
96		else:
97		order = "ASC"
98
99		sql = """
100		SELECT event_id, topological_ordering, stream_ordering
101		FROM event_relations
102		INNER JOIN events USING (event_id)
103		WHERE %s
104		ORDER BY topological_ordering %s, stream_ordering %s
105		LIMIT ?
106		""" % (
107		" AND ".join(where_clause),
108		order,
109		order,
110		)
111
112		def _get_recent_references_for_event_txn(txn):
113		txn.execute(sql, where_args + [limit + 1])
114
115		last_topo_id = None
116		last_stream_id = None
117		events = []
118		for row in txn:
119		events.append({"event_id": row[0]})
120		last_topo_id = row[1]
121		last_stream_id = row[2]
122
123		next_batch = None
124		if len(events) > limit and last_topo_id and last_stream_id:
125		next_batch = RelationPaginationToken(last_topo_id, last_stream_id)
126
127		return PaginationChunk(
128		chunk=list(events[:limit]), next_batch=next_batch, prev_batch=from_token
129		)
130
131		return self.db.runInteraction(
132		"get_recent_references_for_event", _get_recent_references_for_event_txn
133		)
134
135		@cached(tree=True)
136		def get_aggregation_groups_for_event(
137		self,
138		event_id,
139		event_type=None,
140		limit=5,
141		direction="b",
142		from_token=None,
143		to_token=None,
144		):
145		"""Get a list of annotations on the event, grouped by event type and
146		aggregation key, sorted by count.
147
148		This is used e.g. to get the what and how many reactions have happend
149		on an event.
150
151		Args:
152		event_id (str): Fetch events that relate to this event ID.
153		event_type (str\|None): Only fetch events with this event type, if
154		given.
155		limit (int): Only fetch the `limit` groups.
156		direction (str): Whether to fetch the highest count first (`"b"`) or
157		the lowest count first (`"f"`).
158		from_token (AggregationPaginationToken\|None): Fetch rows from the
159		given token, or from the start if None.
160		to_token (AggregationPaginationToken\|None): Fetch rows up to the
161		given token, or up to the end if None.
162
163
164		Returns:
165		Deferred[PaginationChunk]: List of groups of annotations that
166		match. Each row is a dict with `type`, `key` and `count` fields.
167		"""
168
169		where_clause = ["relates_to_id = ?", "relation_type = ?"]
170		where_args = [event_id, RelationTypes.ANNOTATION]
171
172		if event_type:
173		where_clause.append("type = ?")
174		where_args.append(event_type)
175
176		having_clause = generate_pagination_where_clause(
177		direction=direction,
178		column_names=("COUNT(*)", "MAX(stream_ordering)"),
179		from_token=attr.astuple(from_token) if from_token else None,
180		to_token=attr.astuple(to_token) if to_token else None,
181		engine=self.database_engine,
182		)
183
184		if direction == "b":
185		order = "DESC"
186		else:
187		order = "ASC"
188
189		if having_clause:
190		having_clause = "HAVING " + having_clause
191		else:
192		having_clause = ""
193
194		sql = """
195		SELECT type, aggregation_key, COUNT(DISTINCT sender), MAX(stream_ordering)
196		FROM event_relations
197		INNER JOIN events USING (event_id)
198		WHERE {where_clause}
199		GROUP BY relation_type, type, aggregation_key
200		{having_clause}
201		ORDER BY COUNT(*) {order}, MAX(stream_ordering) {order}
202		LIMIT ?
203		""".format(
204		where_clause=" AND ".join(where_clause),
205		order=order,
206		having_clause=having_clause,
207		)
208
209		def _get_aggregation_groups_for_event_txn(txn):
210		txn.execute(sql, where_args + [limit + 1])
211
212		next_batch = None
213		events = []
214		for row in txn:
215		events.append({"type": row[0], "key": row[1], "count": row[2]})
216		next_batch = AggregationPaginationToken(row[2], row[3])
217
218		if len(events) <= limit:
219		next_batch = None
220
221		return PaginationChunk(
222		chunk=list(events[:limit]), next_batch=next_batch, prev_batch=from_token
223		)
224
225		return self.db.runInteraction(
226		"get_aggregation_groups_for_event", _get_aggregation_groups_for_event_txn
227		)
228
229		@cachedInlineCallbacks()
230		def get_applicable_edit(self, event_id):
231		"""Get the most recent edit (if any) that has happened for the given
232		event.
233
234		Correctly handles checking whether edits were allowed to happen.
235
236		Args:
237		event_id (str): The original event ID
238
239		Returns:
240		Deferred[EventBase\|None]: Returns the most recent edit, if any.
241		"""
242
243		# We only allow edits for `m.room.message` events that have the same sender
244		# and event type. We can't assert these things during regular event auth so
245		# we have to do the checks post hoc.
246
247		# Fetches latest edit that has the same type and sender as the
248		# original, and is an `m.room.message`.
249		sql = """
250		SELECT edit.event_id FROM events AS edit
251		INNER JOIN event_relations USING (event_id)
252		INNER JOIN events AS original ON
253		original.event_id = relates_to_id
254		AND edit.type = original.type
255		AND edit.sender = original.sender
256		WHERE
257		relates_to_id = ?
258		AND relation_type = ?
259		AND edit.type = 'm.room.message'
260		ORDER by edit.origin_server_ts DESC, edit.event_id DESC
261		LIMIT 1
262		"""
263
264		def _get_applicable_edit_txn(txn):
265		txn.execute(sql, (event_id, RelationTypes.REPLACE))
266		row = txn.fetchone()
267		if row:
268		return row[0]
269
270		edit_id = yield self.db.runInteraction(
271		"get_applicable_edit", _get_applicable_edit_txn
272		)
273
274		if not edit_id:
275		return
276
277		edit_event = yield self.get_event(edit_id, allow_none=True)
278		return edit_event
279
280		def has_user_annotated_event(self, parent_id, event_type, aggregation_key, sender):
281		"""Check if a user has already annotated an event with the same key
282		(e.g. already liked an event).
283
284		Args:
285		parent_id (str): The event being annotated
286		event_type (str): The event type of the annotation
287		aggregation_key (str): The aggregation key of the annotation
288		sender (str): The sender of the annotation
289
290		Returns:
291		Deferred[bool]
292		"""
293
294		sql = """
295		SELECT 1 FROM event_relations
296		INNER JOIN events USING (event_id)
297		WHERE
298		relates_to_id = ?
299		AND relation_type = ?
300		AND type = ?
301		AND sender = ?
302		AND aggregation_key = ?
303		LIMIT 1;
304		"""
305
306		def _get_if_user_has_annotated_event(txn):
307		txn.execute(
308		sql,
309		(
310		parent_id,
311		RelationTypes.ANNOTATION,
312		event_type,
313		sender,
314		aggregation_key,
315		),
316		)
317
318		return bool(txn.fetchone())
319
320		return self.db.runInteraction(
321		"get_if_user_has_annotated_event", _get_if_user_has_annotated_event
322		)
323
324
325		class RelationsStore(RelationsWorkerStore):
326		pass

+0

-1439

~~synapse/storage/data_stores/main/room.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2019 The Matrix.org Foundation C.I.C.
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		import collections
17		import logging
18		import re
19		from abc import abstractmethod
20		from enum import Enum
21		from typing import Any, Dict, List, Optional, Tuple
22
23		from canonicaljson import json
24
25		from twisted.internet import defer
26
27		from synapse.api.constants import EventTypes
28		from synapse.api.errors import StoreError
29		from synapse.api.room_versions import RoomVersion, RoomVersions
30		from synapse.storage._base import SQLBaseStore, db_to_json
31		from synapse.storage.data_stores.main.search import SearchStore
32		from synapse.storage.database import Database, LoggingTransaction
33		from synapse.types import ThirdPartyInstanceID
34		from synapse.util.caches.descriptors import cached, cachedInlineCallbacks
35
36		logger = logging.getLogger(__name__)
37
38
39		OpsLevel = collections.namedtuple(
40		"OpsLevel", ("ban_level", "kick_level", "redact_level")
41		)
42
43		RatelimitOverride = collections.namedtuple(
44		"RatelimitOverride", ("messages_per_second", "burst_count")
45		)
46
47
48		class RoomSortOrder(Enum):
49		"""
50		Enum to define the sorting method used when returning rooms with get_rooms_paginate
51
52		NAME = sort rooms alphabetically by name
53		JOINED_MEMBERS = sort rooms by membership size, highest to lowest
54		"""
55
56		# ALPHABETICAL and SIZE are deprecated.
57		# ALPHABETICAL is the same as NAME.
58		ALPHABETICAL = "alphabetical"
59		# SIZE is the same as JOINED_MEMBERS.
60		SIZE = "size"
61		NAME = "name"
62		CANONICAL_ALIAS = "canonical_alias"
63		JOINED_MEMBERS = "joined_members"
64		JOINED_LOCAL_MEMBERS = "joined_local_members"
65		VERSION = "version"
66		CREATOR = "creator"
67		ENCRYPTION = "encryption"
68		FEDERATABLE = "federatable"
69		PUBLIC = "public"
70		JOIN_RULES = "join_rules"
71		GUEST_ACCESS = "guest_access"
72		HISTORY_VISIBILITY = "history_visibility"
73		STATE_EVENTS = "state_events"
74
75
76		class RoomWorkerStore(SQLBaseStore):
77		def __init__(self, database: Database, db_conn, hs):
78		super(RoomWorkerStore, self).__init__(database, db_conn, hs)
79
80		self.config = hs.config
81
82		def get_room(self, room_id):
83		"""Retrieve a room.
84
85		Args:
86		room_id (str): The ID of the room to retrieve.
87		Returns:
88		A dict containing the room information, or None if the room is unknown.
89		"""
90		return self.db.simple_select_one(
91		table="rooms",
92		keyvalues={"room_id": room_id},
93		retcols=("room_id", "is_public", "creator"),
94		desc="get_room",
95		allow_none=True,
96		)
97
98		def get_room_with_stats(self, room_id: str):
99		"""Retrieve room with statistics.
100
101		Args:
102		room_id: The ID of the room to retrieve.
103		Returns:
104		A dict containing the room information, or None if the room is unknown.
105		"""
106
107		def get_room_with_stats_txn(txn, room_id):
108		sql = """
109		SELECT room_id, state.name, state.canonical_alias, curr.joined_members,
110		curr.local_users_in_room AS joined_local_members, rooms.room_version AS version,
111		rooms.creator, state.encryption, state.is_federatable AS federatable,
112		rooms.is_public AS public, state.join_rules, state.guest_access,
113		state.history_visibility, curr.current_state_events AS state_events
114		FROM rooms
115		LEFT JOIN room_stats_state state USING (room_id)
116		LEFT JOIN room_stats_current curr USING (room_id)
117		WHERE room_id = ?
118		"""
119		txn.execute(sql, [room_id])
120		# Catch error if sql returns empty result to return "None" instead of an error
121		try:
122		res = self.db.cursor_to_dict(txn)[0]
123		except IndexError:
124		return None
125
126		res["federatable"] = bool(res["federatable"])
127		res["public"] = bool(res["public"])
128		return res
129
130		return self.db.runInteraction(
131		"get_room_with_stats", get_room_with_stats_txn, room_id
132		)
133
134		def get_public_room_ids(self):
135		return self.db.simple_select_onecol(
136		table="rooms",
137		keyvalues={"is_public": True},
138		retcol="room_id",
139		desc="get_public_room_ids",
140		)
141
142		def count_public_rooms(self, network_tuple, ignore_non_federatable):
143		"""Counts the number of public rooms as tracked in the room_stats_current
144		and room_stats_state table.
145
146		Args:
147		network_tuple (ThirdPartyInstanceID\|None)
148		ignore_non_federatable (bool): If true filters out non-federatable rooms
149		"""
150
151		def _count_public_rooms_txn(txn):
152		query_args = []
153
154		if network_tuple:
155		if network_tuple.appservice_id:
156		published_sql = """
157		SELECT room_id from appservice_room_list
158		WHERE appservice_id = ? AND network_id = ?
159		"""
160		query_args.append(network_tuple.appservice_id)
161		query_args.append(network_tuple.network_id)
162		else:
163		published_sql = """
164		SELECT room_id FROM rooms WHERE is_public
165		"""
166		else:
167		published_sql = """
168		SELECT room_id FROM rooms WHERE is_public
169		UNION SELECT room_id from appservice_room_list
170		"""
171
172		sql = """
173		SELECT
174		COALESCE(COUNT(*), 0)
175		FROM (
176		%(published_sql)s
177		) published
178		INNER JOIN room_stats_state USING (room_id)
179		INNER JOIN room_stats_current USING (room_id)
180		WHERE
181		(
182		join_rules = 'public' OR history_visibility = 'world_readable'
183		)
184		AND joined_members > 0
185		""" % {
186		"published_sql": published_sql
187		}
188
189		txn.execute(sql, query_args)
190		return txn.fetchone()[0]
191
192		return self.db.runInteraction("count_public_rooms", _count_public_rooms_txn)
193
194		@defer.inlineCallbacks
195		def get_largest_public_rooms(
196		self,
197		network_tuple: Optional[ThirdPartyInstanceID],
198		search_filter: Optional[dict],
199		limit: Optional[int],
200		bounds: Optional[Tuple[int, str]],
201		forwards: bool,
202		ignore_non_federatable: bool = False,
203		):
204		"""Gets the largest public rooms (where largest is in terms of joined
205		members, as tracked in the statistics table).
206
207		Args:
208		network_tuple
209		search_filter
210		limit: Maxmimum number of rows to return, unlimited otherwise.
211		bounds: An uppoer or lower bound to apply to result set if given,
212		consists of a joined member count and room_id (these are
213		excluded from result set).
214		forwards: true iff going forwards, going backwards otherwise
215		ignore_non_federatable: If true filters out non-federatable rooms.
216
217		Returns:
218		Rooms in order: biggest number of joined users first.
219		We then arbitrarily use the room_id as a tie breaker.
220
221		"""
222
223		where_clauses = []
224		query_args = []
225
226		if network_tuple:
227		if network_tuple.appservice_id:
228		published_sql = """
229		SELECT room_id from appservice_room_list
230		WHERE appservice_id = ? AND network_id = ?
231		"""
232		query_args.append(network_tuple.appservice_id)
233		query_args.append(network_tuple.network_id)
234		else:
235		published_sql = """
236		SELECT room_id FROM rooms WHERE is_public
237		"""
238		else:
239		published_sql = """
240		SELECT room_id FROM rooms WHERE is_public
241		UNION SELECT room_id from appservice_room_list
242		"""
243
244		# Work out the bounds if we're given them, these bounds look slightly
245		# odd, but are designed to help query planner use indices by pulling
246		# out a common bound.
247		if bounds:
248		last_joined_members, last_room_id = bounds
249		if forwards:
250		where_clauses.append(
251		"""
252		joined_members <= ? AND (
253		joined_members < ? OR room_id < ?
254		)
255		"""
256		)
257		else:
258		where_clauses.append(
259		"""
260		joined_members >= ? AND (
261		joined_members > ? OR room_id > ?
262		)
263		"""
264		)
265
266		query_args += [last_joined_members, last_joined_members, last_room_id]
267
268		if ignore_non_federatable:
269		where_clauses.append("is_federatable")
270
271		if search_filter and search_filter.get("generic_search_term", None):
272		search_term = "%" + search_filter["generic_search_term"] + "%"
273
274		where_clauses.append(
275		"""
276		(
277		LOWER(name) LIKE ?
278		OR LOWER(topic) LIKE ?
279		OR LOWER(canonical_alias) LIKE ?
280		)
281		"""
282		)
283		query_args += [
284		search_term.lower(),
285		search_term.lower(),
286		search_term.lower(),
287		]
288
289		where_clause = ""
290		if where_clauses:
291		where_clause = " AND " + " AND ".join(where_clauses)
292
293		sql = """
294		SELECT
295		room_id, name, topic, canonical_alias, joined_members,
296		avatar, history_visibility, joined_members, guest_access
297		FROM (
298		%(published_sql)s
299		) published
300		INNER JOIN room_stats_state USING (room_id)
301		INNER JOIN room_stats_current USING (room_id)
302		WHERE
303		(
304		join_rules = 'public' OR history_visibility = 'world_readable'
305		)
306		AND joined_members > 0
307		%(where_clause)s
308		ORDER BY joined_members %(dir)s, room_id %(dir)s
309		""" % {
310		"published_sql": published_sql,
311		"where_clause": where_clause,
312		"dir": "DESC" if forwards else "ASC",
313		}
314
315		if limit is not None:
316		query_args.append(limit)
317
318		sql += """
319		LIMIT ?
320		"""
321
322		def _get_largest_public_rooms_txn(txn):
323		txn.execute(sql, query_args)
324
325		results = self.db.cursor_to_dict(txn)
326
327		if not forwards:
328		results.reverse()
329
330		return results
331
332		ret_val = yield self.db.runInteraction(
333		"get_largest_public_rooms", _get_largest_public_rooms_txn
334		)
335		defer.returnValue(ret_val)
336
337		@cached(max_entries=10000)
338		def is_room_blocked(self, room_id):
339		return self.db.simple_select_one_onecol(
340		table="blocked_rooms",
341		keyvalues={"room_id": room_id},
342		retcol="1",
343		allow_none=True,
344		desc="is_room_blocked",
345		)
346
347		async def get_rooms_paginate(
348		self,
349		start: int,
350		limit: int,
351		order_by: RoomSortOrder,
352		reverse_order: bool,
353		search_term: Optional[str],
354		) -> Tuple[List[Dict[str, Any]], int]:
355		"""Function to retrieve a paginated list of rooms as json.
356
357		Args:
358		start: offset in the list
359		limit: maximum amount of rooms to retrieve
360		order_by: the sort order of the returned list
361		reverse_order: whether to reverse the room list
362		search_term: a string to filter room names by
363		Returns:
364		A list of room dicts and an integer representing the total number of
365		rooms that exist given this query
366		"""
367		# Filter room names by a string
368		where_statement = ""
369		if search_term:
370		where_statement = "WHERE state.name LIKE ?"
371
372		# Our postgres db driver converts ? -> %s in SQL strings as that's the
373		# placeholder for postgres.
374		# HOWEVER, if you put a % into your SQL then everything goes wibbly.
375		# To get around this, we're going to surround search_term with %'s
376		# before giving it to the database in python instead
377		search_term = "%" + search_term + "%"
378
379		# Set ordering
380		if RoomSortOrder(order_by) == RoomSortOrder.SIZE:
381		# Deprecated in favour of RoomSortOrder.JOINED_MEMBERS
382		order_by_column = "curr.joined_members"
383		order_by_asc = False
384		elif RoomSortOrder(order_by) == RoomSortOrder.ALPHABETICAL:
385		# Deprecated in favour of RoomSortOrder.NAME
386		order_by_column = "state.name"
387		order_by_asc = True
388		elif RoomSortOrder(order_by) == RoomSortOrder.NAME:
389		order_by_column = "state.name"
390		order_by_asc = True
391		elif RoomSortOrder(order_by) == RoomSortOrder.CANONICAL_ALIAS:
392		order_by_column = "state.canonical_alias"
393		order_by_asc = True
394		elif RoomSortOrder(order_by) == RoomSortOrder.JOINED_MEMBERS:
395		order_by_column = "curr.joined_members"
396		order_by_asc = False
397		elif RoomSortOrder(order_by) == RoomSortOrder.JOINED_LOCAL_MEMBERS:
398		order_by_column = "curr.local_users_in_room"
399		order_by_asc = False
400		elif RoomSortOrder(order_by) == RoomSortOrder.VERSION:
401		order_by_column = "rooms.room_version"
402		order_by_asc = False
403		elif RoomSortOrder(order_by) == RoomSortOrder.CREATOR:
404		order_by_column = "rooms.creator"
405		order_by_asc = True
406		elif RoomSortOrder(order_by) == RoomSortOrder.ENCRYPTION:
407		order_by_column = "state.encryption"
408		order_by_asc = True
409		elif RoomSortOrder(order_by) == RoomSortOrder.FEDERATABLE:
410		order_by_column = "state.is_federatable"
411		order_by_asc = True
412		elif RoomSortOrder(order_by) == RoomSortOrder.PUBLIC:
413		order_by_column = "rooms.is_public"
414		order_by_asc = True
415		elif RoomSortOrder(order_by) == RoomSortOrder.JOIN_RULES:
416		order_by_column = "state.join_rules"
417		order_by_asc = True
418		elif RoomSortOrder(order_by) == RoomSortOrder.GUEST_ACCESS:
419		order_by_column = "state.guest_access"
420		order_by_asc = True
421		elif RoomSortOrder(order_by) == RoomSortOrder.HISTORY_VISIBILITY:
422		order_by_column = "state.history_visibility"
423		order_by_asc = True
424		elif RoomSortOrder(order_by) == RoomSortOrder.STATE_EVENTS:
425		order_by_column = "curr.current_state_events"
426		order_by_asc = False
427		else:
428		raise StoreError(
429		500, "Incorrect value for order_by provided: %s" % order_by
430		)
431
432		# Whether to return the list in reverse order
433		if reverse_order:
434		# Flip the boolean
435		order_by_asc = not order_by_asc
436
437		# Create one query for getting the limited number of events that the user asked
438		# for, and another query for getting the total number of events that could be
439		# returned. Thus allowing us to see if there are more events to paginate through
440		info_sql = """
441		SELECT state.room_id, state.name, state.canonical_alias, curr.joined_members,
442		curr.local_users_in_room, rooms.room_version, rooms.creator,
443		state.encryption, state.is_federatable, rooms.is_public, state.join_rules,
444		state.guest_access, state.history_visibility, curr.current_state_events
445		FROM room_stats_state state
446		INNER JOIN room_stats_current curr USING (room_id)
447		INNER JOIN rooms USING (room_id)
448		%s
449		ORDER BY %s %s
450		LIMIT ?
451		OFFSET ?
452		""" % (
453		where_statement,
454		order_by_column,
455		"ASC" if order_by_asc else "DESC",
456		)
457
458		# Use a nested SELECT statement as SQL can't count(*) with an OFFSET
459		count_sql = """
460		SELECT count(*) FROM (
461		SELECT room_id FROM room_stats_state state
462		%s
463		) AS get_room_ids
464		""" % (
465		where_statement,
466		)
467
468		def _get_rooms_paginate_txn(txn):
469		# Execute the data query
470		sql_values = (limit, start)
471		if search_term:
472		# Add the search term into the WHERE clause
473		sql_values = (search_term,) + sql_values
474		txn.execute(info_sql, sql_values)
475
476		# Refactor room query data into a structured dictionary
477		rooms = []
478		for room in txn:
479		rooms.append(
480		{
481		"room_id": room[0],
482		"name": room[1],
483		"canonical_alias": room[2],
484		"joined_members": room[3],
485		"joined_local_members": room[4],
486		"version": room[5],
487		"creator": room[6],
488		"encryption": room[7],
489		"federatable": room[8],
490		"public": room[9],
491		"join_rules": room[10],
492		"guest_access": room[11],
493		"history_visibility": room[12],
494		"state_events": room[13],
495		}
496		)
497
498		# Execute the count query
499
500		# Add the search term into the WHERE clause if present
501		sql_values = (search_term,) if search_term else ()
502		txn.execute(count_sql, sql_values)
503
504		room_count = txn.fetchone()
505		return rooms, room_count[0]
506
507		return await self.db.runInteraction(
508		"get_rooms_paginate", _get_rooms_paginate_txn,
509		)
510
511		@cachedInlineCallbacks(max_entries=10000)
512		def get_ratelimit_for_user(self, user_id):
513		"""Check if there are any overrides for ratelimiting for the given
514		user
515
516		Args:
517		user_id (str)
518
519		Returns:
520		RatelimitOverride if there is an override, else None. If the contents
521		of RatelimitOverride are None or 0 then ratelimitng has been
522		disabled for that user entirely.
523		"""
524		row = yield self.db.simple_select_one(
525		table="ratelimit_override",
526		keyvalues={"user_id": user_id},
527		retcols=("messages_per_second", "burst_count"),
528		allow_none=True,
529		desc="get_ratelimit_for_user",
530		)
531
532		if row:
533		return RatelimitOverride(
534		messages_per_second=row["messages_per_second"],
535		burst_count=row["burst_count"],
536		)
537		else:
538		return None
539
540		@cachedInlineCallbacks()
541		def get_retention_policy_for_room(self, room_id):
542		"""Get the retention policy for a given room.
543
544		If no retention policy has been found for this room, returns a policy defined
545		by the configured default policy (which has None as both the 'min_lifetime' and
546		the 'max_lifetime' if no default policy has been defined in the server's
547		configuration).
548
549		Args:
550		room_id (str): The ID of the room to get the retention policy of.
551
552		Returns:
553		dict[int, int]: "min_lifetime" and "max_lifetime" for this room.
554		"""
555
556		def get_retention_policy_for_room_txn(txn):
557		txn.execute(
558		"""
559		SELECT min_lifetime, max_lifetime FROM room_retention
560		INNER JOIN current_state_events USING (event_id, room_id)
561		WHERE room_id = ?;
562		""",
563		(room_id,),
564		)
565
566		return self.db.cursor_to_dict(txn)
567
568		ret = yield self.db.runInteraction(
569		"get_retention_policy_for_room", get_retention_policy_for_room_txn,
570		)
571
572		# If we don't know this room ID, ret will be None, in this case return the default
573		# policy.
574		if not ret:
575		defer.returnValue(
576		{
577		"min_lifetime": self.config.retention_default_min_lifetime,
578		"max_lifetime": self.config.retention_default_max_lifetime,
579		}
580		)
581
582		row = ret[0]
583
584		# If one of the room's policy's attributes isn't defined, use the matching
585		# attribute from the default policy.
586		# The default values will be None if no default policy has been defined, or if one
587		# of the attributes is missing from the default policy.
588		if row["min_lifetime"] is None:
589		row["min_lifetime"] = self.config.retention_default_min_lifetime
590
591		if row["max_lifetime"] is None:
592		row["max_lifetime"] = self.config.retention_default_max_lifetime
593
594		defer.returnValue(row)
595
596		def get_media_mxcs_in_room(self, room_id):
597		"""Retrieves all the local and remote media MXC URIs in a given room
598
599		Args:
600		room_id (str)
601
602		Returns:
603		The local and remote media as a lists of tuples where the key is
604		the hostname and the value is the media ID.
605		"""
606
607		def _get_media_mxcs_in_room_txn(txn):
608		local_mxcs, remote_mxcs = self._get_media_mxcs_in_room_txn(txn, room_id)
609		local_media_mxcs = []
610		remote_media_mxcs = []
611
612		# Convert the IDs to MXC URIs
613		for media_id in local_mxcs:
614		local_media_mxcs.append("mxc://%s/%s" % (self.hs.hostname, media_id))
615		for hostname, media_id in remote_mxcs:
616		remote_media_mxcs.append("mxc://%s/%s" % (hostname, media_id))
617
618		return local_media_mxcs, remote_media_mxcs
619
620		return self.db.runInteraction(
621		"get_media_ids_in_room", _get_media_mxcs_in_room_txn
622		)
623
624		def quarantine_media_ids_in_room(self, room_id, quarantined_by):
625		"""For a room loops through all events with media and quarantines
626		the associated media
627		"""
628
629		logger.info("Quarantining media in room: %s", room_id)
630
631		def _quarantine_media_in_room_txn(txn):
632		local_mxcs, remote_mxcs = self._get_media_mxcs_in_room_txn(txn, room_id)
633		return self._quarantine_media_txn(
634		txn, local_mxcs, remote_mxcs, quarantined_by
635		)
636
637		return self.db.runInteraction(
638		"quarantine_media_in_room", _quarantine_media_in_room_txn
639		)
640
641		def _get_media_mxcs_in_room_txn(self, txn, room_id):
642		"""Retrieves all the local and remote media MXC URIs in a given room
643
644		Args:
645		txn (cursor)
646		room_id (str)
647
648		Returns:
649		The local and remote media as a lists of tuples where the key is
650		the hostname and the value is the media ID.
651		"""
652		mxc_re = re.compile("^mxc://([^/]+)/([^/#?]+)")
653
654		sql = """
655		SELECT stream_ordering, json FROM events
656		JOIN event_json USING (room_id, event_id)
657		WHERE room_id = ?
658		%(where_clause)s
659		AND contains_url = ? AND outlier = ?
660		ORDER BY stream_ordering DESC
661		LIMIT ?
662		"""
663		txn.execute(sql % {"where_clause": ""}, (room_id, True, False, 100))
664
665		local_media_mxcs = []
666		remote_media_mxcs = []
667
668		while True:
669		next_token = None
670		for stream_ordering, content_json in txn:
671		next_token = stream_ordering
672		event_json = db_to_json(content_json)
673		content = event_json["content"]
674		content_url = content.get("url")
675		thumbnail_url = content.get("info", {}).get("thumbnail_url")
676
677		for url in (content_url, thumbnail_url):
678		if not url:
679		continue
680		matches = mxc_re.match(url)
681		if matches:
682		hostname = matches.group(1)
683		media_id = matches.group(2)
684		if hostname == self.hs.hostname:
685		local_media_mxcs.append(media_id)
686		else:
687		remote_media_mxcs.append((hostname, media_id))
688
689		if next_token is None:
690		# We've gone through the whole room, so we're finished.
691		break
692
693		txn.execute(
694		sql % {"where_clause": "AND stream_ordering < ?"},
695		(room_id, next_token, True, False, 100),
696		)
697
698		return local_media_mxcs, remote_media_mxcs
699
700		def quarantine_media_by_id(
701		self, server_name: str, media_id: str, quarantined_by: str,
702		):
703		"""quarantines a single local or remote media id
704
705		Args:
706		server_name: The name of the server that holds this media
707		media_id: The ID of the media to be quarantined
708		quarantined_by: The user ID that initiated the quarantine request
709		"""
710		logger.info("Quarantining media: %s/%s", server_name, media_id)
711		is_local = server_name == self.config.server_name
712
713		def _quarantine_media_by_id_txn(txn):
714		local_mxcs = [media_id] if is_local else []
715		remote_mxcs = [(server_name, media_id)] if not is_local else []
716
717		return self._quarantine_media_txn(
718		txn, local_mxcs, remote_mxcs, quarantined_by
719		)
720
721		return self.db.runInteraction(
722		"quarantine_media_by_user", _quarantine_media_by_id_txn
723		)
724
725		def quarantine_media_ids_by_user(self, user_id: str, quarantined_by: str):
726		"""quarantines all local media associated with a single user
727
728		Args:
729		user_id: The ID of the user to quarantine media of
730		quarantined_by: The ID of the user who made the quarantine request
731		"""
732
733		def _quarantine_media_by_user_txn(txn):
734		local_media_ids = self._get_media_ids_by_user_txn(txn, user_id)
735		return self._quarantine_media_txn(txn, local_media_ids, [], quarantined_by)
736
737		return self.db.runInteraction(
738		"quarantine_media_by_user", _quarantine_media_by_user_txn
739		)
740
741		def _get_media_ids_by_user_txn(self, txn, user_id: str, filter_quarantined=True):
742		"""Retrieves local media IDs by a given user
743
744		Args:
745		txn (cursor)
746		user_id: The ID of the user to retrieve media IDs of
747
748		Returns:
749		The local and remote media as a lists of tuples where the key is
750		the hostname and the value is the media ID.
751		"""
752		# Local media
753		sql = """
754		SELECT media_id
755		FROM local_media_repository
756		WHERE user_id = ?
757		"""
758		if filter_quarantined:
759		sql += "AND quarantined_by IS NULL"
760		txn.execute(sql, (user_id,))
761
762		local_media_ids = [row[0] for row in txn]
763
764		# TODO: Figure out all remote media a user has referenced in a message
765
766		return local_media_ids
767
768		def _quarantine_media_txn(
769		self,
770		txn,
771		local_mxcs: List[str],
772		remote_mxcs: List[Tuple[str, str]],
773		quarantined_by: str,
774		) -> int:
775		"""Quarantine local and remote media items
776
777		Args:
778		txn (cursor)
779		local_mxcs: A list of local mxc URLs
780		remote_mxcs: A list of (remote server, media id) tuples representing
781		remote mxc URLs
782		quarantined_by: The ID of the user who initiated the quarantine request
783		Returns:
784		The total number of media items quarantined
785		"""
786		# Update all the tables to set the quarantined_by flag
787		txn.executemany(
788		"""
789		UPDATE local_media_repository
790		SET quarantined_by = ?
791		WHERE media_id = ? AND safe_from_quarantine = ?
792		""",
793		((quarantined_by, media_id, False) for media_id in local_mxcs),
794		)
795		# Note that a rowcount of -1 can be used to indicate no rows were affected.
796		total_media_quarantined = txn.rowcount if txn.rowcount > 0 else 0
797
798		txn.executemany(
799		"""
800		UPDATE remote_media_cache
801		SET quarantined_by = ?
802		WHERE media_origin = ? AND media_id = ?
803		""",
804		((quarantined_by, origin, media_id) for origin, media_id in remote_mxcs),
805		)
806		total_media_quarantined += txn.rowcount if txn.rowcount > 0 else 0
807
808		return total_media_quarantined
809
810		async def get_all_new_public_rooms(
811		self, instance_name: str, last_id: int, current_id: int, limit: int
812		) -> Tuple[List[Tuple[int, tuple]], int, bool]:
813		"""Get updates for public rooms replication stream.
814
815		Args:
816		instance_name: The writer we want to fetch updates from. Unused
817		here since there is only ever one writer.
818		last_id: The token to fetch updates from. Exclusive.
819		current_id: The token to fetch updates up to. Inclusive.
820		limit: The requested limit for the number of rows to return. The
821		function may return more or fewer rows.
822
823		Returns:
824		A tuple consisting of: the updates, a token to use to fetch
825		subsequent updates, and whether we returned fewer rows than exists
826		between the requested tokens due to the limit.
827
828		The token returned can be used in a subsequent call to this
829		function to get further updatees.
830
831		The updates are a list of 2-tuples of stream ID and the row data
832		"""
833		if last_id == current_id:
834		return [], current_id, False
835
836		def get_all_new_public_rooms(txn):
837		sql = """
838		SELECT stream_id, room_id, visibility, appservice_id, network_id
839		FROM public_room_list_stream
840		WHERE stream_id > ? AND stream_id <= ?
841		ORDER BY stream_id ASC
842		LIMIT ?
843		"""
844
845		txn.execute(sql, (last_id, current_id, limit))
846		updates = [(row[0], row[1:]) for row in txn]
847		limited = False
848		upto_token = current_id
849		if len(updates) >= limit:
850		upto_token = updates[-1][0]
851		limited = True
852
853		return updates, upto_token, limited
854
855		return await self.db.runInteraction(
856		"get_all_new_public_rooms", get_all_new_public_rooms
857		)
858
859
860		class RoomBackgroundUpdateStore(SQLBaseStore):
861		REMOVE_TOMESTONED_ROOMS_BG_UPDATE = "remove_tombstoned_rooms_from_directory"
862		ADD_ROOMS_ROOM_VERSION_COLUMN = "add_rooms_room_version_column"
863
864		def __init__(self, database: Database, db_conn, hs):
865		super(RoomBackgroundUpdateStore, self).__init__(database, db_conn, hs)
866
867		self.config = hs.config
868
869		self.db.updates.register_background_update_handler(
870		"insert_room_retention", self._background_insert_retention,
871		)
872
873		self.db.updates.register_background_update_handler(
874		self.REMOVE_TOMESTONED_ROOMS_BG_UPDATE,
875		self._remove_tombstoned_rooms_from_directory,
876		)
877
878		self.db.updates.register_background_update_handler(
879		self.ADD_ROOMS_ROOM_VERSION_COLUMN,
880		self._background_add_rooms_room_version_column,
881		)
882
883		@defer.inlineCallbacks
884		def _background_insert_retention(self, progress, batch_size):
885		"""Retrieves a list of all rooms within a range and inserts an entry for each of
886		them into the room_retention table.
887		NULLs the property's columns if missing from the retention event in the room's
888		state (or NULLs all of them if there's no retention event in the room's state),
889		so that we fall back to the server's retention policy.
890		"""
891
892		last_room = progress.get("room_id", "")
893
894		def _background_insert_retention_txn(txn):
895		txn.execute(
896		"""
897		SELECT state.room_id, state.event_id, events.json
898		FROM current_state_events as state
899		LEFT JOIN event_json AS events ON (state.event_id = events.event_id)
900		WHERE state.room_id > ? AND state.type = '%s'
901		ORDER BY state.room_id ASC
902		LIMIT ?;
903		"""
904		% EventTypes.Retention,
905		(last_room, batch_size),
906		)
907
908		rows = self.db.cursor_to_dict(txn)
909
910		if not rows:
911		return True
912
913		for row in rows:
914		if not row["json"]:
915		retention_policy = {}
916		else:
917		ev = db_to_json(row["json"])
918		retention_policy = ev["content"]
919
920		self.db.simple_insert_txn(
921		txn=txn,
922		table="room_retention",
923		values={
924		"room_id": row["room_id"],
925		"event_id": row["event_id"],
926		"min_lifetime": retention_policy.get("min_lifetime"),
927		"max_lifetime": retention_policy.get("max_lifetime"),
928		},
929		)
930
931		logger.info("Inserted %d rows into room_retention", len(rows))
932
933		self.db.updates._background_update_progress_txn(
934		txn, "insert_room_retention", {"room_id": rows[-1]["room_id"]}
935		)
936
937		if batch_size > len(rows):
938		return True
939		else:
940		return False
941
942		end = yield self.db.runInteraction(
943		"insert_room_retention", _background_insert_retention_txn,
944		)
945
946		if end:
947		yield self.db.updates._end_background_update("insert_room_retention")
948
949		defer.returnValue(batch_size)
950
951		async def _background_add_rooms_room_version_column(
952		self, progress: dict, batch_size: int
953		):
954		"""Background update to go and add room version inforamtion to `rooms`
955		table from `current_state_events` table.
956		"""
957
958		last_room_id = progress.get("room_id", "")
959
960		def _background_add_rooms_room_version_column_txn(txn: LoggingTransaction):
961		sql = """
962		SELECT room_id, json FROM current_state_events
963		INNER JOIN event_json USING (room_id, event_id)
964		WHERE room_id > ? AND type = 'm.room.create' AND state_key = ''
965		ORDER BY room_id
966		LIMIT ?
967		"""
968
969		txn.execute(sql, (last_room_id, batch_size))
970
971		updates = []
972		for room_id, event_json in txn:
973		event_dict = db_to_json(event_json)
974		room_version_id = event_dict.get("content", {}).get(
975		"room_version", RoomVersions.V1.identifier
976		)
977
978		creator = event_dict.get("content").get("creator")
979
980		updates.append((room_id, creator, room_version_id))
981
982		if not updates:
983		return True
984
985		new_last_room_id = ""
986		for room_id, creator, room_version_id in updates:
987		# We upsert here just in case we don't already have a row,
988		# mainly for paranoia as much badness would happen if we don't
989		# insert the row and then try and get the room version for the
990		# room.
991		self.db.simple_upsert_txn(
992		txn,
993		table="rooms",
994		keyvalues={"room_id": room_id},
995		values={"room_version": room_version_id},
996		insertion_values={"is_public": False, "creator": creator},
997		)
998		new_last_room_id = room_id
999
1000		self.db.updates._background_update_progress_txn(
1001		txn, self.ADD_ROOMS_ROOM_VERSION_COLUMN, {"room_id": new_last_room_id}
1002		)
1003
1004		return False
1005
1006		end = await self.db.runInteraction(
1007		"_background_add_rooms_room_version_column",
1008		_background_add_rooms_room_version_column_txn,
1009		)
1010
1011		if end:
1012		await self.db.updates._end_background_update(
1013		self.ADD_ROOMS_ROOM_VERSION_COLUMN
1014		)
1015
1016		return batch_size
1017
1018		async def _remove_tombstoned_rooms_from_directory(
1019		self, progress, batch_size
1020		) -> int:
1021		"""Removes any rooms with tombstone events from the room directory
1022
1023		Nowadays this is handled by the room upgrade handler, but we may have some
1024		that got left behind
1025		"""
1026
1027		last_room = progress.get("room_id", "")
1028
1029		def _get_rooms(txn):
1030		txn.execute(
1031		"""
1032		SELECT room_id
1033		FROM rooms r
1034		INNER JOIN current_state_events cse USING (room_id)
1035		WHERE room_id > ? AND r.is_public
1036		AND cse.type = '%s' AND cse.state_key = ''
1037		ORDER BY room_id ASC
1038		LIMIT ?;
1039		"""
1040		% EventTypes.Tombstone,
1041		(last_room, batch_size),
1042		)
1043
1044		return [row[0] for row in txn]
1045
1046		rooms = await self.db.runInteraction(
1047		"get_tombstoned_directory_rooms", _get_rooms
1048		)
1049
1050		if not rooms:
1051		await self.db.updates._end_background_update(
1052		self.REMOVE_TOMESTONED_ROOMS_BG_UPDATE
1053		)
1054		return 0
1055
1056		for room_id in rooms:
1057		logger.info("Removing tombstoned room %s from the directory", room_id)
1058		await self.set_room_is_public(room_id, False)
1059
1060		await self.db.updates._background_update_progress(
1061		self.REMOVE_TOMESTONED_ROOMS_BG_UPDATE, {"room_id": rooms[-1]}
1062		)
1063
1064		return len(rooms)
1065
1066		@abstractmethod
1067		def set_room_is_public(self, room_id, is_public):
1068		# this will need to be implemented if a background update is performed with
1069		# existing (tombstoned, public) rooms in the database.
1070		#
1071		# It's overridden by RoomStore for the synapse master.
1072		raise NotImplementedError()
1073
1074
1075		class RoomStore(RoomBackgroundUpdateStore, RoomWorkerStore, SearchStore):
1076		def __init__(self, database: Database, db_conn, hs):
1077		super(RoomStore, self).__init__(database, db_conn, hs)
1078
1079		self.config = hs.config
1080
1081		async def upsert_room_on_join(self, room_id: str, room_version: RoomVersion):
1082		"""Ensure that the room is stored in the table
1083
1084		Called when we join a room over federation, and overwrites any room version
1085		currently in the table.
1086		"""
1087		await self.db.simple_upsert(
1088		desc="upsert_room_on_join",
1089		table="rooms",
1090		keyvalues={"room_id": room_id},
1091		values={"room_version": room_version.identifier},
1092		insertion_values={"is_public": False, "creator": ""},
1093		# rooms has a unique constraint on room_id, so no need to lock when doing an
1094		# emulated upsert.
1095		lock=False,
1096		)
1097
1098		@defer.inlineCallbacks
1099		def store_room(
1100		self,
1101		room_id: str,
1102		room_creator_user_id: str,
1103		is_public: bool,
1104		room_version: RoomVersion,
1105		):
1106		"""Stores a room.
1107
1108		Args:
1109		room_id: The desired room ID, can be None.
1110		room_creator_user_id: The user ID of the room creator.
1111		is_public: True to indicate that this room should appear in
1112		public room lists.
1113		room_version: The version of the room
1114		Raises:
1115		StoreError if the room could not be stored.
1116		"""
1117		try:
1118
1119		def store_room_txn(txn, next_id):
1120		self.db.simple_insert_txn(
1121		txn,
1122		"rooms",
1123		{
1124		"room_id": room_id,
1125		"creator": room_creator_user_id,
1126		"is_public": is_public,
1127		"room_version": room_version.identifier,
1128		},
1129		)
1130		if is_public:
1131		self.db.simple_insert_txn(
1132		txn,
1133		table="public_room_list_stream",
1134		values={
1135		"stream_id": next_id,
1136		"room_id": room_id,
1137		"visibility": is_public,
1138		},
1139		)
1140
1141		with self._public_room_id_gen.get_next() as next_id:
1142		yield self.db.runInteraction("store_room_txn", store_room_txn, next_id)
1143		except Exception as e:
1144		logger.error("store_room with room_id=%s failed: %s", room_id, e)
1145		raise StoreError(500, "Problem creating room.")
1146
1147		async def maybe_store_room_on_invite(self, room_id: str, room_version: RoomVersion):
1148		"""
1149		When we receive an invite over federation, store the version of the room if we
1150		don't already know the room version.
1151		"""
1152		await self.db.simple_upsert(
1153		desc="maybe_store_room_on_invite",
1154		table="rooms",
1155		keyvalues={"room_id": room_id},
1156		values={},
1157		insertion_values={
1158		"room_version": room_version.identifier,
1159		"is_public": False,
1160		"creator": "",
1161		},
1162		# rooms has a unique constraint on room_id, so no need to lock when doing an
1163		# emulated upsert.
1164		lock=False,
1165		)
1166
1167		@defer.inlineCallbacks
1168		def set_room_is_public(self, room_id, is_public):
1169		def set_room_is_public_txn(txn, next_id):
1170		self.db.simple_update_one_txn(
1171		txn,
1172		table="rooms",
1173		keyvalues={"room_id": room_id},
1174		updatevalues={"is_public": is_public},
1175		)
1176
1177		entries = self.db.simple_select_list_txn(
1178		txn,
1179		table="public_room_list_stream",
1180		keyvalues={
1181		"room_id": room_id,
1182		"appservice_id": None,
1183		"network_id": None,
1184		},
1185		retcols=("stream_id", "visibility"),
1186		)
1187
1188		entries.sort(key=lambda r: r["stream_id"])
1189
1190		add_to_stream = True
1191		if entries:
1192		add_to_stream = bool(entries[-1]["visibility"]) != is_public
1193
1194		if add_to_stream:
1195		self.db.simple_insert_txn(
1196		txn,
1197		table="public_room_list_stream",
1198		values={
1199		"stream_id": next_id,
1200		"room_id": room_id,
1201		"visibility": is_public,
1202		"appservice_id": None,
1203		"network_id": None,
1204		},
1205		)
1206
1207		with self._public_room_id_gen.get_next() as next_id:
1208		yield self.db.runInteraction(
1209		"set_room_is_public", set_room_is_public_txn, next_id
1210		)
1211		self.hs.get_notifier().on_new_replication_data()
1212
1213		@defer.inlineCallbacks
1214		def set_room_is_public_appservice(
1215		self, room_id, appservice_id, network_id, is_public
1216		):
1217		"""Edit the appservice/network specific public room list.
1218
1219		Each appservice can have a number of published room lists associated
1220		with them, keyed off of an appservice defined `network_id`, which
1221		basically represents a single instance of a bridge to a third party
1222		network.
1223
1224		Args:
1225		room_id (str)
1226		appservice_id (str)
1227		network_id (str)
1228		is_public (bool): Whether to publish or unpublish the room from the
1229		list.
1230		"""
1231
1232		def set_room_is_public_appservice_txn(txn, next_id):
1233		if is_public:
1234		try:
1235		self.db.simple_insert_txn(
1236		txn,
1237		table="appservice_room_list",
1238		values={
1239		"appservice_id": appservice_id,
1240		"network_id": network_id,
1241		"room_id": room_id,
1242		},
1243		)
1244		except self.database_engine.module.IntegrityError:
1245		# We've already inserted, nothing to do.
1246		return
1247		else:
1248		self.db.simple_delete_txn(
1249		txn,
1250		table="appservice_room_list",
1251		keyvalues={
1252		"appservice_id": appservice_id,
1253		"network_id": network_id,
1254		"room_id": room_id,
1255		},
1256		)
1257
1258		entries = self.db.simple_select_list_txn(
1259		txn,
1260		table="public_room_list_stream",
1261		keyvalues={
1262		"room_id": room_id,
1263		"appservice_id": appservice_id,
1264		"network_id": network_id,
1265		},
1266		retcols=("stream_id", "visibility"),
1267		)
1268
1269		entries.sort(key=lambda r: r["stream_id"])
1270
1271		add_to_stream = True
1272		if entries:
1273		add_to_stream = bool(entries[-1]["visibility"]) != is_public
1274
1275		if add_to_stream:
1276		self.db.simple_insert_txn(
1277		txn,
1278		table="public_room_list_stream",
1279		values={
1280		"stream_id": next_id,
1281		"room_id": room_id,
1282		"visibility": is_public,
1283		"appservice_id": appservice_id,
1284		"network_id": network_id,
1285		},
1286		)
1287
1288		with self._public_room_id_gen.get_next() as next_id:
1289		yield self.db.runInteraction(
1290		"set_room_is_public_appservice",
1291		set_room_is_public_appservice_txn,
1292		next_id,
1293		)
1294		self.hs.get_notifier().on_new_replication_data()
1295
1296		def get_room_count(self):
1297		"""Retrieve a list of all rooms
1298		"""
1299
1300		def f(txn):
1301		sql = "SELECT count(*) FROM rooms"
1302		txn.execute(sql)
1303		row = txn.fetchone()
1304		return row[0] or 0
1305
1306		return self.db.runInteraction("get_rooms", f)
1307
1308		def add_event_report(
1309		self, room_id, event_id, user_id, reason, content, received_ts
1310		):
1311		next_id = self._event_reports_id_gen.get_next()
1312		return self.db.simple_insert(
1313		table="event_reports",
1314		values={
1315		"id": next_id,
1316		"received_ts": received_ts,
1317		"room_id": room_id,
1318		"event_id": event_id,
1319		"user_id": user_id,
1320		"reason": reason,
1321		"content": json.dumps(content),
1322		},
1323		desc="add_event_report",
1324		)
1325
1326		def get_current_public_room_stream_id(self):
1327		return self._public_room_id_gen.get_current_token()
1328
1329		@defer.inlineCallbacks
1330		def block_room(self, room_id, user_id):
1331		"""Marks the room as blocked. Can be called multiple times.
1332
1333		Args:
1334		room_id (str): Room to block
1335		user_id (str): Who blocked it
1336
1337		Returns:
1338		Deferred
1339		"""
1340		yield self.db.simple_upsert(
1341		table="blocked_rooms",
1342		keyvalues={"room_id": room_id},
1343		values={},
1344		insertion_values={"user_id": user_id},
1345		desc="block_room",
1346		)
1347		yield self.db.runInteraction(
1348		"block_room_invalidation",
1349		self._invalidate_cache_and_stream,
1350		self.is_room_blocked,
1351		(room_id,),
1352		)
1353
1354		@defer.inlineCallbacks
1355		def get_rooms_for_retention_period_in_range(
1356		self, min_ms, max_ms, include_null=False
1357		):
1358		"""Retrieves all of the rooms within the given retention range.
1359
1360		Optionally includes the rooms which don't have a retention policy.
1361
1362		Args:
1363		min_ms (int\|None): Duration in milliseconds that define the lower limit of
1364		the range to handle (exclusive). If None, doesn't set a lower limit.
1365		max_ms (int\|None): Duration in milliseconds that define the upper limit of
1366		the range to handle (inclusive). If None, doesn't set an upper limit.
1367		include_null (bool): Whether to include rooms which retention policy is NULL
1368		in the returned set.
1369
1370		Returns:
1371		dict[str, dict]: The rooms within this range, along with their retention
1372		policy. The key is "room_id", and maps to a dict describing the retention
1373		policy associated with this room ID. The keys for this nested dict are
1374		"min_lifetime" (int\|None), and "max_lifetime" (int\|None).
1375		"""
1376
1377		def get_rooms_for_retention_period_in_range_txn(txn):
1378		range_conditions = []
1379		args = []
1380
1381		if min_ms is not None:
1382		range_conditions.append("max_lifetime > ?")
1383		args.append(min_ms)
1384
1385		if max_ms is not None:
1386		range_conditions.append("max_lifetime <= ?")
1387		args.append(max_ms)
1388
1389		# Do a first query which will retrieve the rooms that have a retention policy
1390		# in their current state.
1391		sql = """
1392		SELECT room_id, min_lifetime, max_lifetime FROM room_retention
1393		INNER JOIN current_state_events USING (event_id, room_id)
1394		"""
1395
1396		if len(range_conditions):
1397		sql += " WHERE (" + " AND ".join(range_conditions) + ")"
1398
1399		if include_null:
1400		sql += " OR max_lifetime IS NULL"
1401
1402		txn.execute(sql, args)
1403
1404		rows = self.db.cursor_to_dict(txn)
1405		rooms_dict = {}
1406
1407		for row in rows:
1408		rooms_dict[row["room_id"]] = {
1409		"min_lifetime": row["min_lifetime"],
1410		"max_lifetime": row["max_lifetime"],
1411		}
1412
1413		if include_null:
1414		# If required, do a second query that retrieves all of the rooms we know
1415		# of so we can handle rooms with no retention policy.
1416		sql = "SELECT DISTINCT room_id FROM current_state_events"
1417
1418		txn.execute(sql)
1419
1420		rows = self.db.cursor_to_dict(txn)
1421
1422		# If a room isn't already in the dict (i.e. it doesn't have a retention
1423		# policy in its state), add it with a null policy.
1424		for row in rows:
1425		if row["room_id"] not in rooms_dict:
1426		rooms_dict[row["room_id"]] = {
1427		"min_lifetime": None,
1428		"max_lifetime": None,
1429		}
1430
1431		return rooms_dict
1432
1433		rooms = yield self.db.runInteraction(
1434		"get_rooms_for_retention_period_in_range",
1435		get_rooms_for_retention_period_in_range_txn,
1436		)
1437
1438		defer.returnValue(rooms)

+0

-1135

~~synapse/storage/data_stores/main/roommember.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2018 New Vector Ltd
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		import logging
17		from typing import Iterable, List, Set
18
19		from twisted.internet import defer
20
21		from synapse.api.constants import EventTypes, Membership
22		from synapse.metrics import LaterGauge
23		from synapse.metrics.background_process_metrics import run_as_background_process
24		from synapse.storage._base import (
25		LoggingTransaction,
26		SQLBaseStore,
27		db_to_json,
28		make_in_list_sql_clause,
29		)
30		from synapse.storage.data_stores.main.events_worker import EventsWorkerStore
31		from synapse.storage.database import Database
32		from synapse.storage.engines import Sqlite3Engine
33		from synapse.storage.roommember import (
34		GetRoomsForUserWithStreamOrdering,
35		MemberSummary,
36		ProfileInfo,
37		RoomsForUser,
38		)
39		from synapse.types import Collection, get_domain_from_id
40		from synapse.util.async_helpers import Linearizer
41		from synapse.util.caches import intern_string
42		from synapse.util.caches.descriptors import cached, cachedInlineCallbacks, cachedList
43		from synapse.util.metrics import Measure
44
45		logger = logging.getLogger(__name__)
46
47
48		_MEMBERSHIP_PROFILE_UPDATE_NAME = "room_membership_profile_update"
49		_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME = "current_state_events_membership"
50
51
52		class RoomMemberWorkerStore(EventsWorkerStore):
53		def __init__(self, database: Database, db_conn, hs):
54		super(RoomMemberWorkerStore, self).__init__(database, db_conn, hs)
55
56		# Is the current_state_events.membership up to date? Or is the
57		# background update still running?
58		self._current_state_events_membership_up_to_date = False
59
60		txn = LoggingTransaction(
61		db_conn.cursor(),
62		name="_check_safe_current_state_events_membership_updated",
63		database_engine=self.database_engine,
64		)
65		self._check_safe_current_state_events_membership_updated_txn(txn)
66		txn.close()
67
68		if self.hs.config.metrics_flags.known_servers:
69		self._known_servers_count = 1
70		self.hs.get_clock().looping_call(
71		run_as_background_process,
72		60 * 1000,
73		"_count_known_servers",
74		self._count_known_servers,
75		)
76		self.hs.get_clock().call_later(
77		1000,
78		run_as_background_process,
79		"_count_known_servers",
80		self._count_known_servers,
81		)
82		LaterGauge(
83		"synapse_federation_known_servers",
84		"",
85		[],
86		lambda: self._known_servers_count,
87		)
88
89		@defer.inlineCallbacks
90		def _count_known_servers(self):
91		"""
92		Count the servers that this server knows about.
93
94		The statistic is stored on the class for the
95		`synapse_federation_known_servers` LaterGauge to collect.
96		"""
97
98		def _transact(txn):
99		if isinstance(self.database_engine, Sqlite3Engine):
100		query = """
101		SELECT COUNT(DISTINCT substr(out.user_id, pos+1))
102		FROM (
103		SELECT rm.user_id as user_id, instr(rm.user_id, ':')
104		AS pos FROM room_memberships as rm
105		INNER JOIN current_state_events as c ON rm.event_id = c.event_id
106		WHERE c.type = 'm.room.member'
107		) as out
108		"""
109		else:
110		query = """
111		SELECT COUNT(DISTINCT split_part(state_key, ':', 2))
112		FROM current_state_events
113		WHERE type = 'm.room.member' AND membership = 'join';
114		"""
115		txn.execute(query)
116		return list(txn)[0][0]
117
118		count = yield self.db.runInteraction("get_known_servers", _transact)
119
120		# We always know about ourselves, even if we have nothing in
121		# room_memberships (for example, the server is new).
122		self._known_servers_count = max([count, 1])
123		return self._known_servers_count
124
125		def _check_safe_current_state_events_membership_updated_txn(self, txn):
126		"""Checks if it is safe to assume the new current_state_events
127		membership column is up to date
128		"""
129
130		pending_update = self.db.simple_select_one_txn(
131		txn,
132		table="background_updates",
133		keyvalues={"update_name": _CURRENT_STATE_MEMBERSHIP_UPDATE_NAME},
134		retcols=["update_name"],
135		allow_none=True,
136		)
137
138		self._current_state_events_membership_up_to_date = not pending_update
139
140		# If the update is still running, reschedule to run.
141		if pending_update:
142		self._clock.call_later(
143		15.0,
144		run_as_background_process,
145		"_check_safe_current_state_events_membership_updated",
146		self.db.runInteraction,
147		"_check_safe_current_state_events_membership_updated",
148		self._check_safe_current_state_events_membership_updated_txn,
149		)
150
151		@cached(max_entries=100000, iterable=True)
152		def get_users_in_room(self, room_id):
153		return self.db.runInteraction(
154		"get_users_in_room", self.get_users_in_room_txn, room_id
155		)
156
157		def get_users_in_room_txn(self, txn, room_id):
158		# If we can assume current_state_events.membership is up to date
159		# then we can avoid a join, which is a Very Good Thing given how
160		# frequently this function gets called.
161		if self._current_state_events_membership_up_to_date:
162		sql = """
163		SELECT state_key FROM current_state_events
164		WHERE type = 'm.room.member' AND room_id = ? AND membership = ?
165		"""
166		else:
167		sql = """
168		SELECT state_key FROM room_memberships as m
169		INNER JOIN current_state_events as c
170		ON m.event_id = c.event_id
171		AND m.room_id = c.room_id
172		AND m.user_id = c.state_key
173		WHERE c.type = 'm.room.member' AND c.room_id = ? AND m.membership = ?
174		"""
175
176		txn.execute(sql, (room_id, Membership.JOIN))
177		return [r[0] for r in txn]
178
179		@cached(max_entries=100000)
180		def get_room_summary(self, room_id):
181		""" Get the details of a room roughly suitable for use by the room
182		summary extension to /sync. Useful when lazy loading room members.
183		Args:
184		room_id (str): The room ID to query
185		Returns:
186		Deferred[dict[str, MemberSummary]:
187		dict of membership states, pointing to a MemberSummary named tuple.
188		"""
189
190		def _get_room_summary_txn(txn):
191		# first get counts.
192		# We do this all in one transaction to keep the cache small.
193		# FIXME: get rid of this when we have room_stats
194
195		# If we can assume current_state_events.membership is up to date
196		# then we can avoid a join, which is a Very Good Thing given how
197		# frequently this function gets called.
198		if self._current_state_events_membership_up_to_date:
199		# Note, rejected events will have a null membership field, so
200		# we we manually filter them out.
201		sql = """
202		SELECT count(*), membership FROM current_state_events
203		WHERE type = 'm.room.member' AND room_id = ?
204		AND membership IS NOT NULL
205		GROUP BY membership
206		"""
207		else:
208		sql = """
209		SELECT count(*), m.membership FROM room_memberships as m
210		INNER JOIN current_state_events as c
211		ON m.event_id = c.event_id
212		AND m.room_id = c.room_id
213		AND m.user_id = c.state_key
214		WHERE c.type = 'm.room.member' AND c.room_id = ?
215		GROUP BY m.membership
216		"""
217
218		txn.execute(sql, (room_id,))
219		res = {}
220		for count, membership in txn:
221		summary = res.setdefault(membership, MemberSummary([], count))
222
223		# we order by membership and then fairly arbitrarily by event_id so
224		# heroes are consistent
225		if self._current_state_events_membership_up_to_date:
226		# Note, rejected events will have a null membership field, so
227		# we we manually filter them out.
228		sql = """
229		SELECT state_key, membership, event_id
230		FROM current_state_events
231		WHERE type = 'm.room.member' AND room_id = ?
232		AND membership IS NOT NULL
233		ORDER BY
234		CASE membership WHEN ? THEN 1 WHEN ? THEN 2 ELSE 3 END ASC,
235		event_id ASC
236		LIMIT ?
237		"""
238		else:
239		sql = """
240		SELECT c.state_key, m.membership, c.event_id
241		FROM room_memberships as m
242		INNER JOIN current_state_events as c USING (room_id, event_id)
243		WHERE c.type = 'm.room.member' AND c.room_id = ?
244		ORDER BY
245		CASE m.membership WHEN ? THEN 1 WHEN ? THEN 2 ELSE 3 END ASC,
246		c.event_id ASC
247		LIMIT ?
248		"""
249
250		# 6 is 5 (number of heroes) plus 1, in case one of them is the calling user.
251		txn.execute(sql, (room_id, Membership.JOIN, Membership.INVITE, 6))
252		for user_id, membership, event_id in txn:
253		summary = res[membership]
254		# we will always have a summary for this membership type at this
255		# point given the summary currently contains the counts.
256		members = summary.members
257		members.append((user_id, event_id))
258
259		return res
260
261		return self.db.runInteraction("get_room_summary", _get_room_summary_txn)
262
263		def _get_user_counts_in_room_txn(self, txn, room_id):
264		"""
265		Get the user count in a room by membership.
266
267		Args:
268		room_id (str)
269		membership (Membership)
270
271		Returns:
272		Deferred[int]
273		"""
274		sql = """
275		SELECT m.membership, count(*) FROM room_memberships as m
276		INNER JOIN current_state_events as c USING(event_id)
277		WHERE c.type = 'm.room.member' AND c.room_id = ?
278		GROUP BY m.membership
279		"""
280
281		txn.execute(sql, (room_id,))
282		return {row[0]: row[1] for row in txn}
283
284		@cached()
285		def get_invited_rooms_for_local_user(self, user_id):
286		""" Get all the rooms the local user is invited to
287
288		Args:
289		user_id (str): The user ID.
290		Returns:
291		A deferred list of RoomsForUser.
292		"""
293
294		return self.get_rooms_for_local_user_where_membership_is(
295		user_id, [Membership.INVITE]
296		)
297
298		@defer.inlineCallbacks
299		def get_invite_for_local_user_in_room(self, user_id, room_id):
300		"""Gets the invite for the given local user and room
301
302		Args:
303		user_id (str)
304		room_id (str)
305
306		Returns:
307		Deferred: Resolves to either a RoomsForUser or None if no invite was
308		found.
309		"""
310		invites = yield self.get_invited_rooms_for_local_user(user_id)
311		for invite in invites:
312		if invite.room_id == room_id:
313		return invite
314		return None
315
316		@defer.inlineCallbacks
317		def get_rooms_for_local_user_where_membership_is(self, user_id, membership_list):
318		""" Get all the rooms for this local user where the membership for this user
319		matches one in the membership list.
320
321		Filters out forgotten rooms.
322
323		Args:
324		user_id (str): The user ID.
325		membership_list (list): A list of synapse.api.constants.Membership
326		values which the user must be in.
327
328		Returns:
329		Deferred[list[RoomsForUser]]
330		"""
331		if not membership_list:
332		return defer.succeed(None)
333
334		rooms = yield self.db.runInteraction(
335		"get_rooms_for_local_user_where_membership_is",
336		self._get_rooms_for_local_user_where_membership_is_txn,
337		user_id,
338		membership_list,
339		)
340
341		# Now we filter out forgotten rooms
342		forgotten_rooms = yield self.get_forgotten_rooms_for_user(user_id)
343		return [room for room in rooms if room.room_id not in forgotten_rooms]
344
345		def _get_rooms_for_local_user_where_membership_is_txn(
346		self, txn, user_id, membership_list
347		):
348		# Paranoia check.
349		if not self.hs.is_mine_id(user_id):
350		raise Exception(
351		"Cannot call 'get_rooms_for_local_user_where_membership_is' on non-local user %r"
352		% (user_id,),
353		)
354
355		clause, args = make_in_list_sql_clause(
356		self.database_engine, "c.membership", membership_list
357		)
358
359		sql = """
360		SELECT room_id, e.sender, c.membership, event_id, e.stream_ordering
361		FROM local_current_membership AS c
362		INNER JOIN events AS e USING (room_id, event_id)
363		WHERE
364		user_id = ?
365		AND %s
366		""" % (
367		clause,
368		)
369
370		txn.execute(sql, (user_id, *args))
371		results = [RoomsForUser(**r) for r in self.db.cursor_to_dict(txn)]
372
373		return results
374
375		@cached(max_entries=500000, iterable=True)
376		def get_rooms_for_user_with_stream_ordering(self, user_id):
377		"""Returns a set of room_ids the user is currently joined to.
378
379		If a remote user only returns rooms this server is currently
380		participating in.
381
382		Args:
383		user_id (str)
384
385		Returns:
386		Deferred[frozenset[GetRoomsForUserWithStreamOrdering]]: Returns
387		the rooms the user is in currently, along with the stream ordering
388		of the most recent join for that user and room.
389		"""
390		return self.db.runInteraction(
391		"get_rooms_for_user_with_stream_ordering",
392		self._get_rooms_for_user_with_stream_ordering_txn,
393		user_id,
394		)
395
396		def _get_rooms_for_user_with_stream_ordering_txn(self, txn, user_id):
397		# We use `current_state_events` here and not `local_current_membership`
398		# as a) this gets called with remote users and b) this only gets called
399		# for rooms the server is participating in.
400		if self._current_state_events_membership_up_to_date:
401		sql = """
402		SELECT room_id, e.stream_ordering
403		FROM current_state_events AS c
404		INNER JOIN events AS e USING (room_id, event_id)
405		WHERE
406		c.type = 'm.room.member'
407		AND state_key = ?
408		AND c.membership = ?
409		"""
410		else:
411		sql = """
412		SELECT room_id, e.stream_ordering
413		FROM current_state_events AS c
414		INNER JOIN room_memberships AS m USING (room_id, event_id)
415		INNER JOIN events AS e USING (room_id, event_id)
416		WHERE
417		c.type = 'm.room.member'
418		AND state_key = ?
419		AND m.membership = ?
420		"""
421
422		txn.execute(sql, (user_id, Membership.JOIN))
423		results = frozenset(GetRoomsForUserWithStreamOrdering(*row) for row in txn)
424
425		return results
426
427		async def get_users_server_still_shares_room_with(
428		self, user_ids: Collection[str]
429		) -> Set[str]:
430		"""Given a list of users return the set that the server still share a
431		room with.
432		"""
433
434		if not user_ids:
435		return set()
436
437		def _get_users_server_still_shares_room_with_txn(txn):
438		sql = """
439		SELECT state_key FROM current_state_events
440		WHERE
441		type = 'm.room.member'
442		AND membership = 'join'
443		AND %s
444		GROUP BY state_key
445		"""
446
447		clause, args = make_in_list_sql_clause(
448		self.database_engine, "state_key", user_ids
449		)
450
451		txn.execute(sql % (clause,), args)
452
453		return {row[0] for row in txn}
454
455		return await self.db.runInteraction(
456		"get_users_server_still_shares_room_with",
457		_get_users_server_still_shares_room_with_txn,
458		)
459
460		@defer.inlineCallbacks
461		def get_rooms_for_user(self, user_id, on_invalidate=None):
462		"""Returns a set of room_ids the user is currently joined to.
463
464		If a remote user only returns rooms this server is currently
465		participating in.
466		"""
467		rooms = yield self.get_rooms_for_user_with_stream_ordering(
468		user_id, on_invalidate=on_invalidate
469		)
470		return frozenset(r.room_id for r in rooms)
471
472		@cachedInlineCallbacks(max_entries=500000, cache_context=True, iterable=True)
473		def get_users_who_share_room_with_user(self, user_id, cache_context):
474		"""Returns the set of users who share a room with `user_id`
475		"""
476		room_ids = yield self.get_rooms_for_user(
477		user_id, on_invalidate=cache_context.invalidate
478		)
479
480		user_who_share_room = set()
481		for room_id in room_ids:
482		user_ids = yield self.get_users_in_room(
483		room_id, on_invalidate=cache_context.invalidate
484		)
485		user_who_share_room.update(user_ids)
486
487		return user_who_share_room
488
489		@defer.inlineCallbacks
490		def get_joined_users_from_context(self, event, context):
491		state_group = context.state_group
492		if not state_group:
493		# If state_group is None it means it has yet to be assigned a
494		# state group, i.e. we need to make sure that calls with a state_group
495		# of None don't hit previous cached calls with a None state_group.
496		# To do this we set the state_group to a new object as object() != object()
497		state_group = object()
498
499		current_state_ids = yield defer.ensureDeferred(context.get_current_state_ids())
500		result = yield self._get_joined_users_from_context(
501		event.room_id, state_group, current_state_ids, event=event, context=context
502		)
503		return result
504
505		@defer.inlineCallbacks
506		def get_joined_users_from_state(self, room_id, state_entry):
507		state_group = state_entry.state_group
508		if not state_group:
509		# If state_group is None it means it has yet to be assigned a
510		# state group, i.e. we need to make sure that calls with a state_group
511		# of None don't hit previous cached calls with a None state_group.
512		# To do this we set the state_group to a new object as object() != object()
513		state_group = object()
514
515		with Measure(self._clock, "get_joined_users_from_state"):
516		return (
517		yield self._get_joined_users_from_context(
518		room_id, state_group, state_entry.state, context=state_entry
519		)
520		)
521
522		@cachedInlineCallbacks(
523		num_args=2, cache_context=True, iterable=True, max_entries=100000
524		)
525		def _get_joined_users_from_context(
526		self,
527		room_id,
528		state_group,
529		current_state_ids,
530		cache_context,
531		event=None,
532		context=None,
533		):
534		# We don't use `state_group`, it's there so that we can cache based
535		# on it. However, it's important that it's never None, since two current_states
536		# with a state_group of None are likely to be different.
537		# See bulk_get_push_rules_for_room for how we work around this.
538		assert state_group is not None
539
540		users_in_room = {}
541		member_event_ids = [
542		e_id
543		for key, e_id in current_state_ids.items()
544		if key[0] == EventTypes.Member
545		]
546
547		if context is not None:
548		# If we have a context with a delta from a previous state group,
549		# check if we also have the result from the previous group in cache.
550		# If we do then we can reuse that result and simply update it with
551		# any membership changes in `delta_ids`
552		if context.prev_group and context.delta_ids:
553		prev_res = self._get_joined_users_from_context.cache.get(
554		(room_id, context.prev_group), None
555		)
556		if prev_res and isinstance(prev_res, dict):
557		users_in_room = dict(prev_res)
558		member_event_ids = [
559		e_id
560		for key, e_id in context.delta_ids.items()
561		if key[0] == EventTypes.Member
562		]
563		for etype, state_key in context.delta_ids:
564		if etype == EventTypes.Member:
565		users_in_room.pop(state_key, None)
566
567		# We check if we have any of the member event ids in the event cache
568		# before we ask the DB
569
570		# We don't update the event cache hit ratio as it completely throws off
571		# the hit ratio counts. After all, we don't populate the cache if we
572		# miss it here
573		event_map = self._get_events_from_cache(
574		member_event_ids, allow_rejected=False, update_metrics=False
575		)
576
577		missing_member_event_ids = []
578		for event_id in member_event_ids:
579		ev_entry = event_map.get(event_id)
580		if ev_entry:
581		if ev_entry.event.membership == Membership.JOIN:
582		users_in_room[ev_entry.event.state_key] = ProfileInfo(
583		display_name=ev_entry.event.content.get("displayname", None),
584		avatar_url=ev_entry.event.content.get("avatar_url", None),
585		)
586		else:
587		missing_member_event_ids.append(event_id)
588
589		if missing_member_event_ids:
590		event_to_memberships = yield self._get_joined_profiles_from_event_ids(
591		missing_member_event_ids
592		)
593		users_in_room.update((row for row in event_to_memberships.values() if row))
594
595		if event is not None and event.type == EventTypes.Member:
596		if event.membership == Membership.JOIN:
597		if event.event_id in member_event_ids:
598		users_in_room[event.state_key] = ProfileInfo(
599		display_name=event.content.get("displayname", None),
600		avatar_url=event.content.get("avatar_url", None),
601		)
602
603		return users_in_room
604
605		@cached(max_entries=10000)
606		def _get_joined_profile_from_event_id(self, event_id):
607		raise NotImplementedError()
608
609		@cachedList(
610		cached_method_name="_get_joined_profile_from_event_id",
611		list_name="event_ids",
612		inlineCallbacks=True,
613		)
614		def _get_joined_profiles_from_event_ids(self, event_ids):
615		"""For given set of member event_ids check if they point to a join
616		event and if so return the associated user and profile info.
617
618		Args:
619		event_ids (Iterable[str]): The member event IDs to lookup
620
621		Returns:
622		Deferred[dict[str, Tuple[str, ProfileInfo]\|None]]: Map from event ID
623		to `user_id` and ProfileInfo (or None if not join event).
624		"""
625
626		rows = yield self.db.simple_select_many_batch(
627		table="room_memberships",
628		column="event_id",
629		iterable=event_ids,
630		retcols=("user_id", "display_name", "avatar_url", "event_id"),
631		keyvalues={"membership": Membership.JOIN},
632		batch_size=500,
633		desc="_get_membership_from_event_ids",
634		)
635
636		return {
637		row["event_id"]: (
638		row["user_id"],
639		ProfileInfo(
640		avatar_url=row["avatar_url"], display_name=row["display_name"]
641		),
642		)
643		for row in rows
644		}
645
646		@cachedInlineCallbacks(max_entries=10000)
647		def is_host_joined(self, room_id, host):
648		if "%" in host or "_" in host:
649		raise Exception("Invalid host name")
650
651		sql = """
652		SELECT state_key FROM current_state_events AS c
653		INNER JOIN room_memberships AS m USING (event_id)
654		WHERE m.membership = 'join'
655		AND type = 'm.room.member'
656		AND c.room_id = ?
657		AND state_key LIKE ?
658		LIMIT 1
659		"""
660
661		# We do need to be careful to ensure that host doesn't have any wild cards
662		# in it, but we checked above for known ones and we'll check below that
663		# the returned user actually has the correct domain.
664		like_clause = "%:" + host
665
666		rows = yield self.db.execute("is_host_joined", None, sql, room_id, like_clause)
667
668		if not rows:
669		return False
670
671		user_id = rows[0][0]
672		if get_domain_from_id(user_id) != host:
673		# This can only happen if the host name has something funky in it
674		raise Exception("Invalid host name")
675
676		return True
677
678		@cachedInlineCallbacks()
679		def was_host_joined(self, room_id, host):
680		"""Check whether the server is or ever was in the room.
681
682		Args:
683		room_id (str)
684		host (str)
685
686		Returns:
687		Deferred: Resolves to True if the host is/was in the room, otherwise
688		False.
689		"""
690		if "%" in host or "_" in host:
691		raise Exception("Invalid host name")
692
693		sql = """
694		SELECT user_id FROM room_memberships
695		WHERE room_id = ?
696		AND user_id LIKE ?
697		AND membership = 'join'
698		LIMIT 1
699		"""
700
701		# We do need to be careful to ensure that host doesn't have any wild cards
702		# in it, but we checked above for known ones and we'll check below that
703		# the returned user actually has the correct domain.
704		like_clause = "%:" + host
705
706		rows = yield self.db.execute("was_host_joined", None, sql, room_id, like_clause)
707
708		if not rows:
709		return False
710
711		user_id = rows[0][0]
712		if get_domain_from_id(user_id) != host:
713		# This can only happen if the host name has something funky in it
714		raise Exception("Invalid host name")
715
716		return True
717
718		@defer.inlineCallbacks
719		def get_joined_hosts(self, room_id, state_entry):
720		state_group = state_entry.state_group
721		if not state_group:
722		# If state_group is None it means it has yet to be assigned a
723		# state group, i.e. we need to make sure that calls with a state_group
724		# of None don't hit previous cached calls with a None state_group.
725		# To do this we set the state_group to a new object as object() != object()
726		state_group = object()
727
728		with Measure(self._clock, "get_joined_hosts"):
729		return (
730		yield self._get_joined_hosts(
731		room_id, state_group, state_entry.state, state_entry=state_entry
732		)
733		)
734
735		@cachedInlineCallbacks(num_args=2, max_entries=10000, iterable=True)
736		# @defer.inlineCallbacks
737		def _get_joined_hosts(self, room_id, state_group, current_state_ids, state_entry):
738		# We don't use `state_group`, its there so that we can cache based
739		# on it. However, its important that its never None, since two current_state's
740		# with a state_group of None are likely to be different.
741		# See bulk_get_push_rules_for_room for how we work around this.
742		assert state_group is not None
743
744		cache = yield self._get_joined_hosts_cache(room_id)
745		joined_hosts = yield cache.get_destinations(state_entry)
746
747		return joined_hosts
748
749		@cached(max_entries=10000)
750		def _get_joined_hosts_cache(self, room_id):
751		return _JoinedHostsCache(self, room_id)
752
753		@cachedInlineCallbacks(num_args=2)
754		def did_forget(self, user_id, room_id):
755		"""Returns whether user_id has elected to discard history for room_id.
756
757		Returns False if they have since re-joined."""
758
759		def f(txn):
760		sql = (
761		"SELECT"
762		" COUNT(*)"
763		" FROM"
764		" room_memberships"
765		" WHERE"
766		" user_id = ?"
767		" AND"
768		" room_id = ?"
769		" AND"
770		" forgotten = 0"
771		)
772		txn.execute(sql, (user_id, room_id))
773		rows = txn.fetchall()
774		return rows[0][0]
775
776		count = yield self.db.runInteraction("did_forget_membership", f)
777		return count == 0
778
779		@cached()
780		def get_forgotten_rooms_for_user(self, user_id):
781		"""Gets all rooms the user has forgotten.
782
783		Args:
784		user_id (str)
785
786		Returns:
787		Deferred[set[str]]
788		"""
789
790		def _get_forgotten_rooms_for_user_txn(txn):
791		# This is a slightly convoluted query that first looks up all rooms
792		# that the user has forgotten in the past, then rechecks that list
793		# to see if any have subsequently been updated. This is done so that
794		# we can use a partial index on `forgotten = 1` on the assumption
795		# that few users will actually forget many rooms.
796		#
797		# Note that a room is considered "forgotten" if all membership
798		# events for that user and room have the forgotten field set (as
799		# when a user forgets a room we update all rows for that user and
800		# room, not just the current one).
801		sql = """
802		SELECT room_id, (
803		SELECT count(*) FROM room_memberships
804		WHERE room_id = m.room_id AND user_id = m.user_id AND forgotten = 0
805		) AS count
806		FROM room_memberships AS m
807		WHERE user_id = ? AND forgotten = 1
808		GROUP BY room_id, user_id;
809		"""
810		txn.execute(sql, (user_id,))
811		return {row[0] for row in txn if row[1] == 0}
812
813		return self.db.runInteraction(
814		"get_forgotten_rooms_for_user", _get_forgotten_rooms_for_user_txn
815		)
816
817		@defer.inlineCallbacks
818		def get_rooms_user_has_been_in(self, user_id):
819		"""Get all rooms that the user has ever been in.
820
821		Args:
822		user_id (str)
823
824		Returns:
825		Deferred[set[str]]: Set of room IDs.
826		"""
827
828		room_ids = yield self.db.simple_select_onecol(
829		table="room_memberships",
830		keyvalues={"membership": Membership.JOIN, "user_id": user_id},
831		retcol="room_id",
832		desc="get_rooms_user_has_been_in",
833		)
834
835		return set(room_ids)
836
837		def get_membership_from_event_ids(
838		self, member_event_ids: Iterable[str]
839		) -> List[dict]:
840		"""Get user_id and membership of a set of event IDs.
841		"""
842
843		return self.db.simple_select_many_batch(
844		table="room_memberships",
845		column="event_id",
846		iterable=member_event_ids,
847		retcols=("user_id", "membership", "event_id"),
848		keyvalues={},
849		batch_size=500,
850		desc="get_membership_from_event_ids",
851		)
852
853		async def is_local_host_in_room_ignoring_users(
854		self, room_id: str, ignore_users: Collection[str]
855		) -> bool:
856		"""Check if there are any local users, excluding those in the given
857		list, in the room.
858		"""
859
860		clause, args = make_in_list_sql_clause(
861		self.database_engine, "user_id", ignore_users
862		)
863
864		sql = """
865		SELECT 1 FROM local_current_membership
866		WHERE
867		room_id = ? AND membership = ?
868		AND NOT (%s)
869		LIMIT 1
870		""" % (
871		clause,
872		)
873
874		def _is_local_host_in_room_ignoring_users_txn(txn):
875		txn.execute(sql, (room_id, Membership.JOIN, *args))
876
877		return bool(txn.fetchone())
878
879		return await self.db.runInteraction(
880		"is_local_host_in_room_ignoring_users",
881		_is_local_host_in_room_ignoring_users_txn,
882		)
883
884
885		class RoomMemberBackgroundUpdateStore(SQLBaseStore):
886		def __init__(self, database: Database, db_conn, hs):
887		super(RoomMemberBackgroundUpdateStore, self).__init__(database, db_conn, hs)
888		self.db.updates.register_background_update_handler(
889		_MEMBERSHIP_PROFILE_UPDATE_NAME, self._background_add_membership_profile
890		)
891		self.db.updates.register_background_update_handler(
892		_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME,
893		self._background_current_state_membership,
894		)
895		self.db.updates.register_background_index_update(
896		"room_membership_forgotten_idx",
897		index_name="room_memberships_user_room_forgotten",
898		table="room_memberships",
899		columns=["user_id", "room_id"],
900		where_clause="forgotten = 1",
901		)
902
903		@defer.inlineCallbacks
904		def _background_add_membership_profile(self, progress, batch_size):
905		target_min_stream_id = progress.get(
906		"target_min_stream_id_inclusive", self._min_stream_order_on_start
907		)
908		max_stream_id = progress.get(
909		"max_stream_id_exclusive", self._stream_order_on_start + 1
910		)
911
912		INSERT_CLUMP_SIZE = 1000
913
914		def add_membership_profile_txn(txn):
915		sql = """
916		SELECT stream_ordering, event_id, events.room_id, event_json.json
917		FROM events
918		INNER JOIN event_json USING (event_id)
919		INNER JOIN room_memberships USING (event_id)
920		WHERE ? <= stream_ordering AND stream_ordering < ?
921		AND type = 'm.room.member'
922		ORDER BY stream_ordering DESC
923		LIMIT ?
924		"""
925
926		txn.execute(sql, (target_min_stream_id, max_stream_id, batch_size))
927
928		rows = self.db.cursor_to_dict(txn)
929		if not rows:
930		return 0
931
932		min_stream_id = rows[-1]["stream_ordering"]
933
934		to_update = []
935		for row in rows:
936		event_id = row["event_id"]
937		room_id = row["room_id"]
938		try:
939		event_json = db_to_json(row["json"])
940		content = event_json["content"]
941		except Exception:
942		continue
943
944		display_name = content.get("displayname", None)
945		avatar_url = content.get("avatar_url", None)
946
947		if display_name or avatar_url:
948		to_update.append((display_name, avatar_url, event_id, room_id))
949
950		to_update_sql = """
951		UPDATE room_memberships SET display_name = ?, avatar_url = ?
952		WHERE event_id = ? AND room_id = ?
953		"""
954		for index in range(0, len(to_update), INSERT_CLUMP_SIZE):
955		clump = to_update[index : index + INSERT_CLUMP_SIZE]
956		txn.executemany(to_update_sql, clump)
957
958		progress = {
959		"target_min_stream_id_inclusive": target_min_stream_id,
960		"max_stream_id_exclusive": min_stream_id,
961		}
962
963		self.db.updates._background_update_progress_txn(
964		txn, _MEMBERSHIP_PROFILE_UPDATE_NAME, progress
965		)
966
967		return len(rows)
968
969		result = yield self.db.runInteraction(
970		_MEMBERSHIP_PROFILE_UPDATE_NAME, add_membership_profile_txn
971		)
972
973		if not result:
974		yield self.db.updates._end_background_update(
975		_MEMBERSHIP_PROFILE_UPDATE_NAME
976		)
977
978		return result
979
980		@defer.inlineCallbacks
981		def _background_current_state_membership(self, progress, batch_size):
982		"""Update the new membership column on current_state_events.
983
984		This works by iterating over all rooms in alphebetical order.
985		"""
986
987		def _background_current_state_membership_txn(txn, last_processed_room):
988		processed = 0
989		while processed < batch_size:
990		txn.execute(
991		"""
992		SELECT MIN(room_id) FROM current_state_events WHERE room_id > ?
993		""",
994		(last_processed_room,),
995		)
996		row = txn.fetchone()
997		if not row or not row[0]:
998		return processed, True
999
1000		(next_room,) = row
1001
1002		sql = """
1003		UPDATE current_state_events
1004		SET membership = (
1005		SELECT membership FROM room_memberships
1006		WHERE event_id = current_state_events.event_id
1007		)
1008		WHERE room_id = ?
1009		"""
1010		txn.execute(sql, (next_room,))
1011		processed += txn.rowcount
1012
1013		last_processed_room = next_room
1014
1015		self.db.updates._background_update_progress_txn(
1016		txn,
1017		_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME,
1018		{"last_processed_room": last_processed_room},
1019		)
1020
1021		return processed, False
1022
1023		# If we haven't got a last processed room then just use the empty
1024		# string, which will compare before all room IDs correctly.
1025		last_processed_room = progress.get("last_processed_room", "")
1026
1027		row_count, finished = yield self.db.runInteraction(
1028		"_background_current_state_membership_update",
1029		_background_current_state_membership_txn,
1030		last_processed_room,
1031		)
1032
1033		if finished:
1034		yield self.db.updates._end_background_update(
1035		_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME
1036		)
1037
1038		return row_count
1039
1040
1041		class RoomMemberStore(RoomMemberWorkerStore, RoomMemberBackgroundUpdateStore):
1042		def __init__(self, database: Database, db_conn, hs):
1043		super(RoomMemberStore, self).__init__(database, db_conn, hs)
1044
1045		def forget(self, user_id, room_id):
1046		"""Indicate that user_id wishes to discard history for room_id."""
1047
1048		def f(txn):
1049		sql = (
1050		"UPDATE"
1051		" room_memberships"
1052		" SET"
1053		" forgotten = 1"
1054		" WHERE"
1055		" user_id = ?"
1056		" AND"
1057		" room_id = ?"
1058		)
1059		txn.execute(sql, (user_id, room_id))
1060
1061		self._invalidate_cache_and_stream(txn, self.did_forget, (user_id, room_id))
1062		self._invalidate_cache_and_stream(
1063		txn, self.get_forgotten_rooms_for_user, (user_id,)
1064		)
1065
1066		return self.db.runInteraction("forget_membership", f)
1067
1068
1069		class _JoinedHostsCache(object):
1070		"""Cache for joined hosts in a room that is optimised to handle updates
1071		via state deltas.
1072		"""
1073
1074		def __init__(self, store, room_id):
1075		self.store = store
1076		self.room_id = room_id
1077
1078		self.hosts_to_joined_users = {}
1079
1080		self.state_group = object()
1081
1082		self.linearizer = Linearizer("_JoinedHostsCache")
1083
1084		self._len = 0
1085
1086		@defer.inlineCallbacks
1087		def get_destinations(self, state_entry):
1088		"""Get set of destinations for a state entry
1089
1090		Args:
1091		state_entry(synapse.state._StateCacheEntry)
1092		"""
1093		if state_entry.state_group == self.state_group:
1094		return frozenset(self.hosts_to_joined_users)
1095
1096		with (yield self.linearizer.queue(())):
1097		if state_entry.state_group == self.state_group:
1098		pass
1099		elif state_entry.prev_group == self.state_group:
1100		for (typ, state_key), event_id in state_entry.delta_ids.items():
1101		if typ != EventTypes.Member:
1102		continue
1103
1104		host = intern_string(get_domain_from_id(state_key))
1105		user_id = state_key
1106		known_joins = self.hosts_to_joined_users.setdefault(host, set())
1107
1108		event = yield self.store.get_event(event_id)
1109		if event.membership == Membership.JOIN:
1110		known_joins.add(user_id)
1111		else:
1112		known_joins.discard(user_id)
1113
1114		if not known_joins:
1115		self.hosts_to_joined_users.pop(host, None)
1116		else:
1117		joined_users = yield self.store.get_joined_users_from_state(
1118		self.room_id, state_entry
1119		)
1120
1121		self.hosts_to_joined_users = {}
1122		for user_id in joined_users:
1123		host = intern_string(get_domain_from_id(user_id))
1124		self.hosts_to_joined_users.setdefault(host, set()).add(user_id)
1125
1126		if state_entry.state_group:
1127		self.state_group = state_entry.state_group
1128		else:
1129		self.state_group = object()
1130		self._len = sum(len(v) for v in self.hosts_to_joined_users.values())
1131		return frozenset(self.hosts_to_joined_users)
1132
1133		def __len__(self):
1134		return self._len

+0

-63

~~synapse/storage/data_stores/main/schema/delta/12/v12.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS rejections(
16		event_id TEXT NOT NULL,
17		reason TEXT NOT NULL,
18		last_check TEXT NOT NULL,
19		UNIQUE (event_id)
20		);
21
22		-- Push notification endpoints that users have configured
23		CREATE TABLE IF NOT EXISTS pushers (
24		id INTEGER PRIMARY KEY AUTOINCREMENT,
25		user_name TEXT NOT NULL,
26		profile_tag VARCHAR(32) NOT NULL,
27		kind VARCHAR(8) NOT NULL,
28		app_id VARCHAR(64) NOT NULL,
29		app_display_name VARCHAR(64) NOT NULL,
30		device_display_name VARCHAR(128) NOT NULL,
31		pushkey VARBINARY(512) NOT NULL,
32		ts BIGINT UNSIGNED NOT NULL,
33		lang VARCHAR(8),
34		data LONGBLOB,
35		last_token TEXT,
36		last_success BIGINT UNSIGNED,
37		failing_since BIGINT UNSIGNED,
38		UNIQUE (app_id, pushkey)
39		);
40
41		CREATE TABLE IF NOT EXISTS push_rules (
42		id INTEGER PRIMARY KEY AUTOINCREMENT,
43		user_name TEXT NOT NULL,
44		rule_id TEXT NOT NULL,
45		priority_class TINYINT NOT NULL,
46		priority INTEGER NOT NULL DEFAULT 0,
47		conditions TEXT NOT NULL,
48		actions TEXT NOT NULL,
49		UNIQUE(user_name, rule_id)
50		);
51
52		CREATE INDEX IF NOT EXISTS push_rules_user_name on push_rules (user_name);
53
54		CREATE TABLE IF NOT EXISTS user_filters(
55		user_id TEXT,
56		filter_id BIGINT UNSIGNED,
57		filter_json LONGBLOB
58		);
59
60		CREATE INDEX IF NOT EXISTS user_filters_by_user_id_filter_id ON user_filters(
61		user_id, filter_id
62		);

+0

-19

~~synapse/storage/data_stores/main/schema/delta/13/v13.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* We used to create a tables called application_services and
16		* application_services_regex, but these are no longer used and are removed in
17		* delta 54.
18		*/

+0

-23

~~synapse/storage/data_stores/main/schema/delta/14/v14.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14		CREATE TABLE IF NOT EXISTS push_rules_enable (
15		id INTEGER PRIMARY KEY AUTOINCREMENT,
16		user_name TEXT NOT NULL,
17		rule_id TEXT NOT NULL,
18		enabled TINYINT,
19		UNIQUE(user_name, rule_id)
20		);
21
22		CREATE INDEX IF NOT EXISTS push_rules_enable_user_name on push_rules_enable (user_name);

+0

-31

~~synapse/storage/data_stores/main/schema/delta/15/appservice_txns.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS application_services_state(
16		as_id TEXT PRIMARY KEY,
17		state VARCHAR(5),
18		last_txn INTEGER
19		);
20
21		CREATE TABLE IF NOT EXISTS application_services_txns(
22		as_id TEXT NOT NULL,
23		txn_id INTEGER NOT NULL,
24		event_ids TEXT NOT NULL,
25		UNIQUE(as_id, txn_id)
26		);
27
28		CREATE INDEX IF NOT EXISTS application_services_txns_id ON application_services_txns (
29		as_id
30		);

+0

-2

~~synapse/storage/data_stores/main/schema/delta/15/presence_indices.sql~~ less more

0
1		CREATE INDEX IF NOT EXISTS presence_list_user_id ON presence_list (user_id);

+0

-24

~~synapse/storage/data_stores/main/schema/delta/15/v15.sql~~ less more

0		-- Drop, copy & recreate pushers table to change unique key
1		-- Also add access_token column at the same time
2		CREATE TABLE IF NOT EXISTS pushers2 (
3		id BIGINT PRIMARY KEY,
4		user_name TEXT NOT NULL,
5		access_token BIGINT DEFAULT NULL,
6		profile_tag VARCHAR(32) NOT NULL,
7		kind VARCHAR(8) NOT NULL,
8		app_id VARCHAR(64) NOT NULL,
9		app_display_name VARCHAR(64) NOT NULL,
10		device_display_name VARCHAR(128) NOT NULL,
11		pushkey bytea NOT NULL,
12		ts BIGINT NOT NULL,
13		lang VARCHAR(8),
14		data bytea,
15		last_token TEXT,
16		last_success BIGINT,
17		failing_since BIGINT,
18		UNIQUE (app_id, pushkey)
19		);
20		INSERT INTO pushers2 (id, user_name, profile_tag, kind, app_id, app_display_name, device_display_name, pushkey, ts, lang, data, last_token, last_success, failing_since)
21		SELECT id, user_name, profile_tag, kind, app_id, app_display_name, device_display_name, pushkey, ts, lang, data, last_token, last_success, failing_since FROM pushers;
22		DROP TABLE pushers;
23		ALTER TABLE pushers2 RENAME TO pushers;

+0

-4

~~synapse/storage/data_stores/main/schema/delta/16/events_order_index.sql~~ less more

0		CREATE INDEX events_order ON events (topological_ordering, stream_ordering);
1		CREATE INDEX events_order_room ON events (
2		room_id, topological_ordering, stream_ordering
3		);

+0

-2

~~synapse/storage/data_stores/main/schema/delta/16/remote_media_cache_index.sql~~ less more

0		CREATE INDEX IF NOT EXISTS remote_media_cache_thumbnails_media_id
1		ON remote_media_cache_thumbnails (media_id);⏎

+0

-9

~~synapse/storage/data_stores/main/schema/delta/16/remove_duplicates.sql~~ less more

0
1
2		DELETE FROM event_to_state_groups WHERE state_group not in (
3		SELECT MAX(state_group) FROM event_to_state_groups GROUP BY event_id
4		);
5
6		DELETE FROM event_to_state_groups WHERE rowid not in (
7		SELECT MIN(rowid) FROM event_to_state_groups GROUP BY event_id
8		);

+0

-3

~~synapse/storage/data_stores/main/schema/delta/16/room_alias_index.sql~~ less more

0
1		CREATE INDEX IF NOT EXISTS room_aliases_id ON room_aliases(room_id);
2		CREATE INDEX IF NOT EXISTS room_alias_servers_alias ON room_alias_servers(room_alias);

+0

-72

~~synapse/storage/data_stores/main/schema/delta/16/unique_constraints.sql~~ less more

0
1		-- We can use SQLite features here, since other db support was only added in v16
2
3		--
4		DELETE FROM current_state_events WHERE rowid not in (
5		SELECT MIN(rowid) FROM current_state_events GROUP BY event_id
6		);
7
8		DROP INDEX IF EXISTS current_state_events_event_id;
9		CREATE UNIQUE INDEX current_state_events_event_id ON current_state_events(event_id);
10
11		--
12		DELETE FROM room_memberships WHERE rowid not in (
13		SELECT MIN(rowid) FROM room_memberships GROUP BY event_id
14		);
15
16		DROP INDEX IF EXISTS room_memberships_event_id;
17		CREATE UNIQUE INDEX room_memberships_event_id ON room_memberships(event_id);
18
19		--
20		DELETE FROM topics WHERE rowid not in (
21		SELECT MIN(rowid) FROM topics GROUP BY event_id
22		);
23
24		DROP INDEX IF EXISTS topics_event_id;
25		CREATE UNIQUE INDEX topics_event_id ON topics(event_id);
26
27		--
28		DELETE FROM room_names WHERE rowid not in (
29		SELECT MIN(rowid) FROM room_names GROUP BY event_id
30		);
31
32		DROP INDEX IF EXISTS room_names_id;
33		CREATE UNIQUE INDEX room_names_id ON room_names(event_id);
34
35		--
36		DELETE FROM presence WHERE rowid not in (
37		SELECT MIN(rowid) FROM presence GROUP BY user_id
38		);
39
40		DROP INDEX IF EXISTS presence_id;
41		CREATE UNIQUE INDEX presence_id ON presence(user_id);
42
43		--
44		DELETE FROM presence_allow_inbound WHERE rowid not in (
45		SELECT MIN(rowid) FROM presence_allow_inbound
46		GROUP BY observed_user_id, observer_user_id
47		);
48
49		DROP INDEX IF EXISTS presence_allow_inbound_observers;
50		CREATE UNIQUE INDEX presence_allow_inbound_observers ON presence_allow_inbound(
51		observed_user_id, observer_user_id
52		);
53
54		--
55		DELETE FROM presence_list WHERE rowid not in (
56		SELECT MIN(rowid) FROM presence_list
57		GROUP BY user_id, observed_user_id
58		);
59
60		DROP INDEX IF EXISTS presence_list_observers;
61		CREATE UNIQUE INDEX presence_list_observers ON presence_list(
62		user_id, observed_user_id
63		);
64
65		--
66		DELETE FROM room_aliases WHERE rowid not in (
67		SELECT MIN(rowid) FROM room_aliases GROUP BY room_alias
68		);
69
70		DROP INDEX IF EXISTS room_aliases_id;
71		CREATE INDEX room_aliases_id ON room_aliases(room_id);

+0

-56

~~synapse/storage/data_stores/main/schema/delta/16/users.sql~~ less more

0		-- Convert `access_tokens`.user from rowids to user strings.
1		-- MUST BE DONE BEFORE REMOVING ID COLUMN FROM USERS TABLE BELOW
2		CREATE TABLE IF NOT EXISTS new_access_tokens(
3		id BIGINT UNSIGNED PRIMARY KEY,
4		user_id TEXT NOT NULL,
5		device_id TEXT,
6		token TEXT NOT NULL,
7		last_used BIGINT UNSIGNED,
8		UNIQUE(token)
9		);
10
11		INSERT INTO new_access_tokens
12		SELECT a.id, u.name, a.device_id, a.token, a.last_used
13		FROM access_tokens as a
14		INNER JOIN users as u ON u.id = a.user_id;
15
16		DROP TABLE access_tokens;
17
18		ALTER TABLE new_access_tokens RENAME TO access_tokens;
19
20		-- Remove ID column from `users` table
21		CREATE TABLE IF NOT EXISTS new_users(
22		name TEXT,
23		password_hash TEXT,
24		creation_ts BIGINT UNSIGNED,
25		admin BOOL DEFAULT 0 NOT NULL,
26		UNIQUE(name)
27		);
28
29		INSERT INTO new_users SELECT name, password_hash, creation_ts, admin FROM users;
30
31		DROP TABLE users;
32
33		ALTER TABLE new_users RENAME TO users;
34
35
36		-- Remove UNIQUE constraint from `user_ips` table
37		CREATE TABLE IF NOT EXISTS new_user_ips (
38		user_id TEXT NOT NULL,
39		access_token TEXT NOT NULL,
40		device_id TEXT,
41		ip TEXT NOT NULL,
42		user_agent TEXT NOT NULL,
43		last_seen BIGINT UNSIGNED NOT NULL
44		);
45
46		INSERT INTO new_user_ips
47		SELECT user, access_token, device_id, ip, user_agent, last_seen FROM user_ips;
48
49		DROP TABLE user_ips;
50
51		ALTER TABLE new_user_ips RENAME TO user_ips;
52
53		CREATE INDEX IF NOT EXISTS user_ips_user ON user_ips(user_id);
54		CREATE INDEX IF NOT EXISTS user_ips_user_ip ON user_ips(user_id, access_token, ip);
55

+0

-18

~~synapse/storage/data_stores/main/schema/delta/17/drop_indexes.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		DROP INDEX IF EXISTS sent_transaction_dest;
16		DROP INDEX IF EXISTS sent_transaction_sent;
17		DROP INDEX IF EXISTS user_ips_user;

+0

-24

~~synapse/storage/data_stores/main/schema/delta/17/server_keys.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS server_keys_json (
16		server_name TEXT, -- Server name.
17		key_id TEXT, -- Requested key id.
18		from_server TEXT, -- Which server the keys were fetched from.
19		ts_added_ms INTEGER, -- When the keys were fetched
20		ts_valid_until_ms INTEGER, -- When this version of the keys exipires.
21		key_json bytea, -- JSON certificate for the remote server.
22		CONSTRAINT uniqueness UNIQUE (server_name, key_id, from_server)
23		);

+0

-9

~~synapse/storage/data_stores/main/schema/delta/17/user_threepids.sql~~ less more

0		CREATE TABLE user_threepids (
1		user_id TEXT NOT NULL,
2		medium TEXT NOT NULL,
3		address TEXT NOT NULL,
4		validated_at BIGINT NOT NULL,
5		added_at BIGINT NOT NULL,
6		CONSTRAINT user_medium_address UNIQUE (user_id, medium, address)
7		);
8		CREATE INDEX user_threepids_user_id ON user_threepids(user_id);

+0

-32

~~synapse/storage/data_stores/main/schema/delta/18/server_keys_bigger_ints.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE IF NOT EXISTS new_server_keys_json (
17		server_name TEXT NOT NULL, -- Server name.
18		key_id TEXT NOT NULL, -- Requested key id.
19		from_server TEXT NOT NULL, -- Which server the keys were fetched from.
20		ts_added_ms BIGINT NOT NULL, -- When the keys were fetched
21		ts_valid_until_ms BIGINT NOT NULL, -- When this version of the keys exipires.
22		key_json bytea NOT NULL, -- JSON certificate for the remote server.
23		CONSTRAINT server_keys_json_uniqueness UNIQUE (server_name, key_id, from_server)
24		);
25
26		INSERT INTO new_server_keys_json
27		SELECT server_name, key_id, from_server,ts_added_ms, ts_valid_until_ms, key_json FROM server_keys_json ;
28
29		DROP TABLE server_keys_json;
30
31		ALTER TABLE new_server_keys_json RENAME TO server_keys_json;

+0

-19

~~synapse/storage/data_stores/main/schema/delta/19/event_index.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE INDEX events_order_topo_stream_room ON events(
17		topological_ordering, stream_ordering, room_id
18		);

+0

-1

~~synapse/storage/data_stores/main/schema/delta/20/dummy.sql~~ less more

0

SELECT 1;

+0

-88

~~synapse/storage/data_stores/main/schema/delta/20/pushers.py~~ less more

0		# Copyright 2015, 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14
15		"""
16		Main purpose of this upgrade is to change the unique key on the
17		pushers table again (it was missed when the v16 full schema was
18		made) but this also changes the pushkey and data columns to text.
19		When selecting a bytea column into a text column, postgres inserts
20		the hex encoded data, and there's no portable way of getting the
21		UTF-8 bytes, so we have to do it in Python.
22		"""
23
24		import logging
25
26		logger = logging.getLogger(__name__)
27
28
29		def run_create(cur, database_engine, args, *kwargs):
30		logger.info("Porting pushers table...")
31		cur.execute(
32		"""
33		CREATE TABLE IF NOT EXISTS pushers2 (
34		id BIGINT PRIMARY KEY,
35		user_name TEXT NOT NULL,
36		access_token BIGINT DEFAULT NULL,
37		profile_tag VARCHAR(32) NOT NULL,
38		kind VARCHAR(8) NOT NULL,
39		app_id VARCHAR(64) NOT NULL,
40		app_display_name VARCHAR(64) NOT NULL,
41		device_display_name VARCHAR(128) NOT NULL,
42		pushkey TEXT NOT NULL,
43		ts BIGINT NOT NULL,
44		lang VARCHAR(8),
45		data TEXT,
46		last_token TEXT,
47		last_success BIGINT,
48		failing_since BIGINT,
49		UNIQUE (app_id, pushkey, user_name)
50		)
51		"""
52		)
53		cur.execute(
54		"""SELECT
55		id, user_name, access_token, profile_tag, kind,
56		app_id, app_display_name, device_display_name,
57		pushkey, ts, lang, data, last_token, last_success,
58		failing_since
59		FROM pushers
60		"""
61		)
62		count = 0
63		for row in cur.fetchall():
64		row = list(row)
65		row[8] = bytes(row[8]).decode("utf-8")
66		row[11] = bytes(row[11]).decode("utf-8")
67		cur.execute(
68		database_engine.convert_param_style(
69		"""
70		INSERT into pushers2 (
71		id, user_name, access_token, profile_tag, kind,
72		app_id, app_display_name, device_display_name,
73		pushkey, ts, lang, data, last_token, last_success,
74		failing_since
75		) values (%s)"""
76		% (",".join(["?" for _ in range(len(row))]))
77		),
78		row,
79		)
80		count += 1
81		cur.execute("DROP TABLE pushers")
82		cur.execute("ALTER TABLE pushers2 RENAME TO pushers")
83		logger.info("Moved %d pushers to new table", count)
84
85
86		def run_upgrade(args, *kwargs):
87		pass

+0

-34

~~synapse/storage/data_stores/main/schema/delta/21/end_to_end_keys.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE IF NOT EXISTS e2e_device_keys_json (
17		user_id TEXT NOT NULL, -- The user these keys are for.
18		device_id TEXT NOT NULL, -- Which of the user's devices these keys are for.
19		ts_added_ms BIGINT NOT NULL, -- When the keys were uploaded.
20		key_json TEXT NOT NULL, -- The keys for the device as a JSON blob.
21		CONSTRAINT e2e_device_keys_json_uniqueness UNIQUE (user_id, device_id)
22		);
23
24
25		CREATE TABLE IF NOT EXISTS e2e_one_time_keys_json (
26		user_id TEXT NOT NULL, -- The user this one-time key is for.
27		device_id TEXT NOT NULL, -- The device this one-time key is for.
28		algorithm TEXT NOT NULL, -- Which algorithm this one-time key is for.
29		key_id TEXT NOT NULL, -- An id for suppressing duplicate uploads.
30		ts_added_ms BIGINT NOT NULL, -- When this key was uploaded.
31		key_json TEXT NOT NULL, -- The key as a JSON blob.
32		CONSTRAINT e2e_one_time_keys_json_uniqueness UNIQUE (user_id, device_id, algorithm, key_id)
33		);

+0

-38

~~synapse/storage/data_stores/main/schema/delta/21/receipts.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE IF NOT EXISTS receipts_graph(
17		room_id TEXT NOT NULL,
18		receipt_type TEXT NOT NULL,
19		user_id TEXT NOT NULL,
20		event_ids TEXT NOT NULL,
21		data TEXT NOT NULL,
22		CONSTRAINT receipts_graph_uniqueness UNIQUE (room_id, receipt_type, user_id)
23		);
24
25		CREATE TABLE IF NOT EXISTS receipts_linearized (
26		stream_id BIGINT NOT NULL,
27		room_id TEXT NOT NULL,
28		receipt_type TEXT NOT NULL,
29		user_id TEXT NOT NULL,
30		event_id TEXT NOT NULL,
31		data TEXT NOT NULL,
32		CONSTRAINT receipts_linearized_uniqueness UNIQUE (room_id, receipt_type, user_id)
33		);
34
35		CREATE INDEX receipts_linearized_id ON receipts_linearized(
36		stream_id
37		);

+0

-22

~~synapse/storage/data_stores/main/schema/delta/22/receipts_index.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/** Using CREATE INDEX directly is deprecated in favour of using background
16		* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
17		* and synapse/storage/registration.py for an example using
18		* "access_tokens_device_index" **/
19		CREATE INDEX receipts_linearized_room_stream ON receipts_linearized(
20		room_id, stream_id
21		);

+0

-19

~~synapse/storage/data_stores/main/schema/delta/22/user_threepids_unique.sql~~ less more

0		CREATE TABLE IF NOT EXISTS user_threepids2 (
1		user_id TEXT NOT NULL,
2		medium TEXT NOT NULL,
3		address TEXT NOT NULL,
4		validated_at BIGINT NOT NULL,
5		added_at BIGINT NOT NULL,
6		CONSTRAINT medium_address UNIQUE (medium, address)
7		);
8
9		INSERT INTO user_threepids2
10		SELECT * FROM user_threepids WHERE added_at IN (
11		SELECT max(added_at) FROM user_threepids GROUP BY medium, address
12		)
13		;
14
15		DROP TABLE user_threepids;
16		ALTER TABLE user_threepids2 RENAME TO user_threepids;
17
18		CREATE INDEX user_threepids_user_id ON user_threepids(user_id);

+0

-18

~~synapse/storage/data_stores/main/schema/delta/24/stats_reporting.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* We used to create a table called stats_reporting, but this is no longer
16		* used and is removed in delta 54.
17		*/⏎

+0

-80

~~synapse/storage/data_stores/main/schema/delta/25/fts.py~~ less more

0		# Copyright 2015, 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13		import json
14		import logging
15
16		from synapse.storage.engines import PostgresEngine, Sqlite3Engine
17		from synapse.storage.prepare_database import get_statements
18
19		logger = logging.getLogger(__name__)
20
21
22		POSTGRES_TABLE = """
23		CREATE TABLE IF NOT EXISTS event_search (
24		event_id TEXT,
25		room_id TEXT,
26		sender TEXT,
27		key TEXT,
28		vector tsvector
29		);
30
31		CREATE INDEX event_search_fts_idx ON event_search USING gin(vector);
32		CREATE INDEX event_search_ev_idx ON event_search(event_id);
33		CREATE INDEX event_search_ev_ridx ON event_search(room_id);
34		"""
35
36
37		SQLITE_TABLE = (
38		"CREATE VIRTUAL TABLE event_search"
39		" USING fts4 ( event_id, room_id, sender, key, value )"
40		)
41
42
43		def run_create(cur, database_engine, args, *kwargs):
44		if isinstance(database_engine, PostgresEngine):
45		for statement in get_statements(POSTGRES_TABLE.splitlines()):
46		cur.execute(statement)
47		elif isinstance(database_engine, Sqlite3Engine):
48		cur.execute(SQLITE_TABLE)
49		else:
50		raise Exception("Unrecognized database engine")
51
52		cur.execute("SELECT MIN(stream_ordering) FROM events")
53		rows = cur.fetchall()
54		min_stream_id = rows[0][0]
55
56		cur.execute("SELECT MAX(stream_ordering) FROM events")
57		rows = cur.fetchall()
58		max_stream_id = rows[0][0]
59
60		if min_stream_id is not None and max_stream_id is not None:
61		progress = {
62		"target_min_stream_id_inclusive": min_stream_id,
63		"max_stream_id_exclusive": max_stream_id + 1,
64		"rows_inserted": 0,
65		}
66		progress_json = json.dumps(progress)
67
68		sql = (
69		"INSERT into background_updates (update_name, progress_json)"
70		" VALUES (?, ?)"
71		)
72
73		sql = database_engine.convert_param_style(sql)
74
75		cur.execute(sql, ("event_search", progress_json))
76
77
78		def run_upgrade(args, *kwargs):
79		pass

+0

-25

~~synapse/storage/data_stores/main/schema/delta/25/guest_access.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/*
16		* This is a manual index of guest_access content of state events,
17		* so that we can join on them in SELECT statements.
18		*/
19		CREATE TABLE IF NOT EXISTS guest_access(
20		event_id TEXT NOT NULL,
21		room_id TEXT NOT NULL,
22		guest_access TEXT NOT NULL,
23		UNIQUE (event_id)
24		);

+0

-25

~~synapse/storage/data_stores/main/schema/delta/25/history_visibility.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/*
16		* This is a manual index of history_visibility content of state events,
17		* so that we can join on them in SELECT statements.
18		*/
19		CREATE TABLE IF NOT EXISTS history_visibility(
20		event_id TEXT NOT NULL,
21		room_id TEXT NOT NULL,
22		history_visibility TEXT NOT NULL,
23		UNIQUE (event_id)
24		);

+0

-38

~~synapse/storage/data_stores/main/schema/delta/25/tags.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE IF NOT EXISTS room_tags(
17		user_id TEXT NOT NULL,
18		room_id TEXT NOT NULL,
19		tag TEXT NOT NULL, -- The name of the tag.
20		content TEXT NOT NULL, -- The JSON content of the tag.
21		CONSTRAINT room_tag_uniqueness UNIQUE (user_id, room_id, tag)
22		);
23
24		CREATE TABLE IF NOT EXISTS room_tags_revisions (
25		user_id TEXT NOT NULL,
26		room_id TEXT NOT NULL,
27		stream_id BIGINT NOT NULL, -- The current version of the room tags.
28		CONSTRAINT room_tag_revisions_uniqueness UNIQUE (user_id, room_id)
29		);
30
31		CREATE TABLE IF NOT EXISTS private_user_data_max_stream_id(
32		Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
33		stream_id BIGINT NOT NULL,
34		CHECK (Lock='X')
35		);
36
37		INSERT INTO private_user_data_max_stream_id (stream_id) VALUES (0);

+0

-17

~~synapse/storage/data_stores/main/schema/delta/26/account_data.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		ALTER TABLE private_user_data_max_stream_id RENAME TO account_data_max_stream_id;

+0

-36

~~synapse/storage/data_stores/main/schema/delta/27/account_data.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS account_data(
16		user_id TEXT NOT NULL,
17		account_data_type TEXT NOT NULL, -- The type of the account_data.
18		stream_id BIGINT NOT NULL, -- The version of the account_data.
19		content TEXT NOT NULL, -- The JSON content of the account_data
20		CONSTRAINT account_data_uniqueness UNIQUE (user_id, account_data_type)
21		);
22
23
24		CREATE TABLE IF NOT EXISTS room_account_data(
25		user_id TEXT NOT NULL,
26		room_id TEXT NOT NULL,
27		account_data_type TEXT NOT NULL, -- The type of the account_data.
28		stream_id BIGINT NOT NULL, -- The version of the account_data.
29		content TEXT NOT NULL, -- The JSON content of the account_data
30		CONSTRAINT room_account_data_uniqueness UNIQUE (user_id, room_id, account_data_type)
31		);
32
33
34		CREATE INDEX account_data_stream_id on account_data(user_id, stream_id);
35		CREATE INDEX room_account_data_stream_id on room_account_data(user_id, stream_id);

+0

-26

~~synapse/storage/data_stores/main/schema/delta/27/forgotten_memberships.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/*
16		* Keeps track of what rooms users have left and don't want to be able to
17		* access again.
18		*
19		* If all users on this server have left a room, we can delete the room
20		* entirely.
21		*
22		* This column should always contain either 0 or 1.
23		*/
24
25		ALTER TABLE room_memberships ADD COLUMN forgotten INTEGER DEFAULT 0;

+0

-59

~~synapse/storage/data_stores/main/schema/delta/27/ts.py~~ less more

0		# Copyright 2015, 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13		import json
14		import logging
15
16		from synapse.storage.prepare_database import get_statements
17
18		logger = logging.getLogger(__name__)
19
20
21		ALTER_TABLE = (
22		"ALTER TABLE events ADD COLUMN origin_server_ts BIGINT;"
23		"CREATE INDEX events_ts ON events(origin_server_ts, stream_ordering);"
24		)
25
26
27		def run_create(cur, database_engine, args, *kwargs):
28		for statement in get_statements(ALTER_TABLE.splitlines()):
29		cur.execute(statement)
30
31		cur.execute("SELECT MIN(stream_ordering) FROM events")
32		rows = cur.fetchall()
33		min_stream_id = rows[0][0]
34
35		cur.execute("SELECT MAX(stream_ordering) FROM events")
36		rows = cur.fetchall()
37		max_stream_id = rows[0][0]
38
39		if min_stream_id is not None and max_stream_id is not None:
40		progress = {
41		"target_min_stream_id_inclusive": min_stream_id,
42		"max_stream_id_exclusive": max_stream_id + 1,
43		"rows_inserted": 0,
44		}
45		progress_json = json.dumps(progress)
46
47		sql = (
48		"INSERT into background_updates (update_name, progress_json)"
49		" VALUES (?, ?)"
50		)
51
52		sql = database_engine.convert_param_style(sql)
53
54		cur.execute(sql, ("event_origin_server_ts", progress_json))
55
56
57		def run_upgrade(args, *kwargs):
58		pass

+0

-27

~~synapse/storage/data_stores/main/schema/delta/28/event_push_actions.sql~~ less more

0		/* Copyright 2015 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS event_push_actions(
16		room_id TEXT NOT NULL,
17		event_id TEXT NOT NULL,
18		user_id TEXT NOT NULL,
19		profile_tag VARCHAR(32),
20		actions TEXT NOT NULL,
21		CONSTRAINT event_id_user_id_profile_tag_uniqueness UNIQUE (room_id, event_id, user_id, profile_tag)
22		);
23
24
25		CREATE INDEX event_push_actions_room_id_event_id_user_id_profile_tag on event_push_actions(room_id, event_id, user_id, profile_tag);
26		CREATE INDEX event_push_actions_room_id_user_id on event_push_actions(room_id, user_id);

+0

-20

~~synapse/storage/data_stores/main/schema/delta/28/events_room_stream.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/** Using CREATE INDEX directly is deprecated in favour of using background
16		* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
17		* and synapse/storage/registration.py for an example using
18		* "access_tokens_device_index" **/
19		CREATE INDEX events_room_stream on events(room_id, stream_ordering);

+0

-20

~~synapse/storage/data_stores/main/schema/delta/28/public_roms_index.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/** Using CREATE INDEX directly is deprecated in favour of using background
16		* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
17		* and synapse/storage/registration.py for an example using
18		* "access_tokens_device_index" **/
19		CREATE INDEX public_room_index on rooms(is_public);

+0

-22

~~synapse/storage/data_stores/main/schema/delta/28/receipts_user_id_index.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/** Using CREATE INDEX directly is deprecated in favour of using background
16		* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
17		* and synapse/storage/registration.py for an example using
18		* "access_tokens_device_index" **/
19		CREATE INDEX receipts_linearized_user ON receipts_linearized(
20		user_id
21		);

+0

-21

~~synapse/storage/data_stores/main/schema/delta/28/upgrade_times.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/*
16		* Stores the timestamp when a user upgraded from a guest to a full user, if
17		* that happened.
18		*/
19
20		ALTER TABLE users ADD COLUMN upgrade_ts BIGINT;

+0

-22

~~synapse/storage/data_stores/main/schema/delta/28/users_is_guest.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE users ADD is_guest SMALLINT DEFAULT 0 NOT NULL;
16		/*
17		* NB: any guest users created between 27 and 28 will be incorrectly
18		* marked as not guests: we don't bother to fill these in correctly
19		* because guest access is not really complete in 27 anyway so it's
20		* very unlikley there will be any guest users created.
21		*/

+0

-35

~~synapse/storage/data_stores/main/schema/delta/29/push_actions.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE event_push_actions ADD COLUMN topological_ordering BIGINT;
16		ALTER TABLE event_push_actions ADD COLUMN stream_ordering BIGINT;
17		ALTER TABLE event_push_actions ADD COLUMN notif SMALLINT;
18		ALTER TABLE event_push_actions ADD COLUMN highlight SMALLINT;
19
20		UPDATE event_push_actions SET stream_ordering = (
21		SELECT stream_ordering FROM events WHERE event_id = event_push_actions.event_id
22		), topological_ordering = (
23		SELECT topological_ordering FROM events WHERE event_id = event_push_actions.event_id
24		);
25
26		UPDATE event_push_actions SET notif = 1, highlight = 0;
27
28		/** Using CREATE INDEX directly is deprecated in favour of using background
29		* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
30		* and synapse/storage/registration.py for an example using
31		* "access_tokens_device_index" **/
32		CREATE INDEX event_push_actions_rm_tokens on event_push_actions(
33		user_id, room_id, topological_ordering, stream_ordering
34		);

+0

-16

~~synapse/storage/data_stores/main/schema/delta/30/alias_creator.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE room_aliases ADD COLUMN creator TEXT;

+0

-67

~~synapse/storage/data_stores/main/schema/delta/30/as_users.py~~ less more

0		# Copyright 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13		import logging
14
15		from synapse.config.appservice import load_appservices
16
17		logger = logging.getLogger(__name__)
18
19
20		def run_create(cur, database_engine, args, *kwargs):
21		# NULL indicates user was not registered by an appservice.
22		try:
23		cur.execute("ALTER TABLE users ADD COLUMN appservice_id TEXT")
24		except Exception:
25		# Maybe we already added the column? Hope so...
26		pass
27
28
29		def run_upgrade(cur, database_engine, config, args, *kwargs):
30		cur.execute("SELECT name FROM users")
31		rows = cur.fetchall()
32
33		config_files = []
34		try:
35		config_files = config.app_service_config_files
36		except AttributeError:
37		logger.warning("Could not get app_service_config_files from config")
38		pass
39
40		appservices = load_appservices(config.server_name, config_files)
41
42		owned = {}
43
44		for row in rows:
45		user_id = row[0]
46		for appservice in appservices:
47		if appservice.is_exclusive_user(user_id):
48		if user_id in owned.keys():
49		logger.error(
50		"user_id %s was owned by more than one application"
51		" service (IDs %s and %s); assigning arbitrarily to %s"
52		% (user_id, owned[user_id], appservice.id, owned[user_id])
53		)
54		owned.setdefault(appservice.id, []).append(user_id)
55
56		for as_id, user_ids in owned.items():
57		n = 100
58		user_chunks = (user_ids[i : i + 100] for i in range(0, len(user_ids), n))
59		for chunk in user_chunks:
60		cur.execute(
61		database_engine.convert_param_style(
62		"UPDATE users SET appservice_id = ? WHERE name IN (%s)"
63		% (",".join("?" for _ in chunk),)
64		),
65		[as_id] + chunk,
66		)

+0

-25

~~synapse/storage/data_stores/main/schema/delta/30/deleted_pushers.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS deleted_pushers(
16		stream_id BIGINT NOT NULL,
17		app_id TEXT NOT NULL,
18		pushkey TEXT NOT NULL,
19		user_id TEXT NOT NULL,
20		/* We only track the most recent delete for each app_id, pushkey and user_id. */
21		UNIQUE (app_id, pushkey, user_id)
22		);
23
24		CREATE INDEX deleted_pushers_stream_id ON deleted_pushers (stream_id);

+0

-30

~~synapse/storage/data_stores/main/schema/delta/30/presence_stream.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE presence_stream(
17		stream_id BIGINT,
18		user_id TEXT,
19		state TEXT,
20		last_active_ts BIGINT,
21		last_federation_update_ts BIGINT,
22		last_user_sync_ts BIGINT,
23		status_msg TEXT,
24		currently_active BOOLEAN
25		);
26
27		CREATE INDEX presence_stream_id ON presence_stream(stream_id, user_id);
28		CREATE INDEX presence_stream_user_id ON presence_stream(user_id);
29		CREATE INDEX presence_stream_state ON presence_stream(state);

+0

-23

~~synapse/storage/data_stores/main/schema/delta/30/public_rooms.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		/* This release removes the restriction that published rooms must have an alias,
17		* so we go back and ensure the only 'public' rooms are ones with an alias.
18		* We use (1 = 0) and (1 = 1) so that it works in both postgres and sqlite
19		*/
20		UPDATE rooms SET is_public = (1 = 0) WHERE is_public = (1 = 1) AND room_id not in (
21		SELECT room_id FROM room_aliases
22		);

+0

-38

~~synapse/storage/data_stores/main/schema/delta/30/push_rule_stream.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16
17		CREATE TABLE push_rules_stream(
18		stream_id BIGINT NOT NULL,
19		event_stream_ordering BIGINT NOT NULL,
20		user_id TEXT NOT NULL,
21		rule_id TEXT NOT NULL,
22		op TEXT NOT NULL, -- One of "ENABLE", "DISABLE", "ACTIONS", "ADD", "DELETE"
23		priority_class SMALLINT,
24		priority INTEGER,
25		conditions TEXT,
26		actions TEXT
27		);
28
29		-- The extra data for each operation is:
30		-- * ENABLE, DISABLE, DELETE: []
31		-- * ACTIONS: ["actions"]
32		-- * ADD: ["priority_class", "priority", "actions", "conditions"]
33
34		-- Index for replication queries.
35		CREATE INDEX push_rules_stream_id ON push_rules_stream(stream_id);
36		-- Index for /sync queries.
37		CREATE INDEX push_rules_stream_user_stream_id on push_rules_stream(user_id, stream_id);

+0

-24

~~synapse/storage/data_stores/main/schema/delta/30/threepid_guest_access_tokens.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Stores guest account access tokens generated for unbound 3pids.
16		CREATE TABLE threepid_guest_access_tokens(
17		medium TEXT, -- The medium of the 3pid. Must be "email".
18		address TEXT, -- The 3pid address.
19		guest_access_token TEXT, -- The access token for a guest user for this 3pid.
20		first_inviter TEXT -- User ID of the first user to invite this 3pid to a room.
21		);
22
23		CREATE UNIQUE INDEX threepid_guest_access_tokens_index ON threepid_guest_access_tokens(medium, address);

+0

-42

~~synapse/storage/data_stores/main/schema/delta/31/invites.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE local_invites(
17		stream_id BIGINT NOT NULL,
18		inviter TEXT NOT NULL,
19		invitee TEXT NOT NULL,
20		event_id TEXT NOT NULL,
21		room_id TEXT NOT NULL,
22		locally_rejected TEXT,
23		replaced_by TEXT
24		);
25
26		-- Insert all invites for local users into new `invites` table
27		INSERT INTO local_invites SELECT
28		stream_ordering as stream_id,
29		sender as inviter,
30		state_key as invitee,
31		event_id,
32		room_id,
33		NULL as locally_rejected,
34		NULL as replaced_by
35		FROM events
36		NATURAL JOIN current_state_events
37		NATURAL JOIN room_memberships
38		WHERE membership = 'invite' AND state_key IN (SELECT name FROM users);
39
40		CREATE INDEX local_invites_id ON local_invites(stream_id);
41		CREATE INDEX local_invites_for_user_idx ON local_invites(invitee, locally_rejected, replaced_by, room_id);

+0

-27

~~synapse/storage/data_stores/main/schema/delta/31/local_media_repository_url_cache.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE local_media_repository_url_cache(
16		url TEXT, -- the URL being cached
17		response_code INTEGER, -- the HTTP response code of this download attempt
18		etag TEXT, -- the etag header of this response
19		expires INTEGER, -- the number of ms this response was valid for
20		og TEXT, -- cache of the OG metadata of this URL as JSON
21		media_id TEXT, -- the media_id, if any, of the URL's content in the repo
22		download_ts BIGINT -- the timestamp of this download attempt
23		);
24
25		CREATE INDEX local_media_repository_url_cache_by_url_download_ts
26		ON local_media_repository_url_cache(url, download_ts);

+0

-87

~~synapse/storage/data_stores/main/schema/delta/31/pushers.py~~ less more

0		# Copyright 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14
15		# Change the last_token to last_stream_ordering now that pushers no longer
16		# listen on an event stream but instead select out of the event_push_actions
17		# table.
18
19
20		import logging
21
22		logger = logging.getLogger(__name__)
23
24
25		def token_to_stream_ordering(token):
26		return int(token[1:].split("_")[0])
27
28
29		def run_create(cur, database_engine, args, *kwargs):
30		logger.info("Porting pushers table, delta 31...")
31		cur.execute(
32		"""
33		CREATE TABLE IF NOT EXISTS pushers2 (
34		id BIGINT PRIMARY KEY,
35		user_name TEXT NOT NULL,
36		access_token BIGINT DEFAULT NULL,
37		profile_tag VARCHAR(32) NOT NULL,
38		kind VARCHAR(8) NOT NULL,
39		app_id VARCHAR(64) NOT NULL,
40		app_display_name VARCHAR(64) NOT NULL,
41		device_display_name VARCHAR(128) NOT NULL,
42		pushkey TEXT NOT NULL,
43		ts BIGINT NOT NULL,
44		lang VARCHAR(8),
45		data TEXT,
46		last_stream_ordering INTEGER,
47		last_success BIGINT,
48		failing_since BIGINT,
49		UNIQUE (app_id, pushkey, user_name)
50		)
51		"""
52		)
53		cur.execute(
54		"""SELECT
55		id, user_name, access_token, profile_tag, kind,
56		app_id, app_display_name, device_display_name,
57		pushkey, ts, lang, data, last_token, last_success,
58		failing_since
59		FROM pushers
60		"""
61		)
62		count = 0
63		for row in cur.fetchall():
64		row = list(row)
65		row[12] = token_to_stream_ordering(row[12])
66		cur.execute(
67		database_engine.convert_param_style(
68		"""
69		INSERT into pushers2 (
70		id, user_name, access_token, profile_tag, kind,
71		app_id, app_display_name, device_display_name,
72		pushkey, ts, lang, data, last_stream_ordering, last_success,
73		failing_since
74		) values (%s)"""
75		% (",".join(["?" for _ in range(len(row))]))
76		),
77		row,
78		)
79		count += 1
80		cur.execute("DROP TABLE pushers")
81		cur.execute("ALTER TABLE pushers2 RENAME TO pushers")
82		logger.info("Moved %d pushers to new table", count)
83
84
85		def run_upgrade(cur, database_engine, args, *kwargs):
86		pass

+0

-22

~~synapse/storage/data_stores/main/schema/delta/31/pushers_index.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/** Using CREATE INDEX directly is deprecated in favour of using background
16		* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
17		* and synapse/storage/registration.py for an example using
18		* "access_tokens_device_index" **/
19		CREATE INDEX event_push_actions_stream_ordering on event_push_actions(
20		stream_ordering, user_id
21		);

+0

-64

~~synapse/storage/data_stores/main/schema/delta/31/search_update.py~~ less more

0		# Copyright 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13		import json
14		import logging
15
16		from synapse.storage.engines import PostgresEngine
17		from synapse.storage.prepare_database import get_statements
18
19		logger = logging.getLogger(__name__)
20
21
22		ALTER_TABLE = """
23		ALTER TABLE event_search ADD COLUMN origin_server_ts BIGINT;
24		ALTER TABLE event_search ADD COLUMN stream_ordering BIGINT;
25		"""
26
27
28		def run_create(cur, database_engine, args, *kwargs):
29		if not isinstance(database_engine, PostgresEngine):
30		return
31
32		for statement in get_statements(ALTER_TABLE.splitlines()):
33		cur.execute(statement)
34
35		cur.execute("SELECT MIN(stream_ordering) FROM events")
36		rows = cur.fetchall()
37		min_stream_id = rows[0][0]
38
39		cur.execute("SELECT MAX(stream_ordering) FROM events")
40		rows = cur.fetchall()
41		max_stream_id = rows[0][0]
42
43		if min_stream_id is not None and max_stream_id is not None:
44		progress = {
45		"target_min_stream_id_inclusive": min_stream_id,
46		"max_stream_id_exclusive": max_stream_id + 1,
47		"rows_inserted": 0,
48		"have_added_indexes": False,
49		}
50		progress_json = json.dumps(progress)
51
52		sql = (
53		"INSERT into background_updates (update_name, progress_json)"
54		" VALUES (?, ?)"
55		)
56
57		sql = database_engine.convert_param_style(sql)
58
59		cur.execute(sql, ("event_search_order", progress_json))
60
61
62		def run_upgrade(cur, database_engine, args, *kwargs):
63		pass

+0

-16

~~synapse/storage/data_stores/main/schema/delta/32/events.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE events ADD COLUMN received_ts BIGINT;

+0

-9

~~synapse/storage/data_stores/main/schema/delta/32/openid.sql~~ less more

0
1		CREATE TABLE open_id_tokens (
2		token TEXT NOT NULL PRIMARY KEY,
3		ts_valid_until_ms bigint NOT NULL,
4		user_id TEXT NOT NULL,
5		UNIQUE (token)
6		);
7
8		CREATE index open_id_tokens_ts_valid_until_ms ON open_id_tokens(ts_valid_until_ms);

+0

-23

~~synapse/storage/data_stores/main/schema/delta/32/pusher_throttle.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE pusher_throttle(
17		pusher BIGINT NOT NULL,
18		room_id TEXT NOT NULL,
19		last_sent_ts BIGINT,
20		throttle_ms BIGINT,
21		PRIMARY KEY (pusher, room_id)
22		);

+0

-33

~~synapse/storage/data_stores/main/schema/delta/32/remove_indices.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		-- The following indices are redundant, other indices are equivalent or
17		-- supersets
18		DROP INDEX IF EXISTS events_room_id; -- Prefix of events_room_stream
19		DROP INDEX IF EXISTS events_order; -- Prefix of events_order_topo_stream_room
20		DROP INDEX IF EXISTS events_topological_ordering; -- Prefix of events_order_topo_stream_room
21		DROP INDEX IF EXISTS events_stream_ordering; -- Duplicate of PRIMARY KEY
22		DROP INDEX IF EXISTS event_to_state_groups_id; -- Duplicate of PRIMARY KEY
23		DROP INDEX IF EXISTS event_push_actions_room_id_event_id_user_id_profile_tag; -- Duplicate of UNIQUE CONSTRAINT
24
25		DROP INDEX IF EXISTS st_extrem_id; -- Prefix of UNIQUE CONSTRAINT
26		DROP INDEX IF EXISTS event_signatures_id; -- Prefix of UNIQUE CONSTRAINT
27		DROP INDEX IF EXISTS redactions_event_id; -- Duplicate of UNIQUE CONSTRAINT
28
29		-- The following indices were unused
30		DROP INDEX IF EXISTS remote_media_cache_thumbnails_media_id;
31		DROP INDEX IF EXISTS evauth_edges_auth_id;
32		DROP INDEX IF EXISTS presence_stream_state;

+0

-25

~~synapse/storage/data_stores/main/schema/delta/32/reports.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE event_reports(
17		id BIGINT NOT NULL PRIMARY KEY,
18		received_ts BIGINT NOT NULL,
19		room_id TEXT NOT NULL,
20		event_id TEXT NOT NULL,
21		user_id TEXT NOT NULL,
22		reason TEXT,
23		content TEXT
24		);

+0

-17

~~synapse/storage/data_stores/main/schema/delta/33/access_tokens_device_index.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT INTO background_updates (update_name, progress_json) VALUES
16		('access_tokens_device_index', '{}');

+0

-21

~~synapse/storage/data_stores/main/schema/delta/33/devices.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE devices (
16		user_id TEXT NOT NULL,
17		device_id TEXT NOT NULL,
18		display_name TEXT,
19		CONSTRAINT device_uniqueness UNIQUE (user_id, device_id)
20		);

+0

-19

~~synapse/storage/data_stores/main/schema/delta/33/devices_for_e2e_keys.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- make sure that we have a device record for each set of E2E keys, so that the
16		-- user can delete them if they like.
17		INSERT INTO devices
18		SELECT user_id, device_id, NULL FROM e2e_device_keys_json;

+0

-20

~~synapse/storage/data_stores/main/schema/delta/33/devices_for_e2e_keys_clear_unknown_device.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- a previous version of the "devices_for_e2e_keys" delta set all the device
16		-- names to "unknown device". This wasn't terribly helpful
17		UPDATE devices
18		SET display_name = NULL
19		WHERE display_name = 'unknown device';

+0

-59

~~synapse/storage/data_stores/main/schema/delta/33/event_fields.py~~ less more

0		# Copyright 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13		import json
14		import logging
15
16		from synapse.storage.prepare_database import get_statements
17
18		logger = logging.getLogger(__name__)
19
20
21		ALTER_TABLE = """
22		ALTER TABLE events ADD COLUMN sender TEXT;
23		ALTER TABLE events ADD COLUMN contains_url BOOLEAN;
24		"""
25
26
27		def run_create(cur, database_engine, args, *kwargs):
28		for statement in get_statements(ALTER_TABLE.splitlines()):
29		cur.execute(statement)
30
31		cur.execute("SELECT MIN(stream_ordering) FROM events")
32		rows = cur.fetchall()
33		min_stream_id = rows[0][0]
34
35		cur.execute("SELECT MAX(stream_ordering) FROM events")
36		rows = cur.fetchall()
37		max_stream_id = rows[0][0]
38
39		if min_stream_id is not None and max_stream_id is not None:
40		progress = {
41		"target_min_stream_id_inclusive": min_stream_id,
42		"max_stream_id_exclusive": max_stream_id + 1,
43		"rows_inserted": 0,
44		}
45		progress_json = json.dumps(progress)
46
47		sql = (
48		"INSERT into background_updates (update_name, progress_json)"
49		" VALUES (?, ?)"
50		)
51
52		sql = database_engine.convert_param_style(sql)
53
54		cur.execute(sql, ("event_fields_sender_url", progress_json))
55
56
57		def run_upgrade(cur, database_engine, args, *kwargs):
58		pass

+0

-30

~~synapse/storage/data_stores/main/schema/delta/33/remote_media_ts.py~~ less more

0		# Copyright 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		import time
15
16		ALTER_TABLE = "ALTER TABLE remote_media_cache ADD COLUMN last_access_ts BIGINT"
17
18
19		def run_create(cur, database_engine, args, *kwargs):
20		cur.execute(ALTER_TABLE)
21
22
23		def run_upgrade(cur, database_engine, args, *kwargs):
24		cur.execute(
25		database_engine.convert_param_style(
26		"UPDATE remote_media_cache SET last_access_ts = ?"
27		),
28		(int(time.time() * 1000),),
29		)

+0

-17

~~synapse/storage/data_stores/main/schema/delta/33/user_ips_index.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT INTO background_updates (update_name, progress_json) VALUES
16		('user_ips_device_index', '{}');

+0

-23

~~synapse/storage/data_stores/main/schema/delta/34/appservice_stream.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS appservice_stream_position(
16		Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
17		stream_ordering BIGINT,
18		CHECK (Lock='X')
19		);
20
21		INSERT INTO appservice_stream_position (stream_ordering)
22		SELECT COALESCE(MAX(stream_ordering), 0) FROM events;

+0

-46

~~synapse/storage/data_stores/main/schema/delta/34/cache_stream.py~~ less more

0		# Copyright 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		import logging
15
16		from synapse.storage.engines import PostgresEngine
17		from synapse.storage.prepare_database import get_statements
18
19		logger = logging.getLogger(__name__)
20
21
22		# This stream is used to notify replication slaves that some caches have
23		# been invalidated that they cannot infer from the other streams.
24		CREATE_TABLE = """
25		CREATE TABLE cache_invalidation_stream (
26		stream_id BIGINT,
27		cache_func TEXT,
28		keys TEXT[],
29		invalidation_ts BIGINT
30		);
31
32		CREATE INDEX cache_invalidation_stream_id ON cache_invalidation_stream(stream_id);
33		"""
34
35
36		def run_create(cur, database_engine, args, *kwargs):
37		if not isinstance(database_engine, PostgresEngine):
38		return
39
40		for statement in get_statements(CREATE_TABLE.splitlines()):
41		cur.execute(statement)
42
43
44		def run_upgrade(cur, database_engine, args, *kwargs):
45		pass

+0

-24

~~synapse/storage/data_stores/main/schema/delta/34/device_inbox.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE device_inbox (
16		user_id TEXT NOT NULL,
17		device_id TEXT NOT NULL,
18		stream_id BIGINT NOT NULL,
19		message_json TEXT NOT NULL -- {"type":, "sender":, "content",}
20		);
21
22		CREATE INDEX device_inbox_user_stream_id ON device_inbox(user_id, device_id, stream_id);
23		CREATE INDEX device_inbox_stream_id ON device_inbox(stream_id);

+0

-20

~~synapse/storage/data_stores/main/schema/delta/34/push_display_name_rename.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		DELETE FROM push_rules WHERE rule_id = 'global/override/.m.rule.contains_display_name';
16		UPDATE push_rules SET rule_id = 'global/override/.m.rule.contains_display_name' WHERE rule_id = 'global/underride/.m.rule.contains_display_name';
17
18		DELETE FROM push_rules_enable WHERE rule_id = 'global/override/.m.rule.contains_display_name';
19		UPDATE push_rules_enable SET rule_id = 'global/override/.m.rule.contains_display_name' WHERE rule_id = 'global/underride/.m.rule.contains_display_name';

+0

-32

~~synapse/storage/data_stores/main/schema/delta/34/received_txn_purge.py~~ less more

0		# Copyright 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		import logging
15
16		from synapse.storage.engines import PostgresEngine
17
18		logger = logging.getLogger(__name__)
19
20
21		def run_create(cur, database_engine, args, *kwargs):
22		if isinstance(database_engine, PostgresEngine):
23		cur.execute("TRUNCATE received_transactions")
24		else:
25		cur.execute("DELETE FROM received_transactions")
26
27		cur.execute("CREATE INDEX received_transactions_ts ON received_transactions(ts)")
28
29
30		def run_upgrade(cur, database_engine, args, *kwargs):
31		pass

+0

-17

~~synapse/storage/data_stores/main/schema/delta/35/contains_url.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT into background_updates (update_name, progress_json)
16		VALUES ('event_contains_url_index', '{}');

+0

-39

~~synapse/storage/data_stores/main/schema/delta/35/device_outbox.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		DROP TABLE IF EXISTS device_federation_outbox;
16		CREATE TABLE device_federation_outbox (
17		destination TEXT NOT NULL,
18		stream_id BIGINT NOT NULL,
19		queued_ts BIGINT NOT NULL,
20		messages_json TEXT NOT NULL
21		);
22
23
24		DROP INDEX IF EXISTS device_federation_outbox_destination_id;
25		CREATE INDEX device_federation_outbox_destination_id
26		ON device_federation_outbox(destination, stream_id);
27
28
29		DROP TABLE IF EXISTS device_federation_inbox;
30		CREATE TABLE device_federation_inbox (
31		origin TEXT NOT NULL,
32		message_id TEXT NOT NULL,
33		received_ts BIGINT NOT NULL
34		);
35
36		DROP INDEX IF EXISTS device_federation_inbox_sender_id;
37		CREATE INDEX device_federation_inbox_sender_id
38		ON device_federation_inbox(origin, message_id);

+0

-21

~~synapse/storage/data_stores/main/schema/delta/35/device_stream_id.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE device_max_stream_id (
16		stream_id BIGINT NOT NULL
17		);
18
19		INSERT INTO device_max_stream_id (stream_id)
20		SELECT COALESCE(MAX(stream_id), 0) FROM device_inbox;

+0

-17

~~synapse/storage/data_stores/main/schema/delta/35/event_push_actions_index.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT into background_updates (update_name, progress_json)
16		VALUES ('epa_highlight_index', '{}');

+0

-33

~~synapse/storage/data_stores/main/schema/delta/35/public_room_list_change_stream.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE public_room_list_stream (
17		stream_id BIGINT NOT NULL,
18		room_id TEXT NOT NULL,
19		visibility BOOLEAN NOT NULL
20		);
21
22		INSERT INTO public_room_list_stream (stream_id, room_id, visibility)
23		SELECT 1, room_id, is_public FROM rooms
24		WHERE is_public = CAST(1 AS BOOLEAN);
25
26		CREATE INDEX public_room_list_stream_idx on public_room_list_stream(
27		stream_id
28		);
29
30		CREATE INDEX public_room_list_stream_rm_idx on public_room_list_stream(
31		room_id, stream_id
32		);

+0

-37

~~synapse/storage/data_stores/main/schema/delta/35/stream_order_to_extrem.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE stream_ordering_to_exterm (
17		stream_ordering BIGINT NOT NULL,
18		room_id TEXT NOT NULL,
19		event_id TEXT NOT NULL
20		);
21
22		INSERT INTO stream_ordering_to_exterm (stream_ordering, room_id, event_id)
23		SELECT stream_ordering, room_id, event_id FROM event_forward_extremities
24		INNER JOIN (
25		SELECT room_id, max(stream_ordering) as stream_ordering FROM events
26		INNER JOIN event_forward_extremities USING (room_id, event_id)
27		GROUP BY room_id
28		) AS rms USING (room_id);
29
30		CREATE INDEX stream_ordering_to_exterm_idx on stream_ordering_to_exterm(
31		stream_ordering
32		);
33
34		CREATE INDEX stream_ordering_to_exterm_rm_idx on stream_ordering_to_exterm(
35		room_id, stream_ordering
36		);

+0

-26

~~synapse/storage/data_stores/main/schema/delta/36/readd_public_rooms.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Re-add some entries to stream_ordering_to_exterm that were incorrectly deleted
16		INSERT INTO stream_ordering_to_exterm (stream_ordering, room_id, event_id)
17		SELECT
18		(SELECT stream_ordering FROM events where event_id = e.event_id) AS stream_ordering,
19		room_id,
20		event_id
21		FROM event_forward_extremities AS e
22		WHERE NOT EXISTS (
23		SELECT room_id FROM stream_ordering_to_exterm AS s
24		WHERE s.room_id = e.room_id
25		);

+0

-85

~~synapse/storage/data_stores/main/schema/delta/37/remove_auth_idx.py~~ less more

0		# Copyright 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		import logging
15
16		from synapse.storage.engines import PostgresEngine
17		from synapse.storage.prepare_database import get_statements
18
19		logger = logging.getLogger(__name__)
20
21		DROP_INDICES = """
22		-- We only ever query based on event_id
23		DROP INDEX IF EXISTS state_events_room_id;
24		DROP INDEX IF EXISTS state_events_type;
25		DROP INDEX IF EXISTS state_events_state_key;
26
27		-- room_id is indexed elsewhere
28		DROP INDEX IF EXISTS current_state_events_room_id;
29		DROP INDEX IF EXISTS current_state_events_state_key;
30		DROP INDEX IF EXISTS current_state_events_type;
31
32		DROP INDEX IF EXISTS transactions_have_ref;
33
34		-- (topological_ordering, stream_ordering, room_id) seems like a strange index,
35		-- and is used incredibly rarely.
36		DROP INDEX IF EXISTS events_order_topo_stream_room;
37
38		-- an equivalent index to this actually gets re-created in delta 41, because it
39		-- turned out that deleting it wasn't a great plan :/. In any case, let's
40		-- delete it here, and delta 41 will create a new one with an added UNIQUE
41		-- constraint
42		DROP INDEX IF EXISTS event_search_ev_idx;
43		"""
44
45		POSTGRES_DROP_CONSTRAINT = """
46		ALTER TABLE event_auth DROP CONSTRAINT IF EXISTS event_auth_event_id_auth_id_room_id_key;
47		"""
48
49		SQLITE_DROP_CONSTRAINT = """
50		DROP INDEX IF EXISTS evauth_edges_id;
51
52		CREATE TABLE IF NOT EXISTS event_auth_new(
53		event_id TEXT NOT NULL,
54		auth_id TEXT NOT NULL,
55		room_id TEXT NOT NULL
56		);
57
58		INSERT INTO event_auth_new
59		SELECT event_id, auth_id, room_id
60		FROM event_auth;
61
62		DROP TABLE event_auth;
63
64		ALTER TABLE event_auth_new RENAME TO event_auth;
65
66		CREATE INDEX evauth_edges_id ON event_auth(event_id);
67		"""
68
69
70		def run_create(cur, database_engine, args, *kwargs):
71		for statement in get_statements(DROP_INDICES.splitlines()):
72		cur.execute(statement)
73
74		if isinstance(database_engine, PostgresEngine):
75		drop_constraint = POSTGRES_DROP_CONSTRAINT
76		else:
77		drop_constraint = SQLITE_DROP_CONSTRAINT
78
79		for statement in get_statements(drop_constraint.splitlines()):
80		cur.execute(statement)
81
82
83		def run_upgrade(cur, database_engine, args, *kwargs):
84		pass

+0

-52

~~synapse/storage/data_stores/main/schema/delta/37/user_threepids.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/*
16		* Update any email addresses that were stored with mixed case into all
17		* lowercase
18		*/
19
20		-- There may be "duplicate" emails (with different case) already in the table,
21		-- so we find them and move all but the most recently used account.
22		UPDATE user_threepids
23		SET medium = 'email_old'
24		WHERE medium = 'email'
25		AND address IN (
26		-- We select all the addresses that are linked to the user_id that is NOT
27		-- the most recently created.
28		SELECT u.address
29		FROM
30		user_threepids AS u,
31		-- `duplicate_addresses` is a table of all the email addresses that
32		-- appear multiple times and when the binding was created
33		(
34		SELECT lower(u1.address) AS address, max(u1.added_at) AS max_ts
35		FROM user_threepids AS u1
36		INNER JOIN user_threepids AS u2 ON u1.medium = u2.medium AND lower(u1.address) = lower(u2.address) AND u1.address != u2.address
37		WHERE u1.medium = 'email' AND u2.medium = 'email'
38		GROUP BY lower(u1.address)
39		) AS duplicate_addresses
40		WHERE
41		lower(u.address) = duplicate_addresses.address
42		AND u.added_at != max_ts -- NOT the most recently created
43		);
44
45
46		-- This update is now safe since we've removed the duplicate addresses.
47		UPDATE user_threepids SET address = LOWER(address) WHERE medium = 'email';
48
49
50		/* Add an index for the select we do on passwored reset */
51		CREATE INDEX user_threepids_medium_address on user_threepids (medium, address);

+0

-19

~~synapse/storage/data_stores/main/schema/delta/38/postgres_fts_gist.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- We no longer do this given we back it out again in schema 47
16
17		-- INSERT into background_updates (update_name, progress_json)
18		-- VALUES ('event_search_postgres_gist', '{}');

+0

-29

~~synapse/storage/data_stores/main/schema/delta/39/appservice_room_list.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE appservice_room_list(
16		appservice_id TEXT NOT NULL,
17		network_id TEXT NOT NULL,
18		room_id TEXT NOT NULL
19		);
20
21		-- Each appservice can have multiple published room lists associated with them,
22		-- keyed of a particular network_id
23		CREATE UNIQUE INDEX appservice_room_list_idx ON appservice_room_list(
24		appservice_id, network_id, room_id
25		);
26
27		ALTER TABLE public_room_list_stream ADD COLUMN appservice_id TEXT;
28		ALTER TABLE public_room_list_stream ADD COLUMN network_id TEXT;

+0

-16

~~synapse/storage/data_stores/main/schema/delta/39/device_federation_stream_idx.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE INDEX device_federation_outbox_id ON device_federation_outbox(stream_id);

+0

-17

~~synapse/storage/data_stores/main/schema/delta/39/event_push_index.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT INTO background_updates (update_name, progress_json) VALUES
16		('event_push_actions_highlights_index', '{}');

+0

-22

~~synapse/storage/data_stores/main/schema/delta/39/federation_out_position.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE federation_stream_position(
16		type TEXT NOT NULL,
17		stream_id INTEGER NOT NULL
18		);
19
20		INSERT INTO federation_stream_position (type, stream_id) VALUES ('federation', -1);
21		INSERT INTO federation_stream_position (type, stream_id) SELECT 'events', coalesce(max(stream_ordering), -1) FROM events;

+0

-20

~~synapse/storage/data_stores/main/schema/delta/39/membership_profile.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE room_memberships ADD COLUMN display_name TEXT;
16		ALTER TABLE room_memberships ADD COLUMN avatar_url TEXT;
17
18		INSERT into background_updates (update_name, progress_json)
19		VALUES ('room_membership_profile_update', '{}');

+0

-17

~~synapse/storage/data_stores/main/schema/delta/40/current_state_idx.sql~~ less more

0		/* Copyright 2017 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT INTO background_updates (update_name, progress_json) VALUES
16		('current_state_members_idx', '{}');

+0

-21

~~synapse/storage/data_stores/main/schema/delta/40/device_inbox.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- turn the pre-fill startup query into a index-only scan on postgresql.
16		INSERT into background_updates (update_name, progress_json)
17		VALUES ('device_inbox_stream_index', '{}');
18
19		INSERT into background_updates (update_name, progress_json, depends_on)
20		VALUES ('device_inbox_stream_drop', '{}', 'device_inbox_stream_index');

+0

-60

~~synapse/storage/data_stores/main/schema/delta/40/device_list_streams.sql~~ less more

0		/* Copyright 2017 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Cache of remote devices.
16		CREATE TABLE device_lists_remote_cache (
17		user_id TEXT NOT NULL,
18		device_id TEXT NOT NULL,
19		content TEXT NOT NULL
20		);
21
22		-- The last update we got for a user. Empty if we're not receiving updates for
23		-- that user.
24		CREATE TABLE device_lists_remote_extremeties (
25		user_id TEXT NOT NULL,
26		stream_id TEXT NOT NULL
27		);
28
29		-- we used to create non-unique indexes on these tables, but as of update 52 we create
30		-- unique indexes concurrently:
31		--
32		-- CREATE INDEX device_lists_remote_cache_id ON device_lists_remote_cache(user_id, device_id);
33		-- CREATE INDEX device_lists_remote_extremeties_id ON device_lists_remote_extremeties(user_id, stream_id);
34
35
36		-- Stream of device lists updates. Includes both local and remotes
37		CREATE TABLE device_lists_stream (
38		stream_id BIGINT NOT NULL,
39		user_id TEXT NOT NULL,
40		device_id TEXT NOT NULL
41		);
42
43		CREATE INDEX device_lists_stream_id ON device_lists_stream(stream_id, user_id);
44
45
46		-- The stream of updates to send to other servers. We keep at least one row
47		-- per user that was sent so that the prev_id for any new updates can be
48		-- calculated
49		CREATE TABLE device_lists_outbound_pokes (
50		destination TEXT NOT NULL,
51		stream_id BIGINT NOT NULL,
52		user_id TEXT NOT NULL,
53		device_id TEXT NOT NULL,
54		sent BOOLEAN NOT NULL,
55		ts BIGINT NOT NULL -- So that in future we can clear out pokes to dead servers
56		);
57
58		CREATE INDEX device_lists_outbound_pokes_id ON device_lists_outbound_pokes(destination, stream_id);
59		CREATE INDEX device_lists_outbound_pokes_user ON device_lists_outbound_pokes(destination, user_id);

+0

-37

~~synapse/storage/data_stores/main/schema/delta/40/event_push_summary.sql~~ less more

0		/* Copyright 2017 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Aggregate of old notification counts that have been deleted out of the
16		-- main event_push_actions table. This count does not include those that were
17		-- highlights, as they remain in the event_push_actions table.
18		CREATE TABLE event_push_summary (
19		user_id TEXT NOT NULL,
20		room_id TEXT NOT NULL,
21		notif_count BIGINT NOT NULL,
22		stream_ordering BIGINT NOT NULL
23		);
24
25		CREATE INDEX event_push_summary_user_rm ON event_push_summary(user_id, room_id);
26
27
28		-- The stream ordering up to which we have aggregated the event_push_actions
29		-- table into event_push_summary
30		CREATE TABLE event_push_summary_stream_ordering (
31		Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
32		stream_ordering BIGINT NOT NULL,
33		CHECK (Lock='X')
34		);
35
36		INSERT INTO event_push_summary_stream_ordering (stream_ordering) VALUES (0);

+0

-39

~~synapse/storage/data_stores/main/schema/delta/40/pushers.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS pushers2 (
16		id BIGINT PRIMARY KEY,
17		user_name TEXT NOT NULL,
18		access_token BIGINT DEFAULT NULL,
19		profile_tag TEXT NOT NULL,
20		kind TEXT NOT NULL,
21		app_id TEXT NOT NULL,
22		app_display_name TEXT NOT NULL,
23		device_display_name TEXT NOT NULL,
24		pushkey TEXT NOT NULL,
25		ts BIGINT NOT NULL,
26		lang TEXT,
27		data TEXT,
28		last_stream_ordering INTEGER,
29		last_success BIGINT,
30		failing_since BIGINT,
31		UNIQUE (app_id, pushkey, user_name)
32		);
33
34		INSERT INTO pushers2 SELECT * FROM PUSHERS;
35
36		DROP TABLE PUSHERS;
37
38		ALTER TABLE pushers2 RENAME TO pushers;

+0

-17

~~synapse/storage/data_stores/main/schema/delta/41/device_list_stream_idx.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT into background_updates (update_name, progress_json)
16		VALUES ('device_lists_stream_idx', '{}');

+0

-16

~~synapse/storage/data_stores/main/schema/delta/41/device_outbound_index.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE INDEX device_lists_outbound_pokes_stream ON device_lists_outbound_pokes(stream_id);

+0

-17

~~synapse/storage/data_stores/main/schema/delta/41/event_search_event_id_idx.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT into background_updates (update_name, progress_json)
16		VALUES ('event_search_event_id_idx', '{}');

+0

-22

~~synapse/storage/data_stores/main/schema/delta/41/ratelimit.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE ratelimit_override (
16		user_id TEXT NOT NULL,
17		messages_per_second BIGINT,
18		burst_count BIGINT
19		);
20
21		CREATE UNIQUE INDEX ratelimit_override_idx ON ratelimit_override(user_id);

+0

-26

~~synapse/storage/data_stores/main/schema/delta/42/current_state_delta.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE current_state_delta_stream (
17		stream_id BIGINT NOT NULL,
18		room_id TEXT NOT NULL,
19		type TEXT NOT NULL,
20		state_key TEXT NOT NULL,
21		event_id TEXT, -- Is null if the key was removed
22		prev_event_id TEXT -- Is null if the key was added
23		);
24
25		CREATE INDEX current_state_delta_stream_idx ON current_state_delta_stream(stream_id);

+0

-33

~~synapse/storage/data_stores/main/schema/delta/42/device_list_last_id.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		-- Table of last stream_id that we sent to destination for user_id. This is
17		-- used to fill out the `prev_id` fields of outbound device list updates.
18		CREATE TABLE device_lists_outbound_last_success (
19		destination TEXT NOT NULL,
20		user_id TEXT NOT NULL,
21		stream_id BIGINT NOT NULL
22		);
23
24		INSERT INTO device_lists_outbound_last_success
25		SELECT destination, user_id, coalesce(max(stream_id), 0) as stream_id
26		FROM device_lists_outbound_pokes
27		WHERE sent = (1 = 1) -- sqlite doesn't have inbuilt boolean values
28		GROUP BY destination, user_id;
29
30		CREATE INDEX device_lists_outbound_last_success_idx ON device_lists_outbound_last_success(
31		destination, user_id, stream_id
32		);

+0

-17

~~synapse/storage/data_stores/main/schema/delta/42/event_auth_state_only.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT INTO background_updates (update_name, progress_json) VALUES
16		('event_auth_state_only', '{}');

+0

-84

~~synapse/storage/data_stores/main/schema/delta/42/user_dir.py~~ less more

0		# Copyright 2017 Vector Creations Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		import logging
15
16		from synapse.storage.engines import PostgresEngine, Sqlite3Engine
17		from synapse.storage.prepare_database import get_statements
18
19		logger = logging.getLogger(__name__)
20
21
22		BOTH_TABLES = """
23		CREATE TABLE user_directory_stream_pos (
24		Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
25		stream_id BIGINT,
26		CHECK (Lock='X')
27		);
28
29		INSERT INTO user_directory_stream_pos (stream_id) VALUES (null);
30
31		CREATE TABLE user_directory (
32		user_id TEXT NOT NULL,
33		room_id TEXT NOT NULL, -- A room_id that we know the user is joined to
34		display_name TEXT,
35		avatar_url TEXT
36		);
37
38		CREATE INDEX user_directory_room_idx ON user_directory(room_id);
39		CREATE UNIQUE INDEX user_directory_user_idx ON user_directory(user_id);
40
41		CREATE TABLE users_in_pubic_room (
42		user_id TEXT NOT NULL,
43		room_id TEXT NOT NULL -- A room_id that we know is public
44		);
45
46		CREATE INDEX users_in_pubic_room_room_idx ON users_in_pubic_room(room_id);
47		CREATE UNIQUE INDEX users_in_pubic_room_user_idx ON users_in_pubic_room(user_id);
48		"""
49
50
51		POSTGRES_TABLE = """
52		CREATE TABLE user_directory_search (
53		user_id TEXT NOT NULL,
54		vector tsvector
55		);
56
57		CREATE INDEX user_directory_search_fts_idx ON user_directory_search USING gin(vector);
58		CREATE UNIQUE INDEX user_directory_search_user_idx ON user_directory_search(user_id);
59		"""
60
61
62		SQLITE_TABLE = """
63		CREATE VIRTUAL TABLE user_directory_search
64		USING fts4 ( user_id, value );
65		"""
66
67
68		def run_create(cur, database_engine, args, *kwargs):
69		for statement in get_statements(BOTH_TABLES.splitlines()):
70		cur.execute(statement)
71
72		if isinstance(database_engine, PostgresEngine):
73		for statement in get_statements(POSTGRES_TABLE.splitlines()):
74		cur.execute(statement)
75		elif isinstance(database_engine, Sqlite3Engine):
76		for statement in get_statements(SQLITE_TABLE.splitlines()):
77		cur.execute(statement)
78		else:
79		raise Exception("Unrecognized database engine")
80
81
82		def run_upgrade(args, *kwargs):
83		pass

+0

-21

~~synapse/storage/data_stores/main/schema/delta/43/blocked_rooms.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE blocked_rooms (
16		room_id TEXT NOT NULL,
17		user_id TEXT NOT NULL -- Admin who blocked the room
18		);
19
20		CREATE UNIQUE INDEX blocked_rooms_idx ON blocked_rooms(room_id);

+0

-17

~~synapse/storage/data_stores/main/schema/delta/43/quarantine_media.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE local_media_repository ADD COLUMN quarantined_by TEXT;
16		ALTER TABLE remote_media_cache ADD COLUMN quarantined_by TEXT;

+0

-16

~~synapse/storage/data_stores/main/schema/delta/43/url_cache.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE local_media_repository ADD COLUMN url_cache TEXT;

+0

-33

~~synapse/storage/data_stores/main/schema/delta/43/user_share.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Table keeping track of who shares a room with who. We only keep track
16		-- of this for local users, so `user_id` is local users only (but we do keep track
17		-- of which remote users share a room)
18		CREATE TABLE users_who_share_rooms (
19		user_id TEXT NOT NULL,
20		other_user_id TEXT NOT NULL,
21		room_id TEXT NOT NULL,
22		share_private BOOLEAN NOT NULL -- is the shared room private? i.e. they share a private room
23		);
24
25
26		CREATE UNIQUE INDEX users_who_share_rooms_u_idx ON users_who_share_rooms(user_id, other_user_id);
27		CREATE INDEX users_who_share_rooms_r_idx ON users_who_share_rooms(room_id);
28		CREATE INDEX users_who_share_rooms_o_idx ON users_who_share_rooms(other_user_id);
29
30
31		-- Make sure that we populate the table initially
32		UPDATE user_directory_stream_pos SET stream_id = NULL;

+0

-41

~~synapse/storage/data_stores/main/schema/delta/44/expire_url_cache.sql~~ less more

0		/* Copyright 2017 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- this didn't work on SQLite 3.7 (because of lack of partial indexes), so was
16		-- removed and replaced with 46/local_media_repository_url_idx.sql.
17		--
18		-- CREATE INDEX local_media_repository_url_idx ON local_media_repository(created_ts) WHERE url_cache IS NOT NULL;
19
20		-- we need to change `expires` to `expires_ts` so that we can index on it. SQLite doesn't support
21		-- indices on expressions until 3.9.
22		CREATE TABLE local_media_repository_url_cache_new(
23		url TEXT,
24		response_code INTEGER,
25		etag TEXT,
26		expires_ts BIGINT,
27		og TEXT,
28		media_id TEXT,
29		download_ts BIGINT
30		);
31
32		INSERT INTO local_media_repository_url_cache_new
33		SELECT url, response_code, etag, expires + download_ts, og, media_id, download_ts FROM local_media_repository_url_cache;
34
35		DROP TABLE local_media_repository_url_cache;
36		ALTER TABLE local_media_repository_url_cache_new RENAME TO local_media_repository_url_cache;
37
38		CREATE INDEX local_media_repository_url_cache_expires_idx ON local_media_repository_url_cache(expires_ts);
39		CREATE INDEX local_media_repository_url_cache_by_url_download_ts ON local_media_repository_url_cache(url, download_ts);
40		CREATE INDEX local_media_repository_url_cache_media_idx ON local_media_repository_url_cache(media_id);

+0

-167

~~synapse/storage/data_stores/main/schema/delta/45/group_server.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE groups (
16		group_id TEXT NOT NULL,
17		name TEXT, -- the display name of the room
18		avatar_url TEXT,
19		short_description TEXT,
20		long_description TEXT
21		);
22
23		CREATE UNIQUE INDEX groups_idx ON groups(group_id);
24
25
26		-- list of users the group server thinks are joined
27		CREATE TABLE group_users (
28		group_id TEXT NOT NULL,
29		user_id TEXT NOT NULL,
30		is_admin BOOLEAN NOT NULL,
31		is_public BOOLEAN NOT NULL -- whether the users membership can be seen by everyone
32		);
33
34
35		CREATE INDEX groups_users_g_idx ON group_users(group_id, user_id);
36		CREATE INDEX groups_users_u_idx ON group_users(user_id);
37
38		-- list of users the group server thinks are invited
39		CREATE TABLE group_invites (
40		group_id TEXT NOT NULL,
41		user_id TEXT NOT NULL
42		);
43
44		CREATE INDEX groups_invites_g_idx ON group_invites(group_id, user_id);
45		CREATE INDEX groups_invites_u_idx ON group_invites(user_id);
46
47
48		CREATE TABLE group_rooms (
49		group_id TEXT NOT NULL,
50		room_id TEXT NOT NULL,
51		is_public BOOLEAN NOT NULL -- whether the room can be seen by everyone
52		);
53
54		CREATE UNIQUE INDEX groups_rooms_g_idx ON group_rooms(group_id, room_id);
55		CREATE INDEX groups_rooms_r_idx ON group_rooms(room_id);
56
57
58		-- Rooms to include in the summary
59		CREATE TABLE group_summary_rooms (
60		group_id TEXT NOT NULL,
61		room_id TEXT NOT NULL,
62		category_id TEXT NOT NULL,
63		room_order BIGINT NOT NULL,
64		is_public BOOLEAN NOT NULL, -- whether the room should be show to everyone
65		UNIQUE (group_id, category_id, room_id, room_order),
66		CHECK (room_order > 0)
67		);
68
69		CREATE UNIQUE INDEX group_summary_rooms_g_idx ON group_summary_rooms(group_id, room_id, category_id);
70
71
72		-- Categories to include in the summary
73		CREATE TABLE group_summary_room_categories (
74		group_id TEXT NOT NULL,
75		category_id TEXT NOT NULL,
76		cat_order BIGINT NOT NULL,
77		UNIQUE (group_id, category_id, cat_order),
78		CHECK (cat_order > 0)
79		);
80
81		-- The categories in the group
82		CREATE TABLE group_room_categories (
83		group_id TEXT NOT NULL,
84		category_id TEXT NOT NULL,
85		profile TEXT NOT NULL,
86		is_public BOOLEAN NOT NULL, -- whether the category should be show to everyone
87		UNIQUE (group_id, category_id)
88		);
89
90		-- The users to include in the group summary
91		CREATE TABLE group_summary_users (
92		group_id TEXT NOT NULL,
93		user_id TEXT NOT NULL,
94		role_id TEXT NOT NULL,
95		user_order BIGINT NOT NULL,
96		is_public BOOLEAN NOT NULL -- whether the user should be show to everyone
97		);
98
99		CREATE INDEX group_summary_users_g_idx ON group_summary_users(group_id);
100
101		-- The roles to include in the group summary
102		CREATE TABLE group_summary_roles (
103		group_id TEXT NOT NULL,
104		role_id TEXT NOT NULL,
105		role_order BIGINT NOT NULL,
106		UNIQUE (group_id, role_id, role_order),
107		CHECK (role_order > 0)
108		);
109
110
111		-- The roles in a groups
112		CREATE TABLE group_roles (
113		group_id TEXT NOT NULL,
114		role_id TEXT NOT NULL,
115		profile TEXT NOT NULL,
116		is_public BOOLEAN NOT NULL, -- whether the role should be show to everyone
117		UNIQUE (group_id, role_id)
118		);
119
120
121		-- List of attestations we've given out and need to renew
122		CREATE TABLE group_attestations_renewals (
123		group_id TEXT NOT NULL,
124		user_id TEXT NOT NULL,
125		valid_until_ms BIGINT NOT NULL
126		);
127
128		CREATE INDEX group_attestations_renewals_g_idx ON group_attestations_renewals(group_id, user_id);
129		CREATE INDEX group_attestations_renewals_u_idx ON group_attestations_renewals(user_id);
130		CREATE INDEX group_attestations_renewals_v_idx ON group_attestations_renewals(valid_until_ms);
131
132
133		-- List of attestations we've received from remotes and are interested in.
134		CREATE TABLE group_attestations_remote (
135		group_id TEXT NOT NULL,
136		user_id TEXT NOT NULL,
137		valid_until_ms BIGINT NOT NULL,
138		attestation_json TEXT NOT NULL
139		);
140
141		CREATE INDEX group_attestations_remote_g_idx ON group_attestations_remote(group_id, user_id);
142		CREATE INDEX group_attestations_remote_u_idx ON group_attestations_remote(user_id);
143		CREATE INDEX group_attestations_remote_v_idx ON group_attestations_remote(valid_until_ms);
144
145
146		-- The group membership for the HS's users
147		CREATE TABLE local_group_membership (
148		group_id TEXT NOT NULL,
149		user_id TEXT NOT NULL,
150		is_admin BOOLEAN NOT NULL,
151		membership TEXT NOT NULL,
152		is_publicised BOOLEAN NOT NULL, -- if the user is publicising their membership
153		content TEXT NOT NULL
154		);
155
156		CREATE INDEX local_group_membership_u_idx ON local_group_membership(user_id, group_id);
157		CREATE INDEX local_group_membership_g_idx ON local_group_membership(group_id);
158
159
160		CREATE TABLE local_group_updates (
161		stream_id BIGINT NOT NULL,
162		group_id TEXT NOT NULL,
163		user_id TEXT NOT NULL,
164		type TEXT NOT NULL,
165		content TEXT NOT NULL
166		);

+0

-28

~~synapse/storage/data_stores/main/schema/delta/45/profile_cache.sql~~ less more

0		/* Copyright 2017 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		-- A subset of remote users whose profiles we have cached.
17		-- Whether a user is in this table or not is defined by the storage function
18		-- `is_subscribed_remote_profile_for_user`
19		CREATE TABLE remote_profile_cache (
20		user_id TEXT NOT NULL,
21		displayname TEXT,
22		avatar_url TEXT,
23		last_check BIGINT NOT NULL
24		);
25
26		CREATE UNIQUE INDEX remote_profile_cache_user_id ON remote_profile_cache(user_id);
27		CREATE INDEX remote_profile_cache_time ON remote_profile_cache(last_check);

+0

-17

~~synapse/storage/data_stores/main/schema/delta/46/drop_refresh_tokens.sql~~ less more

0		/* Copyright 2017 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* we no longer use (or create) the refresh_tokens table */
16		DROP TABLE IF EXISTS refresh_tokens;

+0

-35

~~synapse/storage/data_stores/main/schema/delta/46/drop_unique_deleted_pushers.sql~~ less more

0		/* Copyright 2017 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- drop the unique constraint on deleted_pushers so that we can just insert
16		-- into it rather than upserting.
17
18		CREATE TABLE deleted_pushers2 (
19		stream_id BIGINT NOT NULL,
20		app_id TEXT NOT NULL,
21		pushkey TEXT NOT NULL,
22		user_id TEXT NOT NULL
23		);
24
25		INSERT INTO deleted_pushers2 (stream_id, app_id, pushkey, user_id)
26		SELECT stream_id, app_id, pushkey, user_id from deleted_pushers;
27
28		DROP TABLE deleted_pushers;
29		ALTER TABLE deleted_pushers2 RENAME TO deleted_pushers;
30
31		-- create the index after doing the inserts because that's more efficient.
32		-- it also means we can give it the same name as the old one without renaming.
33		CREATE INDEX deleted_pushers_stream_id ON deleted_pushers (stream_id);
34

+0

-32

~~synapse/storage/data_stores/main/schema/delta/46/group_server.sql~~ less more

0		/* Copyright 2017 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE groups_new (
16		group_id TEXT NOT NULL,
17		name TEXT, -- the display name of the room
18		avatar_url TEXT,
19		short_description TEXT,
20		long_description TEXT,
21		is_public BOOL NOT NULL -- whether non-members can access group APIs
22		);
23
24		-- NB: awful hack to get the default to be true on postgres and 1 on sqlite
25		INSERT INTO groups_new
26		SELECT group_id, name, avatar_url, short_description, long_description, (1=1) FROM groups;
27
28		DROP TABLE groups;
29		ALTER TABLE groups_new RENAME TO groups;
30
31		CREATE UNIQUE INDEX groups_idx ON groups(group_id);

+0

-24

~~synapse/storage/data_stores/main/schema/delta/46/local_media_repository_url_idx.sql~~ less more

0		/* Copyright 2017 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- register a background update which will recreate the
16		-- local_media_repository_url_idx index.
17		--
18		-- We do this as a bg update not because it is a particularly onerous
19		-- operation, but because we'd like it to be a partial index if possible, and
20		-- the background_index_update code will understand whether we are on
21		-- postgres or sqlite and behave accordingly.
22		INSERT INTO background_updates (update_name, progress_json) VALUES
23		('local_media_repository_url_idx', '{}');

+0

-35

~~synapse/storage/data_stores/main/schema/delta/46/user_dir_null_room_ids.sql~~ less more

0		/* Copyright 2017 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- change the user_directory table to also cover global local user profiles
16		-- rather than just profiles within specific rooms.
17
18		CREATE TABLE user_directory2 (
19		user_id TEXT NOT NULL,
20		room_id TEXT,
21		display_name TEXT,
22		avatar_url TEXT
23		);
24
25		INSERT INTO user_directory2(user_id, room_id, display_name, avatar_url)
26		SELECT user_id, room_id, display_name, avatar_url from user_directory;
27
28		DROP TABLE user_directory;
29		ALTER TABLE user_directory2 RENAME TO user_directory;
30
31		-- create indexes after doing the inserts because that's more efficient.
32		-- it also means we can give it the same name as the old one without renaming.
33		CREATE INDEX user_directory_room_idx ON user_directory(room_id);
34		CREATE UNIQUE INDEX user_directory_user_idx ON user_directory(user_id);

+0

-24

~~synapse/storage/data_stores/main/schema/delta/46/user_dir_typos.sql~~ less more

0		/* Copyright 2017 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- this is just embarassing :\|
16		ALTER TABLE users_in_pubic_room RENAME TO users_in_public_rooms;
17
18		-- this is only 300K rows on matrix.org and takes ~3s to generate the index,
19		-- so is hopefully not going to block anyone else for that long...
20		CREATE INDEX users_in_public_rooms_room_idx ON users_in_public_rooms(room_id);
21		CREATE UNIQUE INDEX users_in_public_rooms_user_idx ON users_in_public_rooms(user_id);
22		DROP INDEX users_in_pubic_room_room_idx;
23		DROP INDEX users_in_pubic_room_user_idx;

+0

-16

~~synapse/storage/data_stores/main/schema/delta/47/last_access_media.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE local_media_repository ADD COLUMN last_access_ts BIGINT;

+0

-17

~~synapse/storage/data_stores/main/schema/delta/47/postgres_fts_gin.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT into background_updates (update_name, progress_json)
16		VALUES ('event_search_postgres_gin', '{}');

+0

-28

~~synapse/storage/data_stores/main/schema/delta/47/push_actions_staging.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Temporary staging area for push actions that have been calculated for an
16		-- event, but the event hasn't yet been persisted.
17		-- When the event is persisted the rows are moved over to the
18		-- event_push_actions table.
19		CREATE TABLE event_push_actions_staging (
20		event_id TEXT NOT NULL,
21		user_id TEXT NOT NULL,
22		actions TEXT NOT NULL,
23		notif SMALLINT NOT NULL,
24		highlight SMALLINT NOT NULL
25		);
26
27		CREATE INDEX event_push_actions_staging_id ON event_push_actions_staging(event_id);

+0

-18

~~synapse/storage/data_stores/main/schema/delta/48/add_user_consent.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* record the version of the privacy policy the user has consented to
16		*/
17		ALTER TABLE users ADD COLUMN consent_version TEXT;

+0

-17

~~synapse/storage/data_stores/main/schema/delta/48/add_user_ips_last_seen_index.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT into background_updates (update_name, progress_json)
16		VALUES ('user_ips_last_seen_index', '{}');

+0

-25

~~synapse/storage/data_stores/main/schema/delta/48/deactivated_users.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/*
16		* Store any accounts that have been requested to be deactivated.
17		* We part the account from all the rooms its in when its
18		* deactivated. This can take some time and synapse may be restarted
19		* before it completes, so store the user IDs here until the process
20		* is complete.
21		*/
22		CREATE TABLE users_pending_deactivation (
23		user_id TEXT NOT NULL
24		);

+0

-63

~~synapse/storage/data_stores/main/schema/delta/48/group_unique_indexes.py~~ less more

0		# Copyright 2018 New Vector Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		from synapse.storage.engines import PostgresEngine
15		from synapse.storage.prepare_database import get_statements
16
17		FIX_INDEXES = """
18		-- rebuild indexes as uniques
19		DROP INDEX groups_invites_g_idx;
20		CREATE UNIQUE INDEX group_invites_g_idx ON group_invites(group_id, user_id);
21		DROP INDEX groups_users_g_idx;
22		CREATE UNIQUE INDEX group_users_g_idx ON group_users(group_id, user_id);
23
24		-- rename other indexes to actually match their table names..
25		DROP INDEX groups_users_u_idx;
26		CREATE INDEX group_users_u_idx ON group_users(user_id);
27		DROP INDEX groups_invites_u_idx;
28		CREATE INDEX group_invites_u_idx ON group_invites(user_id);
29		DROP INDEX groups_rooms_g_idx;
30		CREATE UNIQUE INDEX group_rooms_g_idx ON group_rooms(group_id, room_id);
31		DROP INDEX groups_rooms_r_idx;
32		CREATE INDEX group_rooms_r_idx ON group_rooms(room_id);
33		"""
34
35
36		def run_create(cur, database_engine, args, *kwargs):
37		rowid = "ctid" if isinstance(database_engine, PostgresEngine) else "rowid"
38
39		# remove duplicates from group_users & group_invites tables
40		cur.execute(
41		"""
42		DELETE FROM group_users WHERE %s NOT IN (
43		SELECT min(%s) FROM group_users GROUP BY group_id, user_id
44		);
45		"""
46		% (rowid, rowid)
47		)
48		cur.execute(
49		"""
50		DELETE FROM group_invites WHERE %s NOT IN (
51		SELECT min(%s) FROM group_invites GROUP BY group_id, user_id
52		);
53		"""
54		% (rowid, rowid)
55		)
56
57		for statement in get_statements(FIX_INDEXES.splitlines()):
58		cur.execute(statement)
59
60
61		def run_upgrade(args, *kwargs):
62		pass

+0

-22

~~synapse/storage/data_stores/main/schema/delta/48/groups_joinable.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/*
16		* This isn't a real ENUM because sqlite doesn't support it
17		* and we use a default of NULL for inserted rows and interpret
18		* NULL at the python store level as necessary so that existing
19		* rows are given the correct default policy.
20		*/
21		ALTER TABLE groups ADD COLUMN join_policy TEXT NOT NULL DEFAULT 'invite';

+0

-20

~~synapse/storage/data_stores/main/schema/delta/49/add_user_consent_server_notice_sent.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* record whether we have sent a server notice about consenting to the
16		* privacy policy. Specifically records the version of the policy we sent
17		* a message about.
18		*/
19		ALTER TABLE users ADD COLUMN consent_server_notice_sent TEXT;

+0

-21

~~synapse/storage/data_stores/main/schema/delta/49/add_user_daily_visits.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		CREATE TABLE user_daily_visits ( user_id TEXT NOT NULL,
17		device_id TEXT,
18		timestamp BIGINT NOT NULL );
19		CREATE INDEX user_daily_visits_uts_idx ON user_daily_visits(user_id, timestamp);
20		CREATE INDEX user_daily_visits_ts_idx ON user_daily_visits(timestamp);

+0

-17

~~synapse/storage/data_stores/main/schema/delta/49/add_user_ips_last_seen_only_index.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT into background_updates (update_name, progress_json)
16		VALUES ('user_ips_last_seen_only_index', '{}');

+0

-19

~~synapse/storage/data_stores/main/schema/delta/50/add_creation_ts_users_index.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16
17		INSERT into background_updates (update_name, progress_json)
18		VALUES ('users_creation_ts', '{}');

+0

-21

~~synapse/storage/data_stores/main/schema/delta/50/erasure_store.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- a table of users who have requested that their details be erased
16		CREATE TABLE erased_users (
17		user_id TEXT NOT NULL
18		);
19
20		CREATE UNIQUE INDEX erased_users_user ON erased_users(user_id);

+0

-96

~~synapse/storage/data_stores/main/schema/delta/50/make_event_content_nullable.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2018 New Vector Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		"""
16		We want to stop populating 'event.content', so we need to make it nullable.
17
18		If this has to be rolled back, then the following should populate the missing data:
19
20		Postgres:
21
22		UPDATE events SET content=(ej.json::json)->'content' FROM event_json ej
23		WHERE ej.event_id = events.event_id AND
24		stream_ordering < (
25		SELECT stream_ordering FROM events WHERE content IS NOT NULL
26		ORDER BY stream_ordering LIMIT 1
27		);
28
29		UPDATE events SET content=(ej.json::json)->'content' FROM event_json ej
30		WHERE ej.event_id = events.event_id AND
31		stream_ordering > (
32		SELECT stream_ordering FROM events WHERE content IS NOT NULL
33		ORDER BY stream_ordering DESC LIMIT 1
34		);
35
36		SQLite:
37
38		UPDATE events SET content=(
39		SELECT json_extract(json,'$.content') FROM event_json ej
40		WHERE ej.event_id = events.event_id
41		)
42		WHERE
43		stream_ordering < (
44		SELECT stream_ordering FROM events WHERE content IS NOT NULL
45		ORDER BY stream_ordering LIMIT 1
46		)
47		OR stream_ordering > (
48		SELECT stream_ordering FROM events WHERE content IS NOT NULL
49		ORDER BY stream_ordering DESC LIMIT 1
50		);
51
52		"""
53
54		import logging
55
56		from synapse.storage.engines import PostgresEngine
57
58		logger = logging.getLogger(__name__)
59
60
61		def run_create(cur, database_engine, args, *kwargs):
62		pass
63
64
65		def run_upgrade(cur, database_engine, args, *kwargs):
66		if isinstance(database_engine, PostgresEngine):
67		cur.execute(
68		"""
69		ALTER TABLE events ALTER COLUMN content DROP NOT NULL;
70		"""
71		)
72		return
73
74		# sqlite is an arse about this. ref: https://www.sqlite.org/lang_altertable.html
75
76		cur.execute(
77		"SELECT sql FROM sqlite_master WHERE tbl_name='events' AND type='table'"
78		)
79		(oldsql,) = cur.fetchone()
80
81		sql = oldsql.replace("content TEXT NOT NULL", "content TEXT")
82		if sql == oldsql:
83		raise Exception("Couldn't find null constraint to drop in %s" % oldsql)
84
85		logger.info("Replacing definition of 'events' with: %s", sql)
86
87		cur.execute("PRAGMA schema_version")
88		(oldver,) = cur.fetchone()
89		cur.execute("PRAGMA writable_schema=ON")
90		cur.execute(
91		"UPDATE sqlite_master SET sql=? WHERE tbl_name='events' AND type='table'",
92		(sql,),
93		)
94		cur.execute("PRAGMA schema_version=%i" % (oldver + 1,))
95		cur.execute("PRAGMA writable_schema=OFF")

+0

-39

~~synapse/storage/data_stores/main/schema/delta/51/e2e_room_keys.sql~~ less more

0		/* Copyright 2017 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- users' optionally backed up encrypted e2e sessions
16		CREATE TABLE e2e_room_keys (
17		user_id TEXT NOT NULL,
18		room_id TEXT NOT NULL,
19		session_id TEXT NOT NULL,
20		version TEXT NOT NULL,
21		first_message_index INT,
22		forwarded_count INT,
23		is_verified BOOLEAN,
24		session_data TEXT NOT NULL
25		);
26
27		CREATE UNIQUE INDEX e2e_room_keys_idx ON e2e_room_keys(user_id, room_id, session_id);
28
29		-- the metadata for each generation of encrypted e2e session backups
30		CREATE TABLE e2e_room_keys_versions (
31		user_id TEXT NOT NULL,
32		version TEXT NOT NULL,
33		algorithm TEXT NOT NULL,
34		auth_data TEXT NOT NULL,
35		deleted SMALLINT DEFAULT 0 NOT NULL
36		);
37
38		CREATE UNIQUE INDEX e2e_room_keys_versions_idx ON e2e_room_keys_versions(user_id, version);

+0

-27

~~synapse/storage/data_stores/main/schema/delta/51/monthly_active_users.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- a table of monthly active users, for use where blocking based on mau limits
16		CREATE TABLE monthly_active_users (
17		user_id TEXT NOT NULL,
18		-- Last time we saw the user. Not guaranteed to be accurate due to rate limiting
19		-- on updates, Granularity of updates governed by
20		-- synapse.storage.monthly_active_users.LAST_SEEN_GRANULARITY
21		-- Measured in ms since epoch.
22		timestamp BIGINT NOT NULL
23		);
24
25		CREATE UNIQUE INDEX monthly_active_users_users ON monthly_active_users(user_id);
26		CREATE INDEX monthly_active_users_time_stamp ON monthly_active_users(timestamp);

+0

-19

~~synapse/storage/data_stores/main/schema/delta/52/add_event_to_state_group_index.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- This is needed to efficiently check for unreferenced state groups during
16		-- purge. Added events_to_state_group(state_group) index
17		INSERT into background_updates (update_name, progress_json)
18		VALUES ('event_to_state_groups_sg_index', '{}');

+0

-36

~~synapse/storage/data_stores/main/schema/delta/52/device_list_streams_unique_idx.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- register a background update which will create a unique index on
16		-- device_lists_remote_cache
17		INSERT into background_updates (update_name, progress_json)
18		VALUES ('device_lists_remote_cache_unique_idx', '{}');
19
20		-- and one on device_lists_remote_extremeties
21		INSERT into background_updates (update_name, progress_json, depends_on)
22		VALUES (
23		'device_lists_remote_extremeties_unique_idx', '{}',
24
25		-- doesn't really depend on this, but we need to make sure both happen
26		-- before we drop the old indexes.
27		'device_lists_remote_cache_unique_idx'
28		);
29
30		-- once they complete, we can drop the old indexes.
31		INSERT into background_updates (update_name, progress_json, depends_on)
32		VALUES (
33		'drop_device_list_streams_non_unique_indexes', '{}',
34		'device_lists_remote_extremeties_unique_idx'
35		);

+0

-53

~~synapse/storage/data_stores/main/schema/delta/52/e2e_room_keys.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* Change version column to an integer so we can do MAX() sensibly
16		*/
17		CREATE TABLE e2e_room_keys_versions_new (
18		user_id TEXT NOT NULL,
19		version BIGINT NOT NULL,
20		algorithm TEXT NOT NULL,
21		auth_data TEXT NOT NULL,
22		deleted SMALLINT DEFAULT 0 NOT NULL
23		);
24
25		INSERT INTO e2e_room_keys_versions_new
26		SELECT user_id, CAST(version as BIGINT), algorithm, auth_data, deleted FROM e2e_room_keys_versions;
27
28		DROP TABLE e2e_room_keys_versions;
29		ALTER TABLE e2e_room_keys_versions_new RENAME TO e2e_room_keys_versions;
30
31		CREATE UNIQUE INDEX e2e_room_keys_versions_idx ON e2e_room_keys_versions(user_id, version);
32
33		/* Change e2e_rooms_keys to match
34		*/
35		CREATE TABLE e2e_room_keys_new (
36		user_id TEXT NOT NULL,
37		room_id TEXT NOT NULL,
38		session_id TEXT NOT NULL,
39		version BIGINT NOT NULL,
40		first_message_index INT,
41		forwarded_count INT,
42		is_verified BOOLEAN,
43		session_data TEXT NOT NULL
44		);
45
46		INSERT INTO e2e_room_keys_new
47		SELECT user_id, room_id, session_id, CAST(version as BIGINT), first_message_index, forwarded_count, is_verified, session_data FROM e2e_room_keys;
48
49		DROP TABLE e2e_room_keys;
50		ALTER TABLE e2e_room_keys_new RENAME TO e2e_room_keys;
51
52		CREATE UNIQUE INDEX e2e_room_keys_idx ON e2e_room_keys(user_id, room_id, session_id);

+0

-19

~~synapse/storage/data_stores/main/schema/delta/53/add_user_type_to_users.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* The type of the user: NULL for a regular user, or one of the constants in
16		* synapse.api.constants.UserTypes
17		*/
18		ALTER TABLE users ADD COLUMN user_type TEXT DEFAULT NULL;

+0

-16

~~synapse/storage/data_stores/main/schema/delta/53/drop_sent_transactions.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		DROP TABLE IF EXISTS sent_transactions;

+0

-16

~~synapse/storage/data_stores/main/schema/delta/53/event_format_version.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE event_json ADD COLUMN format_version INTEGER;

+0

-30

~~synapse/storage/data_stores/main/schema/delta/53/user_dir_populate.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Set up staging tables
16		INSERT INTO background_updates (update_name, progress_json) VALUES
17		('populate_user_directory_createtables', '{}');
18
19		-- Run through each room and update the user directory according to who is in it
20		INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
21		('populate_user_directory_process_rooms', '{}', 'populate_user_directory_createtables');
22
23		-- Insert all users, if search_all_users is on
24		INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
25		('populate_user_directory_process_users', '{}', 'populate_user_directory_process_rooms');
26
27		-- Clean up staging tables
28		INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
29		('populate_user_directory_cleanup', '{}', 'populate_user_directory_process_users');

+0

-30

~~synapse/storage/data_stores/main/schema/delta/53/user_ips_index.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- analyze user_ips, to help ensure the correct indices are used
16		INSERT INTO background_updates (update_name, progress_json) VALUES
17		('user_ips_analyze', '{}');
18
19		-- delete duplicates
20		INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
21		('user_ips_remove_dupes', '{}', 'user_ips_analyze');
22
23		-- add a new unique index to user_ips table
24		INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
25		('user_ips_device_unique_index', '{}', 'user_ips_remove_dupes');
26
27		-- drop the old original index
28		INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
29		('user_ips_drop_nonunique_index', '{}', 'user_ips_device_unique_index');

+0

-44

~~synapse/storage/data_stores/main/schema/delta/53/user_share.sql~~ less more

0		/* Copyright 2017 Vector Creations Ltd, 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Old disused version of the tables below.
16		DROP TABLE IF EXISTS users_who_share_rooms;
17
18		-- Tables keeping track of what users share rooms. This is a map of local users
19		-- to local or remote users, per room. Remote users cannot be in the user_id
20		-- column, only the other_user_id column. There are two tables, one for public
21		-- rooms and those for private rooms.
22		CREATE TABLE IF NOT EXISTS users_who_share_public_rooms (
23		user_id TEXT NOT NULL,
24		other_user_id TEXT NOT NULL,
25		room_id TEXT NOT NULL
26		);
27
28		CREATE TABLE IF NOT EXISTS users_who_share_private_rooms (
29		user_id TEXT NOT NULL,
30		other_user_id TEXT NOT NULL,
31		room_id TEXT NOT NULL
32		);
33
34		CREATE UNIQUE INDEX users_who_share_public_rooms_u_idx ON users_who_share_public_rooms(user_id, other_user_id, room_id);
35		CREATE INDEX users_who_share_public_rooms_r_idx ON users_who_share_public_rooms(room_id);
36		CREATE INDEX users_who_share_public_rooms_o_idx ON users_who_share_public_rooms(other_user_id);
37
38		CREATE UNIQUE INDEX users_who_share_private_rooms_u_idx ON users_who_share_private_rooms(user_id, other_user_id, room_id);
39		CREATE INDEX users_who_share_private_rooms_r_idx ON users_who_share_private_rooms(room_id);
40		CREATE INDEX users_who_share_private_rooms_o_idx ON users_who_share_private_rooms(other_user_id);
41
42		-- Make sure that we populate the tables initially by resetting the stream ID
43		UPDATE user_directory_stream_pos SET stream_id = NULL;

+0

-29

~~synapse/storage/data_stores/main/schema/delta/53/user_threepid_id.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Tracks which identity server a user bound their threepid via.
16		CREATE TABLE user_threepid_id_server (
17		user_id TEXT NOT NULL,
18		medium TEXT NOT NULL,
19		address TEXT NOT NULL,
20		id_server TEXT NOT NULL
21		);
22
23		CREATE UNIQUE INDEX user_threepid_id_server_idx ON user_threepid_id_server(
24		user_id, medium, address, id_server
25		);
26
27		INSERT INTO background_updates (update_name, progress_json) VALUES
28		('user_threepids_grandfather', '{}');

+0

-28

~~synapse/storage/data_stores/main/schema/delta/53/users_in_public_rooms.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- We don't need the old version of this table.
16		DROP TABLE IF EXISTS users_in_public_rooms;
17
18		-- Old version of users_in_public_rooms
19		DROP TABLE IF EXISTS users_who_share_public_rooms;
20
21		-- Track what users are in public rooms.
22		CREATE TABLE IF NOT EXISTS users_in_public_rooms (
23		user_id TEXT NOT NULL,
24		room_id TEXT NOT NULL
25		);
26
27		CREATE UNIQUE INDEX users_in_public_rooms_u_idx ON users_in_public_rooms(user_id, room_id);

+0

-30

~~synapse/storage/data_stores/main/schema/delta/54/account_validity_with_renewal.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- We previously changed the schema for this table without renaming the file, which means
16		-- that some databases might still be using the old schema. This ensures Synapse uses the
17		-- right schema for the table.
18		DROP TABLE IF EXISTS account_validity;
19
20		-- Track what users are in public rooms.
21		CREATE TABLE IF NOT EXISTS account_validity (
22		user_id TEXT PRIMARY KEY,
23		expiration_ts_ms BIGINT NOT NULL,
24		email_sent BOOLEAN NOT NULL,
25		renewal_token TEXT
26		);
27
28		CREATE INDEX account_validity_email_sent_idx ON account_validity(email_sent, expiration_ts_ms)
29		CREATE UNIQUE INDEX account_validity_renewal_string_idx ON account_validity(renewal_token)

+0

-23

~~synapse/storage/data_stores/main/schema/delta/54/add_validity_to_server_keys.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* When we can use this key until, before we have to refresh it. */
16		ALTER TABLE server_signature_keys ADD COLUMN ts_valid_until_ms BIGINT;
17
18		UPDATE server_signature_keys SET ts_valid_until_ms = (
19		SELECT MAX(ts_valid_until_ms) FROM server_keys_json skj WHERE
20		skj.server_name = server_signature_keys.server_name AND
21		skj.key_id = server_signature_keys.key_id
22		);

+0

-23

~~synapse/storage/data_stores/main/schema/delta/54/delete_forward_extremities.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Start a background job to cleanup extremities that were incorrectly added
16		-- by bug #5269.
17		INSERT INTO background_updates (update_name, progress_json) VALUES
18		('delete_soft_failed_extremities', '{}');
19
20		DROP TABLE IF EXISTS _extremities_to_check; -- To make this delta schema file idempotent.
21		CREATE TABLE _extremities_to_check AS SELECT event_id FROM event_forward_extremities;
22		CREATE INDEX _extremities_to_check_id ON _extremities_to_check(event_id);

+0

-30

~~synapse/storage/data_stores/main/schema/delta/54/drop_legacy_tables.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- we need to do this first due to foreign constraints
16		DROP TABLE IF EXISTS application_services_regex;
17
18		DROP TABLE IF EXISTS application_services;
19		DROP TABLE IF EXISTS transaction_id_to_pdu;
20		DROP TABLE IF EXISTS stats_reporting;
21		DROP TABLE IF EXISTS current_state_resets;
22		DROP TABLE IF EXISTS event_content_hashes;
23		DROP TABLE IF EXISTS event_destinations;
24		DROP TABLE IF EXISTS event_edge_hashes;
25		DROP TABLE IF EXISTS event_signatures;
26		DROP TABLE IF EXISTS feedback;
27		DROP TABLE IF EXISTS room_hosts;
28		DROP TABLE IF EXISTS server_tls_certificates;
29		DROP TABLE IF EXISTS state_forward_extremities;

+0

-16

~~synapse/storage/data_stores/main/schema/delta/54/drop_presence_list.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		DROP TABLE IF EXISTS presence_list;

+0

-27

~~synapse/storage/data_stores/main/schema/delta/54/relations.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Tracks related events, like reactions, replies, edits, etc. Note that things
16		-- in this table are not necessarily "valid", e.g. it may contain edits from
17		-- people who don't have power to edit other peoples events.
18		CREATE TABLE IF NOT EXISTS event_relations (
19		event_id TEXT NOT NULL,
20		relates_to_id TEXT NOT NULL,
21		relation_type TEXT NOT NULL,
22		aggregation_key TEXT
23		);
24
25		CREATE UNIQUE INDEX event_relations_id ON event_relations(event_id);
26		CREATE INDEX event_relations_relates ON event_relations(relates_to_id, relation_type, aggregation_key);

+0

-80

~~synapse/storage/data_stores/main/schema/delta/54/stats.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE stats_stream_pos (
16		Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
17		stream_id BIGINT,
18		CHECK (Lock='X')
19		);
20
21		INSERT INTO stats_stream_pos (stream_id) VALUES (null);
22
23		CREATE TABLE user_stats (
24		user_id TEXT NOT NULL,
25		ts BIGINT NOT NULL,
26		bucket_size INT NOT NULL,
27		public_rooms INT NOT NULL,
28		private_rooms INT NOT NULL
29		);
30
31		CREATE UNIQUE INDEX user_stats_user_ts ON user_stats(user_id, ts);
32
33		CREATE TABLE room_stats (
34		room_id TEXT NOT NULL,
35		ts BIGINT NOT NULL,
36		bucket_size INT NOT NULL,
37		current_state_events INT NOT NULL,
38		joined_members INT NOT NULL,
39		invited_members INT NOT NULL,
40		left_members INT NOT NULL,
41		banned_members INT NOT NULL,
42		state_events INT NOT NULL
43		);
44
45		CREATE UNIQUE INDEX room_stats_room_ts ON room_stats(room_id, ts);
46
47		-- cache of current room state; useful for the publicRooms list
48		CREATE TABLE room_state (
49		room_id TEXT NOT NULL,
50		join_rules TEXT,
51		history_visibility TEXT,
52		encryption TEXT,
53		name TEXT,
54		topic TEXT,
55		avatar TEXT,
56		canonical_alias TEXT
57		-- get aliases straight from the right table
58		);
59
60		CREATE UNIQUE INDEX room_state_room ON room_state(room_id);
61
62		CREATE TABLE room_stats_earliest_token (
63		room_id TEXT NOT NULL,
64		token BIGINT NOT NULL
65		);
66
67		CREATE UNIQUE INDEX room_stats_earliest_token_idx ON room_stats_earliest_token(room_id);
68
69		-- Set up staging tables
70		INSERT INTO background_updates (update_name, progress_json) VALUES
71		('populate_stats_createtables', '{}');
72
73		-- Run through each room and update stats
74		INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
75		('populate_stats_process_rooms', '{}', 'populate_stats_createtables');
76
77		-- Clean up staging tables
78		INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
79		('populate_stats_cleanup', '{}', 'populate_stats_process_rooms');

+0

-28

~~synapse/storage/data_stores/main/schema/delta/54/stats2.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- This delta file gets run after `54/stats.sql` delta.
16
17		-- We want to add some indices to the temporary stats table, so we re-insert
18		-- 'populate_stats_createtables' if we are still processing the rooms update.
19		INSERT INTO background_updates (update_name, progress_json)
20		SELECT 'populate_stats_createtables', '{}'
21		WHERE
22		'populate_stats_process_rooms' IN (
23		SELECT update_name FROM background_updates
24		)
25		AND 'populate_stats_createtables' NOT IN ( -- don't insert if already exists
26		SELECT update_name FROM background_updates
27		);

+0

-18

~~synapse/storage/data_stores/main/schema/delta/55/access_token_expiry.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- when this access token can be used until, in ms since the epoch. NULL means the token
16		-- never expires.
17		ALTER TABLE access_tokens ADD COLUMN valid_until_ms BIGINT;

+0

-31

~~synapse/storage/data_stores/main/schema/delta/55/track_threepid_validations.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14		CREATE TABLE IF NOT EXISTS threepid_validation_session (
15		session_id TEXT PRIMARY KEY,
16		medium TEXT NOT NULL,
17		address TEXT NOT NULL,
18		client_secret TEXT NOT NULL,
19		last_send_attempt BIGINT NOT NULL,
20		validated_at BIGINT
21		);
22
23		CREATE TABLE IF NOT EXISTS threepid_validation_token (
24		token TEXT PRIMARY KEY,
25		session_id TEXT NOT NULL,
26		next_link TEXT,
27		expires BIGINT NOT NULL
28		);
29
30		CREATE INDEX threepid_validation_token_session_id ON threepid_validation_token(session_id);

+0

-19

~~synapse/storage/data_stores/main/schema/delta/55/users_alter_deactivated.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE users ADD deactivated SMALLINT DEFAULT 0 NOT NULL;
16
17		INSERT INTO background_updates (update_name, progress_json) VALUES
18		('users_set_deactivated_flag', '{}');

+0

-20

~~synapse/storage/data_stores/main/schema/delta/56/add_spans_to_device_lists.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/*
16		* Opentracing context data for inclusion in the device_list_update EDUs, as a
17		* json-encoded dictionary. NULL if opentracing is disabled (or not enabled for this destination).
18		*/
19		ALTER TABLE device_lists_outbound_pokes ADD opentracing_context TEXT;

+0

-22

~~synapse/storage/data_stores/main/schema/delta/56/current_state_events_membership.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- We add membership to current state so that we don't need to join against
16		-- room_memberships, which can be surprisingly costly (we do such queries
17		-- very frequently).
18		-- This will be null for non-membership events and the content.membership key
19		-- for membership events. (Will also be null for membership events until the
20		-- background update job has finished).
21		ALTER TABLE current_state_events ADD membership TEXT;

+0

-24

~~synapse/storage/data_stores/main/schema/delta/56/current_state_events_membership_mk2.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- We add membership to current state so that we don't need to join against
16		-- room_memberships, which can be surprisingly costly (we do such queries
17		-- very frequently).
18		-- This will be null for non-membership events and the content.membership key
19		-- for membership events. (Will also be null for membership events until the
20		-- background update job has finished).
21
22		INSERT INTO background_updates (update_name, progress_json) VALUES
23		('current_state_events_membership', '{}');

+0

-25

~~synapse/storage/data_stores/main/schema/delta/56/delete_keys_from_deleted_backups.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* delete room keys that belong to deleted room key version, or to room key
16		* versions that don't exist (anymore)
17		*/
18		DELETE FROM e2e_room_keys
19		WHERE version NOT IN (
20		SELECT version
21		FROM e2e_room_keys_versions
22		WHERE e2e_room_keys.user_id = e2e_room_keys_versions.user_id
23		AND e2e_room_keys_versions.deleted = 0
24		);

+0

-25

~~synapse/storage/data_stores/main/schema/delta/56/destinations_failure_ts.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/*
16		* Record the timestamp when a given server started failing
17		*/
18		ALTER TABLE destinations ADD failure_ts BIGINT;
19
20		/* as a rough approximation, we assume that the server started failing at
21		* retry_interval before the last retry
22		*/
23		UPDATE destinations SET failure_ts = retry_last_ts - retry_interval
24		WHERE retry_last_ts > 0;

+0

-18

~~synapse/storage/data_stores/main/schema/delta/56/destinations_retry_interval_type.sql.postgres~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- We want to store large retry intervals so we upgrade the column from INT
16		-- to BIGINT. We don't need to do this on SQLite.
17		ALTER TABLE destinations ALTER retry_interval SET DATA TYPE BIGINT;

+0

-20

~~synapse/storage/data_stores/main/schema/delta/56/device_stream_id_insert.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- This line already existed in deltas/35/device_stream_id but was not included in the
16		-- 54 full schema SQL. Add some SQL here to insert the missing row if it does not exist
17		INSERT INTO device_max_stream_id (stream_id) SELECT 0 WHERE NOT EXISTS (
18		SELECT * from device_max_stream_id
19		);⏎

+0

-24

~~synapse/storage/data_stores/main/schema/delta/56/devices_last_seen.sql~~ less more

0		/* Copyright 2019 Matrix.org Foundation CIC
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Track last seen information for a device in the devices table, rather
16		-- than relying on it being in the user_ips table (which we want to be able
17		-- to purge old entries from)
18		ALTER TABLE devices ADD COLUMN last_seen BIGINT;
19		ALTER TABLE devices ADD COLUMN ip TEXT;
20		ALTER TABLE devices ADD COLUMN user_agent TEXT;
21
22		INSERT INTO background_updates (update_name, progress_json) VALUES
23		('devices_last_seen', '{}');

+0

-20

~~synapse/storage/data_stores/main/schema/delta/56/drop_unused_event_tables.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- these tables are never used.
16		DROP TABLE IF EXISTS room_names;
17		DROP TABLE IF EXISTS topics;
18		DROP TABLE IF EXISTS history_visibility;
19		DROP TABLE IF EXISTS guest_access;

+0

-21

~~synapse/storage/data_stores/main/schema/delta/56/event_expiry.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS event_expiry (
16		event_id TEXT PRIMARY KEY,
17		expiry_ts BIGINT NOT NULL
18		);
19
20		CREATE INDEX event_expiry_expiry_ts_idx ON event_expiry(expiry_ts);

+0

-30

~~synapse/storage/data_stores/main/schema/delta/56/event_labels.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- room_id and topoligical_ordering are denormalised from the events table in order to
16		-- make the index work.
17		CREATE TABLE IF NOT EXISTS event_labels (
18		event_id TEXT,
19		label TEXT,
20		room_id TEXT NOT NULL,
21		topological_ordering BIGINT NOT NULL,
22		PRIMARY KEY(event_id, label)
23		);
24
25
26		-- This index enables an event pagination looking for a particular label to index the
27		-- event_labels table first, which is much quicker than scanning the events table and then
28		-- filtering by label, if the label is rarely used relative to the size of the room.
29		CREATE INDEX event_labels_room_id_label_idx ON event_labels(room_id, label, topological_ordering);

+0

-17

~~synapse/storage/data_stores/main/schema/delta/56/event_labels_background_update.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT INTO background_updates (update_name, progress_json) VALUES
16		('event_store_labels', '{}');

+0

-18

~~synapse/storage/data_stores/main/schema/delta/56/fix_room_keys_index.sql~~ less more

0		/* Copyright 2019 Matrix.org Foundation CIC
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- version is supposed to be part of the room keys index
16		CREATE UNIQUE INDEX e2e_room_keys_with_version_idx ON e2e_room_keys(user_id, version, room_id, session_id);
17		DROP INDEX IF EXISTS e2e_room_keys_idx;

+0

-18

~~synapse/storage/data_stores/main/schema/delta/56/hidden_devices.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- device list needs to know which ones are "real" devices, and which ones are
16		-- just used to avoid collisions
17		ALTER TABLE devices ADD COLUMN hidden BOOLEAN DEFAULT FALSE;

+0

-42

~~synapse/storage/data_stores/main/schema/delta/56/hidden_devices_fix.sql.sqlite~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* Change the hidden column from a default value of FALSE to a default value of
16		* 0, because sqlite3 prior to 3.23.0 caused the hidden column to contain the
17		* string 'FALSE', which is truthy.
18		*
19		* Since sqlite doesn't allow us to just change the default value, we have to
20		* recreate the table, copy the data, fix the rows that have incorrect data, and
21		* replace the old table with the new table.
22		*/
23
24		CREATE TABLE IF NOT EXISTS devices2 (
25		user_id TEXT NOT NULL,
26		device_id TEXT NOT NULL,
27		display_name TEXT,
28		last_seen BIGINT,
29		ip TEXT,
30		user_agent TEXT,
31		hidden BOOLEAN DEFAULT 0,
32		CONSTRAINT device_uniqueness UNIQUE (user_id, device_id)
33		);
34
35		INSERT INTO devices2 SELECT * FROM devices;
36
37		UPDATE devices2 SET hidden = 0 WHERE hidden = 'FALSE';
38
39		DROP TABLE devices;
40
41		ALTER TABLE devices2 RENAME TO devices;

+0

-29

~~synapse/storage/data_stores/main/schema/delta/56/nuke_empty_communities_from_db.sql~~ less more

0		/* Copyright 2019 Werner Sembach
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Groups/communities now get deleted when the last member leaves. This is a one time cleanup to remove old groups/communities that were already empty before that change was made.
16		DELETE FROM group_attestations_remote WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
17		DELETE FROM group_attestations_renewals WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
18		DELETE FROM group_invites WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
19		DELETE FROM group_roles WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
20		DELETE FROM group_room_categories WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
21		DELETE FROM group_rooms WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
22		DELETE FROM group_summary_roles WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
23		DELETE FROM group_summary_room_categories WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
24		DELETE FROM group_summary_rooms WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
25		DELETE FROM group_summary_users WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
26		DELETE FROM local_group_membership WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
27		DELETE FROM local_group_updates WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
28		DELETE FROM groups WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));

+0

-16

~~synapse/storage/data_stores/main/schema/delta/56/public_room_list_idx.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE INDEX public_room_list_stream_network ON public_room_list_stream (appservice_id, network_id, room_id);

+0

-16

~~synapse/storage/data_stores/main/schema/delta/56/redaction_censor.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE redactions ADD COLUMN have_censored BOOL NOT NULL DEFAULT false;

+0

-22

~~synapse/storage/data_stores/main/schema/delta/56/redaction_censor2.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		ALTER TABLE redactions ADD COLUMN received_ts BIGINT;
16
17		INSERT INTO background_updates (update_name, progress_json) VALUES
18		('redactions_received_ts', '{}');
19
20		INSERT INTO background_updates (update_name, progress_json) VALUES
21		('redactions_have_censored_ts_idx', '{}');

+0

-25

~~synapse/storage/data_stores/main/schema/delta/56/redaction_censor3_fix_update.sql.postgres~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		-- There was a bug where we may have updated censored redactions as bytes,
17		-- which can (somehow) cause json to be inserted hex encoded. These updates go
18		-- and undoes any such hex encoded JSON.
19
20		INSERT into background_updates (update_name, progress_json)
21		VALUES ('event_fix_redactions_bytes_create_index', '{}');
22
23		INSERT into background_updates (update_name, progress_json, depends_on)
24		VALUES ('event_fix_redactions_bytes', '{}', 'event_fix_redactions_bytes_create_index');

+0

-16

~~synapse/storage/data_stores/main/schema/delta/56/redaction_censor4.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		DROP INDEX IF EXISTS redactions_have_censored;

+0

-18

~~synapse/storage/data_stores/main/schema/delta/56/remove_tombstoned_rooms_from_directory.sql~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Now that #6232 is a thing, we can remove old rooms from the directory.
16		INSERT INTO background_updates (update_name, progress_json) VALUES
17		('remove_tombstoned_rooms_from_directory', '{}');

+0

-17

~~synapse/storage/data_stores/main/schema/delta/56/room_key_etag.sql~~ less more

0		/* Copyright 2019 Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- store the current etag of backup version
16		ALTER TABLE e2e_room_keys_versions ADD COLUMN etag BIGINT;

+0

-18

~~synapse/storage/data_stores/main/schema/delta/56/room_membership_idx.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Adds an index on room_memberships for fetching all forgotten rooms for a user
16		INSERT INTO background_updates (update_name, progress_json) VALUES
17		('room_membership_forgotten_idx', '{}');

+0

-33

~~synapse/storage/data_stores/main/schema/delta/56/room_retention.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Tracks the retention policy of a room.
16		-- A NULL max_lifetime or min_lifetime means that the matching property is not defined in
17		-- the room's retention policy state event.
18		-- If a room doesn't have a retention policy state event in its state, both max_lifetime
19		-- and min_lifetime are NULL.
20		CREATE TABLE IF NOT EXISTS room_retention(
21		room_id TEXT,
22		event_id TEXT,
23		min_lifetime BIGINT,
24		max_lifetime BIGINT,
25
26		PRIMARY KEY(room_id, event_id)
27		);
28
29		CREATE INDEX room_retention_max_lifetime_idx on room_retention(max_lifetime);
30
31		INSERT INTO background_updates (update_name, progress_json) VALUES
32		('insert_room_retention', '{}');

+0

-56

~~synapse/storage/data_stores/main/schema/delta/56/signing_keys.sql~~ less more

0		/* Copyright 2019 New Vector Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- cross-signing keys
16		CREATE TABLE IF NOT EXISTS e2e_cross_signing_keys (
17		user_id TEXT NOT NULL,
18		-- the type of cross-signing key (master, user_signing, or self_signing)
19		keytype TEXT NOT NULL,
20		-- the full key information, as a json-encoded dict
21		keydata TEXT NOT NULL,
22		-- for keeping the keys in order, so that we can fetch the latest one
23		stream_id BIGINT NOT NULL
24		);
25
26		CREATE UNIQUE INDEX e2e_cross_signing_keys_idx ON e2e_cross_signing_keys(user_id, keytype, stream_id);
27
28		-- cross-signing signatures
29		CREATE TABLE IF NOT EXISTS e2e_cross_signing_signatures (
30		-- user who did the signing
31		user_id TEXT NOT NULL,
32		-- key used to sign
33		key_id TEXT NOT NULL,
34		-- user who was signed
35		target_user_id TEXT NOT NULL,
36		-- device/key that was signed
37		target_device_id TEXT NOT NULL,
38		-- the actual signature
39		signature TEXT NOT NULL
40		);
41
42		-- replaced by the index created in signing_keys_nonunique_signatures.sql
43		-- CREATE UNIQUE INDEX e2e_cross_signing_signatures_idx ON e2e_cross_signing_signatures(user_id, target_user_id, target_device_id);
44
45		-- stream of user signature updates
46		CREATE TABLE IF NOT EXISTS user_signature_stream (
47		-- uses the same stream ID as device list stream
48		stream_id BIGINT NOT NULL,
49		-- user who did the signing
50		from_user_id TEXT NOT NULL,
51		-- list of users who were signed, as a JSON array
52		user_ids TEXT NOT NULL
53		);
54
55		CREATE UNIQUE INDEX user_signature_stream_idx ON user_signature_stream(stream_id);

+0

-22

~~synapse/storage/data_stores/main/schema/delta/56/signing_keys_nonunique_signatures.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* The cross-signing signatures index should not be a unique index, because a
16		* user may upload multiple signatures for the same target user. The previous
17		* index was unique, so delete it if it's there and create a new non-unique
18		* index. */
19
20		DROP INDEX IF EXISTS e2e_cross_signing_signatures_idx; CREATE INDEX IF NOT
21		EXISTS e2e_cross_signing_signatures2_idx ON e2e_cross_signing_signatures(user_id, target_user_id, target_device_id);

+0

-156

~~synapse/storage/data_stores/main/schema/delta/56/stats_separated.sql~~ less more

0		/* Copyright 2018 New Vector Ltd
1		* Copyright 2019 The Matrix.org Foundation C.I.C.
2		*
3		* Licensed under the Apache License, Version 2.0 (the "License");
4		* you may not use this file except in compliance with the License.
5		* You may obtain a copy of the License at
6		*
7		* http://www.apache.org/licenses/LICENSE-2.0
8		*
9		* Unless required by applicable law or agreed to in writing, software
10		* distributed under the License is distributed on an "AS IS" BASIS,
11		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		* See the License for the specific language governing permissions and
13		* limitations under the License.
14		*/
15
16
17		----- First clean up from previous versions of room stats.
18
19		-- First remove old stats stuff
20		DROP TABLE IF EXISTS room_stats;
21		DROP TABLE IF EXISTS room_state;
22		DROP TABLE IF EXISTS room_stats_state;
23		DROP TABLE IF EXISTS user_stats;
24		DROP TABLE IF EXISTS room_stats_earliest_tokens;
25		DROP TABLE IF EXISTS _temp_populate_stats_position;
26		DROP TABLE IF EXISTS _temp_populate_stats_rooms;
27		DROP TABLE IF EXISTS stats_stream_pos;
28
29		-- Unschedule old background updates if they're still scheduled
30		DELETE FROM background_updates WHERE update_name IN (
31		'populate_stats_createtables',
32		'populate_stats_process_rooms',
33		'populate_stats_process_users',
34		'populate_stats_cleanup'
35		);
36
37		-- this relies on current_state_events.membership having been populated, so add
38		-- a dependency on current_state_events_membership.
39		INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
40		('populate_stats_process_rooms', '{}', 'current_state_events_membership');
41
42		-- this also relies on current_state_events.membership having been populated, but
43		-- we get that as a side-effect of depending on populate_stats_process_rooms.
44		INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
45		('populate_stats_process_users', '{}', 'populate_stats_process_rooms');
46
47		----- Create tables for our version of room stats.
48
49		-- single-row table to track position of incremental updates
50		DROP TABLE IF EXISTS stats_incremental_position;
51		CREATE TABLE stats_incremental_position (
52		Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
53		stream_id BIGINT NOT NULL,
54		CHECK (Lock='X')
55		);
56
57		-- insert a null row and make sure it is the only one.
58		INSERT INTO stats_incremental_position (
59		stream_id
60		) SELECT COALESCE(MAX(stream_ordering), 0) from events;
61
62		-- represents PRESENT room statistics for a room
63		-- only holds absolute fields
64		DROP TABLE IF EXISTS room_stats_current;
65		CREATE TABLE room_stats_current (
66		room_id TEXT NOT NULL PRIMARY KEY,
67
68		-- These are absolute counts
69		current_state_events INT NOT NULL,
70		joined_members INT NOT NULL,
71		invited_members INT NOT NULL,
72		left_members INT NOT NULL,
73		banned_members INT NOT NULL,
74
75		local_users_in_room INT NOT NULL,
76
77		-- The maximum delta stream position that this row takes into account.
78		completed_delta_stream_id BIGINT NOT NULL
79		);
80
81
82		-- represents HISTORICAL room statistics for a room
83		DROP TABLE IF EXISTS room_stats_historical;
84		CREATE TABLE room_stats_historical (
85		room_id TEXT NOT NULL,
86		-- These stats cover the time from (end_ts - bucket_size)...end_ts (in ms).
87		-- Note that end_ts is quantised.
88		end_ts BIGINT NOT NULL,
89		bucket_size BIGINT NOT NULL,
90
91		-- These stats are absolute counts
92		current_state_events BIGINT NOT NULL,
93		joined_members BIGINT NOT NULL,
94		invited_members BIGINT NOT NULL,
95		left_members BIGINT NOT NULL,
96		banned_members BIGINT NOT NULL,
97		local_users_in_room BIGINT NOT NULL,
98
99		-- These stats are per time slice
100		total_events BIGINT NOT NULL,
101		total_event_bytes BIGINT NOT NULL,
102
103		PRIMARY KEY (room_id, end_ts)
104		);
105
106		-- We use this index to speed up deletion of ancient room stats.
107		CREATE INDEX room_stats_historical_end_ts ON room_stats_historical (end_ts);
108
109		-- represents PRESENT statistics for a user
110		-- only holds absolute fields
111		DROP TABLE IF EXISTS user_stats_current;
112		CREATE TABLE user_stats_current (
113		user_id TEXT NOT NULL PRIMARY KEY,
114
115		joined_rooms BIGINT NOT NULL,
116
117		-- The maximum delta stream position that this row takes into account.
118		completed_delta_stream_id BIGINT NOT NULL
119		);
120
121		-- represents HISTORICAL statistics for a user
122		DROP TABLE IF EXISTS user_stats_historical;
123		CREATE TABLE user_stats_historical (
124		user_id TEXT NOT NULL,
125		end_ts BIGINT NOT NULL,
126		bucket_size BIGINT NOT NULL,
127
128		joined_rooms BIGINT NOT NULL,
129
130		invites_sent BIGINT NOT NULL,
131		rooms_created BIGINT NOT NULL,
132		total_events BIGINT NOT NULL,
133		total_event_bytes BIGINT NOT NULL,
134
135		PRIMARY KEY (user_id, end_ts)
136		);
137
138		-- We use this index to speed up deletion of ancient user stats.
139		CREATE INDEX user_stats_historical_end_ts ON user_stats_historical (end_ts);
140
141
142		CREATE TABLE room_stats_state (
143		room_id TEXT NOT NULL,
144		name TEXT,
145		canonical_alias TEXT,
146		join_rules TEXT,
147		history_visibility TEXT,
148		encryption TEXT,
149		avatar TEXT,
150		guest_access TEXT,
151		is_federatable BOOLEAN,
152		topic TEXT
153		);
154
155		CREATE UNIQUE INDEX room_stats_state_room ON room_stats_state(room_id);

+0

-52

~~synapse/storage/data_stores/main/schema/delta/56/unique_user_filter_index.py~~ less more

0		import logging
1
2		from synapse.storage.engines import PostgresEngine
3
4		logger = logging.getLogger(__name__)
5
6
7		"""
8		This migration updates the user_filters table as follows:
9
10		- drops any (user_id, filter_id) duplicates
11		- makes the columns NON-NULLable
12		- turns the index into a UNIQUE index
13		"""
14
15
16		def run_upgrade(cur, database_engine, args, *kwargs):
17		pass
18
19
20		def run_create(cur, database_engine, args, *kwargs):
21		if isinstance(database_engine, PostgresEngine):
22		select_clause = """
23		SELECT DISTINCT ON (user_id, filter_id) user_id, filter_id, filter_json
24		FROM user_filters
25		"""
26		else:
27		select_clause = """
28		SELECT * FROM user_filters GROUP BY user_id, filter_id
29		"""
30		sql = """
31		DROP TABLE IF EXISTS user_filters_migration;
32		DROP INDEX IF EXISTS user_filters_unique;
33		CREATE TABLE user_filters_migration (
34		user_id TEXT NOT NULL,
35		filter_id BIGINT NOT NULL,
36		filter_json BYTEA NOT NULL
37		);
38		INSERT INTO user_filters_migration (user_id, filter_id, filter_json)
39		%s;
40		CREATE UNIQUE INDEX user_filters_unique ON user_filters_migration
41		(user_id, filter_id);
42		DROP TABLE user_filters;
43		ALTER TABLE user_filters_migration RENAME TO user_filters;
44		""" % (
45		select_clause,
46		)
47
48		if isinstance(database_engine, PostgresEngine):
49		cur.execute(sql)
50		else:
51		cur.executescript(sql)

+0

-24

~~synapse/storage/data_stores/main/schema/delta/56/user_external_ids.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/*
16		* a table which records mappings from external auth providers to mxids
17		*/
18		CREATE TABLE IF NOT EXISTS user_external_ids (
19		auth_provider TEXT NOT NULL,
20		external_id TEXT NOT NULL,
21		user_id TEXT NOT NULL,
22		UNIQUE (auth_provider, external_id)
23		);

+0

-17

~~synapse/storage/data_stores/main/schema/delta/56/users_in_public_rooms_idx.sql~~ less more

0		/* Copyright 2019 Matrix.org Foundation CIC
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- this was apparently forgotten when the table was created back in delta 53.
16		CREATE INDEX users_in_public_rooms_r_idx ON users_in_public_rooms(room_id);

+0

-22

~~synapse/storage/data_stores/main/schema/delta/57/delete_old_current_state_events.sql~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Add background update to go and delete current state events for rooms the
16		-- server is no longer in.
17		--
18		-- this relies on the 'membership' column of current_state_events, so make sure
19		-- that's populated first!
20		INSERT into background_updates (update_name, progress_json, depends_on)
21		VALUES ('delete_old_current_state_events', '{}', 'current_state_events_membership');

+0

-25

~~synapse/storage/data_stores/main/schema/delta/57/device_list_remote_cache_stale.sql~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- Records whether the server thinks that the remote users cached device lists
16		-- may be out of date (e.g. if we have received a to device message from a
17		-- device we don't know about).
18		CREATE TABLE IF NOT EXISTS device_lists_remote_resync (
19		user_id TEXT NOT NULL,
20		added_ts BIGINT NOT NULL
21		);
22
23		CREATE UNIQUE INDEX device_lists_remote_resync_idx ON device_lists_remote_resync (user_id);
24		CREATE INDEX device_lists_remote_resync_ts_idx ON device_lists_remote_resync (added_ts);

+0

-98

~~synapse/storage/data_stores/main/schema/delta/57/local_current_membership.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2020 New Vector Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15
16		# We create a new table called `local_current_membership` that stores the latest
17		# membership state of local users in rooms, which helps track leaves/bans/etc
18		# even if the server has left the room (and so has deleted the room from
19		# `current_state_events`). This will also include outstanding invites for local
20		# users for rooms the server isn't in.
21		#
22		# If the server isn't and hasn't been in the room then it will only include
23		# outsstanding invites, and not e.g. pre-emptive bans of local users.
24		#
25		# If the server later rejoins a room `local_current_membership` can simply be
26		# replaced with the new current state of the room (which results in the
27		# equivalent behaviour as if the server had remained in the room).
28
29
30		def run_upgrade(cur, database_engine, config, args, *kwargs):
31		# We need to do the insert in `run_upgrade` section as we don't have access
32		# to `config` in `run_create`.
33
34		# This upgrade may take a bit of time for large servers (e.g. one minute for
35		# matrix.org) but means we avoid a lots of book keeping required to do it as
36		# a background update.
37
38		# We check if the `current_state_events.membership` is up to date by
39		# checking if the relevant background update has finished. If it has
40		# finished we can avoid doing a join against `room_memberships`, which
41		# speesd things up.
42		cur.execute(
43		"""SELECT 1 FROM background_updates
44		WHERE update_name = 'current_state_events_membership'
45		"""
46		)
47		current_state_membership_up_to_date = not bool(cur.fetchone())
48
49		# Cheekily drop and recreate indices, as that is faster.
50		cur.execute("DROP INDEX local_current_membership_idx")
51		cur.execute("DROP INDEX local_current_membership_room_idx")
52
53		if current_state_membership_up_to_date:
54		sql = """
55		INSERT INTO local_current_membership (room_id, user_id, event_id, membership)
56		SELECT c.room_id, state_key AS user_id, event_id, c.membership
57		FROM current_state_events AS c
58		WHERE type = 'm.room.member' AND c.membership IS NOT NULL AND state_key LIKE ?
59		"""
60		else:
61		# We can't rely on the membership column, so we need to join against
62		# `room_memberships`.
63		sql = """
64		INSERT INTO local_current_membership (room_id, user_id, event_id, membership)
65		SELECT c.room_id, state_key AS user_id, event_id, r.membership
66		FROM current_state_events AS c
67		INNER JOIN room_memberships AS r USING (event_id)
68		WHERE type = 'm.room.member' AND state_key LIKE ?
69		"""
70		sql = database_engine.convert_param_style(sql)
71		cur.execute(sql, ("%:" + config.server_name,))
72
73		cur.execute(
74		"CREATE UNIQUE INDEX local_current_membership_idx ON local_current_membership(user_id, room_id)"
75		)
76		cur.execute(
77		"CREATE INDEX local_current_membership_room_idx ON local_current_membership(room_id)"
78		)
79
80
81		def run_create(cur, database_engine, args, *kwargs):
82		cur.execute(
83		"""
84		CREATE TABLE local_current_membership (
85		room_id TEXT NOT NULL,
86		user_id TEXT NOT NULL,
87		event_id TEXT NOT NULL,
88		membership TEXT NOT NULL
89		)"""
90		)
91
92		cur.execute(
93		"CREATE UNIQUE INDEX local_current_membership_idx ON local_current_membership(user_id, room_id)"
94		)
95		cur.execute(
96		"CREATE INDEX local_current_membership_room_idx ON local_current_membership(room_id)"
97		)

+0

-21

~~synapse/storage/data_stores/main/schema/delta/57/remove_sent_outbound_pokes.sql~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- we no longer keep sent outbound device pokes in the db; clear them out
16		-- so that we don't have to worry about them.
17		--
18		-- This is a sequence scan, but it doesn't take too long.
19
20		DELETE FROM device_lists_outbound_pokes WHERE sent;

+0

-24

~~synapse/storage/data_stores/main/schema/delta/57/rooms_version_column.sql~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		-- We want to start storing the room version independently of
17		-- `current_state_events` so that we can delete stale entries from it without
18		-- losing the information.
19		ALTER TABLE rooms ADD COLUMN room_version TEXT;
20
21
22		INSERT into background_updates (update_name, progress_json)
23		VALUES ('add_rooms_room_version_column', '{}');

+0

-35

~~synapse/storage/data_stores/main/schema/delta/57/rooms_version_column_2.sql.postgres~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- when we first added the room_version column, it was populated via a background
16		-- update. We now need it to be populated before synapse starts, so we populate
17		-- any remaining rows with a NULL room version now. For servers which have completed
18		-- the background update, this will be pretty quick.
19
20		-- the following query will set room_version to NULL if no create event is found for
21		-- the room in current_state_events, and will set it to '1' if a create event with no
22		-- room_version is found.
23
24		UPDATE rooms SET room_version=(
25		SELECT COALESCE(json::json->'content'->>'room_version','1')
26		FROM current_state_events cse INNER JOIN event_json ej USING (event_id)
27		WHERE cse.room_id=rooms.room_id AND cse.type='m.room.create' AND cse.state_key=''
28		) WHERE rooms.room_version IS NULL;
29
30		-- we still allow the background update to complete: it has the useful side-effect of
31		-- populating `rooms` with any missing rooms (based on the current_state_events table).
32
33		-- see also rooms_version_column_2.sql.sqlite which has a copy of the above query, using
34		-- sqlite syntax for the json extraction.

+0

-22

~~synapse/storage/data_stores/main/schema/delta/57/rooms_version_column_2.sql.sqlite~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- see rooms_version_column_2.sql.postgres for details of what's going on here.
16
17		UPDATE rooms SET room_version=(
18		SELECT COALESCE(json_extract(ej.json, '$.content.room_version'), '1')
19		FROM current_state_events cse INNER JOIN event_json ej USING (event_id)
20		WHERE cse.room_id=rooms.room_id AND cse.type='m.room.create' AND cse.state_key=''
21		) WHERE rooms.room_version IS NULL;

+0

-39

~~synapse/storage/data_stores/main/schema/delta/57/rooms_version_column_3.sql.postgres~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- When we first added the room_version column to the rooms table, it was populated from
16		-- the current_state_events table. However, there was an issue causing a background
17		-- update to clean up the current_state_events table for rooms where the server is no
18		-- longer participating, before that column could be populated. Therefore, some rooms had
19		-- a NULL room_version.
20
21		-- The rooms_version_column_2.sql.* delta files were introduced to make the populating
22		-- synchronous instead of running it in a background update, which fixed this issue.
23		-- However, all of the instances of Synapse installed or updated in the meantime got
24		-- their rooms table corrupted with NULL room_versions.
25
26		-- This query fishes out the room versions from the create event using the state_events
27		-- table instead of the current_state_events one, as the former still have all of the
28		-- create events.
29
30		UPDATE rooms SET room_version=(
31		SELECT COALESCE(json::json->'content'->>'room_version','1')
32		FROM state_events se INNER JOIN event_json ej USING (event_id)
33		WHERE se.room_id=rooms.room_id AND se.type='m.room.create' AND se.state_key=''
34		LIMIT 1
35		) WHERE rooms.room_version IS NULL;
36
37		-- see also rooms_version_column_3.sql.sqlite which has a copy of the above query, using
38		-- sqlite syntax for the json extraction.

+0

-23

~~synapse/storage/data_stores/main/schema/delta/57/rooms_version_column_3.sql.sqlite~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- see rooms_version_column_3.sql.postgres for details of what's going on here.
16
17		UPDATE rooms SET room_version=(
18		SELECT COALESCE(json_extract(ej.json, '$.content.room_version'), '1')
19		FROM state_events se INNER JOIN event_json ej USING (event_id)
20		WHERE se.room_id=rooms.room_id AND se.type='m.room.create' AND se.state_key=''
21		LIMIT 1
22		) WHERE rooms.room_version IS NULL;

+0

-22

~~synapse/storage/data_stores/main/schema/delta/58/02remove_dup_outbound_pokes.sql~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* for some reason, we have accumulated duplicate entries in
16		* device_lists_outbound_pokes, which makes prune_outbound_device_list_pokes less
17		* efficient.
18		*/
19
20		INSERT INTO background_updates (ordering, update_name, progress_json)
21		VALUES (5800, 'remove_dup_outbound_pokes', '{}');

+0

-36

~~synapse/storage/data_stores/main/schema/delta/58/03persist_ui_auth.sql~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS ui_auth_sessions(
16		session_id TEXT NOT NULL, -- The session ID passed to the client.
17		creation_time BIGINT NOT NULL, -- The time this session was created (epoch time in milliseconds).
18		serverdict TEXT NOT NULL, -- A JSON dictionary of arbitrary data added by Synapse.
19		clientdict TEXT NOT NULL, -- A JSON dictionary of arbitrary data from the client.
20		uri TEXT NOT NULL, -- The URI the UI authentication session is using.
21		method TEXT NOT NULL, -- The HTTP method the UI authentication session is using.
22		-- The clientdict, uri, and method make up an tuple that must be immutable
23		-- throughout the lifetime of the UI Auth session.
24		description TEXT NOT NULL, -- A human readable description of the operation which caused the UI Auth flow to occur.
25		UNIQUE (session_id)
26		);
27
28		CREATE TABLE IF NOT EXISTS ui_auth_sessions_credentials(
29		session_id TEXT NOT NULL, -- The corresponding UI Auth session.
30		stage_type TEXT NOT NULL, -- The stage type.
31		result TEXT NOT NULL, -- The result of the stage verification, stored as JSON.
32		UNIQUE (session_id, stage_type),
33		FOREIGN KEY (session_id)
34		REFERENCES ui_auth_sessions (session_id)
35		);

+0

-30

~~synapse/storage/data_stores/main/schema/delta/58/05cache_instance.sql.postgres~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- We keep the old table here to enable us to roll back. It doesn't matter
16		-- that we have dropped all the data here.
17		TRUNCATE cache_invalidation_stream;
18
19		CREATE TABLE cache_invalidation_stream_by_instance (
20		stream_id BIGINT NOT NULL,
21		instance_name TEXT NOT NULL,
22		cache_func TEXT NOT NULL,
23		keys TEXT[],
24		invalidation_ts BIGINT
25		);
26
27		CREATE UNIQUE INDEX cache_invalidation_stream_by_instance_id ON cache_invalidation_stream_by_instance(stream_id);
28
29		CREATE SEQUENCE cache_invalidation_stream_seq;

+0

-80

~~synapse/storage/data_stores/main/schema/delta/58/06dlols_unique_idx.py~~ less more

0		# Copyright 2020 The Matrix.org Foundation C.I.C.
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		"""
15		This migration rebuilds the device_lists_outbound_last_success table without duplicate
16		entries, and with a UNIQUE index.
17		"""
18
19		import logging
20		from io import StringIO
21
22		from synapse.storage.engines import BaseDatabaseEngine, PostgresEngine
23		from synapse.storage.prepare_database import execute_statements_from_stream
24		from synapse.storage.types import Cursor
25
26		logger = logging.getLogger(__name__)
27
28
29		def run_upgrade(args, *kwargs):
30		pass
31
32
33		def run_create(cur: Cursor, database_engine: BaseDatabaseEngine, args, *kwargs):
34		# some instances might already have this index, in which case we can skip this
35		if isinstance(database_engine, PostgresEngine):
36		cur.execute(
37		"""
38		SELECT 1 FROM pg_class WHERE relkind = 'i'
39		AND relname = 'device_lists_outbound_last_success_unique_idx'
40		"""
41		)
42
43		if cur.rowcount:
44		logger.info(
45		"Unique index exists on device_lists_outbound_last_success: "
46		"skipping rebuild"
47		)
48		return
49
50		logger.info("Rebuilding device_lists_outbound_last_success with unique index")
51		execute_statements_from_stream(cur, StringIO(_rebuild_commands))
52
53
54		# there might be duplicates, so the easiest way to achieve this is to create a new
55		# table with the right data, and renaming it into place
56
57		_rebuild_commands = """
58		DROP TABLE IF EXISTS device_lists_outbound_last_success_new;
59
60		CREATE TABLE device_lists_outbound_last_success_new (
61		destination TEXT NOT NULL,
62		user_id TEXT NOT NULL,
63		stream_id BIGINT NOT NULL
64		);
65
66		-- this took about 30 seconds on matrix.org's 16 million rows.
67		INSERT INTO device_lists_outbound_last_success_new
68		SELECT destination, user_id, MAX(stream_id) FROM device_lists_outbound_last_success
69		GROUP BY destination, user_id;
70
71		-- and this another 30 seconds.
72		CREATE UNIQUE INDEX device_lists_outbound_last_success_unique_idx
73		ON device_lists_outbound_last_success_new (destination, user_id);
74
75		DROP TABLE device_lists_outbound_last_success;
76
77		ALTER TABLE device_lists_outbound_last_success_new
78		RENAME TO device_lists_outbound_last_success;
79		"""

+0

-18

~~synapse/storage/data_stores/main/schema/delta/58/08_media_safe_from_quarantine.sql.postgres~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- The local_media_repository should have files which do not get quarantined,
16		-- e.g. files from sticker packs.
17		ALTER TABLE local_media_repository ADD COLUMN safe_from_quarantine BOOLEAN NOT NULL DEFAULT FALSE;

+0

-18

~~synapse/storage/data_stores/main/schema/delta/58/08_media_safe_from_quarantine.sql.sqlite~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- The local_media_repository should have files which do not get quarantined,
16		-- e.g. files from sticker packs.
17		ALTER TABLE local_media_repository ADD COLUMN safe_from_quarantine BOOLEAN NOT NULL DEFAULT 0;

+0

-22

~~synapse/storage/data_stores/main/schema/delta/58/10drop_local_rejections_stream.sql~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/*
16		The version of synapse 1.16.0 on pypi incorrectly contained a migration which
17		added a table called 'local_rejections_stream'. This table is not used, and
18		we drop it here for anyone who was affected.
19		*/
20
21		DROP TABLE IF EXISTS local_rejections_stream;

+0

-22

~~synapse/storage/data_stores/main/schema/delta/58/10federation_pos_instance_name.sql~~ less more

0		/* Copyright 2020 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- We need to store the stream positions by instance in a sharded config world.
16		--
17		-- We default to master as we want the column to be NOT NULL and we correctly
18		-- reset the instance name to match the config each time we start up.
19		ALTER TABLE federation_stream_position ADD COLUMN instance_name TEXT NOT NULL DEFAULT 'master';
20
21		CREATE UNIQUE INDEX federation_stream_position_instance ON federation_stream_position(type, instance_name);

+0

-34

~~synapse/storage/data_stores/main/schema/delta/58/11user_id_seq.py~~ less more

0		# Copyright 2020 The Matrix.org Foundation C.I.C.
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		"""
15		Adds a postgres SEQUENCE for generating guest user IDs.
16		"""
17
18		from synapse.storage.data_stores.main.registration import (
19		find_max_generated_user_id_localpart,
20		)
21		from synapse.storage.engines import PostgresEngine
22
23
24		def run_create(cur, database_engine, args, *kwargs):
25		if not isinstance(database_engine, PostgresEngine):
26		return
27
28		next_id = find_max_generated_user_id_localpart(cur) + 1
29		cur.execute("CREATE SEQUENCE user_id_seq START WITH %s", (next_id,))
30
31
32		def run_upgrade(args, *kwargs):
33		pass

+0

-37

~~synapse/storage/data_stores/main/schema/full_schemas/16/application_services.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* We used to create tables called application_services and
16		* application_services_regex, but these are no longer used and are removed in
17		* delta 54.
18		*/
19
20
21		CREATE TABLE IF NOT EXISTS application_services_state(
22		as_id TEXT PRIMARY KEY,
23		state VARCHAR(5),
24		last_txn INTEGER
25		);
26
27		CREATE TABLE IF NOT EXISTS application_services_txns(
28		as_id TEXT NOT NULL,
29		txn_id INTEGER NOT NULL,
30		event_ids TEXT NOT NULL,
31		UNIQUE(as_id, txn_id)
32		);
33
34		CREATE INDEX application_services_txns_id ON application_services_txns (
35		as_id
36		);

+0

-70

~~synapse/storage/data_stores/main/schema/full_schemas/16/event_edges.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* We used to create tables called event_destinations and
16		* state_forward_extremities, but these are no longer used and are removed in
17		* delta 54.
18		*/
19
20		CREATE TABLE IF NOT EXISTS event_forward_extremities(
21		event_id TEXT NOT NULL,
22		room_id TEXT NOT NULL,
23		UNIQUE (event_id, room_id)
24		);
25
26		CREATE INDEX ev_extrem_room ON event_forward_extremities(room_id);
27		CREATE INDEX ev_extrem_id ON event_forward_extremities(event_id);
28
29
30		CREATE TABLE IF NOT EXISTS event_backward_extremities(
31		event_id TEXT NOT NULL,
32		room_id TEXT NOT NULL,
33		UNIQUE (event_id, room_id)
34		);
35
36		CREATE INDEX ev_b_extrem_room ON event_backward_extremities(room_id);
37		CREATE INDEX ev_b_extrem_id ON event_backward_extremities(event_id);
38
39
40		CREATE TABLE IF NOT EXISTS event_edges(
41		event_id TEXT NOT NULL,
42		prev_event_id TEXT NOT NULL,
43		room_id TEXT NOT NULL,
44		is_state BOOL NOT NULL, -- true if this is a prev_state edge rather than a regular
45		-- event dag edge.
46		UNIQUE (event_id, prev_event_id, room_id, is_state)
47		);
48
49		CREATE INDEX ev_edges_id ON event_edges(event_id);
50		CREATE INDEX ev_edges_prev_id ON event_edges(prev_event_id);
51
52
53		CREATE TABLE IF NOT EXISTS room_depth(
54		room_id TEXT NOT NULL,
55		min_depth INTEGER NOT NULL,
56		UNIQUE (room_id)
57		);
58
59		CREATE INDEX room_depth_room ON room_depth(room_id);
60
61		CREATE TABLE IF NOT EXISTS event_auth(
62		event_id TEXT NOT NULL,
63		auth_id TEXT NOT NULL,
64		room_id TEXT NOT NULL,
65		UNIQUE (event_id, auth_id, room_id)
66		);
67
68		CREATE INDEX evauth_edges_id ON event_auth(event_id);
69		CREATE INDEX evauth_edges_auth_id ON event_auth(auth_id);

+0

-38

~~synapse/storage/data_stores/main/schema/full_schemas/16/event_signatures.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* We used to create tables called event_content_hashes and event_edge_hashes,
16		* but these are no longer used and are removed in delta 54.
17		*/
18
19		CREATE TABLE IF NOT EXISTS event_reference_hashes (
20		event_id TEXT,
21		algorithm TEXT,
22		hash bytea,
23		UNIQUE (event_id, algorithm)
24		);
25
26		CREATE INDEX event_reference_hashes_id ON event_reference_hashes(event_id);
27
28
29		CREATE TABLE IF NOT EXISTS event_signatures (
30		event_id TEXT,
31		signature_name TEXT,
32		key_id TEXT,
33		signature bytea,
34		UNIQUE (event_id, signature_name, key_id)
35		);
36
37		CREATE INDEX event_signatures_id ON event_signatures(event_id);

+0

-120

~~synapse/storage/data_stores/main/schema/full_schemas/16/im.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		/* We used to create tables called room_hosts and feedback,
16		* but these are no longer used and are removed in delta 54.
17		*/
18
19		CREATE TABLE IF NOT EXISTS events(
20		stream_ordering INTEGER PRIMARY KEY,
21		topological_ordering BIGINT NOT NULL,
22		event_id TEXT NOT NULL,
23		type TEXT NOT NULL,
24		room_id TEXT NOT NULL,
25
26		-- 'content' used to be created NULLable, but as of delta 50 we drop that constraint.
27		-- the hack we use to drop the constraint doesn't work for an in-memory sqlite
28		-- database, which breaks the sytests. Hence, we no longer make it nullable.
29		content TEXT,
30
31		unrecognized_keys TEXT,
32		processed BOOL NOT NULL,
33		outlier BOOL NOT NULL,
34		depth BIGINT DEFAULT 0 NOT NULL,
35		UNIQUE (event_id)
36		);
37
38		CREATE INDEX events_stream_ordering ON events (stream_ordering);
39		CREATE INDEX events_topological_ordering ON events (topological_ordering);
40		CREATE INDEX events_order ON events (topological_ordering, stream_ordering);
41		CREATE INDEX events_room_id ON events (room_id);
42		CREATE INDEX events_order_room ON events (
43		room_id, topological_ordering, stream_ordering
44		);
45
46
47		CREATE TABLE IF NOT EXISTS event_json(
48		event_id TEXT NOT NULL,
49		room_id TEXT NOT NULL,
50		internal_metadata TEXT NOT NULL,
51		json TEXT NOT NULL,
52		UNIQUE (event_id)
53		);
54
55		CREATE INDEX event_json_room_id ON event_json(room_id);
56
57
58		CREATE TABLE IF NOT EXISTS state_events(
59		event_id TEXT NOT NULL,
60		room_id TEXT NOT NULL,
61		type TEXT NOT NULL,
62		state_key TEXT NOT NULL,
63		prev_state TEXT,
64		UNIQUE (event_id)
65		);
66
67		CREATE INDEX state_events_room_id ON state_events (room_id);
68		CREATE INDEX state_events_type ON state_events (type);
69		CREATE INDEX state_events_state_key ON state_events (state_key);
70
71
72		CREATE TABLE IF NOT EXISTS current_state_events(
73		event_id TEXT NOT NULL,
74		room_id TEXT NOT NULL,
75		type TEXT NOT NULL,
76		state_key TEXT NOT NULL,
77		UNIQUE (event_id),
78		UNIQUE (room_id, type, state_key)
79		);
80
81		CREATE INDEX current_state_events_room_id ON current_state_events (room_id);
82		CREATE INDEX current_state_events_type ON current_state_events (type);
83		CREATE INDEX current_state_events_state_key ON current_state_events (state_key);
84
85		CREATE TABLE IF NOT EXISTS room_memberships(
86		event_id TEXT NOT NULL,
87		user_id TEXT NOT NULL,
88		sender TEXT NOT NULL,
89		room_id TEXT NOT NULL,
90		membership TEXT NOT NULL,
91		UNIQUE (event_id)
92		);
93
94		CREATE INDEX room_memberships_room_id ON room_memberships (room_id);
95		CREATE INDEX room_memberships_user_id ON room_memberships (user_id);
96
97		CREATE TABLE IF NOT EXISTS topics(
98		event_id TEXT NOT NULL,
99		room_id TEXT NOT NULL,
100		topic TEXT NOT NULL,
101		UNIQUE (event_id)
102		);
103
104		CREATE INDEX topics_room_id ON topics(room_id);
105
106		CREATE TABLE IF NOT EXISTS room_names(
107		event_id TEXT NOT NULL,
108		room_id TEXT NOT NULL,
109		name TEXT NOT NULL,
110		UNIQUE (event_id)
111		);
112
113		CREATE INDEX room_names_room_id ON room_names(room_id);
114
115		CREATE TABLE IF NOT EXISTS rooms(
116		room_id TEXT PRIMARY KEY NOT NULL,
117		is_public BOOL,
118		creator TEXT
119		);

+0

-26

~~synapse/storage/data_stores/main/schema/full_schemas/16/keys.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		-- we used to create a table called server_tls_certificates, but this is no
16		-- longer used, and is removed in delta 54.
17
18		CREATE TABLE IF NOT EXISTS server_signature_keys(
19		server_name TEXT, -- Server name.
20		key_id TEXT, -- Key version.
21		from_server TEXT, -- Which key server the key was fetched form.
22		ts_added_ms BIGINT, -- When the key was added.
23		verify_key bytea, -- NACL verification key.
24		UNIQUE (server_name, key_id)
25		);

+0

-68

~~synapse/storage/data_stores/main/schema/full_schemas/16/media_repository.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS local_media_repository (
16		media_id TEXT, -- The id used to refer to the media.
17		media_type TEXT, -- The MIME-type of the media.
18		media_length INTEGER, -- Length of the media in bytes.
19		created_ts BIGINT, -- When the content was uploaded in ms.
20		upload_name TEXT, -- The name the media was uploaded with.
21		user_id TEXT, -- The user who uploaded the file.
22		UNIQUE (media_id)
23		);
24
25		CREATE TABLE IF NOT EXISTS local_media_repository_thumbnails (
26		media_id TEXT, -- The id used to refer to the media.
27		thumbnail_width INTEGER, -- The width of the thumbnail in pixels.
28		thumbnail_height INTEGER, -- The height of the thumbnail in pixels.
29		thumbnail_type TEXT, -- The MIME-type of the thumbnail.
30		thumbnail_method TEXT, -- The method used to make the thumbnail.
31		thumbnail_length INTEGER, -- The length of the thumbnail in bytes.
32		UNIQUE (
33		media_id, thumbnail_width, thumbnail_height, thumbnail_type
34		)
35		);
36
37		CREATE INDEX local_media_repository_thumbnails_media_id
38		ON local_media_repository_thumbnails (media_id);
39
40		CREATE TABLE IF NOT EXISTS remote_media_cache (
41		media_origin TEXT, -- The remote HS the media came from.
42		media_id TEXT, -- The id used to refer to the media on that server.
43		media_type TEXT, -- The MIME-type of the media.
44		created_ts BIGINT, -- When the content was uploaded in ms.
45		upload_name TEXT, -- The name the media was uploaded with.
46		media_length INTEGER, -- Length of the media in bytes.
47		filesystem_id TEXT, -- The name used to store the media on disk.
48		UNIQUE (media_origin, media_id)
49		);
50
51		CREATE TABLE IF NOT EXISTS remote_media_cache_thumbnails (
52		media_origin TEXT, -- The remote HS the media came from.
53		media_id TEXT, -- The id used to refer to the media.
54		thumbnail_width INTEGER, -- The width of the thumbnail in pixels.
55		thumbnail_height INTEGER, -- The height of the thumbnail in pixels.
56		thumbnail_method TEXT, -- The method used to make the thumbnail
57		thumbnail_type TEXT, -- The MIME-type of the thumbnail.
58		thumbnail_length INTEGER, -- The length of the thumbnail in bytes.
59		filesystem_id TEXT, -- The name used to store the media on disk.
60		UNIQUE (
61		media_origin, media_id, thumbnail_width, thumbnail_height,
62		thumbnail_type
63		)
64		);
65
66		CREATE INDEX remote_media_cache_thumbnails_media_id
67		ON remote_media_cache_thumbnails (media_id);

+0

-32

~~synapse/storage/data_stores/main/schema/full_schemas/16/presence.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14		CREATE TABLE IF NOT EXISTS presence(
15		user_id TEXT NOT NULL,
16		state VARCHAR(20),
17		status_msg TEXT,
18		mtime BIGINT, -- miliseconds since last state change
19		UNIQUE (user_id)
20		);
21
22		-- For each of /my/ users which possibly-remote users are allowed to see their
23		-- presence state
24		CREATE TABLE IF NOT EXISTS presence_allow_inbound(
25		observed_user_id TEXT NOT NULL,
26		observer_user_id TEXT NOT NULL, -- a UserID,
27		UNIQUE (observed_user_id, observer_user_id)
28		);
29
30		-- We used to create a table called presence_list, but this is no longer used
31		-- and is removed in delta 54.⏎

+0

-20

~~synapse/storage/data_stores/main/schema/full_schemas/16/profiles.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14		CREATE TABLE IF NOT EXISTS profiles(
15		user_id TEXT NOT NULL,
16		displayname TEXT,
17		avatar_url TEXT,
18		UNIQUE(user_id)
19		);

+0

-74

~~synapse/storage/data_stores/main/schema/full_schemas/16/push.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS rejections(
16		event_id TEXT NOT NULL,
17		reason TEXT NOT NULL,
18		last_check TEXT NOT NULL,
19		UNIQUE (event_id)
20		);
21
22		-- Push notification endpoints that users have configured
23		CREATE TABLE IF NOT EXISTS pushers (
24		id BIGINT PRIMARY KEY,
25		user_name TEXT NOT NULL,
26		access_token BIGINT DEFAULT NULL,
27		profile_tag VARCHAR(32) NOT NULL,
28		kind VARCHAR(8) NOT NULL,
29		app_id VARCHAR(64) NOT NULL,
30		app_display_name VARCHAR(64) NOT NULL,
31		device_display_name VARCHAR(128) NOT NULL,
32		pushkey bytea NOT NULL,
33		ts BIGINT NOT NULL,
34		lang VARCHAR(8),
35		data bytea,
36		last_token TEXT,
37		last_success BIGINT,
38		failing_since BIGINT,
39		UNIQUE (app_id, pushkey)
40		);
41
42		CREATE TABLE IF NOT EXISTS push_rules (
43		id BIGINT PRIMARY KEY,
44		user_name TEXT NOT NULL,
45		rule_id TEXT NOT NULL,
46		priority_class SMALLINT NOT NULL,
47		priority INTEGER NOT NULL DEFAULT 0,
48		conditions TEXT NOT NULL,
49		actions TEXT NOT NULL,
50		UNIQUE(user_name, rule_id)
51		);
52
53		CREATE INDEX push_rules_user_name on push_rules (user_name);
54
55		CREATE TABLE IF NOT EXISTS user_filters(
56		user_id TEXT,
57		filter_id BIGINT,
58		filter_json bytea
59		);
60
61		CREATE INDEX user_filters_by_user_id_filter_id ON user_filters(
62		user_id, filter_id
63		);
64
65		CREATE TABLE IF NOT EXISTS push_rules_enable (
66		id BIGINT PRIMARY KEY,
67		user_name TEXT NOT NULL,
68		rule_id TEXT NOT NULL,
69		enabled SMALLINT,
70		UNIQUE(user_name, rule_id)
71		);
72
73		CREATE INDEX push_rules_enable_user_name on push_rules_enable (user_name);

+0

-22

~~synapse/storage/data_stores/main/schema/full_schemas/16/redactions.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14		CREATE TABLE IF NOT EXISTS redactions (
15		event_id TEXT NOT NULL,
16		redacts TEXT NOT NULL,
17		UNIQUE (event_id)
18		);
19
20		CREATE INDEX redactions_event_id ON redactions (event_id);
21		CREATE INDEX redactions_redacts ON redactions (redacts);

+0

-29

~~synapse/storage/data_stores/main/schema/full_schemas/16/room_aliases.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS room_aliases(
16		room_alias TEXT NOT NULL,
17		room_id TEXT NOT NULL,
18		UNIQUE (room_alias)
19		);
20
21		CREATE INDEX room_aliases_id ON room_aliases(room_id);
22
23		CREATE TABLE IF NOT EXISTS room_alias_servers(
24		room_alias TEXT NOT NULL,
25		server TEXT NOT NULL
26		);
27
28		CREATE INDEX room_alias_servers_alias ON room_alias_servers(room_alias);

+0

-40

~~synapse/storage/data_stores/main/schema/full_schemas/16/state.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE IF NOT EXISTS state_groups(
16		id BIGINT PRIMARY KEY,
17		room_id TEXT NOT NULL,
18		event_id TEXT NOT NULL
19		);
20
21		CREATE TABLE IF NOT EXISTS state_groups_state(
22		state_group BIGINT NOT NULL,
23		room_id TEXT NOT NULL,
24		type TEXT NOT NULL,
25		state_key TEXT NOT NULL,
26		event_id TEXT NOT NULL
27		);
28
29		CREATE TABLE IF NOT EXISTS event_to_state_groups(
30		event_id TEXT NOT NULL,
31		state_group BIGINT NOT NULL,
32		UNIQUE (event_id)
33		);
34
35		CREATE INDEX state_groups_id ON state_groups(id);
36
37		CREATE INDEX state_groups_state_id ON state_groups_state(state_group);
38		CREATE INDEX state_groups_state_tuple ON state_groups_state(room_id, type, state_key);
39		CREATE INDEX event_to_state_groups_id ON event_to_state_groups(event_id);

+0

-44

~~synapse/storage/data_stores/main/schema/full_schemas/16/transactions.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14		-- Stores what transaction ids we have received and what our response was
15		CREATE TABLE IF NOT EXISTS received_transactions(
16		transaction_id TEXT,
17		origin TEXT,
18		ts BIGINT,
19		response_code INTEGER,
20		response_json bytea,
21		has_been_referenced smallint default 0, -- Whether thishas been referenced by a prev_tx
22		UNIQUE (transaction_id, origin)
23		);
24
25		CREATE INDEX transactions_have_ref ON received_transactions(origin, has_been_referenced);-- WHERE has_been_referenced = 0;
26
27		-- For sent transactions only.
28		CREATE TABLE IF NOT EXISTS transaction_id_to_pdu(
29		transaction_id INTEGER,
30		destination TEXT,
31		pdu_id TEXT,
32		pdu_origin TEXT,
33		UNIQUE (transaction_id, destination)
34		);
35
36		CREATE INDEX transaction_id_to_pdu_dest ON transaction_id_to_pdu(destination);
37
38		-- To track destination health
39		CREATE TABLE IF NOT EXISTS destinations(
40		destination TEXT PRIMARY KEY,
41		retry_last_ts BIGINT,
42		retry_interval INTEGER
43		);

+0

-42

~~synapse/storage/data_stores/main/schema/full_schemas/16/users.sql~~ less more

0		/* Copyright 2014-2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14		CREATE TABLE IF NOT EXISTS users(
15		name TEXT,
16		password_hash TEXT,
17		creation_ts BIGINT,
18		admin SMALLINT DEFAULT 0 NOT NULL,
19		UNIQUE(name)
20		);
21
22		CREATE TABLE IF NOT EXISTS access_tokens(
23		id BIGINT PRIMARY KEY,
24		user_id TEXT NOT NULL,
25		device_id TEXT,
26		token TEXT NOT NULL,
27		last_used BIGINT,
28		UNIQUE(token)
29		);
30
31		CREATE TABLE IF NOT EXISTS user_ips (
32		user_id TEXT NOT NULL,
33		access_token TEXT NOT NULL,
34		device_id TEXT,
35		ip TEXT NOT NULL,
36		user_agent TEXT NOT NULL,
37		last_seen BIGINT NOT NULL
38		);
39
40		CREATE INDEX user_ips_user ON user_ips(user_id);
41		CREATE INDEX user_ips_user_ip ON user_ips(user_id, access_token, ip);

+0

-1983

~~synapse/storage/data_stores/main/schema/full_schemas/54/full.sql.postgres~~ less more

0
1
2
3
4
5		CREATE TABLE access_tokens (
6		id bigint NOT NULL,
7		user_id text NOT NULL,
8		device_id text,
9		token text NOT NULL,
10		last_used bigint
11		);
12
13
14
15		CREATE TABLE account_data (
16		user_id text NOT NULL,
17		account_data_type text NOT NULL,
18		stream_id bigint NOT NULL,
19		content text NOT NULL
20		);
21
22
23
24		CREATE TABLE account_data_max_stream_id (
25		lock character(1) DEFAULT 'X'::bpchar NOT NULL,
26		stream_id bigint NOT NULL,
27		CONSTRAINT private_user_data_max_stream_id_lock_check CHECK ((lock = 'X'::bpchar))
28		);
29
30
31
32		CREATE TABLE account_validity (
33		user_id text NOT NULL,
34		expiration_ts_ms bigint NOT NULL,
35		email_sent boolean NOT NULL,
36		renewal_token text
37		);
38
39
40
41		CREATE TABLE application_services_state (
42		as_id text NOT NULL,
43		state character varying(5),
44		last_txn integer
45		);
46
47
48
49		CREATE TABLE application_services_txns (
50		as_id text NOT NULL,
51		txn_id integer NOT NULL,
52		event_ids text NOT NULL
53		);
54
55
56
57		CREATE TABLE appservice_room_list (
58		appservice_id text NOT NULL,
59		network_id text NOT NULL,
60		room_id text NOT NULL
61		);
62
63
64
65		CREATE TABLE appservice_stream_position (
66		lock character(1) DEFAULT 'X'::bpchar NOT NULL,
67		stream_ordering bigint,
68		CONSTRAINT appservice_stream_position_lock_check CHECK ((lock = 'X'::bpchar))
69		);
70
71
72		CREATE TABLE blocked_rooms (
73		room_id text NOT NULL,
74		user_id text NOT NULL
75		);
76
77
78
79		CREATE TABLE cache_invalidation_stream (
80		stream_id bigint,
81		cache_func text,
82		keys text[],
83		invalidation_ts bigint
84		);
85
86
87
88		CREATE TABLE current_state_delta_stream (
89		stream_id bigint NOT NULL,
90		room_id text NOT NULL,
91		type text NOT NULL,
92		state_key text NOT NULL,
93		event_id text,
94		prev_event_id text
95		);
96
97
98
99		CREATE TABLE current_state_events (
100		event_id text NOT NULL,
101		room_id text NOT NULL,
102		type text NOT NULL,
103		state_key text NOT NULL
104		);
105
106
107
108		CREATE TABLE deleted_pushers (
109		stream_id bigint NOT NULL,
110		app_id text NOT NULL,
111		pushkey text NOT NULL,
112		user_id text NOT NULL
113		);
114
115
116
117		CREATE TABLE destinations (
118		destination text NOT NULL,
119		retry_last_ts bigint,
120		retry_interval integer
121		);
122
123
124
125		CREATE TABLE device_federation_inbox (
126		origin text NOT NULL,
127		message_id text NOT NULL,
128		received_ts bigint NOT NULL
129		);
130
131
132
133		CREATE TABLE device_federation_outbox (
134		destination text NOT NULL,
135		stream_id bigint NOT NULL,
136		queued_ts bigint NOT NULL,
137		messages_json text NOT NULL
138		);
139
140
141
142		CREATE TABLE device_inbox (
143		user_id text NOT NULL,
144		device_id text NOT NULL,
145		stream_id bigint NOT NULL,
146		message_json text NOT NULL
147		);
148
149
150
151		CREATE TABLE device_lists_outbound_last_success (
152		destination text NOT NULL,
153		user_id text NOT NULL,
154		stream_id bigint NOT NULL
155		);
156
157
158
159		CREATE TABLE device_lists_outbound_pokes (
160		destination text NOT NULL,
161		stream_id bigint NOT NULL,
162		user_id text NOT NULL,
163		device_id text NOT NULL,
164		sent boolean NOT NULL,
165		ts bigint NOT NULL
166		);
167
168
169
170		CREATE TABLE device_lists_remote_cache (
171		user_id text NOT NULL,
172		device_id text NOT NULL,
173		content text NOT NULL
174		);
175
176
177
178		CREATE TABLE device_lists_remote_extremeties (
179		user_id text NOT NULL,
180		stream_id text NOT NULL
181		);
182
183
184
185		CREATE TABLE device_lists_stream (
186		stream_id bigint NOT NULL,
187		user_id text NOT NULL,
188		device_id text NOT NULL
189		);
190
191
192
193		CREATE TABLE device_max_stream_id (
194		stream_id bigint NOT NULL
195		);
196
197
198
199		CREATE TABLE devices (
200		user_id text NOT NULL,
201		device_id text NOT NULL,
202		display_name text
203		);
204
205
206
207		CREATE TABLE e2e_device_keys_json (
208		user_id text NOT NULL,
209		device_id text NOT NULL,
210		ts_added_ms bigint NOT NULL,
211		key_json text NOT NULL
212		);
213
214
215
216		CREATE TABLE e2e_one_time_keys_json (
217		user_id text NOT NULL,
218		device_id text NOT NULL,
219		algorithm text NOT NULL,
220		key_id text NOT NULL,
221		ts_added_ms bigint NOT NULL,
222		key_json text NOT NULL
223		);
224
225
226
227		CREATE TABLE e2e_room_keys (
228		user_id text NOT NULL,
229		room_id text NOT NULL,
230		session_id text NOT NULL,
231		version bigint NOT NULL,
232		first_message_index integer,
233		forwarded_count integer,
234		is_verified boolean,
235		session_data text NOT NULL
236		);
237
238
239
240		CREATE TABLE e2e_room_keys_versions (
241		user_id text NOT NULL,
242		version bigint NOT NULL,
243		algorithm text NOT NULL,
244		auth_data text NOT NULL,
245		deleted smallint DEFAULT 0 NOT NULL
246		);
247
248
249
250		CREATE TABLE erased_users (
251		user_id text NOT NULL
252		);
253
254
255
256		CREATE TABLE event_auth (
257		event_id text NOT NULL,
258		auth_id text NOT NULL,
259		room_id text NOT NULL
260		);
261
262
263
264		CREATE TABLE event_backward_extremities (
265		event_id text NOT NULL,
266		room_id text NOT NULL
267		);
268
269
270
271		CREATE TABLE event_edges (
272		event_id text NOT NULL,
273		prev_event_id text NOT NULL,
274		room_id text NOT NULL,
275		is_state boolean NOT NULL
276		);
277
278
279
280		CREATE TABLE event_forward_extremities (
281		event_id text NOT NULL,
282		room_id text NOT NULL
283		);
284
285
286
287		CREATE TABLE event_json (
288		event_id text NOT NULL,
289		room_id text NOT NULL,
290		internal_metadata text NOT NULL,
291		json text NOT NULL,
292		format_version integer
293		);
294
295
296
297		CREATE TABLE event_push_actions (
298		room_id text NOT NULL,
299		event_id text NOT NULL,
300		user_id text NOT NULL,
301		profile_tag character varying(32),
302		actions text NOT NULL,
303		topological_ordering bigint,
304		stream_ordering bigint,
305		notif smallint,
306		highlight smallint
307		);
308
309
310
311		CREATE TABLE event_push_actions_staging (
312		event_id text NOT NULL,
313		user_id text NOT NULL,
314		actions text NOT NULL,
315		notif smallint NOT NULL,
316		highlight smallint NOT NULL
317		);
318
319
320
321		CREATE TABLE event_push_summary (
322		user_id text NOT NULL,
323		room_id text NOT NULL,
324		notif_count bigint NOT NULL,
325		stream_ordering bigint NOT NULL
326		);
327
328
329
330		CREATE TABLE event_push_summary_stream_ordering (
331		lock character(1) DEFAULT 'X'::bpchar NOT NULL,
332		stream_ordering bigint NOT NULL,
333		CONSTRAINT event_push_summary_stream_ordering_lock_check CHECK ((lock = 'X'::bpchar))
334		);
335
336
337
338		CREATE TABLE event_reference_hashes (
339		event_id text,
340		algorithm text,
341		hash bytea
342		);
343
344
345
346		CREATE TABLE event_relations (
347		event_id text NOT NULL,
348		relates_to_id text NOT NULL,
349		relation_type text NOT NULL,
350		aggregation_key text
351		);
352
353
354
355		CREATE TABLE event_reports (
356		id bigint NOT NULL,
357		received_ts bigint NOT NULL,
358		room_id text NOT NULL,
359		event_id text NOT NULL,
360		user_id text NOT NULL,
361		reason text,
362		content text
363		);
364
365
366
367		CREATE TABLE event_search (
368		event_id text,
369		room_id text,
370		sender text,
371		key text,
372		vector tsvector,
373		origin_server_ts bigint,
374		stream_ordering bigint
375		);
376
377
378
379		CREATE TABLE event_to_state_groups (
380		event_id text NOT NULL,
381		state_group bigint NOT NULL
382		);
383
384
385
386		CREATE TABLE events (
387		stream_ordering integer NOT NULL,
388		topological_ordering bigint NOT NULL,
389		event_id text NOT NULL,
390		type text NOT NULL,
391		room_id text NOT NULL,
392		content text,
393		unrecognized_keys text,
394		processed boolean NOT NULL,
395		outlier boolean NOT NULL,
396		depth bigint DEFAULT 0 NOT NULL,
397		origin_server_ts bigint,
398		received_ts bigint,
399		sender text,
400		contains_url boolean
401		);
402
403
404
405		CREATE TABLE ex_outlier_stream (
406		event_stream_ordering bigint NOT NULL,
407		event_id text NOT NULL,
408		state_group bigint NOT NULL
409		);
410
411
412
413		CREATE TABLE federation_stream_position (
414		type text NOT NULL,
415		stream_id integer NOT NULL
416		);
417
418
419
420		CREATE TABLE group_attestations_remote (
421		group_id text NOT NULL,
422		user_id text NOT NULL,
423		valid_until_ms bigint NOT NULL,
424		attestation_json text NOT NULL
425		);
426
427
428
429		CREATE TABLE group_attestations_renewals (
430		group_id text NOT NULL,
431		user_id text NOT NULL,
432		valid_until_ms bigint NOT NULL
433		);
434
435
436
437		CREATE TABLE group_invites (
438		group_id text NOT NULL,
439		user_id text NOT NULL
440		);
441
442
443
444		CREATE TABLE group_roles (
445		group_id text NOT NULL,
446		role_id text NOT NULL,
447		profile text NOT NULL,
448		is_public boolean NOT NULL
449		);
450
451
452
453		CREATE TABLE group_room_categories (
454		group_id text NOT NULL,
455		category_id text NOT NULL,
456		profile text NOT NULL,
457		is_public boolean NOT NULL
458		);
459
460
461
462		CREATE TABLE group_rooms (
463		group_id text NOT NULL,
464		room_id text NOT NULL,
465		is_public boolean NOT NULL
466		);
467
468
469
470		CREATE TABLE group_summary_roles (
471		group_id text NOT NULL,
472		role_id text NOT NULL,
473		role_order bigint NOT NULL,
474		CONSTRAINT group_summary_roles_role_order_check CHECK ((role_order > 0))
475		);
476
477
478
479		CREATE TABLE group_summary_room_categories (
480		group_id text NOT NULL,
481		category_id text NOT NULL,
482		cat_order bigint NOT NULL,
483		CONSTRAINT group_summary_room_categories_cat_order_check CHECK ((cat_order > 0))
484		);
485
486
487
488		CREATE TABLE group_summary_rooms (
489		group_id text NOT NULL,
490		room_id text NOT NULL,
491		category_id text NOT NULL,
492		room_order bigint NOT NULL,
493		is_public boolean NOT NULL,
494		CONSTRAINT group_summary_rooms_room_order_check CHECK ((room_order > 0))
495		);
496
497
498
499		CREATE TABLE group_summary_users (
500		group_id text NOT NULL,
501		user_id text NOT NULL,
502		role_id text NOT NULL,
503		user_order bigint NOT NULL,
504		is_public boolean NOT NULL
505		);
506
507
508
509		CREATE TABLE group_users (
510		group_id text NOT NULL,
511		user_id text NOT NULL,
512		is_admin boolean NOT NULL,
513		is_public boolean NOT NULL
514		);
515
516
517
518		CREATE TABLE groups (
519		group_id text NOT NULL,
520		name text,
521		avatar_url text,
522		short_description text,
523		long_description text,
524		is_public boolean NOT NULL,
525		join_policy text DEFAULT 'invite'::text NOT NULL
526		);
527
528
529
530		CREATE TABLE guest_access (
531		event_id text NOT NULL,
532		room_id text NOT NULL,
533		guest_access text NOT NULL
534		);
535
536
537
538		CREATE TABLE history_visibility (
539		event_id text NOT NULL,
540		room_id text NOT NULL,
541		history_visibility text NOT NULL
542		);
543
544
545
546		CREATE TABLE local_group_membership (
547		group_id text NOT NULL,
548		user_id text NOT NULL,
549		is_admin boolean NOT NULL,
550		membership text NOT NULL,
551		is_publicised boolean NOT NULL,
552		content text NOT NULL
553		);
554
555
556
557		CREATE TABLE local_group_updates (
558		stream_id bigint NOT NULL,
559		group_id text NOT NULL,
560		user_id text NOT NULL,
561		type text NOT NULL,
562		content text NOT NULL
563		);
564
565
566
567		CREATE TABLE local_invites (
568		stream_id bigint NOT NULL,
569		inviter text NOT NULL,
570		invitee text NOT NULL,
571		event_id text NOT NULL,
572		room_id text NOT NULL,
573		locally_rejected text,
574		replaced_by text
575		);
576
577
578
579		CREATE TABLE local_media_repository (
580		media_id text,
581		media_type text,
582		media_length integer,
583		created_ts bigint,
584		upload_name text,
585		user_id text,
586		quarantined_by text,
587		url_cache text,
588		last_access_ts bigint
589		);
590
591
592
593		CREATE TABLE local_media_repository_thumbnails (
594		media_id text,
595		thumbnail_width integer,
596		thumbnail_height integer,
597		thumbnail_type text,
598		thumbnail_method text,
599		thumbnail_length integer
600		);
601
602
603
604		CREATE TABLE local_media_repository_url_cache (
605		url text,
606		response_code integer,
607		etag text,
608		expires_ts bigint,
609		og text,
610		media_id text,
611		download_ts bigint
612		);
613
614
615
616		CREATE TABLE monthly_active_users (
617		user_id text NOT NULL,
618		"timestamp" bigint NOT NULL
619		);
620
621
622
623		CREATE TABLE open_id_tokens (
624		token text NOT NULL,
625		ts_valid_until_ms bigint NOT NULL,
626		user_id text NOT NULL
627		);
628
629
630
631		CREATE TABLE presence (
632		user_id text NOT NULL,
633		state character varying(20),
634		status_msg text,
635		mtime bigint
636		);
637
638
639
640		CREATE TABLE presence_allow_inbound (
641		observed_user_id text NOT NULL,
642		observer_user_id text NOT NULL
643		);
644
645
646
647		CREATE TABLE presence_stream (
648		stream_id bigint,
649		user_id text,
650		state text,
651		last_active_ts bigint,
652		last_federation_update_ts bigint,
653		last_user_sync_ts bigint,
654		status_msg text,
655		currently_active boolean
656		);
657
658
659
660		CREATE TABLE profiles (
661		user_id text NOT NULL,
662		displayname text,
663		avatar_url text
664		);
665
666
667
668		CREATE TABLE public_room_list_stream (
669		stream_id bigint NOT NULL,
670		room_id text NOT NULL,
671		visibility boolean NOT NULL,
672		appservice_id text,
673		network_id text
674		);
675
676
677
678		CREATE TABLE push_rules (
679		id bigint NOT NULL,
680		user_name text NOT NULL,
681		rule_id text NOT NULL,
682		priority_class smallint NOT NULL,
683		priority integer DEFAULT 0 NOT NULL,
684		conditions text NOT NULL,
685		actions text NOT NULL
686		);
687
688
689
690		CREATE TABLE push_rules_enable (
691		id bigint NOT NULL,
692		user_name text NOT NULL,
693		rule_id text NOT NULL,
694		enabled smallint
695		);
696
697
698
699		CREATE TABLE push_rules_stream (
700		stream_id bigint NOT NULL,
701		event_stream_ordering bigint NOT NULL,
702		user_id text NOT NULL,
703		rule_id text NOT NULL,
704		op text NOT NULL,
705		priority_class smallint,
706		priority integer,
707		conditions text,
708		actions text
709		);
710
711
712
713		CREATE TABLE pusher_throttle (
714		pusher bigint NOT NULL,
715		room_id text NOT NULL,
716		last_sent_ts bigint,
717		throttle_ms bigint
718		);
719
720
721
722		CREATE TABLE pushers (
723		id bigint NOT NULL,
724		user_name text NOT NULL,
725		access_token bigint,
726		profile_tag text NOT NULL,
727		kind text NOT NULL,
728		app_id text NOT NULL,
729		app_display_name text NOT NULL,
730		device_display_name text NOT NULL,
731		pushkey text NOT NULL,
732		ts bigint NOT NULL,
733		lang text,
734		data text,
735		last_stream_ordering integer,
736		last_success bigint,
737		failing_since bigint
738		);
739
740
741
742		CREATE TABLE ratelimit_override (
743		user_id text NOT NULL,
744		messages_per_second bigint,
745		burst_count bigint
746		);
747
748
749
750		CREATE TABLE receipts_graph (
751		room_id text NOT NULL,
752		receipt_type text NOT NULL,
753		user_id text NOT NULL,
754		event_ids text NOT NULL,
755		data text NOT NULL
756		);
757
758
759
760		CREATE TABLE receipts_linearized (
761		stream_id bigint NOT NULL,
762		room_id text NOT NULL,
763		receipt_type text NOT NULL,
764		user_id text NOT NULL,
765		event_id text NOT NULL,
766		data text NOT NULL
767		);
768
769
770
771		CREATE TABLE received_transactions (
772		transaction_id text,
773		origin text,
774		ts bigint,
775		response_code integer,
776		response_json bytea,
777		has_been_referenced smallint DEFAULT 0
778		);
779
780
781
782		CREATE TABLE redactions (
783		event_id text NOT NULL,
784		redacts text NOT NULL
785		);
786
787
788
789		CREATE TABLE rejections (
790		event_id text NOT NULL,
791		reason text NOT NULL,
792		last_check text NOT NULL
793		);
794
795
796
797		CREATE TABLE remote_media_cache (
798		media_origin text,
799		media_id text,
800		media_type text,
801		created_ts bigint,
802		upload_name text,
803		media_length integer,
804		filesystem_id text,
805		last_access_ts bigint,
806		quarantined_by text
807		);
808
809
810
811		CREATE TABLE remote_media_cache_thumbnails (
812		media_origin text,
813		media_id text,
814		thumbnail_width integer,
815		thumbnail_height integer,
816		thumbnail_method text,
817		thumbnail_type text,
818		thumbnail_length integer,
819		filesystem_id text
820		);
821
822
823
824		CREATE TABLE remote_profile_cache (
825		user_id text NOT NULL,
826		displayname text,
827		avatar_url text,
828		last_check bigint NOT NULL
829		);
830
831
832
833		CREATE TABLE room_account_data (
834		user_id text NOT NULL,
835		room_id text NOT NULL,
836		account_data_type text NOT NULL,
837		stream_id bigint NOT NULL,
838		content text NOT NULL
839		);
840
841
842
843		CREATE TABLE room_alias_servers (
844		room_alias text NOT NULL,
845		server text NOT NULL
846		);
847
848
849
850		CREATE TABLE room_aliases (
851		room_alias text NOT NULL,
852		room_id text NOT NULL,
853		creator text
854		);
855
856
857
858		CREATE TABLE room_depth (
859		room_id text NOT NULL,
860		min_depth integer NOT NULL
861		);
862
863
864
865		CREATE TABLE room_memberships (
866		event_id text NOT NULL,
867		user_id text NOT NULL,
868		sender text NOT NULL,
869		room_id text NOT NULL,
870		membership text NOT NULL,
871		forgotten integer DEFAULT 0,
872		display_name text,
873		avatar_url text
874		);
875
876
877
878		CREATE TABLE room_names (
879		event_id text NOT NULL,
880		room_id text NOT NULL,
881		name text NOT NULL
882		);
883
884
885
886		CREATE TABLE room_state (
887		room_id text NOT NULL,
888		join_rules text,
889		history_visibility text,
890		encryption text,
891		name text,
892		topic text,
893		avatar text,
894		canonical_alias text
895		);
896
897
898
899		CREATE TABLE room_stats (
900		room_id text NOT NULL,
901		ts bigint NOT NULL,
902		bucket_size integer NOT NULL,
903		current_state_events integer NOT NULL,
904		joined_members integer NOT NULL,
905		invited_members integer NOT NULL,
906		left_members integer NOT NULL,
907		banned_members integer NOT NULL,
908		state_events integer NOT NULL
909		);
910
911
912
913		CREATE TABLE room_stats_earliest_token (
914		room_id text NOT NULL,
915		token bigint NOT NULL
916		);
917
918
919
920		CREATE TABLE room_tags (
921		user_id text NOT NULL,
922		room_id text NOT NULL,
923		tag text NOT NULL,
924		content text NOT NULL
925		);
926
927
928
929		CREATE TABLE room_tags_revisions (
930		user_id text NOT NULL,
931		room_id text NOT NULL,
932		stream_id bigint NOT NULL
933		);
934
935
936
937		CREATE TABLE rooms (
938		room_id text NOT NULL,
939		is_public boolean,
940		creator text
941		);
942
943
944
945		CREATE TABLE server_keys_json (
946		server_name text NOT NULL,
947		key_id text NOT NULL,
948		from_server text NOT NULL,
949		ts_added_ms bigint NOT NULL,
950		ts_valid_until_ms bigint NOT NULL,
951		key_json bytea NOT NULL
952		);
953
954
955
956		CREATE TABLE server_signature_keys (
957		server_name text,
958		key_id text,
959		from_server text,
960		ts_added_ms bigint,
961		verify_key bytea,
962		ts_valid_until_ms bigint
963		);
964
965
966
967		CREATE TABLE state_events (
968		event_id text NOT NULL,
969		room_id text NOT NULL,
970		type text NOT NULL,
971		state_key text NOT NULL,
972		prev_state text
973		);
974
975
976
977		CREATE TABLE stats_stream_pos (
978		lock character(1) DEFAULT 'X'::bpchar NOT NULL,
979		stream_id bigint,
980		CONSTRAINT stats_stream_pos_lock_check CHECK ((lock = 'X'::bpchar))
981		);
982
983
984
985		CREATE TABLE stream_ordering_to_exterm (
986		stream_ordering bigint NOT NULL,
987		room_id text NOT NULL,
988		event_id text NOT NULL
989		);
990
991
992
993		CREATE TABLE threepid_guest_access_tokens (
994		medium text,
995		address text,
996		guest_access_token text,
997		first_inviter text
998		);
999
1000
1001
1002		CREATE TABLE topics (
1003		event_id text NOT NULL,
1004		room_id text NOT NULL,
1005		topic text NOT NULL
1006		);
1007
1008
1009
1010		CREATE TABLE user_daily_visits (
1011		user_id text NOT NULL,
1012		device_id text,
1013		"timestamp" bigint NOT NULL
1014		);
1015
1016
1017
1018		CREATE TABLE user_directory (
1019		user_id text NOT NULL,
1020		room_id text,
1021		display_name text,
1022		avatar_url text
1023		);
1024
1025
1026
1027		CREATE TABLE user_directory_search (
1028		user_id text NOT NULL,
1029		vector tsvector
1030		);
1031
1032
1033
1034		CREATE TABLE user_directory_stream_pos (
1035		lock character(1) DEFAULT 'X'::bpchar NOT NULL,
1036		stream_id bigint,
1037		CONSTRAINT user_directory_stream_pos_lock_check CHECK ((lock = 'X'::bpchar))
1038		);
1039
1040
1041
1042		CREATE TABLE user_filters (
1043		user_id text,
1044		filter_id bigint,
1045		filter_json bytea
1046		);
1047
1048
1049
1050		CREATE TABLE user_ips (
1051		user_id text NOT NULL,
1052		access_token text NOT NULL,
1053		device_id text,
1054		ip text NOT NULL,
1055		user_agent text NOT NULL,
1056		last_seen bigint NOT NULL
1057		);
1058
1059
1060
1061		CREATE TABLE user_stats (
1062		user_id text NOT NULL,
1063		ts bigint NOT NULL,
1064		bucket_size integer NOT NULL,
1065		public_rooms integer NOT NULL,
1066		private_rooms integer NOT NULL
1067		);
1068
1069
1070
1071		CREATE TABLE user_threepid_id_server (
1072		user_id text NOT NULL,
1073		medium text NOT NULL,
1074		address text NOT NULL,
1075		id_server text NOT NULL
1076		);
1077
1078
1079
1080		CREATE TABLE user_threepids (
1081		user_id text NOT NULL,
1082		medium text NOT NULL,
1083		address text NOT NULL,
1084		validated_at bigint NOT NULL,
1085		added_at bigint NOT NULL
1086		);
1087
1088
1089
1090		CREATE TABLE users (
1091		name text,
1092		password_hash text,
1093		creation_ts bigint,
1094		admin smallint DEFAULT 0 NOT NULL,
1095		upgrade_ts bigint,
1096		is_guest smallint DEFAULT 0 NOT NULL,
1097		appservice_id text,
1098		consent_version text,
1099		consent_server_notice_sent text,
1100		user_type text
1101		);
1102
1103
1104
1105		CREATE TABLE users_in_public_rooms (
1106		user_id text NOT NULL,
1107		room_id text NOT NULL
1108		);
1109
1110
1111
1112		CREATE TABLE users_pending_deactivation (
1113		user_id text NOT NULL
1114		);
1115
1116
1117
1118		CREATE TABLE users_who_share_private_rooms (
1119		user_id text NOT NULL,
1120		other_user_id text NOT NULL,
1121		room_id text NOT NULL
1122		);
1123
1124
1125
1126		ALTER TABLE ONLY access_tokens
1127		ADD CONSTRAINT access_tokens_pkey PRIMARY KEY (id);
1128
1129
1130
1131		ALTER TABLE ONLY access_tokens
1132		ADD CONSTRAINT access_tokens_token_key UNIQUE (token);
1133
1134
1135
1136		ALTER TABLE ONLY account_data
1137		ADD CONSTRAINT account_data_uniqueness UNIQUE (user_id, account_data_type);
1138
1139
1140
1141		ALTER TABLE ONLY account_validity
1142		ADD CONSTRAINT account_validity_pkey PRIMARY KEY (user_id);
1143
1144
1145
1146		ALTER TABLE ONLY application_services_state
1147		ADD CONSTRAINT application_services_state_pkey PRIMARY KEY (as_id);
1148
1149
1150
1151		ALTER TABLE ONLY application_services_txns
1152		ADD CONSTRAINT application_services_txns_as_id_txn_id_key UNIQUE (as_id, txn_id);
1153
1154
1155
1156		ALTER TABLE ONLY appservice_stream_position
1157		ADD CONSTRAINT appservice_stream_position_lock_key UNIQUE (lock);
1158
1159
1160
1161		ALTER TABLE ONLY current_state_events
1162		ADD CONSTRAINT current_state_events_event_id_key UNIQUE (event_id);
1163
1164
1165
1166		ALTER TABLE ONLY current_state_events
1167		ADD CONSTRAINT current_state_events_room_id_type_state_key_key UNIQUE (room_id, type, state_key);
1168
1169
1170
1171		ALTER TABLE ONLY destinations
1172		ADD CONSTRAINT destinations_pkey PRIMARY KEY (destination);
1173
1174
1175
1176		ALTER TABLE ONLY devices
1177		ADD CONSTRAINT device_uniqueness UNIQUE (user_id, device_id);
1178
1179
1180
1181		ALTER TABLE ONLY e2e_device_keys_json
1182		ADD CONSTRAINT e2e_device_keys_json_uniqueness UNIQUE (user_id, device_id);
1183
1184
1185
1186		ALTER TABLE ONLY e2e_one_time_keys_json
1187		ADD CONSTRAINT e2e_one_time_keys_json_uniqueness UNIQUE (user_id, device_id, algorithm, key_id);
1188
1189
1190
1191		ALTER TABLE ONLY event_backward_extremities
1192		ADD CONSTRAINT event_backward_extremities_event_id_room_id_key UNIQUE (event_id, room_id);
1193
1194
1195
1196		ALTER TABLE ONLY event_edges
1197		ADD CONSTRAINT event_edges_event_id_prev_event_id_room_id_is_state_key UNIQUE (event_id, prev_event_id, room_id, is_state);
1198
1199
1200
1201		ALTER TABLE ONLY event_forward_extremities
1202		ADD CONSTRAINT event_forward_extremities_event_id_room_id_key UNIQUE (event_id, room_id);
1203
1204
1205
1206		ALTER TABLE ONLY event_push_actions
1207		ADD CONSTRAINT event_id_user_id_profile_tag_uniqueness UNIQUE (room_id, event_id, user_id, profile_tag);
1208
1209
1210
1211		ALTER TABLE ONLY event_json
1212		ADD CONSTRAINT event_json_event_id_key UNIQUE (event_id);
1213
1214
1215
1216		ALTER TABLE ONLY event_push_summary_stream_ordering
1217		ADD CONSTRAINT event_push_summary_stream_ordering_lock_key UNIQUE (lock);
1218
1219
1220
1221		ALTER TABLE ONLY event_reference_hashes
1222		ADD CONSTRAINT event_reference_hashes_event_id_algorithm_key UNIQUE (event_id, algorithm);
1223
1224
1225
1226		ALTER TABLE ONLY event_reports
1227		ADD CONSTRAINT event_reports_pkey PRIMARY KEY (id);
1228
1229
1230
1231		ALTER TABLE ONLY event_to_state_groups
1232		ADD CONSTRAINT event_to_state_groups_event_id_key UNIQUE (event_id);
1233
1234
1235
1236		ALTER TABLE ONLY events
1237		ADD CONSTRAINT events_event_id_key UNIQUE (event_id);
1238
1239
1240
1241		ALTER TABLE ONLY events
1242		ADD CONSTRAINT events_pkey PRIMARY KEY (stream_ordering);
1243
1244
1245
1246		ALTER TABLE ONLY ex_outlier_stream
1247		ADD CONSTRAINT ex_outlier_stream_pkey PRIMARY KEY (event_stream_ordering);
1248
1249
1250
1251		ALTER TABLE ONLY group_roles
1252		ADD CONSTRAINT group_roles_group_id_role_id_key UNIQUE (group_id, role_id);
1253
1254
1255
1256		ALTER TABLE ONLY group_room_categories
1257		ADD CONSTRAINT group_room_categories_group_id_category_id_key UNIQUE (group_id, category_id);
1258
1259
1260
1261		ALTER TABLE ONLY group_summary_roles
1262		ADD CONSTRAINT group_summary_roles_group_id_role_id_role_order_key UNIQUE (group_id, role_id, role_order);
1263
1264
1265
1266		ALTER TABLE ONLY group_summary_room_categories
1267		ADD CONSTRAINT group_summary_room_categories_group_id_category_id_cat_orde_key UNIQUE (group_id, category_id, cat_order);
1268
1269
1270
1271		ALTER TABLE ONLY group_summary_rooms
1272		ADD CONSTRAINT group_summary_rooms_group_id_category_id_room_id_room_order_key UNIQUE (group_id, category_id, room_id, room_order);
1273
1274
1275
1276		ALTER TABLE ONLY guest_access
1277		ADD CONSTRAINT guest_access_event_id_key UNIQUE (event_id);
1278
1279
1280
1281		ALTER TABLE ONLY history_visibility
1282		ADD CONSTRAINT history_visibility_event_id_key UNIQUE (event_id);
1283
1284
1285
1286		ALTER TABLE ONLY local_media_repository
1287		ADD CONSTRAINT local_media_repository_media_id_key UNIQUE (media_id);
1288
1289
1290
1291		ALTER TABLE ONLY local_media_repository_thumbnails
1292		ADD CONSTRAINT local_media_repository_thumbn_media_id_thumbnail_width_thum_key UNIQUE (media_id, thumbnail_width, thumbnail_height, thumbnail_type);
1293
1294
1295
1296		ALTER TABLE ONLY user_threepids
1297		ADD CONSTRAINT medium_address UNIQUE (medium, address);
1298
1299
1300
1301		ALTER TABLE ONLY open_id_tokens
1302		ADD CONSTRAINT open_id_tokens_pkey PRIMARY KEY (token);
1303
1304
1305
1306		ALTER TABLE ONLY presence_allow_inbound
1307		ADD CONSTRAINT presence_allow_inbound_observed_user_id_observer_user_id_key UNIQUE (observed_user_id, observer_user_id);
1308
1309
1310
1311		ALTER TABLE ONLY presence
1312		ADD CONSTRAINT presence_user_id_key UNIQUE (user_id);
1313
1314
1315
1316		ALTER TABLE ONLY account_data_max_stream_id
1317		ADD CONSTRAINT private_user_data_max_stream_id_lock_key UNIQUE (lock);
1318
1319
1320
1321		ALTER TABLE ONLY profiles
1322		ADD CONSTRAINT profiles_user_id_key UNIQUE (user_id);
1323
1324
1325
1326		ALTER TABLE ONLY push_rules_enable
1327		ADD CONSTRAINT push_rules_enable_pkey PRIMARY KEY (id);
1328
1329
1330
1331		ALTER TABLE ONLY push_rules_enable
1332		ADD CONSTRAINT push_rules_enable_user_name_rule_id_key UNIQUE (user_name, rule_id);
1333
1334
1335
1336		ALTER TABLE ONLY push_rules
1337		ADD CONSTRAINT push_rules_pkey PRIMARY KEY (id);
1338
1339
1340
1341		ALTER TABLE ONLY push_rules
1342		ADD CONSTRAINT push_rules_user_name_rule_id_key UNIQUE (user_name, rule_id);
1343
1344
1345
1346		ALTER TABLE ONLY pusher_throttle
1347		ADD CONSTRAINT pusher_throttle_pkey PRIMARY KEY (pusher, room_id);
1348
1349
1350
1351		ALTER TABLE ONLY pushers
1352		ADD CONSTRAINT pushers2_app_id_pushkey_user_name_key UNIQUE (app_id, pushkey, user_name);
1353
1354
1355
1356		ALTER TABLE ONLY pushers
1357		ADD CONSTRAINT pushers2_pkey PRIMARY KEY (id);
1358
1359
1360
1361		ALTER TABLE ONLY receipts_graph
1362		ADD CONSTRAINT receipts_graph_uniqueness UNIQUE (room_id, receipt_type, user_id);
1363
1364
1365
1366		ALTER TABLE ONLY receipts_linearized
1367		ADD CONSTRAINT receipts_linearized_uniqueness UNIQUE (room_id, receipt_type, user_id);
1368
1369
1370
1371		ALTER TABLE ONLY received_transactions
1372		ADD CONSTRAINT received_transactions_transaction_id_origin_key UNIQUE (transaction_id, origin);
1373
1374
1375
1376		ALTER TABLE ONLY redactions
1377		ADD CONSTRAINT redactions_event_id_key UNIQUE (event_id);
1378
1379
1380
1381		ALTER TABLE ONLY rejections
1382		ADD CONSTRAINT rejections_event_id_key UNIQUE (event_id);
1383
1384
1385
1386		ALTER TABLE ONLY remote_media_cache
1387		ADD CONSTRAINT remote_media_cache_media_origin_media_id_key UNIQUE (media_origin, media_id);
1388
1389
1390
1391		ALTER TABLE ONLY remote_media_cache_thumbnails
1392		ADD CONSTRAINT remote_media_cache_thumbnails_media_origin_media_id_thumbna_key UNIQUE (media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type);
1393
1394
1395
1396		ALTER TABLE ONLY room_account_data
1397		ADD CONSTRAINT room_account_data_uniqueness UNIQUE (user_id, room_id, account_data_type);
1398
1399
1400
1401		ALTER TABLE ONLY room_aliases
1402		ADD CONSTRAINT room_aliases_room_alias_key UNIQUE (room_alias);
1403
1404
1405
1406		ALTER TABLE ONLY room_depth
1407		ADD CONSTRAINT room_depth_room_id_key UNIQUE (room_id);
1408
1409
1410
1411		ALTER TABLE ONLY room_memberships
1412		ADD CONSTRAINT room_memberships_event_id_key UNIQUE (event_id);
1413
1414
1415
1416		ALTER TABLE ONLY room_names
1417		ADD CONSTRAINT room_names_event_id_key UNIQUE (event_id);
1418
1419
1420
1421		ALTER TABLE ONLY room_tags_revisions
1422		ADD CONSTRAINT room_tag_revisions_uniqueness UNIQUE (user_id, room_id);
1423
1424
1425
1426		ALTER TABLE ONLY room_tags
1427		ADD CONSTRAINT room_tag_uniqueness UNIQUE (user_id, room_id, tag);
1428
1429
1430
1431		ALTER TABLE ONLY rooms
1432		ADD CONSTRAINT rooms_pkey PRIMARY KEY (room_id);
1433
1434
1435
1436		ALTER TABLE ONLY server_keys_json
1437		ADD CONSTRAINT server_keys_json_uniqueness UNIQUE (server_name, key_id, from_server);
1438
1439
1440
1441		ALTER TABLE ONLY server_signature_keys
1442		ADD CONSTRAINT server_signature_keys_server_name_key_id_key UNIQUE (server_name, key_id);
1443
1444
1445
1446		ALTER TABLE ONLY state_events
1447		ADD CONSTRAINT state_events_event_id_key UNIQUE (event_id);
1448
1449
1450		ALTER TABLE ONLY stats_stream_pos
1451		ADD CONSTRAINT stats_stream_pos_lock_key UNIQUE (lock);
1452
1453
1454
1455		ALTER TABLE ONLY topics
1456		ADD CONSTRAINT topics_event_id_key UNIQUE (event_id);
1457
1458
1459
1460		ALTER TABLE ONLY user_directory_stream_pos
1461		ADD CONSTRAINT user_directory_stream_pos_lock_key UNIQUE (lock);
1462
1463
1464
1465		ALTER TABLE ONLY users
1466		ADD CONSTRAINT users_name_key UNIQUE (name);
1467
1468
1469
1470		CREATE INDEX access_tokens_device_id ON access_tokens USING btree (user_id, device_id);
1471
1472
1473
1474		CREATE INDEX account_data_stream_id ON account_data USING btree (user_id, stream_id);
1475
1476
1477
1478		CREATE INDEX application_services_txns_id ON application_services_txns USING btree (as_id);
1479
1480
1481
1482		CREATE UNIQUE INDEX appservice_room_list_idx ON appservice_room_list USING btree (appservice_id, network_id, room_id);
1483
1484
1485
1486		CREATE UNIQUE INDEX blocked_rooms_idx ON blocked_rooms USING btree (room_id);
1487
1488
1489
1490		CREATE INDEX cache_invalidation_stream_id ON cache_invalidation_stream USING btree (stream_id);
1491
1492
1493
1494		CREATE INDEX current_state_delta_stream_idx ON current_state_delta_stream USING btree (stream_id);
1495
1496
1497
1498		CREATE INDEX current_state_events_member_index ON current_state_events USING btree (state_key) WHERE (type = 'm.room.member'::text);
1499
1500
1501
1502		CREATE INDEX deleted_pushers_stream_id ON deleted_pushers USING btree (stream_id);
1503
1504
1505
1506		CREATE INDEX device_federation_inbox_sender_id ON device_federation_inbox USING btree (origin, message_id);
1507
1508
1509
1510		CREATE INDEX device_federation_outbox_destination_id ON device_federation_outbox USING btree (destination, stream_id);
1511
1512
1513
1514		CREATE INDEX device_federation_outbox_id ON device_federation_outbox USING btree (stream_id);
1515
1516
1517
1518		CREATE INDEX device_inbox_stream_id_user_id ON device_inbox USING btree (stream_id, user_id);
1519
1520
1521
1522		CREATE INDEX device_inbox_user_stream_id ON device_inbox USING btree (user_id, device_id, stream_id);
1523
1524
1525
1526		CREATE INDEX device_lists_outbound_last_success_idx ON device_lists_outbound_last_success USING btree (destination, user_id, stream_id);
1527
1528
1529
1530		CREATE INDEX device_lists_outbound_pokes_id ON device_lists_outbound_pokes USING btree (destination, stream_id);
1531
1532
1533
1534		CREATE INDEX device_lists_outbound_pokes_stream ON device_lists_outbound_pokes USING btree (stream_id);
1535
1536
1537
1538		CREATE INDEX device_lists_outbound_pokes_user ON device_lists_outbound_pokes USING btree (destination, user_id);
1539
1540
1541
1542		CREATE UNIQUE INDEX device_lists_remote_cache_unique_id ON device_lists_remote_cache USING btree (user_id, device_id);
1543
1544
1545
1546		CREATE UNIQUE INDEX device_lists_remote_extremeties_unique_idx ON device_lists_remote_extremeties USING btree (user_id);
1547
1548
1549
1550		CREATE INDEX device_lists_stream_id ON device_lists_stream USING btree (stream_id, user_id);
1551
1552
1553
1554		CREATE INDEX device_lists_stream_user_id ON device_lists_stream USING btree (user_id, device_id);
1555
1556
1557
1558		CREATE UNIQUE INDEX e2e_room_keys_idx ON e2e_room_keys USING btree (user_id, room_id, session_id);
1559
1560
1561
1562		CREATE UNIQUE INDEX e2e_room_keys_versions_idx ON e2e_room_keys_versions USING btree (user_id, version);
1563
1564
1565
1566		CREATE UNIQUE INDEX erased_users_user ON erased_users USING btree (user_id);
1567
1568
1569
1570		CREATE INDEX ev_b_extrem_id ON event_backward_extremities USING btree (event_id);
1571
1572
1573
1574		CREATE INDEX ev_b_extrem_room ON event_backward_extremities USING btree (room_id);
1575
1576
1577
1578		CREATE INDEX ev_edges_id ON event_edges USING btree (event_id);
1579
1580
1581
1582		CREATE INDEX ev_edges_prev_id ON event_edges USING btree (prev_event_id);
1583
1584
1585
1586		CREATE INDEX ev_extrem_id ON event_forward_extremities USING btree (event_id);
1587
1588
1589
1590		CREATE INDEX ev_extrem_room ON event_forward_extremities USING btree (room_id);
1591
1592
1593
1594		CREATE INDEX evauth_edges_id ON event_auth USING btree (event_id);
1595
1596
1597
1598		CREATE INDEX event_contains_url_index ON events USING btree (room_id, topological_ordering, stream_ordering) WHERE ((contains_url = true) AND (outlier = false));
1599
1600
1601
1602		CREATE INDEX event_json_room_id ON event_json USING btree (room_id);
1603
1604
1605
1606		CREATE INDEX event_push_actions_highlights_index ON event_push_actions USING btree (user_id, room_id, topological_ordering, stream_ordering) WHERE (highlight = 1);
1607
1608
1609
1610		CREATE INDEX event_push_actions_rm_tokens ON event_push_actions USING btree (user_id, room_id, topological_ordering, stream_ordering);
1611
1612
1613
1614		CREATE INDEX event_push_actions_room_id_user_id ON event_push_actions USING btree (room_id, user_id);
1615
1616
1617
1618		CREATE INDEX event_push_actions_staging_id ON event_push_actions_staging USING btree (event_id);
1619
1620
1621
1622		CREATE INDEX event_push_actions_stream_ordering ON event_push_actions USING btree (stream_ordering, user_id);
1623
1624
1625
1626		CREATE INDEX event_push_actions_u_highlight ON event_push_actions USING btree (user_id, stream_ordering);
1627
1628
1629
1630		CREATE INDEX event_push_summary_user_rm ON event_push_summary USING btree (user_id, room_id);
1631
1632
1633
1634		CREATE INDEX event_reference_hashes_id ON event_reference_hashes USING btree (event_id);
1635
1636
1637
1638		CREATE UNIQUE INDEX event_relations_id ON event_relations USING btree (event_id);
1639
1640
1641
1642		CREATE INDEX event_relations_relates ON event_relations USING btree (relates_to_id, relation_type, aggregation_key);
1643
1644
1645
1646		CREATE INDEX event_search_ev_ridx ON event_search USING btree (room_id);
1647
1648
1649
1650		CREATE UNIQUE INDEX event_search_event_id_idx ON event_search USING btree (event_id);
1651
1652
1653
1654		CREATE INDEX event_search_fts_idx ON event_search USING gin (vector);
1655
1656
1657
1658		CREATE INDEX event_to_state_groups_sg_index ON event_to_state_groups USING btree (state_group);
1659
1660
1661
1662		CREATE INDEX events_order_room ON events USING btree (room_id, topological_ordering, stream_ordering);
1663
1664
1665
1666		CREATE INDEX events_room_stream ON events USING btree (room_id, stream_ordering);
1667
1668
1669
1670		CREATE INDEX events_ts ON events USING btree (origin_server_ts, stream_ordering);
1671
1672
1673
1674		CREATE INDEX group_attestations_remote_g_idx ON group_attestations_remote USING btree (group_id, user_id);
1675
1676
1677
1678		CREATE INDEX group_attestations_remote_u_idx ON group_attestations_remote USING btree (user_id);
1679
1680
1681
1682		CREATE INDEX group_attestations_remote_v_idx ON group_attestations_remote USING btree (valid_until_ms);
1683
1684
1685
1686		CREATE INDEX group_attestations_renewals_g_idx ON group_attestations_renewals USING btree (group_id, user_id);
1687
1688
1689
1690		CREATE INDEX group_attestations_renewals_u_idx ON group_attestations_renewals USING btree (user_id);
1691
1692
1693
1694		CREATE INDEX group_attestations_renewals_v_idx ON group_attestations_renewals USING btree (valid_until_ms);
1695
1696
1697
1698		CREATE UNIQUE INDEX group_invites_g_idx ON group_invites USING btree (group_id, user_id);
1699
1700
1701
1702		CREATE INDEX group_invites_u_idx ON group_invites USING btree (user_id);
1703
1704
1705
1706		CREATE UNIQUE INDEX group_rooms_g_idx ON group_rooms USING btree (group_id, room_id);
1707
1708
1709
1710		CREATE INDEX group_rooms_r_idx ON group_rooms USING btree (room_id);
1711
1712
1713
1714		CREATE UNIQUE INDEX group_summary_rooms_g_idx ON group_summary_rooms USING btree (group_id, room_id, category_id);
1715
1716
1717
1718		CREATE INDEX group_summary_users_g_idx ON group_summary_users USING btree (group_id);
1719
1720
1721
1722		CREATE UNIQUE INDEX group_users_g_idx ON group_users USING btree (group_id, user_id);
1723
1724
1725
1726		CREATE INDEX group_users_u_idx ON group_users USING btree (user_id);
1727
1728
1729
1730		CREATE UNIQUE INDEX groups_idx ON groups USING btree (group_id);
1731
1732
1733
1734		CREATE INDEX local_group_membership_g_idx ON local_group_membership USING btree (group_id);
1735
1736
1737
1738		CREATE INDEX local_group_membership_u_idx ON local_group_membership USING btree (user_id, group_id);
1739
1740
1741
1742		CREATE INDEX local_invites_for_user_idx ON local_invites USING btree (invitee, locally_rejected, replaced_by, room_id);
1743
1744
1745
1746		CREATE INDEX local_invites_id ON local_invites USING btree (stream_id);
1747
1748
1749
1750		CREATE INDEX local_media_repository_thumbnails_media_id ON local_media_repository_thumbnails USING btree (media_id);
1751
1752
1753
1754		CREATE INDEX local_media_repository_url_cache_by_url_download_ts ON local_media_repository_url_cache USING btree (url, download_ts);
1755
1756
1757
1758		CREATE INDEX local_media_repository_url_cache_expires_idx ON local_media_repository_url_cache USING btree (expires_ts);
1759
1760
1761
1762		CREATE INDEX local_media_repository_url_cache_media_idx ON local_media_repository_url_cache USING btree (media_id);
1763
1764
1765
1766		CREATE INDEX local_media_repository_url_idx ON local_media_repository USING btree (created_ts) WHERE (url_cache IS NOT NULL);
1767
1768
1769
1770		CREATE INDEX monthly_active_users_time_stamp ON monthly_active_users USING btree ("timestamp");
1771
1772
1773
1774		CREATE UNIQUE INDEX monthly_active_users_users ON monthly_active_users USING btree (user_id);
1775
1776
1777
1778		CREATE INDEX open_id_tokens_ts_valid_until_ms ON open_id_tokens USING btree (ts_valid_until_ms);
1779
1780
1781
1782		CREATE INDEX presence_stream_id ON presence_stream USING btree (stream_id, user_id);
1783
1784
1785
1786		CREATE INDEX presence_stream_user_id ON presence_stream USING btree (user_id);
1787
1788
1789
1790		CREATE INDEX public_room_index ON rooms USING btree (is_public);
1791
1792
1793
1794		CREATE INDEX public_room_list_stream_idx ON public_room_list_stream USING btree (stream_id);
1795
1796
1797
1798		CREATE INDEX public_room_list_stream_rm_idx ON public_room_list_stream USING btree (room_id, stream_id);
1799
1800
1801
1802		CREATE INDEX push_rules_enable_user_name ON push_rules_enable USING btree (user_name);
1803
1804
1805
1806		CREATE INDEX push_rules_stream_id ON push_rules_stream USING btree (stream_id);
1807
1808
1809
1810		CREATE INDEX push_rules_stream_user_stream_id ON push_rules_stream USING btree (user_id, stream_id);
1811
1812
1813
1814		CREATE INDEX push_rules_user_name ON push_rules USING btree (user_name);
1815
1816
1817
1818		CREATE UNIQUE INDEX ratelimit_override_idx ON ratelimit_override USING btree (user_id);
1819
1820
1821
1822		CREATE INDEX receipts_linearized_id ON receipts_linearized USING btree (stream_id);
1823
1824
1825
1826		CREATE INDEX receipts_linearized_room_stream ON receipts_linearized USING btree (room_id, stream_id);
1827
1828
1829
1830		CREATE INDEX receipts_linearized_user ON receipts_linearized USING btree (user_id);
1831
1832
1833
1834		CREATE INDEX received_transactions_ts ON received_transactions USING btree (ts);
1835
1836
1837
1838		CREATE INDEX redactions_redacts ON redactions USING btree (redacts);
1839
1840
1841
1842		CREATE INDEX remote_profile_cache_time ON remote_profile_cache USING btree (last_check);
1843
1844
1845
1846		CREATE UNIQUE INDEX remote_profile_cache_user_id ON remote_profile_cache USING btree (user_id);
1847
1848
1849
1850		CREATE INDEX room_account_data_stream_id ON room_account_data USING btree (user_id, stream_id);
1851
1852
1853
1854		CREATE INDEX room_alias_servers_alias ON room_alias_servers USING btree (room_alias);
1855
1856
1857
1858		CREATE INDEX room_aliases_id ON room_aliases USING btree (room_id);
1859
1860
1861
1862		CREATE INDEX room_depth_room ON room_depth USING btree (room_id);
1863
1864
1865
1866		CREATE INDEX room_memberships_room_id ON room_memberships USING btree (room_id);
1867
1868
1869
1870		CREATE INDEX room_memberships_user_id ON room_memberships USING btree (user_id);
1871
1872
1873
1874		CREATE INDEX room_names_room_id ON room_names USING btree (room_id);
1875
1876
1877
1878		CREATE UNIQUE INDEX room_state_room ON room_state USING btree (room_id);
1879
1880
1881
1882		CREATE UNIQUE INDEX room_stats_earliest_token_idx ON room_stats_earliest_token USING btree (room_id);
1883
1884
1885
1886		CREATE UNIQUE INDEX room_stats_room_ts ON room_stats USING btree (room_id, ts);
1887
1888
1889
1890		CREATE INDEX stream_ordering_to_exterm_idx ON stream_ordering_to_exterm USING btree (stream_ordering);
1891
1892
1893
1894		CREATE INDEX stream_ordering_to_exterm_rm_idx ON stream_ordering_to_exterm USING btree (room_id, stream_ordering);
1895
1896
1897
1898		CREATE UNIQUE INDEX threepid_guest_access_tokens_index ON threepid_guest_access_tokens USING btree (medium, address);
1899
1900
1901
1902		CREATE INDEX topics_room_id ON topics USING btree (room_id);
1903
1904
1905
1906		CREATE INDEX user_daily_visits_ts_idx ON user_daily_visits USING btree ("timestamp");
1907
1908
1909
1910		CREATE INDEX user_daily_visits_uts_idx ON user_daily_visits USING btree (user_id, "timestamp");
1911
1912
1913
1914		CREATE INDEX user_directory_room_idx ON user_directory USING btree (room_id);
1915
1916
1917
1918		CREATE INDEX user_directory_search_fts_idx ON user_directory_search USING gin (vector);
1919
1920
1921
1922		CREATE UNIQUE INDEX user_directory_search_user_idx ON user_directory_search USING btree (user_id);
1923
1924
1925
1926		CREATE UNIQUE INDEX user_directory_user_idx ON user_directory USING btree (user_id);
1927
1928
1929
1930		CREATE INDEX user_filters_by_user_id_filter_id ON user_filters USING btree (user_id, filter_id);
1931
1932
1933
1934		CREATE INDEX user_ips_device_id ON user_ips USING btree (user_id, device_id, last_seen);
1935
1936
1937
1938		CREATE INDEX user_ips_last_seen ON user_ips USING btree (user_id, last_seen);
1939
1940
1941
1942		CREATE INDEX user_ips_last_seen_only ON user_ips USING btree (last_seen);
1943
1944
1945
1946		CREATE UNIQUE INDEX user_ips_user_token_ip_unique_index ON user_ips USING btree (user_id, access_token, ip);
1947
1948
1949
1950		CREATE UNIQUE INDEX user_stats_user_ts ON user_stats USING btree (user_id, ts);
1951
1952
1953
1954		CREATE UNIQUE INDEX user_threepid_id_server_idx ON user_threepid_id_server USING btree (user_id, medium, address, id_server);
1955
1956
1957
1958		CREATE INDEX user_threepids_medium_address ON user_threepids USING btree (medium, address);
1959
1960
1961
1962		CREATE INDEX user_threepids_user_id ON user_threepids USING btree (user_id);
1963
1964
1965
1966		CREATE INDEX users_creation_ts ON users USING btree (creation_ts);
1967
1968
1969
1970		CREATE UNIQUE INDEX users_in_public_rooms_u_idx ON users_in_public_rooms USING btree (user_id, room_id);
1971
1972
1973
1974		CREATE INDEX users_who_share_private_rooms_o_idx ON users_who_share_private_rooms USING btree (other_user_id);
1975
1976
1977
1978		CREATE INDEX users_who_share_private_rooms_r_idx ON users_who_share_private_rooms USING btree (room_id);
1979
1980
1981
1982		CREATE UNIQUE INDEX users_who_share_private_rooms_u_idx ON users_who_share_private_rooms USING btree (user_id, other_user_id, room_id);

+0

-253

~~synapse/storage/data_stores/main/schema/full_schemas/54/full.sql.sqlite~~ less more

0		CREATE TABLE application_services_state( as_id TEXT PRIMARY KEY, state VARCHAR(5), last_txn INTEGER );
1		CREATE TABLE application_services_txns( as_id TEXT NOT NULL, txn_id INTEGER NOT NULL, event_ids TEXT NOT NULL, UNIQUE(as_id, txn_id) );
2		CREATE INDEX application_services_txns_id ON application_services_txns ( as_id );
3		CREATE TABLE presence( user_id TEXT NOT NULL, state VARCHAR(20), status_msg TEXT, mtime BIGINT, UNIQUE (user_id) );
4		CREATE TABLE presence_allow_inbound( observed_user_id TEXT NOT NULL, observer_user_id TEXT NOT NULL, UNIQUE (observed_user_id, observer_user_id) );
5		CREATE TABLE users( name TEXT, password_hash TEXT, creation_ts BIGINT, admin SMALLINT DEFAULT 0 NOT NULL, upgrade_ts BIGINT, is_guest SMALLINT DEFAULT 0 NOT NULL, appservice_id TEXT, consent_version TEXT, consent_server_notice_sent TEXT, user_type TEXT DEFAULT NULL, UNIQUE(name) );
6		CREATE TABLE access_tokens( id BIGINT PRIMARY KEY, user_id TEXT NOT NULL, device_id TEXT, token TEXT NOT NULL, last_used BIGINT, UNIQUE(token) );
7		CREATE TABLE user_ips ( user_id TEXT NOT NULL, access_token TEXT NOT NULL, device_id TEXT, ip TEXT NOT NULL, user_agent TEXT NOT NULL, last_seen BIGINT NOT NULL );
8		CREATE TABLE profiles( user_id TEXT NOT NULL, displayname TEXT, avatar_url TEXT, UNIQUE(user_id) );
9		CREATE TABLE received_transactions( transaction_id TEXT, origin TEXT, ts BIGINT, response_code INTEGER, response_json bytea, has_been_referenced smallint default 0, UNIQUE (transaction_id, origin) );
10		CREATE TABLE destinations( destination TEXT PRIMARY KEY, retry_last_ts BIGINT, retry_interval INTEGER );
11		CREATE TABLE events( stream_ordering INTEGER PRIMARY KEY, topological_ordering BIGINT NOT NULL, event_id TEXT NOT NULL, type TEXT NOT NULL, room_id TEXT NOT NULL, content TEXT, unrecognized_keys TEXT, processed BOOL NOT NULL, outlier BOOL NOT NULL, depth BIGINT DEFAULT 0 NOT NULL, origin_server_ts BIGINT, received_ts BIGINT, sender TEXT, contains_url BOOLEAN, UNIQUE (event_id) );
12		CREATE INDEX events_order_room ON events ( room_id, topological_ordering, stream_ordering );
13		CREATE TABLE event_json( event_id TEXT NOT NULL, room_id TEXT NOT NULL, internal_metadata TEXT NOT NULL, json TEXT NOT NULL, format_version INTEGER, UNIQUE (event_id) );
14		CREATE INDEX event_json_room_id ON event_json(room_id);
15		CREATE TABLE state_events( event_id TEXT NOT NULL, room_id TEXT NOT NULL, type TEXT NOT NULL, state_key TEXT NOT NULL, prev_state TEXT, UNIQUE (event_id) );
16		CREATE TABLE current_state_events( event_id TEXT NOT NULL, room_id TEXT NOT NULL, type TEXT NOT NULL, state_key TEXT NOT NULL, UNIQUE (event_id), UNIQUE (room_id, type, state_key) );
17		CREATE TABLE room_memberships( event_id TEXT NOT NULL, user_id TEXT NOT NULL, sender TEXT NOT NULL, room_id TEXT NOT NULL, membership TEXT NOT NULL, forgotten INTEGER DEFAULT 0, display_name TEXT, avatar_url TEXT, UNIQUE (event_id) );
18		CREATE INDEX room_memberships_room_id ON room_memberships (room_id);
19		CREATE INDEX room_memberships_user_id ON room_memberships (user_id);
20		CREATE TABLE topics( event_id TEXT NOT NULL, room_id TEXT NOT NULL, topic TEXT NOT NULL, UNIQUE (event_id) );
21		CREATE INDEX topics_room_id ON topics(room_id);
22		CREATE TABLE room_names( event_id TEXT NOT NULL, room_id TEXT NOT NULL, name TEXT NOT NULL, UNIQUE (event_id) );
23		CREATE INDEX room_names_room_id ON room_names(room_id);
24		CREATE TABLE rooms( room_id TEXT PRIMARY KEY NOT NULL, is_public BOOL, creator TEXT );
25		CREATE TABLE server_signature_keys( server_name TEXT, key_id TEXT, from_server TEXT, ts_added_ms BIGINT, verify_key bytea, ts_valid_until_ms BIGINT, UNIQUE (server_name, key_id) );
26		CREATE TABLE rejections( event_id TEXT NOT NULL, reason TEXT NOT NULL, last_check TEXT NOT NULL, UNIQUE (event_id) );
27		CREATE TABLE push_rules ( id BIGINT PRIMARY KEY, user_name TEXT NOT NULL, rule_id TEXT NOT NULL, priority_class SMALLINT NOT NULL, priority INTEGER NOT NULL DEFAULT 0, conditions TEXT NOT NULL, actions TEXT NOT NULL, UNIQUE(user_name, rule_id) );
28		CREATE INDEX push_rules_user_name on push_rules (user_name);
29		CREATE TABLE user_filters( user_id TEXT, filter_id BIGINT, filter_json bytea );
30		CREATE INDEX user_filters_by_user_id_filter_id ON user_filters( user_id, filter_id );
31		CREATE TABLE push_rules_enable ( id BIGINT PRIMARY KEY, user_name TEXT NOT NULL, rule_id TEXT NOT NULL, enabled SMALLINT, UNIQUE(user_name, rule_id) );
32		CREATE INDEX push_rules_enable_user_name on push_rules_enable (user_name);
33		CREATE TABLE event_forward_extremities( event_id TEXT NOT NULL, room_id TEXT NOT NULL, UNIQUE (event_id, room_id) );
34		CREATE INDEX ev_extrem_room ON event_forward_extremities(room_id);
35		CREATE INDEX ev_extrem_id ON event_forward_extremities(event_id);
36		CREATE TABLE event_backward_extremities( event_id TEXT NOT NULL, room_id TEXT NOT NULL, UNIQUE (event_id, room_id) );
37		CREATE INDEX ev_b_extrem_room ON event_backward_extremities(room_id);
38		CREATE INDEX ev_b_extrem_id ON event_backward_extremities(event_id);
39		CREATE TABLE event_edges( event_id TEXT NOT NULL, prev_event_id TEXT NOT NULL, room_id TEXT NOT NULL, is_state BOOL NOT NULL, UNIQUE (event_id, prev_event_id, room_id, is_state) );
40		CREATE INDEX ev_edges_id ON event_edges(event_id);
41		CREATE INDEX ev_edges_prev_id ON event_edges(prev_event_id);
42		CREATE TABLE room_depth( room_id TEXT NOT NULL, min_depth INTEGER NOT NULL, UNIQUE (room_id) );
43		CREATE INDEX room_depth_room ON room_depth(room_id);
44		CREATE TABLE event_to_state_groups( event_id TEXT NOT NULL, state_group BIGINT NOT NULL, UNIQUE (event_id) );
45		CREATE TABLE local_media_repository ( media_id TEXT, media_type TEXT, media_length INTEGER, created_ts BIGINT, upload_name TEXT, user_id TEXT, quarantined_by TEXT, url_cache TEXT, last_access_ts BIGINT, UNIQUE (media_id) );
46		CREATE TABLE local_media_repository_thumbnails ( media_id TEXT, thumbnail_width INTEGER, thumbnail_height INTEGER, thumbnail_type TEXT, thumbnail_method TEXT, thumbnail_length INTEGER, UNIQUE ( media_id, thumbnail_width, thumbnail_height, thumbnail_type ) );
47		CREATE INDEX local_media_repository_thumbnails_media_id ON local_media_repository_thumbnails (media_id);
48		CREATE TABLE remote_media_cache ( media_origin TEXT, media_id TEXT, media_type TEXT, created_ts BIGINT, upload_name TEXT, media_length INTEGER, filesystem_id TEXT, last_access_ts BIGINT, quarantined_by TEXT, UNIQUE (media_origin, media_id) );
49		CREATE TABLE remote_media_cache_thumbnails ( media_origin TEXT, media_id TEXT, thumbnail_width INTEGER, thumbnail_height INTEGER, thumbnail_method TEXT, thumbnail_type TEXT, thumbnail_length INTEGER, filesystem_id TEXT, UNIQUE ( media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type ) );
50		CREATE TABLE redactions ( event_id TEXT NOT NULL, redacts TEXT NOT NULL, UNIQUE (event_id) );
51		CREATE INDEX redactions_redacts ON redactions (redacts);
52		CREATE TABLE room_aliases( room_alias TEXT NOT NULL, room_id TEXT NOT NULL, creator TEXT, UNIQUE (room_alias) );
53		CREATE INDEX room_aliases_id ON room_aliases(room_id);
54		CREATE TABLE room_alias_servers( room_alias TEXT NOT NULL, server TEXT NOT NULL );
55		CREATE INDEX room_alias_servers_alias ON room_alias_servers(room_alias);
56		CREATE TABLE event_reference_hashes ( event_id TEXT, algorithm TEXT, hash bytea, UNIQUE (event_id, algorithm) );
57		CREATE INDEX event_reference_hashes_id ON event_reference_hashes(event_id);
58		CREATE TABLE IF NOT EXISTS "server_keys_json" ( server_name TEXT NOT NULL, key_id TEXT NOT NULL, from_server TEXT NOT NULL, ts_added_ms BIGINT NOT NULL, ts_valid_until_ms BIGINT NOT NULL, key_json bytea NOT NULL, CONSTRAINT server_keys_json_uniqueness UNIQUE (server_name, key_id, from_server) );
59		CREATE TABLE e2e_device_keys_json ( user_id TEXT NOT NULL, device_id TEXT NOT NULL, ts_added_ms BIGINT NOT NULL, key_json TEXT NOT NULL, CONSTRAINT e2e_device_keys_json_uniqueness UNIQUE (user_id, device_id) );
60		CREATE TABLE e2e_one_time_keys_json ( user_id TEXT NOT NULL, device_id TEXT NOT NULL, algorithm TEXT NOT NULL, key_id TEXT NOT NULL, ts_added_ms BIGINT NOT NULL, key_json TEXT NOT NULL, CONSTRAINT e2e_one_time_keys_json_uniqueness UNIQUE (user_id, device_id, algorithm, key_id) );
61		CREATE TABLE receipts_graph( room_id TEXT NOT NULL, receipt_type TEXT NOT NULL, user_id TEXT NOT NULL, event_ids TEXT NOT NULL, data TEXT NOT NULL, CONSTRAINT receipts_graph_uniqueness UNIQUE (room_id, receipt_type, user_id) );
62		CREATE TABLE receipts_linearized ( stream_id BIGINT NOT NULL, room_id TEXT NOT NULL, receipt_type TEXT NOT NULL, user_id TEXT NOT NULL, event_id TEXT NOT NULL, data TEXT NOT NULL, CONSTRAINT receipts_linearized_uniqueness UNIQUE (room_id, receipt_type, user_id) );
63		CREATE INDEX receipts_linearized_id ON receipts_linearized( stream_id );
64		CREATE INDEX receipts_linearized_room_stream ON receipts_linearized( room_id, stream_id );
65		CREATE TABLE IF NOT EXISTS "user_threepids" ( user_id TEXT NOT NULL, medium TEXT NOT NULL, address TEXT NOT NULL, validated_at BIGINT NOT NULL, added_at BIGINT NOT NULL, CONSTRAINT medium_address UNIQUE (medium, address) );
66		CREATE INDEX user_threepids_user_id ON user_threepids(user_id);
67		CREATE VIRTUAL TABLE event_search USING fts4 ( event_id, room_id, sender, key, value )
68		/* event_search(event_id,room_id,sender,"key",value) */;
69		CREATE TABLE IF NOT EXISTS 'event_search_content'(docid INTEGER PRIMARY KEY, 'c0event_id', 'c1room_id', 'c2sender', 'c3key', 'c4value');
70		CREATE TABLE IF NOT EXISTS 'event_search_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
71		CREATE TABLE IF NOT EXISTS 'event_search_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));
72		CREATE TABLE IF NOT EXISTS 'event_search_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
73		CREATE TABLE IF NOT EXISTS 'event_search_stat'(id INTEGER PRIMARY KEY, value BLOB);
74		CREATE TABLE guest_access( event_id TEXT NOT NULL, room_id TEXT NOT NULL, guest_access TEXT NOT NULL, UNIQUE (event_id) );
75		CREATE TABLE history_visibility( event_id TEXT NOT NULL, room_id TEXT NOT NULL, history_visibility TEXT NOT NULL, UNIQUE (event_id) );
76		CREATE TABLE room_tags( user_id TEXT NOT NULL, room_id TEXT NOT NULL, tag TEXT NOT NULL, content TEXT NOT NULL, CONSTRAINT room_tag_uniqueness UNIQUE (user_id, room_id, tag) );
77		CREATE TABLE room_tags_revisions ( user_id TEXT NOT NULL, room_id TEXT NOT NULL, stream_id BIGINT NOT NULL, CONSTRAINT room_tag_revisions_uniqueness UNIQUE (user_id, room_id) );
78		CREATE TABLE IF NOT EXISTS "account_data_max_stream_id"( Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, stream_id BIGINT NOT NULL, CHECK (Lock='X') );
79		CREATE TABLE account_data( user_id TEXT NOT NULL, account_data_type TEXT NOT NULL, stream_id BIGINT NOT NULL, content TEXT NOT NULL, CONSTRAINT account_data_uniqueness UNIQUE (user_id, account_data_type) );
80		CREATE TABLE room_account_data( user_id TEXT NOT NULL, room_id TEXT NOT NULL, account_data_type TEXT NOT NULL, stream_id BIGINT NOT NULL, content TEXT NOT NULL, CONSTRAINT room_account_data_uniqueness UNIQUE (user_id, room_id, account_data_type) );
81		CREATE INDEX account_data_stream_id on account_data(user_id, stream_id);
82		CREATE INDEX room_account_data_stream_id on room_account_data(user_id, stream_id);
83		CREATE INDEX events_ts ON events(origin_server_ts, stream_ordering);
84		CREATE TABLE event_push_actions( room_id TEXT NOT NULL, event_id TEXT NOT NULL, user_id TEXT NOT NULL, profile_tag VARCHAR(32), actions TEXT NOT NULL, topological_ordering BIGINT, stream_ordering BIGINT, notif SMALLINT, highlight SMALLINT, CONSTRAINT event_id_user_id_profile_tag_uniqueness UNIQUE (room_id, event_id, user_id, profile_tag) );
85		CREATE INDEX event_push_actions_room_id_user_id on event_push_actions(room_id, user_id);
86		CREATE INDEX events_room_stream on events(room_id, stream_ordering);
87		CREATE INDEX public_room_index on rooms(is_public);
88		CREATE INDEX receipts_linearized_user ON receipts_linearized( user_id );
89		CREATE INDEX event_push_actions_rm_tokens on event_push_actions( user_id, room_id, topological_ordering, stream_ordering );
90		CREATE TABLE presence_stream( stream_id BIGINT, user_id TEXT, state TEXT, last_active_ts BIGINT, last_federation_update_ts BIGINT, last_user_sync_ts BIGINT, status_msg TEXT, currently_active BOOLEAN );
91		CREATE INDEX presence_stream_id ON presence_stream(stream_id, user_id);
92		CREATE INDEX presence_stream_user_id ON presence_stream(user_id);
93		CREATE TABLE push_rules_stream( stream_id BIGINT NOT NULL, event_stream_ordering BIGINT NOT NULL, user_id TEXT NOT NULL, rule_id TEXT NOT NULL, op TEXT NOT NULL, priority_class SMALLINT, priority INTEGER, conditions TEXT, actions TEXT );
94		CREATE INDEX push_rules_stream_id ON push_rules_stream(stream_id);
95		CREATE INDEX push_rules_stream_user_stream_id on push_rules_stream(user_id, stream_id);
96		CREATE TABLE ex_outlier_stream( event_stream_ordering BIGINT PRIMARY KEY NOT NULL, event_id TEXT NOT NULL, state_group BIGINT NOT NULL );
97		CREATE TABLE threepid_guest_access_tokens( medium TEXT, address TEXT, guest_access_token TEXT, first_inviter TEXT );
98		CREATE UNIQUE INDEX threepid_guest_access_tokens_index ON threepid_guest_access_tokens(medium, address);
99		CREATE TABLE local_invites( stream_id BIGINT NOT NULL, inviter TEXT NOT NULL, invitee TEXT NOT NULL, event_id TEXT NOT NULL, room_id TEXT NOT NULL, locally_rejected TEXT, replaced_by TEXT );
100		CREATE INDEX local_invites_id ON local_invites(stream_id);
101		CREATE INDEX local_invites_for_user_idx ON local_invites(invitee, locally_rejected, replaced_by, room_id);
102		CREATE INDEX event_push_actions_stream_ordering on event_push_actions( stream_ordering, user_id );
103		CREATE TABLE open_id_tokens ( token TEXT NOT NULL PRIMARY KEY, ts_valid_until_ms bigint NOT NULL, user_id TEXT NOT NULL, UNIQUE (token) );
104		CREATE INDEX open_id_tokens_ts_valid_until_ms ON open_id_tokens(ts_valid_until_ms);
105		CREATE TABLE pusher_throttle( pusher BIGINT NOT NULL, room_id TEXT NOT NULL, last_sent_ts BIGINT, throttle_ms BIGINT, PRIMARY KEY (pusher, room_id) );
106		CREATE TABLE event_reports( id BIGINT NOT NULL PRIMARY KEY, received_ts BIGINT NOT NULL, room_id TEXT NOT NULL, event_id TEXT NOT NULL, user_id TEXT NOT NULL, reason TEXT, content TEXT );
107		CREATE TABLE devices ( user_id TEXT NOT NULL, device_id TEXT NOT NULL, display_name TEXT, CONSTRAINT device_uniqueness UNIQUE (user_id, device_id) );
108		CREATE TABLE appservice_stream_position( Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, stream_ordering BIGINT, CHECK (Lock='X') );
109		CREATE TABLE device_inbox ( user_id TEXT NOT NULL, device_id TEXT NOT NULL, stream_id BIGINT NOT NULL, message_json TEXT NOT NULL );
110		CREATE INDEX device_inbox_user_stream_id ON device_inbox(user_id, device_id, stream_id);
111		CREATE INDEX received_transactions_ts ON received_transactions(ts);
112		CREATE TABLE device_federation_outbox ( destination TEXT NOT NULL, stream_id BIGINT NOT NULL, queued_ts BIGINT NOT NULL, messages_json TEXT NOT NULL );
113		CREATE INDEX device_federation_outbox_destination_id ON device_federation_outbox(destination, stream_id);
114		CREATE TABLE device_federation_inbox ( origin TEXT NOT NULL, message_id TEXT NOT NULL, received_ts BIGINT NOT NULL );
115		CREATE INDEX device_federation_inbox_sender_id ON device_federation_inbox(origin, message_id);
116		CREATE TABLE device_max_stream_id ( stream_id BIGINT NOT NULL );
117		CREATE TABLE public_room_list_stream ( stream_id BIGINT NOT NULL, room_id TEXT NOT NULL, visibility BOOLEAN NOT NULL , appservice_id TEXT, network_id TEXT);
118		CREATE INDEX public_room_list_stream_idx on public_room_list_stream( stream_id );
119		CREATE INDEX public_room_list_stream_rm_idx on public_room_list_stream( room_id, stream_id );
120		CREATE TABLE stream_ordering_to_exterm ( stream_ordering BIGINT NOT NULL, room_id TEXT NOT NULL, event_id TEXT NOT NULL );
121		CREATE INDEX stream_ordering_to_exterm_idx on stream_ordering_to_exterm( stream_ordering );
122		CREATE INDEX stream_ordering_to_exterm_rm_idx on stream_ordering_to_exterm( room_id, stream_ordering );
123		CREATE TABLE IF NOT EXISTS "event_auth"( event_id TEXT NOT NULL, auth_id TEXT NOT NULL, room_id TEXT NOT NULL );
124		CREATE INDEX evauth_edges_id ON event_auth(event_id);
125		CREATE INDEX user_threepids_medium_address on user_threepids (medium, address);
126		CREATE TABLE appservice_room_list( appservice_id TEXT NOT NULL, network_id TEXT NOT NULL, room_id TEXT NOT NULL );
127		CREATE UNIQUE INDEX appservice_room_list_idx ON appservice_room_list( appservice_id, network_id, room_id );
128		CREATE INDEX device_federation_outbox_id ON device_federation_outbox(stream_id);
129		CREATE TABLE federation_stream_position( type TEXT NOT NULL, stream_id INTEGER NOT NULL );
130		CREATE TABLE device_lists_remote_cache ( user_id TEXT NOT NULL, device_id TEXT NOT NULL, content TEXT NOT NULL );
131		CREATE TABLE device_lists_remote_extremeties ( user_id TEXT NOT NULL, stream_id TEXT NOT NULL );
132		CREATE TABLE device_lists_stream ( stream_id BIGINT NOT NULL, user_id TEXT NOT NULL, device_id TEXT NOT NULL );
133		CREATE INDEX device_lists_stream_id ON device_lists_stream(stream_id, user_id);
134		CREATE TABLE device_lists_outbound_pokes ( destination TEXT NOT NULL, stream_id BIGINT NOT NULL, user_id TEXT NOT NULL, device_id TEXT NOT NULL, sent BOOLEAN NOT NULL, ts BIGINT NOT NULL );
135		CREATE INDEX device_lists_outbound_pokes_id ON device_lists_outbound_pokes(destination, stream_id);
136		CREATE INDEX device_lists_outbound_pokes_user ON device_lists_outbound_pokes(destination, user_id);
137		CREATE TABLE event_push_summary ( user_id TEXT NOT NULL, room_id TEXT NOT NULL, notif_count BIGINT NOT NULL, stream_ordering BIGINT NOT NULL );
138		CREATE INDEX event_push_summary_user_rm ON event_push_summary(user_id, room_id);
139		CREATE TABLE event_push_summary_stream_ordering ( Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, stream_ordering BIGINT NOT NULL, CHECK (Lock='X') );
140		CREATE TABLE IF NOT EXISTS "pushers" ( id BIGINT PRIMARY KEY, user_name TEXT NOT NULL, access_token BIGINT DEFAULT NULL, profile_tag TEXT NOT NULL, kind TEXT NOT NULL, app_id TEXT NOT NULL, app_display_name TEXT NOT NULL, device_display_name TEXT NOT NULL, pushkey TEXT NOT NULL, ts BIGINT NOT NULL, lang TEXT, data TEXT, last_stream_ordering INTEGER, last_success BIGINT, failing_since BIGINT, UNIQUE (app_id, pushkey, user_name) );
141		CREATE INDEX device_lists_outbound_pokes_stream ON device_lists_outbound_pokes(stream_id);
142		CREATE TABLE ratelimit_override ( user_id TEXT NOT NULL, messages_per_second BIGINT, burst_count BIGINT );
143		CREATE UNIQUE INDEX ratelimit_override_idx ON ratelimit_override(user_id);
144		CREATE TABLE current_state_delta_stream ( stream_id BIGINT NOT NULL, room_id TEXT NOT NULL, type TEXT NOT NULL, state_key TEXT NOT NULL, event_id TEXT, prev_event_id TEXT );
145		CREATE INDEX current_state_delta_stream_idx ON current_state_delta_stream(stream_id);
146		CREATE TABLE device_lists_outbound_last_success ( destination TEXT NOT NULL, user_id TEXT NOT NULL, stream_id BIGINT NOT NULL );
147		CREATE INDEX device_lists_outbound_last_success_idx ON device_lists_outbound_last_success( destination, user_id, stream_id );
148		CREATE TABLE user_directory_stream_pos ( Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, stream_id BIGINT, CHECK (Lock='X') );
149		CREATE VIRTUAL TABLE user_directory_search USING fts4 ( user_id, value )
150		/* user_directory_search(user_id,value) */;
151		CREATE TABLE IF NOT EXISTS 'user_directory_search_content'(docid INTEGER PRIMARY KEY, 'c0user_id', 'c1value');
152		CREATE TABLE IF NOT EXISTS 'user_directory_search_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
153		CREATE TABLE IF NOT EXISTS 'user_directory_search_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));
154		CREATE TABLE IF NOT EXISTS 'user_directory_search_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
155		CREATE TABLE IF NOT EXISTS 'user_directory_search_stat'(id INTEGER PRIMARY KEY, value BLOB);
156		CREATE TABLE blocked_rooms ( room_id TEXT NOT NULL, user_id TEXT NOT NULL );
157		CREATE UNIQUE INDEX blocked_rooms_idx ON blocked_rooms(room_id);
158		CREATE TABLE IF NOT EXISTS "local_media_repository_url_cache"( url TEXT, response_code INTEGER, etag TEXT, expires_ts BIGINT, og TEXT, media_id TEXT, download_ts BIGINT );
159		CREATE INDEX local_media_repository_url_cache_expires_idx ON local_media_repository_url_cache(expires_ts);
160		CREATE INDEX local_media_repository_url_cache_by_url_download_ts ON local_media_repository_url_cache(url, download_ts);
161		CREATE INDEX local_media_repository_url_cache_media_idx ON local_media_repository_url_cache(media_id);
162		CREATE TABLE group_users ( group_id TEXT NOT NULL, user_id TEXT NOT NULL, is_admin BOOLEAN NOT NULL, is_public BOOLEAN NOT NULL );
163		CREATE TABLE group_invites ( group_id TEXT NOT NULL, user_id TEXT NOT NULL );
164		CREATE TABLE group_rooms ( group_id TEXT NOT NULL, room_id TEXT NOT NULL, is_public BOOLEAN NOT NULL );
165		CREATE TABLE group_summary_rooms ( group_id TEXT NOT NULL, room_id TEXT NOT NULL, category_id TEXT NOT NULL, room_order BIGINT NOT NULL, is_public BOOLEAN NOT NULL, UNIQUE (group_id, category_id, room_id, room_order), CHECK (room_order > 0) );
166		CREATE UNIQUE INDEX group_summary_rooms_g_idx ON group_summary_rooms(group_id, room_id, category_id);
167		CREATE TABLE group_summary_room_categories ( group_id TEXT NOT NULL, category_id TEXT NOT NULL, cat_order BIGINT NOT NULL, UNIQUE (group_id, category_id, cat_order), CHECK (cat_order > 0) );
168		CREATE TABLE group_room_categories ( group_id TEXT NOT NULL, category_id TEXT NOT NULL, profile TEXT NOT NULL, is_public BOOLEAN NOT NULL, UNIQUE (group_id, category_id) );
169		CREATE TABLE group_summary_users ( group_id TEXT NOT NULL, user_id TEXT NOT NULL, role_id TEXT NOT NULL, user_order BIGINT NOT NULL, is_public BOOLEAN NOT NULL );
170		CREATE INDEX group_summary_users_g_idx ON group_summary_users(group_id);
171		CREATE TABLE group_summary_roles ( group_id TEXT NOT NULL, role_id TEXT NOT NULL, role_order BIGINT NOT NULL, UNIQUE (group_id, role_id, role_order), CHECK (role_order > 0) );
172		CREATE TABLE group_roles ( group_id TEXT NOT NULL, role_id TEXT NOT NULL, profile TEXT NOT NULL, is_public BOOLEAN NOT NULL, UNIQUE (group_id, role_id) );
173		CREATE TABLE group_attestations_renewals ( group_id TEXT NOT NULL, user_id TEXT NOT NULL, valid_until_ms BIGINT NOT NULL );
174		CREATE INDEX group_attestations_renewals_g_idx ON group_attestations_renewals(group_id, user_id);
175		CREATE INDEX group_attestations_renewals_u_idx ON group_attestations_renewals(user_id);
176		CREATE INDEX group_attestations_renewals_v_idx ON group_attestations_renewals(valid_until_ms);
177		CREATE TABLE group_attestations_remote ( group_id TEXT NOT NULL, user_id TEXT NOT NULL, valid_until_ms BIGINT NOT NULL, attestation_json TEXT NOT NULL );
178		CREATE INDEX group_attestations_remote_g_idx ON group_attestations_remote(group_id, user_id);
179		CREATE INDEX group_attestations_remote_u_idx ON group_attestations_remote(user_id);
180		CREATE INDEX group_attestations_remote_v_idx ON group_attestations_remote(valid_until_ms);
181		CREATE TABLE local_group_membership ( group_id TEXT NOT NULL, user_id TEXT NOT NULL, is_admin BOOLEAN NOT NULL, membership TEXT NOT NULL, is_publicised BOOLEAN NOT NULL, content TEXT NOT NULL );
182		CREATE INDEX local_group_membership_u_idx ON local_group_membership(user_id, group_id);
183		CREATE INDEX local_group_membership_g_idx ON local_group_membership(group_id);
184		CREATE TABLE local_group_updates ( stream_id BIGINT NOT NULL, group_id TEXT NOT NULL, user_id TEXT NOT NULL, type TEXT NOT NULL, content TEXT NOT NULL );
185		CREATE TABLE remote_profile_cache ( user_id TEXT NOT NULL, displayname TEXT, avatar_url TEXT, last_check BIGINT NOT NULL );
186		CREATE UNIQUE INDEX remote_profile_cache_user_id ON remote_profile_cache(user_id);
187		CREATE INDEX remote_profile_cache_time ON remote_profile_cache(last_check);
188		CREATE TABLE IF NOT EXISTS "deleted_pushers" ( stream_id BIGINT NOT NULL, app_id TEXT NOT NULL, pushkey TEXT NOT NULL, user_id TEXT NOT NULL );
189		CREATE INDEX deleted_pushers_stream_id ON deleted_pushers (stream_id);
190		CREATE TABLE IF NOT EXISTS "groups" ( group_id TEXT NOT NULL, name TEXT, avatar_url TEXT, short_description TEXT, long_description TEXT, is_public BOOL NOT NULL , join_policy TEXT NOT NULL DEFAULT 'invite');
191		CREATE UNIQUE INDEX groups_idx ON groups(group_id);
192		CREATE TABLE IF NOT EXISTS "user_directory" ( user_id TEXT NOT NULL, room_id TEXT, display_name TEXT, avatar_url TEXT );
193		CREATE INDEX user_directory_room_idx ON user_directory(room_id);
194		CREATE UNIQUE INDEX user_directory_user_idx ON user_directory(user_id);
195		CREATE TABLE event_push_actions_staging ( event_id TEXT NOT NULL, user_id TEXT NOT NULL, actions TEXT NOT NULL, notif SMALLINT NOT NULL, highlight SMALLINT NOT NULL );
196		CREATE INDEX event_push_actions_staging_id ON event_push_actions_staging(event_id);
197		CREATE TABLE users_pending_deactivation ( user_id TEXT NOT NULL );
198		CREATE UNIQUE INDEX group_invites_g_idx ON group_invites(group_id, user_id);
199		CREATE UNIQUE INDEX group_users_g_idx ON group_users(group_id, user_id);
200		CREATE INDEX group_users_u_idx ON group_users(user_id);
201		CREATE INDEX group_invites_u_idx ON group_invites(user_id);
202		CREATE UNIQUE INDEX group_rooms_g_idx ON group_rooms(group_id, room_id);
203		CREATE INDEX group_rooms_r_idx ON group_rooms(room_id);
204		CREATE TABLE user_daily_visits ( user_id TEXT NOT NULL, device_id TEXT, timestamp BIGINT NOT NULL );
205		CREATE INDEX user_daily_visits_uts_idx ON user_daily_visits(user_id, timestamp);
206		CREATE INDEX user_daily_visits_ts_idx ON user_daily_visits(timestamp);
207		CREATE TABLE erased_users ( user_id TEXT NOT NULL );
208		CREATE UNIQUE INDEX erased_users_user ON erased_users(user_id);
209		CREATE TABLE monthly_active_users ( user_id TEXT NOT NULL, timestamp BIGINT NOT NULL );
210		CREATE UNIQUE INDEX monthly_active_users_users ON monthly_active_users(user_id);
211		CREATE INDEX monthly_active_users_time_stamp ON monthly_active_users(timestamp);
212		CREATE TABLE IF NOT EXISTS "e2e_room_keys_versions" ( user_id TEXT NOT NULL, version BIGINT NOT NULL, algorithm TEXT NOT NULL, auth_data TEXT NOT NULL, deleted SMALLINT DEFAULT 0 NOT NULL );
213		CREATE UNIQUE INDEX e2e_room_keys_versions_idx ON e2e_room_keys_versions(user_id, version);
214		CREATE TABLE IF NOT EXISTS "e2e_room_keys" ( user_id TEXT NOT NULL, room_id TEXT NOT NULL, session_id TEXT NOT NULL, version BIGINT NOT NULL, first_message_index INT, forwarded_count INT, is_verified BOOLEAN, session_data TEXT NOT NULL );
215		CREATE UNIQUE INDEX e2e_room_keys_idx ON e2e_room_keys(user_id, room_id, session_id);
216		CREATE TABLE users_who_share_private_rooms ( user_id TEXT NOT NULL, other_user_id TEXT NOT NULL, room_id TEXT NOT NULL );
217		CREATE UNIQUE INDEX users_who_share_private_rooms_u_idx ON users_who_share_private_rooms(user_id, other_user_id, room_id);
218		CREATE INDEX users_who_share_private_rooms_r_idx ON users_who_share_private_rooms(room_id);
219		CREATE INDEX users_who_share_private_rooms_o_idx ON users_who_share_private_rooms(other_user_id);
220		CREATE TABLE user_threepid_id_server ( user_id TEXT NOT NULL, medium TEXT NOT NULL, address TEXT NOT NULL, id_server TEXT NOT NULL );
221		CREATE UNIQUE INDEX user_threepid_id_server_idx ON user_threepid_id_server( user_id, medium, address, id_server );
222		CREATE TABLE users_in_public_rooms ( user_id TEXT NOT NULL, room_id TEXT NOT NULL );
223		CREATE UNIQUE INDEX users_in_public_rooms_u_idx ON users_in_public_rooms(user_id, room_id);
224		CREATE TABLE account_validity ( user_id TEXT PRIMARY KEY, expiration_ts_ms BIGINT NOT NULL, email_sent BOOLEAN NOT NULL, renewal_token TEXT );
225		CREATE TABLE event_relations ( event_id TEXT NOT NULL, relates_to_id TEXT NOT NULL, relation_type TEXT NOT NULL, aggregation_key TEXT );
226		CREATE UNIQUE INDEX event_relations_id ON event_relations(event_id);
227		CREATE INDEX event_relations_relates ON event_relations(relates_to_id, relation_type, aggregation_key);
228		CREATE TABLE stats_stream_pos ( Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, stream_id BIGINT, CHECK (Lock='X') );
229		CREATE TABLE user_stats ( user_id TEXT NOT NULL, ts BIGINT NOT NULL, bucket_size INT NOT NULL, public_rooms INT NOT NULL, private_rooms INT NOT NULL );
230		CREATE UNIQUE INDEX user_stats_user_ts ON user_stats(user_id, ts);
231		CREATE TABLE room_stats ( room_id TEXT NOT NULL, ts BIGINT NOT NULL, bucket_size INT NOT NULL, current_state_events INT NOT NULL, joined_members INT NOT NULL, invited_members INT NOT NULL, left_members INT NOT NULL, banned_members INT NOT NULL, state_events INT NOT NULL );
232		CREATE UNIQUE INDEX room_stats_room_ts ON room_stats(room_id, ts);
233		CREATE TABLE room_state ( room_id TEXT NOT NULL, join_rules TEXT, history_visibility TEXT, encryption TEXT, name TEXT, topic TEXT, avatar TEXT, canonical_alias TEXT );
234		CREATE UNIQUE INDEX room_state_room ON room_state(room_id);
235		CREATE TABLE room_stats_earliest_token ( room_id TEXT NOT NULL, token BIGINT NOT NULL );
236		CREATE UNIQUE INDEX room_stats_earliest_token_idx ON room_stats_earliest_token(room_id);
237		CREATE INDEX access_tokens_device_id ON access_tokens (user_id, device_id);
238		CREATE INDEX user_ips_device_id ON user_ips (user_id, device_id, last_seen);
239		CREATE INDEX event_contains_url_index ON events (room_id, topological_ordering, stream_ordering);
240		CREATE INDEX event_push_actions_u_highlight ON event_push_actions (user_id, stream_ordering);
241		CREATE INDEX event_push_actions_highlights_index ON event_push_actions (user_id, room_id, topological_ordering, stream_ordering);
242		CREATE INDEX current_state_events_member_index ON current_state_events (state_key);
243		CREATE INDEX device_inbox_stream_id_user_id ON device_inbox (stream_id, user_id);
244		CREATE INDEX device_lists_stream_user_id ON device_lists_stream (user_id, device_id);
245		CREATE INDEX local_media_repository_url_idx ON local_media_repository (created_ts);
246		CREATE INDEX user_ips_last_seen ON user_ips (user_id, last_seen);
247		CREATE INDEX user_ips_last_seen_only ON user_ips (last_seen);
248		CREATE INDEX users_creation_ts ON users (creation_ts);
249		CREATE INDEX event_to_state_groups_sg_index ON event_to_state_groups (state_group);
250		CREATE UNIQUE INDEX device_lists_remote_cache_unique_id ON device_lists_remote_cache (user_id, device_id);
251		CREATE UNIQUE INDEX device_lists_remote_extremeties_unique_idx ON device_lists_remote_extremeties (user_id);
252		CREATE UNIQUE INDEX user_ips_user_token_ip_unique_index ON user_ips (user_id, access_token, ip);

+0

-8

~~synapse/storage/data_stores/main/schema/full_schemas/54/stream_positions.sql~~ less more

0
1		INSERT INTO appservice_stream_position (stream_ordering) SELECT COALESCE(MAX(stream_ordering), 0) FROM events;
2		INSERT INTO federation_stream_position (type, stream_id) VALUES ('federation', -1);
3		INSERT INTO federation_stream_position (type, stream_id) SELECT 'events', coalesce(max(stream_ordering), -1) FROM events;
4		INSERT INTO user_directory_stream_pos (stream_id) VALUES (0);
5		INSERT INTO stats_stream_pos (stream_id) VALUES (0);
6		INSERT INTO event_push_summary_stream_ordering (stream_ordering) VALUES (0);
7		-- device_max_stream_id is handled separately in 56/device_stream_id_insert.sql⏎

+0

-21

~~synapse/storage/data_stores/main/schema/full_schemas/README.md~~ less more

0		# Synapse Database Schemas
1
2		These schemas are used as a basis to create brand new Synapse databases, on both
3		SQLite3 and Postgres.
4
5		## Building full schema dumps
6
7		If you want to recreate these schemas, they need to be made from a database that
8		has had all background updates run.
9
10		To do so, use `scripts-dev/make_full_schema.sh`. This will produce new
11		`full.sql.postgres ` and `full.sql.sqlite` files.
12
13		Ensure postgres is installed and your user has the ability to run bash commands
14		such as `createdb`, then call
15
16		./scripts-dev/make_full_schema.sh -p postgres_username -o output_dir/
17
18		There are currently two folders with full-schema snapshots. `16` is a snapshot
19		from 2015, for historical reference. The other contains the most recent full
20		schema snapshot.

+0

-708

~~synapse/storage/data_stores/main/search.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2015, 2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16		import re
17		from collections import namedtuple
18
19		from twisted.internet import defer
20
21		from synapse.api.errors import SynapseError
22		from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
23		from synapse.storage.data_stores.main.events_worker import EventRedactBehaviour
24		from synapse.storage.database import Database
25		from synapse.storage.engines import PostgresEngine, Sqlite3Engine
26
27		logger = logging.getLogger(__name__)
28
29		SearchEntry = namedtuple(
30		"SearchEntry",
31		["key", "value", "event_id", "room_id", "stream_ordering", "origin_server_ts"],
32		)
33
34
35		class SearchWorkerStore(SQLBaseStore):
36		def store_search_entries_txn(self, txn, entries):
37		"""Add entries to the search table
38
39		Args:
40		txn (cursor):
41		entries (iterable[SearchEntry]):
42		entries to be added to the table
43		"""
44		if not self.hs.config.enable_search:
45		return
46		if isinstance(self.database_engine, PostgresEngine):
47		sql = (
48		"INSERT INTO event_search"
49		" (event_id, room_id, key, vector, stream_ordering, origin_server_ts)"
50		" VALUES (?,?,?,to_tsvector('english', ?),?,?)"
51		)
52
53		args = (
54		(
55		entry.event_id,
56		entry.room_id,
57		entry.key,
58		entry.value,
59		entry.stream_ordering,
60		entry.origin_server_ts,
61		)
62		for entry in entries
63		)
64
65		txn.executemany(sql, args)
66
67		elif isinstance(self.database_engine, Sqlite3Engine):
68		sql = (
69		"INSERT INTO event_search (event_id, room_id, key, value)"
70		" VALUES (?,?,?,?)"
71		)
72		args = (
73		(entry.event_id, entry.room_id, entry.key, entry.value)
74		for entry in entries
75		)
76
77		txn.executemany(sql, args)
78		else:
79		# This should be unreachable.
80		raise Exception("Unrecognized database engine")
81
82
83		class SearchBackgroundUpdateStore(SearchWorkerStore):
84
85		EVENT_SEARCH_UPDATE_NAME = "event_search"
86		EVENT_SEARCH_ORDER_UPDATE_NAME = "event_search_order"
87		EVENT_SEARCH_USE_GIST_POSTGRES_NAME = "event_search_postgres_gist"
88		EVENT_SEARCH_USE_GIN_POSTGRES_NAME = "event_search_postgres_gin"
89
90		def __init__(self, database: Database, db_conn, hs):
91		super(SearchBackgroundUpdateStore, self).__init__(database, db_conn, hs)
92
93		if not hs.config.enable_search:
94		return
95
96		self.db.updates.register_background_update_handler(
97		self.EVENT_SEARCH_UPDATE_NAME, self._background_reindex_search
98		)
99		self.db.updates.register_background_update_handler(
100		self.EVENT_SEARCH_ORDER_UPDATE_NAME, self._background_reindex_search_order
101		)
102
103		# we used to have a background update to turn the GIN index into a
104		# GIST one; we no longer do that (obviously) because we actually want
105		# a GIN index. However, it's possible that some people might still have
106		# the background update queued, so we register a handler to clear the
107		# background update.
108		self.db.updates.register_noop_background_update(
109		self.EVENT_SEARCH_USE_GIST_POSTGRES_NAME
110		)
111
112		self.db.updates.register_background_update_handler(
113		self.EVENT_SEARCH_USE_GIN_POSTGRES_NAME, self._background_reindex_gin_search
114		)
115
116		@defer.inlineCallbacks
117		def _background_reindex_search(self, progress, batch_size):
118		# we work through the events table from highest stream id to lowest
119		target_min_stream_id = progress["target_min_stream_id_inclusive"]
120		max_stream_id = progress["max_stream_id_exclusive"]
121		rows_inserted = progress.get("rows_inserted", 0)
122
123		TYPES = ["m.room.name", "m.room.message", "m.room.topic"]
124
125		def reindex_search_txn(txn):
126		sql = (
127		"SELECT stream_ordering, event_id, room_id, type, json, "
128		" origin_server_ts FROM events"
129		" JOIN event_json USING (room_id, event_id)"
130		" WHERE ? <= stream_ordering AND stream_ordering < ?"
131		" AND (%s)"
132		" ORDER BY stream_ordering DESC"
133		" LIMIT ?"
134		) % (" OR ".join("type = '%s'" % (t,) for t in TYPES),)
135
136		txn.execute(sql, (target_min_stream_id, max_stream_id, batch_size))
137
138		# we could stream straight from the results into
139		# store_search_entries_txn with a generator function, but that
140		# would mean having two cursors open on the database at once.
141		# Instead we just build a list of results.
142		rows = self.db.cursor_to_dict(txn)
143		if not rows:
144		return 0
145
146		min_stream_id = rows[-1]["stream_ordering"]
147
148		event_search_rows = []
149		for row in rows:
150		try:
151		event_id = row["event_id"]
152		room_id = row["room_id"]
153		etype = row["type"]
154		stream_ordering = row["stream_ordering"]
155		origin_server_ts = row["origin_server_ts"]
156		try:
157		event_json = db_to_json(row["json"])
158		content = event_json["content"]
159		except Exception:
160		continue
161
162		if etype == "m.room.message":
163		key = "content.body"
164		value = content["body"]
165		elif etype == "m.room.topic":
166		key = "content.topic"
167		value = content["topic"]
168		elif etype == "m.room.name":
169		key = "content.name"
170		value = content["name"]
171		else:
172		raise Exception("unexpected event type %s" % etype)
173		except (KeyError, AttributeError):
174		# If the event is missing a necessary field then
175		# skip over it.
176		continue
177
178		if not isinstance(value, str):
179		# If the event body, name or topic isn't a string
180		# then skip over it
181		continue
182
183		event_search_rows.append(
184		SearchEntry(
185		key=key,
186		value=value,
187		event_id=event_id,
188		room_id=room_id,
189		stream_ordering=stream_ordering,
190		origin_server_ts=origin_server_ts,
191		)
192		)
193
194		self.store_search_entries_txn(txn, event_search_rows)
195
196		progress = {
197		"target_min_stream_id_inclusive": target_min_stream_id,
198		"max_stream_id_exclusive": min_stream_id,
199		"rows_inserted": rows_inserted + len(event_search_rows),
200		}
201
202		self.db.updates._background_update_progress_txn(
203		txn, self.EVENT_SEARCH_UPDATE_NAME, progress
204		)
205
206		return len(event_search_rows)
207
208		result = yield self.db.runInteraction(
209		self.EVENT_SEARCH_UPDATE_NAME, reindex_search_txn
210		)
211
212		if not result:
213		yield self.db.updates._end_background_update(self.EVENT_SEARCH_UPDATE_NAME)
214
215		return result
216
217		@defer.inlineCallbacks
218		def _background_reindex_gin_search(self, progress, batch_size):
219		"""This handles old synapses which used GIST indexes, if any;
220		converting them back to be GIN as per the actual schema.
221		"""
222
223		def create_index(conn):
224		conn.rollback()
225
226		# we have to set autocommit, because postgres refuses to
227		# CREATE INDEX CONCURRENTLY without it.
228		conn.set_session(autocommit=True)
229
230		try:
231		c = conn.cursor()
232
233		# if we skipped the conversion to GIST, we may already/still
234		# have an event_search_fts_idx; unfortunately postgres 9.4
235		# doesn't support CREATE INDEX IF EXISTS so we just catch the
236		# exception and ignore it.
237		import psycopg2
238
239		try:
240		c.execute(
241		"CREATE INDEX CONCURRENTLY event_search_fts_idx"
242		" ON event_search USING GIN (vector)"
243		)
244		except psycopg2.ProgrammingError as e:
245		logger.warning(
246		"Ignoring error %r when trying to switch from GIST to GIN", e
247		)
248
249		# we should now be able to delete the GIST index.
250		c.execute("DROP INDEX IF EXISTS event_search_fts_idx_gist")
251		finally:
252		conn.set_session(autocommit=False)
253
254		if isinstance(self.database_engine, PostgresEngine):
255		yield self.db.runWithConnection(create_index)
256
257		yield self.db.updates._end_background_update(
258		self.EVENT_SEARCH_USE_GIN_POSTGRES_NAME
259		)
260		return 1
261
262		@defer.inlineCallbacks
263		def _background_reindex_search_order(self, progress, batch_size):
264		target_min_stream_id = progress["target_min_stream_id_inclusive"]
265		max_stream_id = progress["max_stream_id_exclusive"]
266		rows_inserted = progress.get("rows_inserted", 0)
267		have_added_index = progress["have_added_indexes"]
268
269		if not have_added_index:
270
271		def create_index(conn):
272		conn.rollback()
273		conn.set_session(autocommit=True)
274		c = conn.cursor()
275
276		# We create with NULLS FIRST so that when we search backwards
277		# we get the ones with non null origin_server_ts first
278		c.execute(
279		"CREATE INDEX CONCURRENTLY event_search_room_order ON event_search("
280		"room_id, origin_server_ts NULLS FIRST, stream_ordering NULLS FIRST)"
281		)
282		c.execute(
283		"CREATE INDEX CONCURRENTLY event_search_order ON event_search("
284		"origin_server_ts NULLS FIRST, stream_ordering NULLS FIRST)"
285		)
286		conn.set_session(autocommit=False)
287
288		yield self.db.runWithConnection(create_index)
289
290		pg = dict(progress)
291		pg["have_added_indexes"] = True
292
293		yield self.db.runInteraction(
294		self.EVENT_SEARCH_ORDER_UPDATE_NAME,
295		self.db.updates._background_update_progress_txn,
296		self.EVENT_SEARCH_ORDER_UPDATE_NAME,
297		pg,
298		)
299
300		def reindex_search_txn(txn):
301		sql = (
302		"UPDATE event_search AS es SET stream_ordering = e.stream_ordering,"
303		" origin_server_ts = e.origin_server_ts"
304		" FROM events AS e"
305		" WHERE e.event_id = es.event_id"
306		" AND ? <= e.stream_ordering AND e.stream_ordering < ?"
307		" RETURNING es.stream_ordering"
308		)
309
310		min_stream_id = max_stream_id - batch_size
311		txn.execute(sql, (min_stream_id, max_stream_id))
312		rows = txn.fetchall()
313
314		if min_stream_id < target_min_stream_id:
315		# We've recached the end.
316		return len(rows), False
317
318		progress = {
319		"target_min_stream_id_inclusive": target_min_stream_id,
320		"max_stream_id_exclusive": min_stream_id,
321		"rows_inserted": rows_inserted + len(rows),
322		"have_added_indexes": True,
323		}
324
325		self.db.updates._background_update_progress_txn(
326		txn, self.EVENT_SEARCH_ORDER_UPDATE_NAME, progress
327		)
328
329		return len(rows), True
330
331		num_rows, finished = yield self.db.runInteraction(
332		self.EVENT_SEARCH_ORDER_UPDATE_NAME, reindex_search_txn
333		)
334
335		if not finished:
336		yield self.db.updates._end_background_update(
337		self.EVENT_SEARCH_ORDER_UPDATE_NAME
338		)
339
340		return num_rows
341
342
343		class SearchStore(SearchBackgroundUpdateStore):
344		def __init__(self, database: Database, db_conn, hs):
345		super(SearchStore, self).__init__(database, db_conn, hs)
346
347		@defer.inlineCallbacks
348		def search_msgs(self, room_ids, search_term, keys):
349		"""Performs a full text search over events with given keys.
350
351		Args:
352		room_ids (list): List of room ids to search in
353		search_term (str): Search term to search for
354		keys (list): List of keys to search in, currently supports
355		"content.body", "content.name", "content.topic"
356
357		Returns:
358		list of dicts
359		"""
360		clauses = []
361
362		search_query = _parse_query(self.database_engine, search_term)
363
364		args = []
365
366		# Make sure we don't explode because the person is in too many rooms.
367		# We filter the results below regardless.
368		if len(room_ids) < 500:
369		clause, args = make_in_list_sql_clause(
370		self.database_engine, "room_id", room_ids
371		)
372		clauses = [clause]
373
374		local_clauses = []
375		for key in keys:
376		local_clauses.append("key = ?")
377		args.append(key)
378
379		clauses.append("(%s)" % (" OR ".join(local_clauses),))
380
381		count_args = args
382		count_clauses = clauses
383
384		if isinstance(self.database_engine, PostgresEngine):
385		sql = (
386		"SELECT ts_rank_cd(vector, to_tsquery('english', ?)) AS rank,"
387		" room_id, event_id"
388		" FROM event_search"
389		" WHERE vector @@ to_tsquery('english', ?)"
390		)
391		args = [search_query, search_query] + args
392
393		count_sql = (
394		"SELECT room_id, count(*) as count FROM event_search"
395		" WHERE vector @@ to_tsquery('english', ?)"
396		)
397		count_args = [search_query] + count_args
398		elif isinstance(self.database_engine, Sqlite3Engine):
399		sql = (
400		"SELECT rank(matchinfo(event_search)) as rank, room_id, event_id"
401		" FROM event_search"
402		" WHERE value MATCH ?"
403		)
404		args = [search_query] + args
405
406		count_sql = (
407		"SELECT room_id, count(*) as count FROM event_search"
408		" WHERE value MATCH ?"
409		)
410		count_args = [search_term] + count_args
411		else:
412		# This should be unreachable.
413		raise Exception("Unrecognized database engine")
414
415		for clause in clauses:
416		sql += " AND " + clause
417
418		for clause in count_clauses:
419		count_sql += " AND " + clause
420
421		# We add an arbitrary limit here to ensure we don't try to pull the
422		# entire table from the database.
423		sql += " ORDER BY rank DESC LIMIT 500"
424
425		results = yield self.db.execute(
426		"search_msgs", self.db.cursor_to_dict, sql, *args
427		)
428
429		results = list(filter(lambda row: row["room_id"] in room_ids, results))
430
431		# We set redact_behaviour to BLOCK here to prevent redacted events being returned in
432		# search results (which is a data leak)
433		events = yield self.get_events_as_list(
434		[r["event_id"] for r in results],
435		redact_behaviour=EventRedactBehaviour.BLOCK,
436		)
437
438		event_map = {ev.event_id: ev for ev in events}
439
440		highlights = None
441		if isinstance(self.database_engine, PostgresEngine):
442		highlights = yield self._find_highlights_in_postgres(search_query, events)
443
444		count_sql += " GROUP BY room_id"
445
446		count_results = yield self.db.execute(
447		"search_rooms_count", self.db.cursor_to_dict, count_sql, *count_args
448		)
449
450		count = sum(row["count"] for row in count_results if row["room_id"] in room_ids)
451
452		return {
453		"results": [
454		{"event": event_map[r["event_id"]], "rank": r["rank"]}
455		for r in results
456		if r["event_id"] in event_map
457		],
458		"highlights": highlights,
459		"count": count,
460		}
461
462		@defer.inlineCallbacks
463		def search_rooms(self, room_ids, search_term, keys, limit, pagination_token=None):
464		"""Performs a full text search over events with given keys.
465
466		Args:
467		room_id (list): The room_ids to search in
468		search_term (str): Search term to search for
469		keys (list): List of keys to search in, currently supports
470		"content.body", "content.name", "content.topic"
471		pagination_token (str): A pagination token previously returned
472
473		Returns:
474		list of dicts
475		"""
476		clauses = []
477
478		search_query = _parse_query(self.database_engine, search_term)
479
480		args = []
481
482		# Make sure we don't explode because the person is in too many rooms.
483		# We filter the results below regardless.
484		if len(room_ids) < 500:
485		clause, args = make_in_list_sql_clause(
486		self.database_engine, "room_id", room_ids
487		)
488		clauses = [clause]
489
490		local_clauses = []
491		for key in keys:
492		local_clauses.append("key = ?")
493		args.append(key)
494
495		clauses.append("(%s)" % (" OR ".join(local_clauses),))
496
497		# take copies of the current args and clauses lists, before adding
498		# pagination clauses to main query.
499		count_args = list(args)
500		count_clauses = list(clauses)
501
502		if pagination_token:
503		try:
504		origin_server_ts, stream = pagination_token.split(",")
505		origin_server_ts = int(origin_server_ts)
506		stream = int(stream)
507		except Exception:
508		raise SynapseError(400, "Invalid pagination token")
509
510		clauses.append(
511		"(origin_server_ts < ?"
512		" OR (origin_server_ts = ? AND stream_ordering < ?))"
513		)
514		args.extend([origin_server_ts, origin_server_ts, stream])
515
516		if isinstance(self.database_engine, PostgresEngine):
517		sql = (
518		"SELECT ts_rank_cd(vector, to_tsquery('english', ?)) as rank,"
519		" origin_server_ts, stream_ordering, room_id, event_id"
520		" FROM event_search"
521		" WHERE vector @@ to_tsquery('english', ?) AND "
522		)
523		args = [search_query, search_query] + args
524
525		count_sql = (
526		"SELECT room_id, count(*) as count FROM event_search"
527		" WHERE vector @@ to_tsquery('english', ?) AND "
528		)
529		count_args = [search_query] + count_args
530		elif isinstance(self.database_engine, Sqlite3Engine):
531		# We use CROSS JOIN here to ensure we use the right indexes.
532		# https://sqlite.org/optoverview.html#crossjoin
533		#
534		# We want to use the full text search index on event_search to
535		# extract all possible matches first, then lookup those matches
536		# in the events table to get the topological ordering. We need
537		# to use the indexes in this order because sqlite refuses to
538		# MATCH unless it uses the full text search index
539		sql = (
540		"SELECT rank(matchinfo) as rank, room_id, event_id,"
541		" origin_server_ts, stream_ordering"
542		" FROM (SELECT key, event_id, matchinfo(event_search) as matchinfo"
543		" FROM event_search"
544		" WHERE value MATCH ?"
545		" )"
546		" CROSS JOIN events USING (event_id)"
547		" WHERE "
548		)
549		args = [search_query] + args
550
551		count_sql = (
552		"SELECT room_id, count(*) as count FROM event_search"
553		" WHERE value MATCH ? AND "
554		)
555		count_args = [search_term] + count_args
556		else:
557		# This should be unreachable.
558		raise Exception("Unrecognized database engine")
559
560		sql += " AND ".join(clauses)
561		count_sql += " AND ".join(count_clauses)
562
563		# We add an arbitrary limit here to ensure we don't try to pull the
564		# entire table from the database.
565		if isinstance(self.database_engine, PostgresEngine):
566		sql += (
567		" ORDER BY origin_server_ts DESC NULLS LAST,"
568		" stream_ordering DESC NULLS LAST LIMIT ?"
569		)
570		elif isinstance(self.database_engine, Sqlite3Engine):
571		sql += " ORDER BY origin_server_ts DESC, stream_ordering DESC LIMIT ?"
572		else:
573		raise Exception("Unrecognized database engine")
574
575		args.append(limit)
576
577		results = yield self.db.execute(
578		"search_rooms", self.db.cursor_to_dict, sql, *args
579		)
580
581		results = list(filter(lambda row: row["room_id"] in room_ids, results))
582
583		# We set redact_behaviour to BLOCK here to prevent redacted events being returned in
584		# search results (which is a data leak)
585		events = yield self.get_events_as_list(
586		[r["event_id"] for r in results],
587		redact_behaviour=EventRedactBehaviour.BLOCK,
588		)
589
590		event_map = {ev.event_id: ev for ev in events}
591
592		highlights = None
593		if isinstance(self.database_engine, PostgresEngine):
594		highlights = yield self._find_highlights_in_postgres(search_query, events)
595
596		count_sql += " GROUP BY room_id"
597
598		count_results = yield self.db.execute(
599		"search_rooms_count", self.db.cursor_to_dict, count_sql, *count_args
600		)
601
602		count = sum(row["count"] for row in count_results if row["room_id"] in room_ids)
603
604		return {
605		"results": [
606		{
607		"event": event_map[r["event_id"]],
608		"rank": r["rank"],
609		"pagination_token": "%s,%s"
610		% (r["origin_server_ts"], r["stream_ordering"]),
611		}
612		for r in results
613		if r["event_id"] in event_map
614		],
615		"highlights": highlights,
616		"count": count,
617		}
618
619		def _find_highlights_in_postgres(self, search_query, events):
620		"""Given a list of events and a search term, return a list of words
621		that match from the content of the event.
622
623		This is used to give a list of words that clients can match against to
624		highlight the matching parts.
625
626		Args:
627		search_query (str)
628		events (list): A list of events
629
630		Returns:
631		deferred : A set of strings.
632		"""
633
634		def f(txn):
635		highlight_words = set()
636		for event in events:
637		# As a hack we simply join values of all possible keys. This is
638		# fine since we're only using them to find possible highlights.
639		values = []
640		for key in ("body", "name", "topic"):
641		v = event.content.get(key, None)
642		if v:
643		values.append(v)
644
645		if not values:
646		continue
647
648		value = " ".join(values)
649
650		# We need to find some values for StartSel and StopSel that
651		# aren't in the value so that we can pick results out.
652		start_sel = "<"
653		stop_sel = ">"
654
655		while start_sel in value:
656		start_sel += "<"
657		while stop_sel in value:
658		stop_sel += ">"
659
660		query = "SELECT ts_headline(?, to_tsquery('english', ?), %s)" % (
661		_to_postgres_options(
662		{
663		"StartSel": start_sel,
664		"StopSel": stop_sel,
665		"MaxFragments": "50",
666		}
667		)
668		)
669		txn.execute(query, (value, search_query))
670		(headline,) = txn.fetchall()[0]
671
672		# Now we need to pick the possible highlights out of the haedline
673		# result.
674		matcher_regex = "%s(.*?)%s" % (
675		re.escape(start_sel),
676		re.escape(stop_sel),
677		)
678
679		res = re.findall(matcher_regex, headline)
680		highlight_words.update([r.lower() for r in res])
681
682		return highlight_words
683
684		return self.db.runInteraction("_find_highlights", f)
685
686
687		def _to_postgres_options(options_dict):
688		return "'%s'" % (",".join("%s=%s" % (k, v) for k, v in options_dict.items()),)
689
690
691		def _parse_query(database_engine, search_term):
692		"""Takes a plain unicode string from the user and converts it into a form
693		that can be passed to database.
694		We use this so that we can add prefix matching, which isn't something
695		that is supported by default.
696		"""
697
698		# Pull out the individual words, discarding any non-word characters.
699		results = re.findall(r"([\w\-]+)", search_term, re.UNICODE)
700
701		if isinstance(database_engine, PostgresEngine):
702		return " & ".join(result + ":*" for result in results)
703		elif isinstance(database_engine, Sqlite3Engine):
704		return " & ".join(result + "*" for result in results)
705		else:
706		# This should be unreachable.
707		raise Exception("Unrecognized database engine")

+0

-71

~~synapse/storage/data_stores/main/signatures.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		from unpaddedbase64 import encode_base64
16
17		from twisted.internet import defer
18
19		from synapse.storage._base import SQLBaseStore
20		from synapse.util.caches.descriptors import cached, cachedList
21
22
23		class SignatureWorkerStore(SQLBaseStore):
24		@cached()
25		def get_event_reference_hash(self, event_id):
26		# This is a dummy function to allow get_event_reference_hashes
27		# to use its cache
28		raise NotImplementedError()
29
30		@cachedList(
31		cached_method_name="get_event_reference_hash", list_name="event_ids", num_args=1
32		)
33		def get_event_reference_hashes(self, event_ids):
34		def f(txn):
35		return {
36		event_id: self._get_event_reference_hashes_txn(txn, event_id)
37		for event_id in event_ids
38		}
39
40		return self.db.runInteraction("get_event_reference_hashes", f)
41
42		@defer.inlineCallbacks
43		def add_event_hashes(self, event_ids):
44		hashes = yield self.get_event_reference_hashes(event_ids)
45		hashes = {
46		e_id: {k: encode_base64(v) for k, v in h.items() if k == "sha256"}
47		for e_id, h in hashes.items()
48		}
49
50		return list(hashes.items())
51
52		def _get_event_reference_hashes_txn(self, txn, event_id):
53		"""Get all the hashes for a given PDU.
54		Args:
55		txn (cursor):
56		event_id (str): Id for the Event.
57		Returns:
58		A dict[unicode, bytes] of algorithm -> hash.
59		"""
60		query = (
61		"SELECT algorithm, hash"
62		" FROM event_reference_hashes"
63		" WHERE event_id = ?"
64		)
65		txn.execute(query, (event_id,))
66		return {k: v for k, v in txn}
67
68
69		class SignatureStore(SignatureWorkerStore):
70		"""Persistence for event signatures and hashes"""

+0

-512

~~synapse/storage/data_stores/main/state.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2020 The Matrix.org Foundation C.I.C.
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15		import collections.abc
16		import logging
17		from collections import namedtuple
18
19		from twisted.internet import defer
20
21		from synapse.api.constants import EventTypes, Membership
22		from synapse.api.errors import NotFoundError, UnsupportedRoomVersionError
23		from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, RoomVersion
24		from synapse.storage._base import SQLBaseStore
25		from synapse.storage.data_stores.main.events_worker import EventsWorkerStore
26		from synapse.storage.data_stores.main.roommember import RoomMemberWorkerStore
27		from synapse.storage.database import Database
28		from synapse.storage.state import StateFilter
29		from synapse.util.caches import intern_string
30		from synapse.util.caches.descriptors import cached, cachedList
31
32		logger = logging.getLogger(__name__)
33
34
35		MAX_STATE_DELTA_HOPS = 100
36
37
38		class _GetStateGroupDelta(
39		namedtuple("_GetStateGroupDelta", ("prev_group", "delta_ids"))
40		):
41		"""Return type of get_state_group_delta that implements __len__, which lets
42		us use the itrable flag when caching
43		"""
44
45		__slots__ = []
46
47		def __len__(self):
48		return len(self.delta_ids) if self.delta_ids else 0
49
50
51		# this inherits from EventsWorkerStore because it calls self.get_events
52		class StateGroupWorkerStore(EventsWorkerStore, SQLBaseStore):
53		"""The parts of StateGroupStore that can be called from workers.
54		"""
55
56		def __init__(self, database: Database, db_conn, hs):
57		super(StateGroupWorkerStore, self).__init__(database, db_conn, hs)
58
59		async def get_room_version(self, room_id: str) -> RoomVersion:
60		"""Get the room_version of a given room
61
62		Raises:
63		NotFoundError: if the room is unknown
64
65		UnsupportedRoomVersionError: if the room uses an unknown room version.
66		Typically this happens if support for the room's version has been
67		removed from Synapse.
68		"""
69		room_version_id = await self.get_room_version_id(room_id)
70		v = KNOWN_ROOM_VERSIONS.get(room_version_id)
71
72		if not v:
73		raise UnsupportedRoomVersionError(
74		"Room %s uses a room version %s which is no longer supported"
75		% (room_id, room_version_id)
76		)
77
78		return v
79
80		@cached(max_entries=10000)
81		async def get_room_version_id(self, room_id: str) -> str:
82		"""Get the room_version of a given room
83
84		Raises:
85		NotFoundError: if the room is unknown
86		"""
87
88		# First we try looking up room version from the database, but for old
89		# rooms we might not have added the room version to it yet so we fall
90		# back to previous behaviour and look in current state events.
91
92		# We really should have an entry in the rooms table for every room we
93		# care about, but let's be a bit paranoid (at least while the background
94		# update is happening) to avoid breaking existing rooms.
95		version = await self.db.simple_select_one_onecol(
96		table="rooms",
97		keyvalues={"room_id": room_id},
98		retcol="room_version",
99		desc="get_room_version",
100		allow_none=True,
101		)
102
103		if version is not None:
104		return version
105
106		# Retrieve the room's create event
107		create_event = await self.get_create_event_for_room(room_id)
108		return create_event.content.get("room_version", "1")
109
110		@defer.inlineCallbacks
111		def get_room_predecessor(self, room_id):
112		"""Get the predecessor of an upgraded room if it exists.
113		Otherwise return None.
114
115		Args:
116		room_id (str)
117
118		Returns:
119		Deferred[dict\|None]: A dictionary containing the structure of the predecessor
120		field from the room's create event. The structure is subject to other servers,
121		but it is expected to be:
122		* room_id (str): The room ID of the predecessor room
123		* event_id (str): The ID of the tombstone event in the predecessor room
124
125		None if a predecessor key is not found, or is not a dictionary.
126
127		Raises:
128		NotFoundError if the given room is unknown
129		"""
130		# Retrieve the room's create event
131		create_event = yield self.get_create_event_for_room(room_id)
132
133		# Retrieve the predecessor key of the create event
134		predecessor = create_event.content.get("predecessor", None)
135
136		# Ensure the key is a dictionary
137		if not isinstance(predecessor, collections.abc.Mapping):
138		return None
139
140		return predecessor
141
142		@defer.inlineCallbacks
143		def get_create_event_for_room(self, room_id):
144		"""Get the create state event for a room.
145
146		Args:
147		room_id (str)
148
149		Returns:
150		Deferred[EventBase]: The room creation event.
151
152		Raises:
153		NotFoundError if the room is unknown
154		"""
155		state_ids = yield self.get_current_state_ids(room_id)
156		create_id = state_ids.get((EventTypes.Create, ""))
157
158		# If we can't find the create event, assume we've hit a dead end
159		if not create_id:
160		raise NotFoundError("Unknown room %s" % (room_id,))
161
162		# Retrieve the room's create event and return
163		create_event = yield self.get_event(create_id)
164		return create_event
165
166		@cached(max_entries=100000, iterable=True)
167		def get_current_state_ids(self, room_id):
168		"""Get the current state event ids for a room based on the
169		current_state_events table.
170
171		Args:
172		room_id (str)
173
174		Returns:
175		deferred: dict of (type, state_key) -> event_id
176		"""
177
178		def _get_current_state_ids_txn(txn):
179		txn.execute(
180		"""SELECT type, state_key, event_id FROM current_state_events
181		WHERE room_id = ?
182		""",
183		(room_id,),
184		)
185
186		return {(intern_string(r[0]), intern_string(r[1])): r[2] for r in txn}
187
188		return self.db.runInteraction(
189		"get_current_state_ids", _get_current_state_ids_txn
190		)
191
192		# FIXME: how should this be cached?
193		def get_filtered_current_state_ids(
194		self, room_id: str, state_filter: StateFilter = StateFilter.all()
195		):
196		"""Get the current state event of a given type for a room based on the
197		current_state_events table. This may not be as up-to-date as the result
198		of doing a fresh state resolution as per state_handler.get_current_state
199
200		Args:
201		room_id
202		state_filter: The state filter used to fetch state
203		from the database.
204
205		Returns:
206		defer.Deferred[StateMap[str]]: Map from type/state_key to event ID.
207		"""
208
209		where_clause, where_args = state_filter.make_sql_filter_clause()
210
211		if not where_clause:
212		# We delegate to the cached version
213		return self.get_current_state_ids(room_id)
214
215		def _get_filtered_current_state_ids_txn(txn):
216		results = {}
217		sql = """
218		SELECT type, state_key, event_id FROM current_state_events
219		WHERE room_id = ?
220		"""
221
222		if where_clause:
223		sql += " AND (%s)" % (where_clause,)
224
225		args = [room_id]
226		args.extend(where_args)
227		txn.execute(sql, args)
228		for row in txn:
229		typ, state_key, event_id = row
230		key = (intern_string(typ), intern_string(state_key))
231		results[key] = event_id
232
233		return results
234
235		return self.db.runInteraction(
236		"get_filtered_current_state_ids", _get_filtered_current_state_ids_txn
237		)
238
239		@defer.inlineCallbacks
240		def get_canonical_alias_for_room(self, room_id):
241		"""Get canonical alias for room, if any
242
243		Args:
244		room_id (str)
245
246		Returns:
247		Deferred[str\|None]: The canonical alias, if any
248		"""
249
250		state = yield self.get_filtered_current_state_ids(
251		room_id, StateFilter.from_types([(EventTypes.CanonicalAlias, "")])
252		)
253
254		event_id = state.get((EventTypes.CanonicalAlias, ""))
255		if not event_id:
256		return
257
258		event = yield self.get_event(event_id, allow_none=True)
259		if not event:
260		return
261
262		return event.content.get("canonical_alias")
263
264		@cached(max_entries=50000)
265		def _get_state_group_for_event(self, event_id):
266		return self.db.simple_select_one_onecol(
267		table="event_to_state_groups",
268		keyvalues={"event_id": event_id},
269		retcol="state_group",
270		allow_none=True,
271		desc="_get_state_group_for_event",
272		)
273
274		@cachedList(
275		cached_method_name="_get_state_group_for_event",
276		list_name="event_ids",
277		num_args=1,
278		inlineCallbacks=True,
279		)
280		def _get_state_group_for_events(self, event_ids):
281		"""Returns mapping event_id -> state_group
282		"""
283		rows = yield self.db.simple_select_many_batch(
284		table="event_to_state_groups",
285		column="event_id",
286		iterable=event_ids,
287		keyvalues={},
288		retcols=("event_id", "state_group"),
289		desc="_get_state_group_for_events",
290		)
291
292		return {row["event_id"]: row["state_group"] for row in rows}
293
294		@defer.inlineCallbacks
295		def get_referenced_state_groups(self, state_groups):
296		"""Check if the state groups are referenced by events.
297
298		Args:
299		state_groups (Iterable[int])
300
301		Returns:
302		Deferred[set[int]]: The subset of state groups that are
303		referenced.
304		"""
305
306		rows = yield self.db.simple_select_many_batch(
307		table="event_to_state_groups",
308		column="state_group",
309		iterable=state_groups,
310		keyvalues={},
311		retcols=("DISTINCT state_group",),
312		desc="get_referenced_state_groups",
313		)
314
315		return {row["state_group"] for row in rows}
316
317
318		class MainStateBackgroundUpdateStore(RoomMemberWorkerStore):
319
320		CURRENT_STATE_INDEX_UPDATE_NAME = "current_state_members_idx"
321		EVENT_STATE_GROUP_INDEX_UPDATE_NAME = "event_to_state_groups_sg_index"
322		DELETE_CURRENT_STATE_UPDATE_NAME = "delete_old_current_state_events"
323
324		def __init__(self, database: Database, db_conn, hs):
325		super(MainStateBackgroundUpdateStore, self).__init__(database, db_conn, hs)
326
327		self.server_name = hs.hostname
328
329		self.db.updates.register_background_index_update(
330		self.CURRENT_STATE_INDEX_UPDATE_NAME,
331		index_name="current_state_events_member_index",
332		table="current_state_events",
333		columns=["state_key"],
334		where_clause="type='m.room.member'",
335		)
336		self.db.updates.register_background_index_update(
337		self.EVENT_STATE_GROUP_INDEX_UPDATE_NAME,
338		index_name="event_to_state_groups_sg_index",
339		table="event_to_state_groups",
340		columns=["state_group"],
341		)
342		self.db.updates.register_background_update_handler(
343		self.DELETE_CURRENT_STATE_UPDATE_NAME, self._background_remove_left_rooms,
344		)
345
346		async def _background_remove_left_rooms(self, progress, batch_size):
347		"""Background update to delete rows from `current_state_events` and
348		`event_forward_extremities` tables of rooms that the server is no
349		longer joined to.
350		"""
351
352		last_room_id = progress.get("last_room_id", "")
353
354		def _background_remove_left_rooms_txn(txn):
355		# get a batch of room ids to consider
356		sql = """
357		SELECT DISTINCT room_id FROM current_state_events
358		WHERE room_id > ? ORDER BY room_id LIMIT ?
359		"""
360
361		txn.execute(sql, (last_room_id, batch_size))
362		room_ids = [row[0] for row in txn]
363		if not room_ids:
364		return True, set()
365
366		###########################################################################
367		#
368		# exclude rooms where we have active members
369
370		sql = """
371		SELECT room_id
372		FROM local_current_membership
373		WHERE
374		room_id > ? AND room_id <= ?
375		AND membership = 'join'
376		GROUP BY room_id
377		"""
378
379		txn.execute(sql, (last_room_id, room_ids[-1]))
380		joined_room_ids = {row[0] for row in txn}
381		to_delete = set(room_ids) - joined_room_ids
382
383		###########################################################################
384		#
385		# exclude rooms which we are in the process of constructing; these otherwise
386		# qualify as "rooms with no local users", and would have their
387		# forward extremities cleaned up.
388
389		# the following query will return a list of rooms which have forward
390		# extremities that are not also the create event in the room - ie
391		# those that are not being created currently.
392
393		sql = """
394		SELECT DISTINCT efe.room_id
395		FROM event_forward_extremities efe
396		LEFT JOIN current_state_events cse ON
397		cse.event_id = efe.event_id
398		AND cse.type = 'm.room.create'
399		AND cse.state_key = ''
400		WHERE
401		cse.event_id IS NULL
402		AND efe.room_id > ? AND efe.room_id <= ?
403		"""
404
405		txn.execute(sql, (last_room_id, room_ids[-1]))
406
407		# build a set of those rooms within `to_delete` that do not appear in
408		# the above, leaving us with the rooms in `to_delete` that are being
409		# created.
410		creating_rooms = to_delete.difference(row[0] for row in txn)
411		logger.info("skipping rooms which are being created: %s", creating_rooms)
412
413		# now remove the rooms being created from the list of those to delete.
414		#
415		# (we could have just taken the intersection of `to_delete` with the result
416		# of the sql query, but it's useful to be able to log `creating_rooms`; and
417		# having done so, it's quicker to remove the (few) creating rooms from
418		# `to_delete` than it is to form the intersection with the (larger) list of
419		# not-creating-rooms)
420
421		to_delete -= creating_rooms
422
423		###########################################################################
424		#
425		# now clear the state for the rooms
426
427		logger.info("Deleting current state left rooms: %r", to_delete)
428
429		# First we get all users that we still think were joined to the
430		# room. This is so that we can mark those device lists as
431		# potentially stale, since there may have been a period where the
432		# server didn't share a room with the remote user and therefore may
433		# have missed any device updates.
434		rows = self.db.simple_select_many_txn(
435		txn,
436		table="current_state_events",
437		column="room_id",
438		iterable=to_delete,
439		keyvalues={"type": EventTypes.Member, "membership": Membership.JOIN},
440		retcols=("state_key",),
441		)
442
443		potentially_left_users = {row["state_key"] for row in rows}
444
445		# Now lets actually delete the rooms from the DB.
446		self.db.simple_delete_many_txn(
447		txn,
448		table="current_state_events",
449		column="room_id",
450		iterable=to_delete,
451		keyvalues={},
452		)
453
454		self.db.simple_delete_many_txn(
455		txn,
456		table="event_forward_extremities",
457		column="room_id",
458		iterable=to_delete,
459		keyvalues={},
460		)
461
462		self.db.updates._background_update_progress_txn(
463		txn,
464		self.DELETE_CURRENT_STATE_UPDATE_NAME,
465		{"last_room_id": room_ids[-1]},
466		)
467
468		return False, potentially_left_users
469
470		finished, potentially_left_users = await self.db.runInteraction(
471		"_background_remove_left_rooms", _background_remove_left_rooms_txn
472		)
473
474		if finished:
475		await self.db.updates._end_background_update(
476		self.DELETE_CURRENT_STATE_UPDATE_NAME
477		)
478
479		# Now go and check if we still share a room with the remote users in
480		# the deleted rooms. If not mark their device lists as stale.
481		joined_users = await self.get_users_server_still_shares_room_with(
482		potentially_left_users
483		)
484
485		for user_id in potentially_left_users - joined_users:
486		await self.mark_remote_user_device_list_as_unsubscribed(user_id)
487
488		return batch_size
489
490
491		class StateStore(StateGroupWorkerStore, MainStateBackgroundUpdateStore):
492		""" Keeps track of the state at a given event.
493
494		This is done by the concept of `state groups`. Every event is a assigned
495		a state group (identified by an arbitrary string), which references a
496		collection of state events. The current state of an event is then the
497		collection of state events referenced by the event's state group.
498
499		Hence, every change in the current state causes a new state group to be
500		generated. However, if no change happens (e.g., if we get a message event
501		with only one parent it inherits the state group from its parent.)
502
503		There are three tables:
504		* `state_groups`: Stores group name, first event with in the group and
505		room id.
506		* `event_to_state_groups`: Maps events to state groups.
507		* `state_groups_state`: Maps state group to state events.
508		"""
509
510		def __init__(self, database: Database, db_conn, hs):
511		super(StateStore, self).__init__(database, db_conn, hs)

+0

-121

~~synapse/storage/data_stores/main/state_deltas.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2018 Vector Creations Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16
17		from twisted.internet import defer
18
19		from synapse.storage._base import SQLBaseStore
20
21		logger = logging.getLogger(__name__)
22
23
24		class StateDeltasStore(SQLBaseStore):
25		def get_current_state_deltas(self, prev_stream_id: int, max_stream_id: int):
26		"""Fetch a list of room state changes since the given stream id
27
28		Each entry in the result contains the following fields:
29		- stream_id (int)
30		- room_id (str)
31		- type (str): event type
32		- state_key (str):
33		- event_id (str\|None): new event_id for this state key. None if the
34		state has been deleted.
35		- prev_event_id (str\|None): previous event_id for this state key. None
36		if it's new state.
37
38		Args:
39		prev_stream_id (int): point to get changes since (exclusive)
40		max_stream_id (int): the point that we know has been correctly persisted
41		- ie, an upper limit to return changes from.
42
43		Returns:
44		Deferred[tuple[int, list[dict]]: A tuple consisting of:
45		- the stream id which these results go up to
46		- list of current_state_delta_stream rows. If it is empty, we are
47		up to date.
48		"""
49		prev_stream_id = int(prev_stream_id)
50
51		# check we're not going backwards
52		assert prev_stream_id <= max_stream_id
53
54		if not self._curr_state_delta_stream_cache.has_any_entity_changed(
55		prev_stream_id
56		):
57		# if the CSDs haven't changed between prev_stream_id and now, we
58		# know for certain that they haven't changed between prev_stream_id and
59		# max_stream_id.
60		return defer.succeed((max_stream_id, []))
61
62		def get_current_state_deltas_txn(txn):
63		# First we calculate the max stream id that will give us less than
64		# N results.
65		# We arbitarily limit to 100 stream_id entries to ensure we don't
66		# select toooo many.
67		sql = """
68		SELECT stream_id, count(*)
69		FROM current_state_delta_stream
70		WHERE stream_id > ? AND stream_id <= ?
71		GROUP BY stream_id
72		ORDER BY stream_id ASC
73		LIMIT 100
74		"""
75		txn.execute(sql, (prev_stream_id, max_stream_id))
76
77		total = 0
78
79		for stream_id, count in txn:
80		total += count
81		if total > 100:
82		# We arbitarily limit to 100 entries to ensure we don't
83		# select toooo many.
84		logger.debug(
85		"Clipping current_state_delta_stream rows to stream_id %i",
86		stream_id,
87		)
88		clipped_stream_id = stream_id
89		break
90		else:
91		# if there's no problem, we may as well go right up to the max_stream_id
92		clipped_stream_id = max_stream_id
93
94		# Now actually get the deltas
95		sql = """
96		SELECT stream_id, room_id, type, state_key, event_id, prev_event_id
97		FROM current_state_delta_stream
98		WHERE ? < stream_id AND stream_id <= ?
99		ORDER BY stream_id ASC
100		"""
101		txn.execute(sql, (prev_stream_id, clipped_stream_id))
102		return clipped_stream_id, self.db.cursor_to_dict(txn)
103
104		return self.db.runInteraction(
105		"get_current_state_deltas", get_current_state_deltas_txn
106		)
107
108		def _get_max_stream_id_in_current_state_deltas_txn(self, txn):
109		return self.db.simple_select_one_onecol_txn(
110		txn,
111		table="current_state_delta_stream",
112		keyvalues={},
113		retcol="COALESCE(MAX(stream_id), -1)",
114		)
115
116		def get_max_stream_id_in_current_state_deltas(self):
117		return self.db.runInteraction(
118		"get_max_stream_id_in_current_state_deltas",
119		self._get_max_stream_id_in_current_state_deltas_txn,
120		)

+0

-857

~~synapse/storage/data_stores/main/stats.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2018, 2019 New Vector Ltd
2		# Copyright 2019 The Matrix.org Foundation C.I.C.
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		import logging
17		from itertools import chain
18
19		from twisted.internet import defer
20		from twisted.internet.defer import DeferredLock
21
22		from synapse.api.constants import EventTypes, Membership
23		from synapse.storage.data_stores.main.state_deltas import StateDeltasStore
24		from synapse.storage.database import Database
25		from synapse.storage.engines import PostgresEngine
26		from synapse.util.caches.descriptors import cached
27
28		logger = logging.getLogger(__name__)
29
30		# these fields track absolutes (e.g. total number of rooms on the server)
31		# You can think of these as Prometheus Gauges.
32		# You can draw these stats on a line graph.
33		# Example: number of users in a room
34		ABSOLUTE_STATS_FIELDS = {
35		"room": (
36		"current_state_events",
37		"joined_members",
38		"invited_members",
39		"left_members",
40		"banned_members",
41		"local_users_in_room",
42		),
43		"user": ("joined_rooms",),
44		}
45
46		# these fields are per-timeslice and so should be reset to 0 upon a new slice
47		# You can draw these stats on a histogram.
48		# Example: number of events sent locally during a time slice
49		PER_SLICE_FIELDS = {
50		"room": ("total_events", "total_event_bytes"),
51		"user": ("invites_sent", "rooms_created", "total_events", "total_event_bytes"),
52		}
53
54		TYPE_TO_TABLE = {"room": ("room_stats", "room_id"), "user": ("user_stats", "user_id")}
55
56		# these are the tables (& ID columns) which contain our actual subjects
57		TYPE_TO_ORIGIN_TABLE = {"room": ("rooms", "room_id"), "user": ("users", "name")}
58
59
60		class StatsStore(StateDeltasStore):
61		def __init__(self, database: Database, db_conn, hs):
62		super(StatsStore, self).__init__(database, db_conn, hs)
63
64		self.server_name = hs.hostname
65		self.clock = self.hs.get_clock()
66		self.stats_enabled = hs.config.stats_enabled
67		self.stats_bucket_size = hs.config.stats_bucket_size
68
69		self.stats_delta_processing_lock = DeferredLock()
70
71		self.db.updates.register_background_update_handler(
72		"populate_stats_process_rooms", self._populate_stats_process_rooms
73		)
74		self.db.updates.register_background_update_handler(
75		"populate_stats_process_users", self._populate_stats_process_users
76		)
77		# we no longer need to perform clean-up, but we will give ourselves
78		# the potential to reintroduce it in the future – so documentation
79		# will still encourage the use of this no-op handler.
80		self.db.updates.register_noop_background_update("populate_stats_cleanup")
81		self.db.updates.register_noop_background_update("populate_stats_prepare")
82
83		def quantise_stats_time(self, ts):
84		"""
85		Quantises a timestamp to be a multiple of the bucket size.
86
87		Args:
88		ts (int): the timestamp to quantise, in milliseconds since the Unix
89		Epoch
90
91		Returns:
92		int: a timestamp which
93		- is divisible by the bucket size;
94		- is no later than `ts`; and
95		- is the largest such timestamp.
96		"""
97		return (ts // self.stats_bucket_size) * self.stats_bucket_size
98
99		@defer.inlineCallbacks
100		def _populate_stats_process_users(self, progress, batch_size):
101		"""
102		This is a background update which regenerates statistics for users.
103		"""
104		if not self.stats_enabled:
105		yield self.db.updates._end_background_update("populate_stats_process_users")
106		return 1
107
108		last_user_id = progress.get("last_user_id", "")
109
110		def _get_next_batch(txn):
111		sql = """
112		SELECT DISTINCT name FROM users
113		WHERE name > ?
114		ORDER BY name ASC
115		LIMIT ?
116		"""
117		txn.execute(sql, (last_user_id, batch_size))
118		return [r for r, in txn]
119
120		users_to_work_on = yield self.db.runInteraction(
121		"_populate_stats_process_users", _get_next_batch
122		)
123
124		# No more rooms -- complete the transaction.
125		if not users_to_work_on:
126		yield self.db.updates._end_background_update("populate_stats_process_users")
127		return 1
128
129		for user_id in users_to_work_on:
130		yield self._calculate_and_set_initial_state_for_user(user_id)
131		progress["last_user_id"] = user_id
132
133		yield self.db.runInteraction(
134		"populate_stats_process_users",
135		self.db.updates._background_update_progress_txn,
136		"populate_stats_process_users",
137		progress,
138		)
139
140		return len(users_to_work_on)
141
142		@defer.inlineCallbacks
143		def _populate_stats_process_rooms(self, progress, batch_size):
144		"""
145		This is a background update which regenerates statistics for rooms.
146		"""
147		if not self.stats_enabled:
148		yield self.db.updates._end_background_update("populate_stats_process_rooms")
149		return 1
150
151		last_room_id = progress.get("last_room_id", "")
152
153		def _get_next_batch(txn):
154		sql = """
155		SELECT DISTINCT room_id FROM current_state_events
156		WHERE room_id > ?
157		ORDER BY room_id ASC
158		LIMIT ?
159		"""
160		txn.execute(sql, (last_room_id, batch_size))
161		return [r for r, in txn]
162
163		rooms_to_work_on = yield self.db.runInteraction(
164		"populate_stats_rooms_get_batch", _get_next_batch
165		)
166
167		# No more rooms -- complete the transaction.
168		if not rooms_to_work_on:
169		yield self.db.updates._end_background_update("populate_stats_process_rooms")
170		return 1
171
172		for room_id in rooms_to_work_on:
173		yield self._calculate_and_set_initial_state_for_room(room_id)
174		progress["last_room_id"] = room_id
175
176		yield self.db.runInteraction(
177		"_populate_stats_process_rooms",
178		self.db.updates._background_update_progress_txn,
179		"populate_stats_process_rooms",
180		progress,
181		)
182
183		return len(rooms_to_work_on)
184
185		def get_stats_positions(self):
186		"""
187		Returns the stats processor positions.
188		"""
189		return self.db.simple_select_one_onecol(
190		table="stats_incremental_position",
191		keyvalues={},
192		retcol="stream_id",
193		desc="stats_incremental_position",
194		)
195
196		def update_room_state(self, room_id, fields):
197		"""
198		Args:
199		room_id (str)
200		fields (dict[str:Any])
201		"""
202
203		# For whatever reason some of the fields may contain null bytes, which
204		# postgres isn't a fan of, so we replace those fields with null.
205		for col in (
206		"join_rules",
207		"history_visibility",
208		"encryption",
209		"name",
210		"topic",
211		"avatar",
212		"canonical_alias",
213		):
214		field = fields.get(col)
215		if field and "\0" in field:
216		fields[col] = None
217
218		return self.db.simple_upsert(
219		table="room_stats_state",
220		keyvalues={"room_id": room_id},
221		values=fields,
222		desc="update_room_state",
223		)
224
225		def get_statistics_for_subject(self, stats_type, stats_id, start, size=100):
226		"""
227		Get statistics for a given subject.
228
229		Args:
230		stats_type (str): The type of subject
231		stats_id (str): The ID of the subject (e.g. room_id or user_id)
232		start (int): Pagination start. Number of entries, not timestamp.
233		size (int): How many entries to return.
234
235		Returns:
236		Deferred[list[dict]], where the dict has the keys of
237		ABSOLUTE_STATS_FIELDS[stats_type], and "bucket_size" and "end_ts".
238		"""
239		return self.db.runInteraction(
240		"get_statistics_for_subject",
241		self._get_statistics_for_subject_txn,
242		stats_type,
243		stats_id,
244		start,
245		size,
246		)
247
248		def _get_statistics_for_subject_txn(
249		self, txn, stats_type, stats_id, start, size=100
250		):
251		"""
252		Transaction-bound version of L{get_statistics_for_subject}.
253		"""
254
255		table, id_col = TYPE_TO_TABLE[stats_type]
256		selected_columns = list(
257		ABSOLUTE_STATS_FIELDS[stats_type] + PER_SLICE_FIELDS[stats_type]
258		)
259
260		slice_list = self.db.simple_select_list_paginate_txn(
261		txn,
262		table + "_historical",
263		"end_ts",
264		start,
265		size,
266		retcols=selected_columns + ["bucket_size", "end_ts"],
267		keyvalues={id_col: stats_id},
268		order_direction="DESC",
269		)
270
271		return slice_list
272
273		@cached()
274		def get_earliest_token_for_stats(self, stats_type, id):
275		"""
276		Fetch the "earliest token". This is used by the room stats delta
277		processor to ignore deltas that have been processed between the
278		start of the background task and any particular room's stats
279		being calculated.
280
281		Returns:
282		Deferred[int]
283		"""
284		table, id_col = TYPE_TO_TABLE[stats_type]
285
286		return self.db.simple_select_one_onecol(
287		"%s_current" % (table,),
288		keyvalues={id_col: id},
289		retcol="completed_delta_stream_id",
290		allow_none=True,
291		)
292
293		def bulk_update_stats_delta(self, ts, updates, stream_id):
294		"""Bulk update stats tables for a given stream_id and updates the stats
295		incremental position.
296
297		Args:
298		ts (int): Current timestamp in ms
299		updates(dict[str, dict[str, dict[str, Counter]]]): The updates to
300		commit as a mapping stats_type -> stats_id -> field -> delta.
301		stream_id (int): Current position.
302
303		Returns:
304		Deferred
305		"""
306
307		def _bulk_update_stats_delta_txn(txn):
308		for stats_type, stats_updates in updates.items():
309		for stats_id, fields in stats_updates.items():
310		logger.debug(
311		"Updating %s stats for %s: %s", stats_type, stats_id, fields
312		)
313		self._update_stats_delta_txn(
314		txn,
315		ts=ts,
316		stats_type=stats_type,
317		stats_id=stats_id,
318		fields=fields,
319		complete_with_stream_id=stream_id,
320		)
321
322		self.db.simple_update_one_txn(
323		txn,
324		table="stats_incremental_position",
325		keyvalues={},
326		updatevalues={"stream_id": stream_id},
327		)
328
329		return self.db.runInteraction(
330		"bulk_update_stats_delta", _bulk_update_stats_delta_txn
331		)
332
333		def update_stats_delta(
334		self,
335		ts,
336		stats_type,
337		stats_id,
338		fields,
339		complete_with_stream_id,
340		absolute_field_overrides=None,
341		):
342		"""
343		Updates the statistics for a subject, with a delta (difference/relative
344		change).
345
346		Args:
347		ts (int): timestamp of the change
348		stats_type (str): "room" or "user" – the kind of subject
349		stats_id (str): the subject's ID (room ID or user ID)
350		fields (dict[str, int]): Deltas of stats values.
351		complete_with_stream_id (int, optional):
352		If supplied, converts an incomplete row into a complete row,
353		with the supplied stream_id marked as the stream_id where the
354		row was completed.
355		absolute_field_overrides (dict[str, int]): Current stats values
356		(i.e. not deltas) of absolute fields.
357		Does not work with per-slice fields.
358		"""
359
360		return self.db.runInteraction(
361		"update_stats_delta",
362		self._update_stats_delta_txn,
363		ts,
364		stats_type,
365		stats_id,
366		fields,
367		complete_with_stream_id=complete_with_stream_id,
368		absolute_field_overrides=absolute_field_overrides,
369		)
370
371		def _update_stats_delta_txn(
372		self,
373		txn,
374		ts,
375		stats_type,
376		stats_id,
377		fields,
378		complete_with_stream_id,
379		absolute_field_overrides=None,
380		):
381		if absolute_field_overrides is None:
382		absolute_field_overrides = {}
383
384		table, id_col = TYPE_TO_TABLE[stats_type]
385
386		quantised_ts = self.quantise_stats_time(int(ts))
387		end_ts = quantised_ts + self.stats_bucket_size
388
389		# Lets be paranoid and check that all the given field names are known
390		abs_field_names = ABSOLUTE_STATS_FIELDS[stats_type]
391		slice_field_names = PER_SLICE_FIELDS[stats_type]
392		for field in chain(fields.keys(), absolute_field_overrides.keys()):
393		if field not in abs_field_names and field not in slice_field_names:
394		# guard against potential SQL injection dodginess
395		raise ValueError(
396		"%s is not a recognised field"
397		" for stats type %s" % (field, stats_type)
398		)
399
400		# Per slice fields do not get added to the _current table
401
402		# This calculates the deltas (`field = field + ?` values)
403		# for absolute fields,
404		# * defaulting to 0 if not specified
405		# (required for the INSERT part of upserting to work)
406		# * omitting overrides specified in `absolute_field_overrides`
407		deltas_of_absolute_fields = {
408		key: fields.get(key, 0)
409		for key in abs_field_names
410		if key not in absolute_field_overrides
411		}
412
413		# Keep the delta stream ID field up to date
414		absolute_field_overrides = absolute_field_overrides.copy()
415		absolute_field_overrides["completed_delta_stream_id"] = complete_with_stream_id
416
417		# first upsert the `_current` table
418		self._upsert_with_additive_relatives_txn(
419		txn=txn,
420		table=table + "_current",
421		keyvalues={id_col: stats_id},
422		absolutes=absolute_field_overrides,
423		additive_relatives=deltas_of_absolute_fields,
424		)
425
426		per_slice_additive_relatives = {
427		key: fields.get(key, 0) for key in slice_field_names
428		}
429		self._upsert_copy_from_table_with_additive_relatives_txn(
430		txn=txn,
431		into_table=table + "_historical",
432		keyvalues={id_col: stats_id},
433		extra_dst_insvalues={"bucket_size": self.stats_bucket_size},
434		extra_dst_keyvalues={"end_ts": end_ts},
435		additive_relatives=per_slice_additive_relatives,
436		src_table=table + "_current",
437		copy_columns=abs_field_names,
438		)
439
440		def _upsert_with_additive_relatives_txn(
441		self, txn, table, keyvalues, absolutes, additive_relatives
442		):
443		"""Used to update values in the stats tables.
444
445		This is basically a slightly convoluted upsert that adds to any
446		existing rows.
447
448		Args:
449		txn
450		table (str): Table name
451		keyvalues (dict[str, any]): Row-identifying key values
452		absolutes (dict[str, any]): Absolute (set) fields
453		additive_relatives (dict[str, int]): Fields that will be added onto
454		if existing row present.
455		"""
456		if self.database_engine.can_native_upsert:
457		absolute_updates = [
458		"%(field)s = EXCLUDED.%(field)s" % {"field": field}
459		for field in absolutes.keys()
460		]
461
462		relative_updates = [
463		"%(field)s = EXCLUDED.%(field)s + %(table)s.%(field)s"
464		% {"table": table, "field": field}
465		for field in additive_relatives.keys()
466		]
467
468		insert_cols = []
469		qargs = []
470
471		for (key, val) in chain(
472		keyvalues.items(), absolutes.items(), additive_relatives.items()
473		):
474		insert_cols.append(key)
475		qargs.append(val)
476
477		sql = """
478		INSERT INTO %(table)s (%(insert_cols_cs)s)
479		VALUES (%(insert_vals_qs)s)
480		ON CONFLICT (%(key_columns)s) DO UPDATE SET %(updates)s
481		""" % {
482		"table": table,
483		"insert_cols_cs": ", ".join(insert_cols),
484		"insert_vals_qs": ", ".join(
485		["?"] * (len(keyvalues) + len(absolutes) + len(additive_relatives))
486		),
487		"key_columns": ", ".join(keyvalues),
488		"updates": ", ".join(chain(absolute_updates, relative_updates)),
489		}
490
491		txn.execute(sql, qargs)
492		else:
493		self.database_engine.lock_table(txn, table)
494		retcols = list(chain(absolutes.keys(), additive_relatives.keys()))
495		current_row = self.db.simple_select_one_txn(
496		txn, table, keyvalues, retcols, allow_none=True
497		)
498		if current_row is None:
499		merged_dict = {keyvalues, absolutes, **additive_relatives}
500		self.db.simple_insert_txn(txn, table, merged_dict)
501		else:
502		for (key, val) in additive_relatives.items():
503		current_row[key] += val
504		current_row.update(absolutes)
505		self.db.simple_update_one_txn(txn, table, keyvalues, current_row)
506
507		def _upsert_copy_from_table_with_additive_relatives_txn(
508		self,
509		txn,
510		into_table,
511		keyvalues,
512		extra_dst_keyvalues,
513		extra_dst_insvalues,
514		additive_relatives,
515		src_table,
516		copy_columns,
517		):
518		"""Updates the historic stats table with latest updates.
519
520		This involves copying "absolute" fields from the `_current` table, and
521		adding relative fields to any existing values.
522
523		Args:
524		txn: Transaction
525		into_table (str): The destination table to UPSERT the row into
526		keyvalues (dict[str, any]): Row-identifying key values
527		extra_dst_keyvalues (dict[str, any]): Additional keyvalues
528		for `into_table`.
529		extra_dst_insvalues (dict[str, any]): Additional values to insert
530		on new row creation for `into_table`.
531		additive_relatives (dict[str, any]): Fields that will be added onto
532		if existing row present. (Must be disjoint from copy_columns.)
533		src_table (str): The source table to copy from
534		copy_columns (iterable[str]): The list of columns to copy
535		"""
536		if self.database_engine.can_native_upsert:
537		ins_columns = chain(
538		keyvalues,
539		copy_columns,
540		additive_relatives,
541		extra_dst_keyvalues,
542		extra_dst_insvalues,
543		)
544		sel_exprs = chain(
545		keyvalues,
546		copy_columns,
547		(
548		"?"
549		for _ in chain(
550		additive_relatives, extra_dst_keyvalues, extra_dst_insvalues
551		)
552		),
553		)
554		keyvalues_where = ("%s = ?" % f for f in keyvalues)
555
556		sets_cc = ("%s = EXCLUDED.%s" % (f, f) for f in copy_columns)
557		sets_ar = (
558		"%s = EXCLUDED.%s + %s.%s" % (f, f, into_table, f)
559		for f in additive_relatives
560		)
561
562		sql = """
563		INSERT INTO %(into_table)s (%(ins_columns)s)
564		SELECT %(sel_exprs)s
565		FROM %(src_table)s
566		WHERE %(keyvalues_where)s
567		ON CONFLICT (%(keyvalues)s)
568		DO UPDATE SET %(sets)s
569		""" % {
570		"into_table": into_table,
571		"ins_columns": ", ".join(ins_columns),
572		"sel_exprs": ", ".join(sel_exprs),
573		"keyvalues_where": " AND ".join(keyvalues_where),
574		"src_table": src_table,
575		"keyvalues": ", ".join(
576		chain(keyvalues.keys(), extra_dst_keyvalues.keys())
577		),
578		"sets": ", ".join(chain(sets_cc, sets_ar)),
579		}
580
581		qargs = list(
582		chain(
583		additive_relatives.values(),
584		extra_dst_keyvalues.values(),
585		extra_dst_insvalues.values(),
586		keyvalues.values(),
587		)
588		)
589		txn.execute(sql, qargs)
590		else:
591		self.database_engine.lock_table(txn, into_table)
592		src_row = self.db.simple_select_one_txn(
593		txn, src_table, keyvalues, copy_columns
594		)
595		all_dest_keyvalues = {keyvalues, extra_dst_keyvalues}
596		dest_current_row = self.db.simple_select_one_txn(
597		txn,
598		into_table,
599		keyvalues=all_dest_keyvalues,
600		retcols=list(chain(additive_relatives.keys(), copy_columns)),
601		allow_none=True,
602		)
603
604		if dest_current_row is None:
605		merged_dict = {
606		**keyvalues,
607		**extra_dst_keyvalues,
608		**extra_dst_insvalues,
609		**src_row,
610		**additive_relatives,
611		}
612		self.db.simple_insert_txn(txn, into_table, merged_dict)
613		else:
614		for (key, val) in additive_relatives.items():
615		src_row[key] = dest_current_row[key] + val
616		self.db.simple_update_txn(txn, into_table, all_dest_keyvalues, src_row)
617
618		def get_changes_room_total_events_and_bytes(self, min_pos, max_pos):
619		"""Fetches the counts of events in the given range of stream IDs.
620
621		Args:
622		min_pos (int)
623		max_pos (int)
624
625		Returns:
626		Deferred[dict[str, dict[str, int]]]: Mapping of room ID to field
627		changes.
628		"""
629
630		return self.db.runInteraction(
631		"stats_incremental_total_events_and_bytes",
632		self.get_changes_room_total_events_and_bytes_txn,
633		min_pos,
634		max_pos,
635		)
636
637		def get_changes_room_total_events_and_bytes_txn(self, txn, low_pos, high_pos):
638		"""Gets the total_events and total_event_bytes counts for rooms and
639		senders, in a range of stream_orderings (including backfilled events).
640
641		Args:
642		txn
643		low_pos (int): Low stream ordering
644		high_pos (int): High stream ordering
645
646		Returns:
647		tuple[dict[str, dict[str, int]], dict[str, dict[str, int]]]: The
648		room and user deltas for total_events/total_event_bytes in the
649		format of `stats_id` -> fields
650		"""
651
652		if low_pos >= high_pos:
653		# nothing to do here.
654		return {}, {}
655
656		if isinstance(self.database_engine, PostgresEngine):
657		new_bytes_expression = "OCTET_LENGTH(json)"
658		else:
659		new_bytes_expression = "LENGTH(CAST(json AS BLOB))"
660
661		sql = """
662		SELECT events.room_id, COUNT(*) AS new_events, SUM(%s) AS new_bytes
663		FROM events INNER JOIN event_json USING (event_id)
664		WHERE (? < stream_ordering AND stream_ordering <= ?)
665		OR (? <= stream_ordering AND stream_ordering <= ?)
666		GROUP BY events.room_id
667		""" % (
668		new_bytes_expression,
669		)
670
671		txn.execute(sql, (low_pos, high_pos, -high_pos, -low_pos))
672
673		room_deltas = {
674		room_id: {"total_events": new_events, "total_event_bytes": new_bytes}
675		for room_id, new_events, new_bytes in txn
676		}
677
678		sql = """
679		SELECT events.sender, COUNT(*) AS new_events, SUM(%s) AS new_bytes
680		FROM events INNER JOIN event_json USING (event_id)
681		WHERE (? < stream_ordering AND stream_ordering <= ?)
682		OR (? <= stream_ordering AND stream_ordering <= ?)
683		GROUP BY events.sender
684		""" % (
685		new_bytes_expression,
686		)
687
688		txn.execute(sql, (low_pos, high_pos, -high_pos, -low_pos))
689
690		user_deltas = {
691		user_id: {"total_events": new_events, "total_event_bytes": new_bytes}
692		for user_id, new_events, new_bytes in txn
693		if self.hs.is_mine_id(user_id)
694		}
695
696		return room_deltas, user_deltas
697
698		@defer.inlineCallbacks
699		def _calculate_and_set_initial_state_for_room(self, room_id):
700		"""Calculate and insert an entry into room_stats_current.
701
702		Args:
703		room_id (str)
704
705		Returns:
706		Deferred[tuple[dict, dict, int]]: A tuple of room state, membership
707		counts and stream position.
708		"""
709
710		def _fetch_current_state_stats(txn):
711		pos = self.get_room_max_stream_ordering()
712
713		rows = self.db.simple_select_many_txn(
714		txn,
715		table="current_state_events",
716		column="type",
717		iterable=[
718		EventTypes.Create,
719		EventTypes.JoinRules,
720		EventTypes.RoomHistoryVisibility,
721		EventTypes.RoomEncryption,
722		EventTypes.Name,
723		EventTypes.Topic,
724		EventTypes.RoomAvatar,
725		EventTypes.CanonicalAlias,
726		],
727		keyvalues={"room_id": room_id, "state_key": ""},
728		retcols=["event_id"],
729		)
730
731		event_ids = [row["event_id"] for row in rows]
732
733		txn.execute(
734		"""
735		SELECT membership, count(*) FROM current_state_events
736		WHERE room_id = ? AND type = 'm.room.member'
737		GROUP BY membership
738		""",
739		(room_id,),
740		)
741		membership_counts = {membership: cnt for membership, cnt in txn}
742
743		txn.execute(
744		"""
745		SELECT COALESCE(count(*), 0) FROM current_state_events
746		WHERE room_id = ?
747		""",
748		(room_id,),
749		)
750
751		(current_state_events_count,) = txn.fetchone()
752
753		users_in_room = self.get_users_in_room_txn(txn, room_id)
754
755		return (
756		event_ids,
757		membership_counts,
758		current_state_events_count,
759		users_in_room,
760		pos,
761		)
762
763		(
764		event_ids,
765		membership_counts,
766		current_state_events_count,
767		users_in_room,
768		pos,
769		) = yield self.db.runInteraction(
770		"get_initial_state_for_room", _fetch_current_state_stats
771		)
772
773		state_event_map = yield self.get_events(event_ids, get_prev_content=False)
774
775		room_state = {
776		"join_rules": None,
777		"history_visibility": None,
778		"encryption": None,
779		"name": None,
780		"topic": None,
781		"avatar": None,
782		"canonical_alias": None,
783		"is_federatable": True,
784		}
785
786		for event in state_event_map.values():
787		if event.type == EventTypes.JoinRules:
788		room_state["join_rules"] = event.content.get("join_rule")
789		elif event.type == EventTypes.RoomHistoryVisibility:
790		room_state["history_visibility"] = event.content.get(
791		"history_visibility"
792		)
793		elif event.type == EventTypes.RoomEncryption:
794		room_state["encryption"] = event.content.get("algorithm")
795		elif event.type == EventTypes.Name:
796		room_state["name"] = event.content.get("name")
797		elif event.type == EventTypes.Topic:
798		room_state["topic"] = event.content.get("topic")
799		elif event.type == EventTypes.RoomAvatar:
800		room_state["avatar"] = event.content.get("url")
801		elif event.type == EventTypes.CanonicalAlias:
802		room_state["canonical_alias"] = event.content.get("alias")
803		elif event.type == EventTypes.Create:
804		room_state["is_federatable"] = (
805		event.content.get("m.federate", True) is True
806		)
807
808		yield self.update_room_state(room_id, room_state)
809
810		local_users_in_room = [u for u in users_in_room if self.hs.is_mine_id(u)]
811
812		yield self.update_stats_delta(
813		ts=self.clock.time_msec(),
814		stats_type="room",
815		stats_id=room_id,
816		fields={},
817		complete_with_stream_id=pos,
818		absolute_field_overrides={
819		"current_state_events": current_state_events_count,
820		"joined_members": membership_counts.get(Membership.JOIN, 0),
821		"invited_members": membership_counts.get(Membership.INVITE, 0),
822		"left_members": membership_counts.get(Membership.LEAVE, 0),
823		"banned_members": membership_counts.get(Membership.BAN, 0),
824		"local_users_in_room": len(local_users_in_room),
825		},
826		)
827
828		@defer.inlineCallbacks
829		def _calculate_and_set_initial_state_for_user(self, user_id):
830		def _calculate_and_set_initial_state_for_user_txn(txn):
831		pos = self._get_max_stream_id_in_current_state_deltas_txn(txn)
832
833		txn.execute(
834		"""
835		SELECT COUNT(distinct room_id) FROM current_state_events
836		WHERE type = 'm.room.member' AND state_key = ?
837		AND membership = 'join'
838		""",
839		(user_id,),
840		)
841		(count,) = txn.fetchone()
842		return count, pos
843
844		joined_rooms, pos = yield self.db.runInteraction(
845		"calculate_and_set_initial_state_for_user",
846		_calculate_and_set_initial_state_for_user_txn,
847		)
848
849		yield self.update_stats_delta(
850		ts=self.clock.time_msec(),
851		stats_type="user",
852		stats_id=user_id,
853		fields={},
854		complete_with_stream_id=pos,
855		absolute_field_overrides={"joined_rooms": joined_rooms},
856		)

+0

-1064

~~synapse/storage/data_stores/main/stream.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2017 Vector Creations Ltd
3		# Copyright 2018-2019 New Vector Ltd
4		# Copyright 2019 The Matrix.org Foundation C.I.C.
5		#
6		# Licensed under the Apache License, Version 2.0 (the "License");
7		# you may not use this file except in compliance with the License.
8		# You may obtain a copy of the License at
9		#
10		# http://www.apache.org/licenses/LICENSE-2.0
11		#
12		# Unless required by applicable law or agreed to in writing, software
13		# distributed under the License is distributed on an "AS IS" BASIS,
14		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15		# See the License for the specific language governing permissions and
16		# limitations under the License.
17
18		""" This module is responsible for getting events from the DB for pagination
19		and event streaming.
20
21		The order it returns events in depend on whether we are streaming forwards or
22		are paginating backwards. We do this because we want to handle out of order
23		messages nicely, while still returning them in the correct order when we
24		paginate bacwards.
25
26		This is implemented by keeping two ordering columns: stream_ordering and
27		topological_ordering. Stream ordering is basically insertion/received order
28		(except for events from backfill requests). The topological_ordering is a
29		weak ordering of events based on the pdu graph.
30
31		This means that we have to have two different types of tokens, depending on
32		what sort order was used:
33		- stream tokens are of the form: "s%d", which maps directly to the column
34		- topological tokems: "t%d-%d", where the integers map to the topological
35		and stream ordering columns respectively.
36		"""
37
38		import abc
39		import logging
40		from collections import namedtuple
41
42		from twisted.internet import defer
43
44		from synapse.logging.context import make_deferred_yieldable, run_in_background
45		from synapse.storage._base import SQLBaseStore
46		from synapse.storage.data_stores.main.events_worker import EventsWorkerStore
47		from synapse.storage.database import Database, make_in_list_sql_clause
48		from synapse.storage.engines import PostgresEngine
49		from synapse.types import RoomStreamToken
50		from synapse.util.caches.stream_change_cache import StreamChangeCache
51
52		logger = logging.getLogger(__name__)
53
54
55		MAX_STREAM_SIZE = 1000
56
57
58		_STREAM_TOKEN = "stream"
59		_TOPOLOGICAL_TOKEN = "topological"
60
61
62		# Used as return values for pagination APIs
63		_EventDictReturn = namedtuple(
64		"_EventDictReturn", ("event_id", "topological_ordering", "stream_ordering")
65		)
66
67
68		def generate_pagination_where_clause(
69		direction, column_names, from_token, to_token, engine
70		):
71		"""Creates an SQL expression to bound the columns by the pagination
72		tokens.
73
74		For example creates an SQL expression like:
75
76		(6, 7) >= (topological_ordering, stream_ordering)
77		AND (5, 3) < (topological_ordering, stream_ordering)
78
79		would be generated for dir=b, from_token=(6, 7) and to_token=(5, 3).
80
81		Note that tokens are considered to be after the row they are in, e.g. if
82		a row A has a token T, then we consider A to be before T. This convention
83		is important when figuring out inequalities for the generated SQL, and
84		produces the following result:
85		- If paginating forwards then we exclude any rows matching the from
86		token, but include those that match the to token.
87		- If paginating backwards then we include any rows matching the from
88		token, but include those that match the to token.
89
90		Args:
91		direction (str): Whether we're paginating backwards("b") or
92		forwards ("f").
93		column_names (tuple[str, str]): The column names to bound. Must not
94		be user defined as these get inserted directly into the SQL
95		statement without escapes.
96		from_token (tuple[int, int]\|None): The start point for the pagination.
97		This is an exclusive minimum bound if direction is "f", and an
98		inclusive maximum bound if direction is "b".
99		to_token (tuple[int, int]\|None): The endpoint point for the pagination.
100		This is an inclusive maximum bound if direction is "f", and an
101		exclusive minimum bound if direction is "b".
102		engine: The database engine to generate the clauses for
103
104		Returns:
105		str: The sql expression
106		"""
107		assert direction in ("b", "f")
108
109		where_clause = []
110		if from_token:
111		where_clause.append(
112		_make_generic_sql_bound(
113		bound=">=" if direction == "b" else "<",
114		column_names=column_names,
115		values=from_token,
116		engine=engine,
117		)
118		)
119
120		if to_token:
121		where_clause.append(
122		_make_generic_sql_bound(
123		bound="<" if direction == "b" else ">=",
124		column_names=column_names,
125		values=to_token,
126		engine=engine,
127		)
128		)
129
130		return " AND ".join(where_clause)
131
132
133		def _make_generic_sql_bound(bound, column_names, values, engine):
134		"""Create an SQL expression that bounds the given column names by the
135		values, e.g. create the equivalent of `(1, 2) < (col1, col2)`.
136
137		Only works with two columns.
138
139		Older versions of SQLite don't support that syntax so we have to expand it
140		out manually.
141
142		Args:
143		bound (str): The comparison operator to use. One of ">", "<", ">=",
144		"<=", where the values are on the left and columns on the right.
145		names (tuple[str, str]): The column names. Must not be user defined
146		as these get inserted directly into the SQL statement without
147		escapes.
148		values (tuple[int\|None, int]): The values to bound the columns by. If
149		the first value is None then only creates a bound on the second
150		column.
151		engine: The database engine to generate the SQL for
152
153		Returns:
154		str
155		"""
156
157		assert bound in (">", "<", ">=", "<=")
158
159		name1, name2 = column_names
160		val1, val2 = values
161
162		if val1 is None:
163		val2 = int(val2)
164		return "(%d %s %s)" % (val2, bound, name2)
165
166		val1 = int(val1)
167		val2 = int(val2)
168
169		if isinstance(engine, PostgresEngine):
170		# Postgres doesn't optimise ``(x < a) OR (x=a AND y<b)`` as well
171		# as it optimises ``(x,y) < (a,b)`` on multicolumn indexes. So we
172		# use the later form when running against postgres.
173		return "((%d,%d) %s (%s,%s))" % (val1, val2, bound, name1, name2)
174
175		# We want to generate queries of e.g. the form:
176		#
177		# (val1 < name1 OR (val1 = name1 AND val2 <= name2))
178		#
179		# which is equivalent to (val1, val2) < (name1, name2)
180
181		return """(
182		{val1:d} {strict_bound} {name1}
183		OR ({val1:d} = {name1} AND {val2:d} {bound} {name2})
184		)""".format(
185		name1=name1,
186		val1=val1,
187		name2=name2,
188		val2=val2,
189		strict_bound=bound[0], # The first bound must always be strict equality here
190		bound=bound,
191		)
192
193
194		def filter_to_clause(event_filter):
195		# NB: This may create SQL clauses that don't optimise well (and we don't
196		# have indices on all possible clauses). E.g. it may create
197		# "room_id == X AND room_id != X", which postgres doesn't optimise.
198
199		if not event_filter:
200		return "", []
201
202		clauses = []
203		args = []
204
205		if event_filter.types:
206		clauses.append("(%s)" % " OR ".join("type = ?" for _ in event_filter.types))
207		args.extend(event_filter.types)
208
209		for typ in event_filter.not_types:
210		clauses.append("type != ?")
211		args.append(typ)
212
213		if event_filter.senders:
214		clauses.append("(%s)" % " OR ".join("sender = ?" for _ in event_filter.senders))
215		args.extend(event_filter.senders)
216
217		for sender in event_filter.not_senders:
218		clauses.append("sender != ?")
219		args.append(sender)
220
221		if event_filter.rooms:
222		clauses.append("(%s)" % " OR ".join("room_id = ?" for _ in event_filter.rooms))
223		args.extend(event_filter.rooms)
224
225		for room_id in event_filter.not_rooms:
226		clauses.append("room_id != ?")
227		args.append(room_id)
228
229		if event_filter.contains_url:
230		clauses.append("contains_url = ?")
231		args.append(event_filter.contains_url)
232
233		# We're only applying the "labels" filter on the database query, because applying the
234		# "not_labels" filter via a SQL query is non-trivial. Instead, we let
235		# event_filter.check_fields apply it, which is not as efficient but makes the
236		# implementation simpler.
237		if event_filter.labels:
238		clauses.append("(%s)" % " OR ".join("label = ?" for _ in event_filter.labels))
239		args.extend(event_filter.labels)
240
241		return " AND ".join(clauses), args
242
243
244		class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
245		"""This is an abstract base class where subclasses must implement
246		`get_room_max_stream_ordering` and `get_room_min_stream_ordering`
247		which can be called in the initializer.
248		"""
249
250		__metaclass__ = abc.ABCMeta
251
252		def __init__(self, database: Database, db_conn, hs):
253		super(StreamWorkerStore, self).__init__(database, db_conn, hs)
254
255		self._instance_name = hs.get_instance_name()
256		self._send_federation = hs.should_send_federation()
257		self._federation_shard_config = hs.config.worker.federation_shard_config
258
259		# If we're a process that sends federation we may need to reset the
260		# `federation_stream_position` table to match the current sharding
261		# config. We don't do this now as otherwise two processes could conflict
262		# during startup which would cause one to die.
263		self._need_to_reset_federation_stream_positions = self._send_federation
264
265		events_max = self.get_room_max_stream_ordering()
266		event_cache_prefill, min_event_val = self.db.get_cache_dict(
267		db_conn,
268		"events",
269		entity_column="room_id",
270		stream_column="stream_ordering",
271		max_value=events_max,
272		)
273		self._events_stream_cache = StreamChangeCache(
274		"EventsRoomStreamChangeCache",
275		min_event_val,
276		prefilled_cache=event_cache_prefill,
277		)
278		self._membership_stream_cache = StreamChangeCache(
279		"MembershipStreamChangeCache", events_max
280		)
281
282		self._stream_order_on_start = self.get_room_max_stream_ordering()
283
284		@abc.abstractmethod
285		def get_room_max_stream_ordering(self):
286		raise NotImplementedError()
287
288		@abc.abstractmethod
289		def get_room_min_stream_ordering(self):
290		raise NotImplementedError()
291
292		@defer.inlineCallbacks
293		def get_room_events_stream_for_rooms(
294		self, room_ids, from_key, to_key, limit=0, order="DESC"
295		):
296		"""Get new room events in stream ordering since `from_key`.
297
298		Args:
299		room_id (str)
300		from_key (str): Token from which no events are returned before
301		to_key (str): Token from which no events are returned after. (This
302		is typically the current stream token)
303		limit (int): Maximum number of events to return
304		order (str): Either "DESC" or "ASC". Determines which events are
305		returned when the result is limited. If "DESC" then the most
306		recent `limit` events are returned, otherwise returns the
307		oldest `limit` events.
308
309		Returns:
310		Deferred[dict[str,tuple[list[FrozenEvent], str]]]
311		A map from room id to a tuple containing:
312		- list of recent events in the room
313		- stream ordering key for the start of the chunk of events returned.
314		"""
315		from_id = RoomStreamToken.parse_stream_token(from_key).stream
316
317		room_ids = yield self._events_stream_cache.get_entities_changed(
318		room_ids, from_id
319		)
320
321		if not room_ids:
322		return {}
323
324		results = {}
325		room_ids = list(room_ids)
326		for rm_ids in (room_ids[i : i + 20] for i in range(0, len(room_ids), 20)):
327		res = yield make_deferred_yieldable(
328		defer.gatherResults(
329		[
330		run_in_background(
331		self.get_room_events_stream_for_room,
332		room_id,
333		from_key,
334		to_key,
335		limit,
336		order=order,
337		)
338		for room_id in rm_ids
339		],
340		consumeErrors=True,
341		)
342		)
343		results.update(dict(zip(rm_ids, res)))
344
345		return results
346
347		def get_rooms_that_changed(self, room_ids, from_key):
348		"""Given a list of rooms and a token, return rooms where there may have
349		been changes.
350
351		Args:
352		room_ids (list)
353		from_key (str): The room_key portion of a StreamToken
354		"""
355		from_key = RoomStreamToken.parse_stream_token(from_key).stream
356		return {
357		room_id
358		for room_id in room_ids
359		if self._events_stream_cache.has_entity_changed(room_id, from_key)
360		}
361
362		@defer.inlineCallbacks
363		def get_room_events_stream_for_room(
364		self, room_id, from_key, to_key, limit=0, order="DESC"
365		):
366
367		"""Get new room events in stream ordering since `from_key`.
368
369		Args:
370		room_id (str)
371		from_key (str): Token from which no events are returned before
372		to_key (str): Token from which no events are returned after. (This
373		is typically the current stream token)
374		limit (int): Maximum number of events to return
375		order (str): Either "DESC" or "ASC". Determines which events are
376		returned when the result is limited. If "DESC" then the most
377		recent `limit` events are returned, otherwise returns the
378		oldest `limit` events.
379
380		Returns:
381		Deferred[tuple[list[FrozenEvent], str]]: Returns the list of
382		events (in ascending order) and the token from the start of
383		the chunk of events returned.
384		"""
385		if from_key == to_key:
386		return [], from_key
387
388		from_id = RoomStreamToken.parse_stream_token(from_key).stream
389		to_id = RoomStreamToken.parse_stream_token(to_key).stream
390
391		has_changed = yield self._events_stream_cache.has_entity_changed(
392		room_id, from_id
393		)
394
395		if not has_changed:
396		return [], from_key
397
398		def f(txn):
399		sql = (
400		"SELECT event_id, stream_ordering FROM events WHERE"
401		" room_id = ?"
402		" AND not outlier"
403		" AND stream_ordering > ? AND stream_ordering <= ?"
404		" ORDER BY stream_ordering %s LIMIT ?"
405		) % (order,)
406		txn.execute(sql, (room_id, from_id, to_id, limit))
407
408		rows = [_EventDictReturn(row[0], None, row[1]) for row in txn]
409		return rows
410
411		rows = yield self.db.runInteraction("get_room_events_stream_for_room", f)
412
413		ret = yield self.get_events_as_list(
414		[r.event_id for r in rows], get_prev_content=True
415		)
416
417		self._set_before_and_after(ret, rows, topo_order=from_id is None)
418
419		if order.lower() == "desc":
420		ret.reverse()
421
422		if rows:
423		key = "s%d" % min(r.stream_ordering for r in rows)
424		else:
425		# Assume we didn't get anything because there was nothing to
426		# get.
427		key = from_key
428
429		return ret, key
430
431		@defer.inlineCallbacks
432		def get_membership_changes_for_user(self, user_id, from_key, to_key):
433		from_id = RoomStreamToken.parse_stream_token(from_key).stream
434		to_id = RoomStreamToken.parse_stream_token(to_key).stream
435
436		if from_key == to_key:
437		return []
438
439		if from_id:
440		has_changed = self._membership_stream_cache.has_entity_changed(
441		user_id, int(from_id)
442		)
443		if not has_changed:
444		return []
445
446		def f(txn):
447		sql = (
448		"SELECT m.event_id, stream_ordering FROM events AS e,"
449		" room_memberships AS m"
450		" WHERE e.event_id = m.event_id"
451		" AND m.user_id = ?"
452		" AND e.stream_ordering > ? AND e.stream_ordering <= ?"
453		" ORDER BY e.stream_ordering ASC"
454		)
455		txn.execute(sql, (user_id, from_id, to_id))
456
457		rows = [_EventDictReturn(row[0], None, row[1]) for row in txn]
458
459		return rows
460
461		rows = yield self.db.runInteraction("get_membership_changes_for_user", f)
462
463		ret = yield self.get_events_as_list(
464		[r.event_id for r in rows], get_prev_content=True
465		)
466
467		self._set_before_and_after(ret, rows, topo_order=False)
468
469		return ret
470
471		@defer.inlineCallbacks
472		def get_recent_events_for_room(self, room_id, limit, end_token):
473		"""Get the most recent events in the room in topological ordering.
474
475		Args:
476		room_id (str)
477		limit (int)
478		end_token (str): The stream token representing now.
479
480		Returns:
481		Deferred[tuple[list[FrozenEvent], str]]: Returns a list of
482		events and a token pointing to the start of the returned
483		events.
484		The events returned are in ascending order.
485		"""
486
487		rows, token = yield self.get_recent_event_ids_for_room(
488		room_id, limit, end_token
489		)
490
491		events = yield self.get_events_as_list(
492		[r.event_id for r in rows], get_prev_content=True
493		)
494
495		self._set_before_and_after(events, rows)
496
497		return (events, token)
498
499		@defer.inlineCallbacks
500		def get_recent_event_ids_for_room(self, room_id, limit, end_token):
501		"""Get the most recent events in the room in topological ordering.
502
503		Args:
504		room_id (str)
505		limit (int)
506		end_token (str): The stream token representing now.
507
508		Returns:
509		Deferred[tuple[list[_EventDictReturn], str]]: Returns a list of
510		_EventDictReturn and a token pointing to the start of the returned
511		events.
512		The events returned are in ascending order.
513		"""
514		# Allow a zero limit here, and no-op.
515		if limit == 0:
516		return [], end_token
517
518		end_token = RoomStreamToken.parse(end_token)
519
520		rows, token = yield self.db.runInteraction(
521		"get_recent_event_ids_for_room",
522		self._paginate_room_events_txn,
523		room_id,
524		from_token=end_token,
525		limit=limit,
526		)
527
528		# We want to return the results in ascending order.
529		rows.reverse()
530
531		return rows, token
532
533		def get_room_event_before_stream_ordering(self, room_id, stream_ordering):
534		"""Gets details of the first event in a room at or before a stream ordering
535
536		Args:
537		room_id (str):
538		stream_ordering (int):
539
540		Returns:
541		Deferred[(int, int, str)]:
542		(stream ordering, topological ordering, event_id)
543		"""
544
545		def _f(txn):
546		sql = (
547		"SELECT stream_ordering, topological_ordering, event_id"
548		" FROM events"
549		" WHERE room_id = ? AND stream_ordering <= ?"
550		" AND NOT outlier"
551		" ORDER BY stream_ordering DESC"
552		" LIMIT 1"
553		)
554		txn.execute(sql, (room_id, stream_ordering))
555		return txn.fetchone()
556
557		return self.db.runInteraction("get_room_event_before_stream_ordering", _f)
558
559		@defer.inlineCallbacks
560		def get_room_events_max_id(self, room_id=None):
561		"""Returns the current token for rooms stream.
562
563		By default, it returns the current global stream token. Specifying a
564		`room_id` causes it to return the current room specific topological
565		token.
566		"""
567		token = yield self.get_room_max_stream_ordering()
568		if room_id is None:
569		return "s%d" % (token,)
570		else:
571		topo = yield self.db.runInteraction(
572		"_get_max_topological_txn", self._get_max_topological_txn, room_id
573		)
574		return "t%d-%d" % (topo, token)
575
576		def get_stream_token_for_event(self, event_id):
577		"""The stream token for an event
578		Args:
579		event_id(str): The id of the event to look up a stream token for.
580		Raises:
581		StoreError if the event wasn't in the database.
582		Returns:
583		A deferred "s%d" stream token.
584		"""
585		return self.db.simple_select_one_onecol(
586		table="events", keyvalues={"event_id": event_id}, retcol="stream_ordering"
587		).addCallback(lambda row: "s%d" % (row,))
588
589		def get_topological_token_for_event(self, event_id):
590		"""The stream token for an event
591		Args:
592		event_id(str): The id of the event to look up a stream token for.
593		Raises:
594		StoreError if the event wasn't in the database.
595		Returns:
596		A deferred "t%d-%d" topological token.
597		"""
598		return self.db.simple_select_one(
599		table="events",
600		keyvalues={"event_id": event_id},
601		retcols=("stream_ordering", "topological_ordering"),
602		desc="get_topological_token_for_event",
603		).addCallback(
604		lambda row: "t%d-%d" % (row["topological_ordering"], row["stream_ordering"])
605		)
606
607		def get_max_topological_token(self, room_id, stream_key):
608		"""Get the max topological token in a room before the given stream
609		ordering.
610
611		Args:
612		room_id (str)
613		stream_key (int)
614
615		Returns:
616		Deferred[int]
617		"""
618		sql = (
619		"SELECT coalesce(max(topological_ordering), 0) FROM events"
620		" WHERE room_id = ? AND stream_ordering < ?"
621		)
622		return self.db.execute(
623		"get_max_topological_token", None, sql, room_id, stream_key
624		).addCallback(lambda r: r[0][0] if r else 0)
625
626		def _get_max_topological_txn(self, txn, room_id):
627		txn.execute(
628		"SELECT MAX(topological_ordering) FROM events WHERE room_id = ?",
629		(room_id,),
630		)
631
632		rows = txn.fetchall()
633		return rows[0][0] if rows else 0
634
635		@staticmethod
636		def _set_before_and_after(events, rows, topo_order=True):
637		"""Inserts ordering information to events' internal metadata from
638		the DB rows.
639
640		Args:
641		events (list[FrozenEvent])
642		rows (list[_EventDictReturn])
643		topo_order (bool): Whether the events were ordered topologically
644		or by stream ordering. If true then all rows should have a non
645		null topological_ordering.
646		"""
647		for event, row in zip(events, rows):
648		stream = row.stream_ordering
649		if topo_order and row.topological_ordering:
650		topo = row.topological_ordering
651		else:
652		topo = None
653		internal = event.internal_metadata
654		internal.before = str(RoomStreamToken(topo, stream - 1))
655		internal.after = str(RoomStreamToken(topo, stream))
656		internal.order = (int(topo) if topo else 0, int(stream))
657
658		@defer.inlineCallbacks
659		def get_events_around(
660		self, room_id, event_id, before_limit, after_limit, event_filter=None
661		):
662		"""Retrieve events and pagination tokens around a given event in a
663		room.
664
665		Args:
666		room_id (str)
667		event_id (str)
668		before_limit (int)
669		after_limit (int)
670		event_filter (Filter\|None)
671
672		Returns:
673		dict
674		"""
675
676		results = yield self.db.runInteraction(
677		"get_events_around",
678		self._get_events_around_txn,
679		room_id,
680		event_id,
681		before_limit,
682		after_limit,
683		event_filter,
684		)
685
686		events_before = yield self.get_events_as_list(
687		list(results["before"]["event_ids"]), get_prev_content=True
688		)
689
690		events_after = yield self.get_events_as_list(
691		list(results["after"]["event_ids"]), get_prev_content=True
692		)
693
694		return {
695		"events_before": events_before,
696		"events_after": events_after,
697		"start": results["before"]["token"],
698		"end": results["after"]["token"],
699		}
700
701		def _get_events_around_txn(
702		self, txn, room_id, event_id, before_limit, after_limit, event_filter
703		):
704		"""Retrieves event_ids and pagination tokens around a given event in a
705		room.
706
707		Args:
708		room_id (str)
709		event_id (str)
710		before_limit (int)
711		after_limit (int)
712		event_filter (Filter\|None)
713
714		Returns:
715		dict
716		"""
717
718		results = self.db.simple_select_one_txn(
719		txn,
720		"events",
721		keyvalues={"event_id": event_id, "room_id": room_id},
722		retcols=["stream_ordering", "topological_ordering"],
723		)
724
725		# Paginating backwards includes the event at the token, but paginating
726		# forward doesn't.
727		before_token = RoomStreamToken(
728		results["topological_ordering"] - 1, results["stream_ordering"]
729		)
730
731		after_token = RoomStreamToken(
732		results["topological_ordering"], results["stream_ordering"]
733		)
734
735		rows, start_token = self._paginate_room_events_txn(
736		txn,
737		room_id,
738		before_token,
739		direction="b",
740		limit=before_limit,
741		event_filter=event_filter,
742		)
743		events_before = [r.event_id for r in rows]
744
745		rows, end_token = self._paginate_room_events_txn(
746		txn,
747		room_id,
748		after_token,
749		direction="f",
750		limit=after_limit,
751		event_filter=event_filter,
752		)
753		events_after = [r.event_id for r in rows]
754
755		return {
756		"before": {"event_ids": events_before, "token": start_token},
757		"after": {"event_ids": events_after, "token": end_token},
758		}
759
760		@defer.inlineCallbacks
761		def get_all_new_events_stream(self, from_id, current_id, limit):
762		"""Get all new events
763
764		Returns all events with from_id < stream_ordering <= current_id.
765
766		Args:
767		from_id (int): the stream_ordering of the last event we processed
768		current_id (int): the stream_ordering of the most recently processed event
769		limit (int): the maximum number of events to return
770
771		Returns:
772		Deferred[Tuple[int, list[FrozenEvent]]]: A tuple of (next_id, events), where
773		`next_id` is the next value to pass as `from_id` (it will either be the
774		stream_ordering of the last returned event, or, if fewer than `limit` events
775		were found, `current_id`.
776		"""
777
778		def get_all_new_events_stream_txn(txn):
779		sql = (
780		"SELECT e.stream_ordering, e.event_id"
781		" FROM events AS e"
782		" WHERE"
783		" ? < e.stream_ordering AND e.stream_ordering <= ?"
784		" ORDER BY e.stream_ordering ASC"
785		" LIMIT ?"
786		)
787
788		txn.execute(sql, (from_id, current_id, limit))
789		rows = txn.fetchall()
790
791		upper_bound = current_id
792		if len(rows) == limit:
793		upper_bound = rows[-1][0]
794
795		return upper_bound, [row[1] for row in rows]
796
797		upper_bound, event_ids = yield self.db.runInteraction(
798		"get_all_new_events_stream", get_all_new_events_stream_txn
799		)
800
801		events = yield self.get_events_as_list(event_ids)
802
803		return upper_bound, events
804
805		async def get_federation_out_pos(self, typ: str) -> int:
806		if self._need_to_reset_federation_stream_positions:
807		await self.db.runInteraction(
808		"_reset_federation_positions_txn", self._reset_federation_positions_txn
809		)
810		self._need_to_reset_federation_stream_positions = False
811
812		return await self.db.simple_select_one_onecol(
813		table="federation_stream_position",
814		retcol="stream_id",
815		keyvalues={"type": typ, "instance_name": self._instance_name},
816		desc="get_federation_out_pos",
817		)
818
819		async def update_federation_out_pos(self, typ, stream_id):
820		if self._need_to_reset_federation_stream_positions:
821		await self.db.runInteraction(
822		"_reset_federation_positions_txn", self._reset_federation_positions_txn
823		)
824		self._need_to_reset_federation_stream_positions = False
825
826		return await self.db.simple_update_one(
827		table="federation_stream_position",
828		keyvalues={"type": typ, "instance_name": self._instance_name},
829		updatevalues={"stream_id": stream_id},
830		desc="update_federation_out_pos",
831		)
832
833		def _reset_federation_positions_txn(self, txn):
834		"""Fiddles with the `federation_stream_position` table to make it match
835		the configured federation sender instances during start up.
836		"""
837
838		# The federation sender instances may have changed, so we need to
839		# massage the `federation_stream_position` table to have a row per type
840		# per instance sending federation. If there is a mismatch we update the
841		# table with the correct rows using the minimum stream ID seen. This
842		# may result in resending of events/EDUs to remote servers, but that is
843		# preferable to dropping them.
844
845		if not self._send_federation:
846		return
847
848		# Pull out the configured instances. If we don't have a shard config then
849		# we assume that we're the only instance sending.
850		configured_instances = self._federation_shard_config.instances
851		if not configured_instances:
852		configured_instances = [self._instance_name]
853		elif self._instance_name not in configured_instances:
854		return
855
856		instances_in_table = self.db.simple_select_onecol_txn(
857		txn,
858		table="federation_stream_position",
859		keyvalues={},
860		retcol="instance_name",
861		)
862
863		if set(instances_in_table) == set(configured_instances):
864		# Nothing to do
865		return
866
867		sql = """
868		SELECT type, MIN(stream_id) FROM federation_stream_position
869		GROUP BY type
870		"""
871		txn.execute(sql)
872		min_positions = dict(txn) # Map from type -> min position
873
874		# Ensure we do actually have some values here
875		assert set(min_positions) == {"federation", "events"}
876
877		sql = """
878		DELETE FROM federation_stream_position
879		WHERE NOT (%s)
880		"""
881		clause, args = make_in_list_sql_clause(
882		txn.database_engine, "instance_name", configured_instances
883		)
884		txn.execute(sql % (clause,), args)
885
886		for typ, stream_id in min_positions.items():
887		self.db.simple_upsert_txn(
888		txn,
889		table="federation_stream_position",
890		keyvalues={"type": typ, "instance_name": self._instance_name},
891		values={"stream_id": stream_id},
892		)
893
894		def has_room_changed_since(self, room_id, stream_id):
895		return self._events_stream_cache.has_entity_changed(room_id, stream_id)
896
897		def _paginate_room_events_txn(
898		self,
899		txn,
900		room_id,
901		from_token,
902		to_token=None,
903		direction="b",
904		limit=-1,
905		event_filter=None,
906		):
907		"""Returns list of events before or after a given token.
908
909		Args:
910		txn
911		room_id (str)
912		from_token (RoomStreamToken): The token used to stream from
913		to_token (RoomStreamToken\|None): A token which if given limits the
914		results to only those before
915		direction(char): Either 'b' or 'f' to indicate whether we are
916		paginating forwards or backwards from `from_key`.
917		limit (int): The maximum number of events to return.
918		event_filter (Filter\|None): If provided filters the events to
919		those that match the filter.
920
921		Returns:
922		Deferred[tuple[list[_EventDictReturn], str]]: Returns the results
923		as a list of _EventDictReturn and a token that points to the end
924		of the result set. If no events are returned then the end of the
925		stream has been reached (i.e. there are no events between
926		`from_token` and `to_token`), or `limit` is zero.
927		"""
928
929		assert int(limit) >= 0
930
931		# Tokens really represent positions between elements, but we use
932		# the convention of pointing to the event before the gap. Hence
933		# we have a bit of asymmetry when it comes to equalities.
934		args = [False, room_id]
935		if direction == "b":
936		order = "DESC"
937		else:
938		order = "ASC"
939
940		bounds = generate_pagination_where_clause(
941		direction=direction,
942		column_names=("topological_ordering", "stream_ordering"),
943		from_token=from_token,
944		to_token=to_token,
945		engine=self.database_engine,
946		)
947
948		filter_clause, filter_args = filter_to_clause(event_filter)
949
950		if filter_clause:
951		bounds += " AND " + filter_clause
952		args.extend(filter_args)
953
954		args.append(int(limit))
955
956		select_keywords = "SELECT"
957		join_clause = ""
958		if event_filter and event_filter.labels:
959		# If we're not filtering on a label, then joining on event_labels will
960		# return as many row for a single event as the number of labels it has. To
961		# avoid this, only join if we're filtering on at least one label.
962		join_clause = """
963		LEFT JOIN event_labels
964		USING (event_id, room_id, topological_ordering)
965		"""
966		if len(event_filter.labels) > 1:
967		# Using DISTINCT in this SELECT query is quite expensive, because it
968		# requires the engine to sort on the entire (not limited) result set,
969		# i.e. the entire events table. We only need to use it when we're
970		# filtering on more than two labels, because that's the only scenario
971		# in which we can possibly to get multiple times the same event ID in
972		# the results.
973		select_keywords += "DISTINCT"
974
975		sql = """
976		%(select_keywords)s event_id, topological_ordering, stream_ordering
977		FROM events
978		%(join_clause)s
979		WHERE outlier = ? AND room_id = ? AND %(bounds)s
980		ORDER BY topological_ordering %(order)s,
981		stream_ordering %(order)s LIMIT ?
982		""" % {
983		"select_keywords": select_keywords,
984		"join_clause": join_clause,
985		"bounds": bounds,
986		"order": order,
987		}
988
989		txn.execute(sql, args)
990
991		rows = [_EventDictReturn(row[0], row[1], row[2]) for row in txn]
992
993		if rows:
994		topo = rows[-1].topological_ordering
995		toke = rows[-1].stream_ordering
996		if direction == "b":
997		# Tokens are positions between events.
998		# This token points after the last event in the chunk.
999		# We need it to point to the event before it in the chunk
1000		# when we are going backwards so we subtract one from the
1001		# stream part.
1002		toke -= 1
1003		next_token = RoomStreamToken(topo, toke)
1004		else:
1005		# TODO (erikj): We should work out what to do here instead.
1006		next_token = to_token if to_token else from_token
1007
1008		return rows, str(next_token)
1009
1010		@defer.inlineCallbacks
1011		def paginate_room_events(
1012		self, room_id, from_key, to_key=None, direction="b", limit=-1, event_filter=None
1013		):
1014		"""Returns list of events before or after a given token.
1015
1016		Args:
1017		room_id (str)
1018		from_key (str): The token used to stream from
1019		to_key (str\|None): A token which if given limits the results to
1020		only those before
1021		direction(char): Either 'b' or 'f' to indicate whether we are
1022		paginating forwards or backwards from `from_key`.
1023		limit (int): The maximum number of events to return.
1024		event_filter (Filter\|None): If provided filters the events to
1025		those that match the filter.
1026
1027		Returns:
1028		tuple[list[FrozenEvent], str]: Returns the results as a list of
1029		events and a token that points to the end of the result set. If no
1030		events are returned then the end of the stream has been reached
1031		(i.e. there are no events between `from_key` and `to_key`).
1032		"""
1033
1034		from_key = RoomStreamToken.parse(from_key)
1035		if to_key:
1036		to_key = RoomStreamToken.parse(to_key)
1037
1038		rows, token = yield self.db.runInteraction(
1039		"paginate_room_events",
1040		self._paginate_room_events_txn,
1041		room_id,
1042		from_key,
1043		to_key,
1044		direction,
1045		limit,
1046		event_filter,
1047		)
1048
1049		events = yield self.get_events_as_list(
1050		[r.event_id for r in rows], get_prev_content=True
1051		)
1052
1053		self._set_before_and_after(events, rows)
1054
1055		return (events, token)
1056
1057
1058		class StreamStore(StreamWorkerStore):
1059		def get_room_max_stream_ordering(self):
1060		return self._stream_id_gen.get_current_token()
1061
1062		def get_room_min_stream_ordering(self):
1063		return self._backfill_id_gen.get_current_token()

+0

-288

~~synapse/storage/data_stores/main/tags.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2018 New Vector Ltd
3		#
4		# Licensed under the Apache License, Version 2.0 (the "License");
5		# you may not use this file except in compliance with the License.
6		# You may obtain a copy of the License at
7		#
8		# http://www.apache.org/licenses/LICENSE-2.0
9		#
10		# Unless required by applicable law or agreed to in writing, software
11		# distributed under the License is distributed on an "AS IS" BASIS,
12		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13		# See the License for the specific language governing permissions and
14		# limitations under the License.
15
16		import logging
17		from typing import List, Tuple
18
19		from canonicaljson import json
20
21		from twisted.internet import defer
22
23		from synapse.storage._base import db_to_json
24		from synapse.storage.data_stores.main.account_data import AccountDataWorkerStore
25		from synapse.util.caches.descriptors import cached
26
27		logger = logging.getLogger(__name__)
28
29
30		class TagsWorkerStore(AccountDataWorkerStore):
31		@cached()
32		def get_tags_for_user(self, user_id):
33		"""Get all the tags for a user.
34
35
36		Args:
37		user_id(str): The user to get the tags for.
38		Returns:
39		A deferred dict mapping from room_id strings to dicts mapping from
40		tag strings to tag content.
41		"""
42
43		deferred = self.db.simple_select_list(
44		"room_tags", {"user_id": user_id}, ["room_id", "tag", "content"]
45		)
46
47		@deferred.addCallback
48		def tags_by_room(rows):
49		tags_by_room = {}
50		for row in rows:
51		room_tags = tags_by_room.setdefault(row["room_id"], {})
52		room_tags[row["tag"]] = db_to_json(row["content"])
53		return tags_by_room
54
55		return deferred
56
57		async def get_all_updated_tags(
58		self, instance_name: str, last_id: int, current_id: int, limit: int
59		) -> Tuple[List[Tuple[int, tuple]], int, bool]:
60		"""Get updates for tags replication stream.
61
62		Args:
63		instance_name: The writer we want to fetch updates from. Unused
64		here since there is only ever one writer.
65		last_id: The token to fetch updates from. Exclusive.
66		current_id: The token to fetch updates up to. Inclusive.
67		limit: The requested limit for the number of rows to return. The
68		function may return more or fewer rows.
69
70		Returns:
71		A tuple consisting of: the updates, a token to use to fetch
72		subsequent updates, and whether we returned fewer rows than exists
73		between the requested tokens due to the limit.
74
75		The token returned can be used in a subsequent call to this
76		function to get further updatees.
77
78		The updates are a list of 2-tuples of stream ID and the row data
79		"""
80
81		if last_id == current_id:
82		return [], current_id, False
83
84		def get_all_updated_tags_txn(txn):
85		sql = (
86		"SELECT stream_id, user_id, room_id"
87		" FROM room_tags_revisions as r"
88		" WHERE ? < stream_id AND stream_id <= ?"
89		" ORDER BY stream_id ASC LIMIT ?"
90		)
91		txn.execute(sql, (last_id, current_id, limit))
92		return txn.fetchall()
93
94		tag_ids = await self.db.runInteraction(
95		"get_all_updated_tags", get_all_updated_tags_txn
96		)
97
98		def get_tag_content(txn, tag_ids):
99		sql = "SELECT tag, content FROM room_tags WHERE user_id=? AND room_id=?"
100		results = []
101		for stream_id, user_id, room_id in tag_ids:
102		txn.execute(sql, (user_id, room_id))
103		tags = []
104		for tag, content in txn:
105		tags.append(json.dumps(tag) + ":" + content)
106		tag_json = "{" + ",".join(tags) + "}"
107		results.append((stream_id, (user_id, room_id, tag_json)))
108
109		return results
110
111		batch_size = 50
112		results = []
113		for i in range(0, len(tag_ids), batch_size):
114		tags = await self.db.runInteraction(
115		"get_all_updated_tag_content",
116		get_tag_content,
117		tag_ids[i : i + batch_size],
118		)
119		results.extend(tags)
120
121		limited = False
122		upto_token = current_id
123		if len(results) >= limit:
124		upto_token = results[-1][0]
125		limited = True
126
127		return results, upto_token, limited
128
129		@defer.inlineCallbacks
130		def get_updated_tags(self, user_id, stream_id):
131		"""Get all the tags for the rooms where the tags have changed since the
132		given version
133
134		Args:
135		user_id(str): The user to get the tags for.
136		stream_id(int): The earliest update to get for the user.
137		Returns:
138		A deferred dict mapping from room_id strings to lists of tag
139		strings for all the rooms that changed since the stream_id token.
140		"""
141
142		def get_updated_tags_txn(txn):
143		sql = (
144		"SELECT room_id from room_tags_revisions"
145		" WHERE user_id = ? AND stream_id > ?"
146		)
147		txn.execute(sql, (user_id, stream_id))
148		room_ids = [row[0] for row in txn]
149		return room_ids
150
151		changed = self._account_data_stream_cache.has_entity_changed(
152		user_id, int(stream_id)
153		)
154		if not changed:
155		return {}
156
157		room_ids = yield self.db.runInteraction(
158		"get_updated_tags", get_updated_tags_txn
159		)
160
161		results = {}
162		if room_ids:
163		tags_by_room = yield self.get_tags_for_user(user_id)
164		for room_id in room_ids:
165		results[room_id] = tags_by_room.get(room_id, {})
166
167		return results
168
169		def get_tags_for_room(self, user_id, room_id):
170		"""Get all the tags for the given room
171		Args:
172		user_id(str): The user to get tags for
173		room_id(str): The room to get tags for
174		Returns:
175		A deferred list of string tags.
176		"""
177		return self.db.simple_select_list(
178		table="room_tags",
179		keyvalues={"user_id": user_id, "room_id": room_id},
180		retcols=("tag", "content"),
181		desc="get_tags_for_room",
182		).addCallback(
183		lambda rows: {row["tag"]: db_to_json(row["content"]) for row in rows}
184		)
185
186
187		class TagsStore(TagsWorkerStore):
188		@defer.inlineCallbacks
189		def add_tag_to_room(self, user_id, room_id, tag, content):
190		"""Add a tag to a room for a user.
191		Args:
192		user_id(str): The user to add a tag for.
193		room_id(str): The room to add a tag for.
194		tag(str): The tag name to add.
195		content(dict): A json object to associate with the tag.
196		Returns:
197		A deferred that completes once the tag has been added.
198		"""
199		content_json = json.dumps(content)
200
201		def add_tag_txn(txn, next_id):
202		self.db.simple_upsert_txn(
203		txn,
204		table="room_tags",
205		keyvalues={"user_id": user_id, "room_id": room_id, "tag": tag},
206		values={"content": content_json},
207		)
208		self._update_revision_txn(txn, user_id, room_id, next_id)
209
210		with self._account_data_id_gen.get_next() as next_id:
211		yield self.db.runInteraction("add_tag", add_tag_txn, next_id)
212
213		self.get_tags_for_user.invalidate((user_id,))
214
215		result = self._account_data_id_gen.get_current_token()
216		return result
217
218		@defer.inlineCallbacks
219		def remove_tag_from_room(self, user_id, room_id, tag):
220		"""Remove a tag from a room for a user.
221		Returns:
222		A deferred that completes once the tag has been removed
223		"""
224
225		def remove_tag_txn(txn, next_id):
226		sql = (
227		"DELETE FROM room_tags "
228		" WHERE user_id = ? AND room_id = ? AND tag = ?"
229		)
230		txn.execute(sql, (user_id, room_id, tag))
231		self._update_revision_txn(txn, user_id, room_id, next_id)
232
233		with self._account_data_id_gen.get_next() as next_id:
234		yield self.db.runInteraction("remove_tag", remove_tag_txn, next_id)
235
236		self.get_tags_for_user.invalidate((user_id,))
237
238		result = self._account_data_id_gen.get_current_token()
239		return result
240
241		def _update_revision_txn(self, txn, user_id, room_id, next_id):
242		"""Update the latest revision of the tags for the given user and room.
243
244		Args:
245		txn: The database cursor
246		user_id(str): The ID of the user.
247		room_id(str): The ID of the room.
248		next_id(int): The the revision to advance to.
249		"""
250
251		txn.call_after(
252		self._account_data_stream_cache.entity_has_changed, user_id, next_id
253		)
254
255		# Note: This is only here for backwards compat to allow admins to
256		# roll back to a previous Synapse version. Next time we update the
257		# database version we can remove this table.
258		update_max_id_sql = (
259		"UPDATE account_data_max_stream_id"
260		" SET stream_id = ?"
261		" WHERE stream_id < ?"
262		)
263		txn.execute(update_max_id_sql, (next_id, next_id))
264
265		update_sql = (
266		"UPDATE room_tags_revisions"
267		" SET stream_id = ?"
268		" WHERE user_id = ?"
269		" AND room_id = ?"
270		)
271		txn.execute(update_sql, (next_id, user_id, room_id))
272
273		if txn.rowcount == 0:
274		insert_sql = (
275		"INSERT INTO room_tags_revisions (user_id, room_id, stream_id)"
276		" VALUES (?, ?, ?)"
277		)
278		try:
279		txn.execute(insert_sql, (user_id, room_id, next_id))
280		except self.database_engine.module.IntegrityError:
281		# Ignore insertion errors. It doesn't matter if the row wasn't
282		# inserted because if two updates happend concurrently the one
283		# with the higher stream_id will not be reported to a client
284		# unless the previous update has completed. It doesn't matter
285		# which stream_id ends up in the table, as long as it is higher
286		# than the id that the client has.
287		pass

+0

-269

~~synapse/storage/data_stores/main/transactions.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16		from collections import namedtuple
17
18		from canonicaljson import encode_canonical_json
19
20		from twisted.internet import defer
21
22		from synapse.metrics.background_process_metrics import run_as_background_process
23		from synapse.storage._base import SQLBaseStore, db_to_json
24		from synapse.storage.database import Database
25		from synapse.util.caches.expiringcache import ExpiringCache
26
27		db_binary_type = memoryview
28
29		logger = logging.getLogger(__name__)
30
31
32		_TransactionRow = namedtuple(
33		"_TransactionRow",
34		("id", "transaction_id", "destination", "ts", "response_code", "response_json"),
35		)
36
37		_UpdateTransactionRow = namedtuple(
38		"_TransactionRow", ("response_code", "response_json")
39		)
40
41		SENTINEL = object()
42
43
44		class TransactionStore(SQLBaseStore):
45		"""A collection of queries for handling PDUs.
46		"""
47
48		def __init__(self, database: Database, db_conn, hs):
49		super(TransactionStore, self).__init__(database, db_conn, hs)
50
51		self._clock.looping_call(self._start_cleanup_transactions, 30 * 60 * 1000)
52
53		self._destination_retry_cache = ExpiringCache(
54		cache_name="get_destination_retry_timings",
55		clock=self._clock,
56		expiry_ms=5 * 60 * 1000,
57		)
58
59		def get_received_txn_response(self, transaction_id, origin):
60		"""For an incoming transaction from a given origin, check if we have
61		already responded to it. If so, return the response code and response
62		body (as a dict).
63
64		Args:
65		transaction_id (str)
66		origin(str)
67
68		Returns:
69		tuple: None if we have not previously responded to
70		this transaction or a 2-tuple of (int, dict)
71		"""
72
73		return self.db.runInteraction(
74		"get_received_txn_response",
75		self._get_received_txn_response,
76		transaction_id,
77		origin,
78		)
79
80		def _get_received_txn_response(self, txn, transaction_id, origin):
81		result = self.db.simple_select_one_txn(
82		txn,
83		table="received_transactions",
84		keyvalues={"transaction_id": transaction_id, "origin": origin},
85		retcols=(
86		"transaction_id",
87		"origin",
88		"ts",
89		"response_code",
90		"response_json",
91		"has_been_referenced",
92		),
93		allow_none=True,
94		)
95
96		if result and result["response_code"]:
97		return result["response_code"], db_to_json(result["response_json"])
98
99		else:
100		return None
101
102		def set_received_txn_response(self, transaction_id, origin, code, response_dict):
103		"""Persist the response we returened for an incoming transaction, and
104		should return for subsequent transactions with the same transaction_id
105		and origin.
106
107		Args:
108		txn
109		transaction_id (str)
110		origin (str)
111		code (int)
112		response_json (str)
113		"""
114
115		return self.db.simple_insert(
116		table="received_transactions",
117		values={
118		"transaction_id": transaction_id,
119		"origin": origin,
120		"response_code": code,
121		"response_json": db_binary_type(encode_canonical_json(response_dict)),
122		"ts": self._clock.time_msec(),
123		},
124		or_ignore=True,
125		desc="set_received_txn_response",
126		)
127
128		@defer.inlineCallbacks
129		def get_destination_retry_timings(self, destination):
130		"""Gets the current retry timings (if any) for a given destination.
131
132		Args:
133		destination (str)
134
135		Returns:
136		None if not retrying
137		Otherwise a dict for the retry scheme
138		"""
139
140		result = self._destination_retry_cache.get(destination, SENTINEL)
141		if result is not SENTINEL:
142		return result
143
144		result = yield self.db.runInteraction(
145		"get_destination_retry_timings",
146		self._get_destination_retry_timings,
147		destination,
148		)
149
150		# We don't hugely care about race conditions between getting and
151		# invalidating the cache, since we time out fairly quickly anyway.
152		self._destination_retry_cache[destination] = result
153		return result
154
155		def _get_destination_retry_timings(self, txn, destination):
156		result = self.db.simple_select_one_txn(
157		txn,
158		table="destinations",
159		keyvalues={"destination": destination},
160		retcols=("destination", "failure_ts", "retry_last_ts", "retry_interval"),
161		allow_none=True,
162		)
163
164		if result and result["retry_last_ts"] > 0:
165		return result
166		else:
167		return None
168
169		def set_destination_retry_timings(
170		self, destination, failure_ts, retry_last_ts, retry_interval
171		):
172		"""Sets the current retry timings for a given destination.
173		Both timings should be zero if retrying is no longer occuring.
174
175		Args:
176		destination (str)
177		failure_ts (int\|None) - when the server started failing (ms since epoch)
178		retry_last_ts (int) - time of last retry attempt in unix epoch ms
179		retry_interval (int) - how long until next retry in ms
180		"""
181
182		self._destination_retry_cache.pop(destination, None)
183		return self.db.runInteraction(
184		"set_destination_retry_timings",
185		self._set_destination_retry_timings,
186		destination,
187		failure_ts,
188		retry_last_ts,
189		retry_interval,
190		)
191
192		def _set_destination_retry_timings(
193		self, txn, destination, failure_ts, retry_last_ts, retry_interval
194		):
195
196		if self.database_engine.can_native_upsert:
197		# Upsert retry time interval if retry_interval is zero (i.e. we're
198		# resetting it) or greater than the existing retry interval.
199
200		sql = """
201		INSERT INTO destinations (
202		destination, failure_ts, retry_last_ts, retry_interval
203		)
204		VALUES (?, ?, ?, ?)
205		ON CONFLICT (destination) DO UPDATE SET
206		failure_ts = EXCLUDED.failure_ts,
207		retry_last_ts = EXCLUDED.retry_last_ts,
208		retry_interval = EXCLUDED.retry_interval
209		WHERE
210		EXCLUDED.retry_interval = 0
211		OR destinations.retry_interval < EXCLUDED.retry_interval
212		"""
213
214		txn.execute(sql, (destination, failure_ts, retry_last_ts, retry_interval))
215
216		return
217
218		self.database_engine.lock_table(txn, "destinations")
219
220		# We need to be careful here as the data may have changed from under us
221		# due to a worker setting the timings.
222
223		prev_row = self.db.simple_select_one_txn(
224		txn,
225		table="destinations",
226		keyvalues={"destination": destination},
227		retcols=("failure_ts", "retry_last_ts", "retry_interval"),
228		allow_none=True,
229		)
230
231		if not prev_row:
232		self.db.simple_insert_txn(
233		txn,
234		table="destinations",
235		values={
236		"destination": destination,
237		"failure_ts": failure_ts,
238		"retry_last_ts": retry_last_ts,
239		"retry_interval": retry_interval,
240		},
241		)
242		elif retry_interval == 0 or prev_row["retry_interval"] < retry_interval:
243		self.db.simple_update_one_txn(
244		txn,
245		"destinations",
246		keyvalues={"destination": destination},
247		updatevalues={
248		"failure_ts": failure_ts,
249		"retry_last_ts": retry_last_ts,
250		"retry_interval": retry_interval,
251		},
252		)
253
254		def _start_cleanup_transactions(self):
255		return run_as_background_process(
256		"cleanup_transactions", self._cleanup_transactions
257		)
258
259		def _cleanup_transactions(self):
260		now = self._clock.time_msec()
261		month_ago = now - 30 * 24 * 60 * 60 * 1000
262
263		def _cleanup_transactions_txn(txn):
264		txn.execute("DELETE FROM received_transactions WHERE ts < ?", (month_ago,))
265
266		return self.db.runInteraction(
267		"_cleanup_transactions", _cleanup_transactions_txn
268		)

+0

-300

~~synapse/storage/data_stores/main/ui_auth.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2020 Matrix.org Foundation C.I.C.
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14		from typing import Any, Dict, Optional, Union
15
16		import attr
17		from canonicaljson import json
18
19		from synapse.api.errors import StoreError
20		from synapse.storage._base import SQLBaseStore, db_to_json
21		from synapse.types import JsonDict
22		from synapse.util import stringutils as stringutils
23
24
25		@attr.s
26		class UIAuthSessionData:
27		session_id = attr.ib(type=str)
28		# The dictionary from the client root level, not the 'auth' key.
29		clientdict = attr.ib(type=JsonDict)
30		# The URI and method the session was intiatied with. These are checked at
31		# each stage of the authentication to ensure that the asked for operation
32		# has not changed.
33		uri = attr.ib(type=str)
34		method = attr.ib(type=str)
35		# A string description of the operation that the current authentication is
36		# authorising.
37		description = attr.ib(type=str)
38
39
40		class UIAuthWorkerStore(SQLBaseStore):
41		"""
42		Manage user interactive authentication sessions.
43		"""
44
45		async def create_ui_auth_session(
46		self, clientdict: JsonDict, uri: str, method: str, description: str,
47		) -> UIAuthSessionData:
48		"""
49		Creates a new user interactive authentication session.
50
51		The session can be used to track the stages necessary to authenticate a
52		user across multiple HTTP requests.
53
54		Args:
55		clientdict:
56		The dictionary from the client root level, not the 'auth' key.
57		uri:
58		The URI this session was initiated with, this is checked at each
59		stage of the authentication to ensure that the asked for
60		operation has not changed.
61		method:
62		The method this session was initiated with, this is checked at each
63		stage of the authentication to ensure that the asked for
64		operation has not changed.
65		description:
66		A string description of the operation that the current
67		authentication is authorising.
68		Returns:
69		The newly created session.
70		Raises:
71		StoreError if a unique session ID cannot be generated.
72		"""
73		# The clientdict gets stored as JSON.
74		clientdict_json = json.dumps(clientdict)
75
76		# autogen a session ID and try to create it. We may clash, so just
77		# try a few times till one goes through, giving up eventually.
78		attempts = 0
79		while attempts < 5:
80		session_id = stringutils.random_string(24)
81
82		try:
83		await self.db.simple_insert(
84		table="ui_auth_sessions",
85		values={
86		"session_id": session_id,
87		"clientdict": clientdict_json,
88		"uri": uri,
89		"method": method,
90		"description": description,
91		"serverdict": "{}",
92		"creation_time": self.hs.get_clock().time_msec(),
93		},
94		desc="create_ui_auth_session",
95		)
96		return UIAuthSessionData(
97		session_id, clientdict, uri, method, description
98		)
99		except self.db.engine.module.IntegrityError:
100		attempts += 1
101		raise StoreError(500, "Couldn't generate a session ID.")
102
103		async def get_ui_auth_session(self, session_id: str) -> UIAuthSessionData:
104		"""Retrieve a UI auth session.
105
106		Args:
107		session_id: The ID of the session.
108		Returns:
109		A dict containing the device information.
110		Raises:
111		StoreError if the session is not found.
112		"""
113		result = await self.db.simple_select_one(
114		table="ui_auth_sessions",
115		keyvalues={"session_id": session_id},
116		retcols=("clientdict", "uri", "method", "description"),
117		desc="get_ui_auth_session",
118		)
119
120		result["clientdict"] = db_to_json(result["clientdict"])
121
122		return UIAuthSessionData(session_id, **result)
123
124		async def mark_ui_auth_stage_complete(
125		self, session_id: str, stage_type: str, result: Union[str, bool, JsonDict],
126		):
127		"""
128		Mark a session stage as completed.
129
130		Args:
131		session_id: The ID of the corresponding session.
132		stage_type: The completed stage type.
133		result: The result of the stage verification.
134		Raises:
135		StoreError if the session cannot be found.
136		"""
137		# Add (or update) the results of the current stage to the database.
138		#
139		# Note that we need to allow for the same stage to complete multiple
140		# times here so that registration is idempotent.
141		try:
142		await self.db.simple_upsert(
143		table="ui_auth_sessions_credentials",
144		keyvalues={"session_id": session_id, "stage_type": stage_type},
145		values={"result": json.dumps(result)},
146		desc="mark_ui_auth_stage_complete",
147		)
148		except self.db.engine.module.IntegrityError:
149		raise StoreError(400, "Unknown session ID: %s" % (session_id,))
150
151		async def get_completed_ui_auth_stages(
152		self, session_id: str
153		) -> Dict[str, Union[str, bool, JsonDict]]:
154		"""
155		Retrieve the completed stages of a UI authentication session.
156
157		Args:
158		session_id: The ID of the session.
159		Returns:
160		The completed stages mapped to the result of the verification of
161		that auth-type.
162		"""
163		results = {}
164		for row in await self.db.simple_select_list(
165		table="ui_auth_sessions_credentials",
166		keyvalues={"session_id": session_id},
167		retcols=("stage_type", "result"),
168		desc="get_completed_ui_auth_stages",
169		):
170		results[row["stage_type"]] = db_to_json(row["result"])
171
172		return results
173
174		async def set_ui_auth_clientdict(
175		self, session_id: str, clientdict: JsonDict
176		) -> None:
177		"""
178		Store an updated clientdict for a given session ID.
179
180		Args:
181		session_id: The ID of this session as returned from check_auth
182		clientdict:
183		The dictionary from the client root level, not the 'auth' key.
184		"""
185		# The clientdict gets stored as JSON.
186		clientdict_json = json.dumps(clientdict)
187
188		await self.db.simple_update_one(
189		table="ui_auth_sessions",
190		keyvalues={"session_id": session_id},
191		updatevalues={"clientdict": clientdict_json},
192		desc="set_ui_auth_client_dict",
193		)
194
195		async def set_ui_auth_session_data(self, session_id: str, key: str, value: Any):
196		"""
197		Store a key-value pair into the sessions data associated with this
198		request. This data is stored server-side and cannot be modified by
199		the client.
200
201		Args:
202		session_id: The ID of this session as returned from check_auth
203		key: The key to store the data under
204		value: The data to store
205		Raises:
206		StoreError if the session cannot be found.
207		"""
208		await self.db.runInteraction(
209		"set_ui_auth_session_data",
210		self._set_ui_auth_session_data_txn,
211		session_id,
212		key,
213		value,
214		)
215
216		def _set_ui_auth_session_data_txn(self, txn, session_id: str, key: str, value: Any):
217		# Get the current value.
218		result = self.db.simple_select_one_txn(
219		txn,
220		table="ui_auth_sessions",
221		keyvalues={"session_id": session_id},
222		retcols=("serverdict",),
223		)
224
225		# Update it and add it back to the database.
226		serverdict = db_to_json(result["serverdict"])
227		serverdict[key] = value
228
229		self.db.simple_update_one_txn(
230		txn,
231		table="ui_auth_sessions",
232		keyvalues={"session_id": session_id},
233		updatevalues={"serverdict": json.dumps(serverdict)},
234		)
235
236		async def get_ui_auth_session_data(
237		self, session_id: str, key: str, default: Optional[Any] = None
238		) -> Any:
239		"""
240		Retrieve data stored with set_session_data
241
242		Args:
243		session_id: The ID of this session as returned from check_auth
244		key: The key to store the data under
245		default: Value to return if the key has not been set
246		Raises:
247		StoreError if the session cannot be found.
248		"""
249		result = await self.db.simple_select_one(
250		table="ui_auth_sessions",
251		keyvalues={"session_id": session_id},
252		retcols=("serverdict",),
253		desc="get_ui_auth_session_data",
254		)
255
256		serverdict = db_to_json(result["serverdict"])
257
258		return serverdict.get(key, default)
259
260
261		class UIAuthStore(UIAuthWorkerStore):
262		def delete_old_ui_auth_sessions(self, expiration_time: int):
263		"""
264		Remove sessions which were last used earlier than the expiration time.
265
266		Args:
267		expiration_time: The latest time that is still considered valid.
268		This is an epoch time in milliseconds.
269
270		"""
271		return self.db.runInteraction(
272		"delete_old_ui_auth_sessions",
273		self._delete_old_ui_auth_sessions_txn,
274		expiration_time,
275		)
276
277		def _delete_old_ui_auth_sessions_txn(self, txn, expiration_time: int):
278		# Get the expired sessions.
279		sql = "SELECT session_id FROM ui_auth_sessions WHERE creation_time <= ?"
280		txn.execute(sql, [expiration_time])
281		session_ids = [r[0] for r in txn.fetchall()]
282
283		# Delete the corresponding completed credentials.
284		self.db.simple_delete_many_txn(
285		txn,
286		table="ui_auth_sessions_credentials",
287		column="session_id",
288		iterable=session_ids,
289		keyvalues={},
290		)
291
292		# Finally, delete the sessions.
293		self.db.simple_delete_many_txn(
294		txn,
295		table="ui_auth_sessions",
296		column="session_id",
297		iterable=session_ids,
298		keyvalues={},
299		)

+0

-837

~~synapse/storage/data_stores/main/user_directory.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2017 Vector Creations Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16		import re
17
18		from twisted.internet import defer
19
20		from synapse.api.constants import EventTypes, JoinRules
21		from synapse.storage.data_stores.main.state import StateFilter
22		from synapse.storage.data_stores.main.state_deltas import StateDeltasStore
23		from synapse.storage.database import Database
24		from synapse.storage.engines import PostgresEngine, Sqlite3Engine
25		from synapse.types import get_domain_from_id, get_localpart_from_id
26		from synapse.util.caches.descriptors import cached
27
28		logger = logging.getLogger(__name__)
29
30
31		TEMP_TABLE = "_temp_populate_user_directory"
32
33
34		class UserDirectoryBackgroundUpdateStore(StateDeltasStore):
35
36		# How many records do we calculate before sending it to
37		# add_users_who_share_private_rooms?
38		SHARE_PRIVATE_WORKING_SET = 500
39
40		def __init__(self, database: Database, db_conn, hs):
41		super(UserDirectoryBackgroundUpdateStore, self).__init__(database, db_conn, hs)
42
43		self.server_name = hs.hostname
44
45		self.db.updates.register_background_update_handler(
46		"populate_user_directory_createtables",
47		self._populate_user_directory_createtables,
48		)
49		self.db.updates.register_background_update_handler(
50		"populate_user_directory_process_rooms",
51		self._populate_user_directory_process_rooms,
52		)
53		self.db.updates.register_background_update_handler(
54		"populate_user_directory_process_users",
55		self._populate_user_directory_process_users,
56		)
57		self.db.updates.register_background_update_handler(
58		"populate_user_directory_cleanup", self._populate_user_directory_cleanup
59		)
60
61		@defer.inlineCallbacks
62		def _populate_user_directory_createtables(self, progress, batch_size):
63
64		# Get all the rooms that we want to process.
65		def _make_staging_area(txn):
66		sql = (
67		"CREATE TABLE IF NOT EXISTS "
68		+ TEMP_TABLE
69		+ "_rooms(room_id TEXT NOT NULL, events BIGINT NOT NULL)"
70		)
71		txn.execute(sql)
72
73		sql = (
74		"CREATE TABLE IF NOT EXISTS "
75		+ TEMP_TABLE
76		+ "_position(position TEXT NOT NULL)"
77		)
78		txn.execute(sql)
79
80		# Get rooms we want to process from the database
81		sql = """
82		SELECT room_id, count(*) FROM current_state_events
83		GROUP BY room_id
84		"""
85		txn.execute(sql)
86		rooms = [{"room_id": x[0], "events": x[1]} for x in txn.fetchall()]
87		self.db.simple_insert_many_txn(txn, TEMP_TABLE + "_rooms", rooms)
88		del rooms
89
90		# If search all users is on, get all the users we want to add.
91		if self.hs.config.user_directory_search_all_users:
92		sql = (
93		"CREATE TABLE IF NOT EXISTS "
94		+ TEMP_TABLE
95		+ "_users(user_id TEXT NOT NULL)"
96		)
97		txn.execute(sql)
98
99		txn.execute("SELECT name FROM users")
100		users = [{"user_id": x[0]} for x in txn.fetchall()]
101
102		self.db.simple_insert_many_txn(txn, TEMP_TABLE + "_users", users)
103
104		new_pos = yield self.get_max_stream_id_in_current_state_deltas()
105		yield self.db.runInteraction(
106		"populate_user_directory_temp_build", _make_staging_area
107		)
108		yield self.db.simple_insert(TEMP_TABLE + "_position", {"position": new_pos})
109
110		yield self.db.updates._end_background_update(
111		"populate_user_directory_createtables"
112		)
113		return 1
114
115		@defer.inlineCallbacks
116		def _populate_user_directory_cleanup(self, progress, batch_size):
117		"""
118		Update the user directory stream position, then clean up the old tables.
119		"""
120		position = yield self.db.simple_select_one_onecol(
121		TEMP_TABLE + "_position", None, "position"
122		)
123		yield self.update_user_directory_stream_pos(position)
124
125		def _delete_staging_area(txn):
126		txn.execute("DROP TABLE IF EXISTS " + TEMP_TABLE + "_rooms")
127		txn.execute("DROP TABLE IF EXISTS " + TEMP_TABLE + "_users")
128		txn.execute("DROP TABLE IF EXISTS " + TEMP_TABLE + "_position")
129
130		yield self.db.runInteraction(
131		"populate_user_directory_cleanup", _delete_staging_area
132		)
133
134		yield self.db.updates._end_background_update("populate_user_directory_cleanup")
135		return 1
136
137		@defer.inlineCallbacks
138		def _populate_user_directory_process_rooms(self, progress, batch_size):
139		"""
140		Args:
141		progress (dict)
142		batch_size (int): Maximum number of state events to process
143		per cycle.
144		"""
145		state = self.hs.get_state_handler()
146
147		# If we don't have progress filed, delete everything.
148		if not progress:
149		yield self.delete_all_from_user_dir()
150
151		def _get_next_batch(txn):
152		# Only fetch 250 rooms, so we don't fetch too many at once, even
153		# if those 250 rooms have less than batch_size state events.
154		sql = """
155		SELECT room_id, events FROM %s
156		ORDER BY events DESC
157		LIMIT 250
158		""" % (
159		TEMP_TABLE + "_rooms",
160		)
161		txn.execute(sql)
162		rooms_to_work_on = txn.fetchall()
163
164		if not rooms_to_work_on:
165		return None
166
167		# Get how many are left to process, so we can give status on how
168		# far we are in processing
169		txn.execute("SELECT COUNT(*) FROM " + TEMP_TABLE + "_rooms")
170		progress["remaining"] = txn.fetchone()[0]
171
172		return rooms_to_work_on
173
174		rooms_to_work_on = yield self.db.runInteraction(
175		"populate_user_directory_temp_read", _get_next_batch
176		)
177
178		# No more rooms -- complete the transaction.
179		if not rooms_to_work_on:
180		yield self.db.updates._end_background_update(
181		"populate_user_directory_process_rooms"
182		)
183		return 1
184
185		logger.debug(
186		"Processing the next %d rooms of %d remaining"
187		% (len(rooms_to_work_on), progress["remaining"])
188		)
189
190		processed_event_count = 0
191
192		for room_id, event_count in rooms_to_work_on:
193		is_in_room = yield self.is_host_joined(room_id, self.server_name)
194
195		if is_in_room:
196		is_public = yield self.is_room_world_readable_or_publicly_joinable(
197		room_id
198		)
199
200		users_with_profile = yield defer.ensureDeferred(
201		state.get_current_users_in_room(room_id)
202		)
203		user_ids = set(users_with_profile)
204
205		# Update each user in the user directory.
206		for user_id, profile in users_with_profile.items():
207		yield self.update_profile_in_user_dir(
208		user_id, profile.display_name, profile.avatar_url
209		)
210
211		to_insert = set()
212
213		if is_public:
214		for user_id in user_ids:
215		if self.get_if_app_services_interested_in_user(user_id):
216		continue
217
218		to_insert.add(user_id)
219
220		if to_insert:
221		yield self.add_users_in_public_rooms(room_id, to_insert)
222		to_insert.clear()
223		else:
224		for user_id in user_ids:
225		if not self.hs.is_mine_id(user_id):
226		continue
227
228		if self.get_if_app_services_interested_in_user(user_id):
229		continue
230
231		for other_user_id in user_ids:
232		if user_id == other_user_id:
233		continue
234
235		user_set = (user_id, other_user_id)
236		to_insert.add(user_set)
237
238		# If it gets too big, stop and write to the database
239		# to prevent storing too much in RAM.
240		if len(to_insert) >= self.SHARE_PRIVATE_WORKING_SET:
241		yield self.add_users_who_share_private_room(
242		room_id, to_insert
243		)
244		to_insert.clear()
245
246		if to_insert:
247		yield self.add_users_who_share_private_room(room_id, to_insert)
248		to_insert.clear()
249
250		# We've finished a room. Delete it from the table.
251		yield self.db.simple_delete_one(TEMP_TABLE + "_rooms", {"room_id": room_id})
252		# Update the remaining counter.
253		progress["remaining"] -= 1
254		yield self.db.runInteraction(
255		"populate_user_directory",
256		self.db.updates._background_update_progress_txn,
257		"populate_user_directory_process_rooms",
258		progress,
259		)
260
261		processed_event_count += event_count
262
263		if processed_event_count > batch_size:
264		# Don't process any more rooms, we've hit our batch size.
265		return processed_event_count
266
267		return processed_event_count
268
269		@defer.inlineCallbacks
270		def _populate_user_directory_process_users(self, progress, batch_size):
271		"""
272		If search_all_users is enabled, add all of the users to the user directory.
273		"""
274		if not self.hs.config.user_directory_search_all_users:
275		yield self.db.updates._end_background_update(
276		"populate_user_directory_process_users"
277		)
278		return 1
279
280		def _get_next_batch(txn):
281		sql = "SELECT user_id FROM %s LIMIT %s" % (
282		TEMP_TABLE + "_users",
283		str(batch_size),
284		)
285		txn.execute(sql)
286		users_to_work_on = txn.fetchall()
287
288		if not users_to_work_on:
289		return None
290
291		users_to_work_on = [x[0] for x in users_to_work_on]
292
293		# Get how many are left to process, so we can give status on how
294		# far we are in processing
295		sql = "SELECT COUNT(*) FROM " + TEMP_TABLE + "_users"
296		txn.execute(sql)
297		progress["remaining"] = txn.fetchone()[0]
298
299		return users_to_work_on
300
301		users_to_work_on = yield self.db.runInteraction(
302		"populate_user_directory_temp_read", _get_next_batch
303		)
304
305		# No more users -- complete the transaction.
306		if not users_to_work_on:
307		yield self.db.updates._end_background_update(
308		"populate_user_directory_process_users"
309		)
310		return 1
311
312		logger.debug(
313		"Processing the next %d users of %d remaining"
314		% (len(users_to_work_on), progress["remaining"])
315		)
316
317		for user_id in users_to_work_on:
318		profile = yield self.get_profileinfo(get_localpart_from_id(user_id))
319		yield self.update_profile_in_user_dir(
320		user_id, profile.display_name, profile.avatar_url
321		)
322
323		# We've finished processing a user. Delete it from the table.
324		yield self.db.simple_delete_one(TEMP_TABLE + "_users", {"user_id": user_id})
325		# Update the remaining counter.
326		progress["remaining"] -= 1
327		yield self.db.runInteraction(
328		"populate_user_directory",
329		self.db.updates._background_update_progress_txn,
330		"populate_user_directory_process_users",
331		progress,
332		)
333
334		return len(users_to_work_on)
335
336		@defer.inlineCallbacks
337		def is_room_world_readable_or_publicly_joinable(self, room_id):
338		"""Check if the room is either world_readable or publically joinable
339		"""
340
341		# Create a state filter that only queries join and history state event
342		types_to_filter = (
343		(EventTypes.JoinRules, ""),
344		(EventTypes.RoomHistoryVisibility, ""),
345		)
346
347		current_state_ids = yield self.get_filtered_current_state_ids(
348		room_id, StateFilter.from_types(types_to_filter)
349		)
350
351		join_rules_id = current_state_ids.get((EventTypes.JoinRules, ""))
352		if join_rules_id:
353		join_rule_ev = yield self.get_event(join_rules_id, allow_none=True)
354		if join_rule_ev:
355		if join_rule_ev.content.get("join_rule") == JoinRules.PUBLIC:
356		return True
357
358		hist_vis_id = current_state_ids.get((EventTypes.RoomHistoryVisibility, ""))
359		if hist_vis_id:
360		hist_vis_ev = yield self.get_event(hist_vis_id, allow_none=True)
361		if hist_vis_ev:
362		if hist_vis_ev.content.get("history_visibility") == "world_readable":
363		return True
364
365		return False
366
367		def update_profile_in_user_dir(self, user_id, display_name, avatar_url):
368		"""
369		Update or add a user's profile in the user directory.
370		"""
371
372		def _update_profile_in_user_dir_txn(txn):
373		new_entry = self.db.simple_upsert_txn(
374		txn,
375		table="user_directory",
376		keyvalues={"user_id": user_id},
377		values={"display_name": display_name, "avatar_url": avatar_url},
378		lock=False, # We're only inserter
379		)
380
381		if isinstance(self.database_engine, PostgresEngine):
382		# We weight the localpart most highly, then display name and finally
383		# server name
384		if self.database_engine.can_native_upsert:
385		sql = """
386		INSERT INTO user_directory_search(user_id, vector)
387		VALUES (?,
388		setweight(to_tsvector('english', ?), 'A')
389		\|\| setweight(to_tsvector('english', ?), 'D')
390		\|\| setweight(to_tsvector('english', COALESCE(?, '')), 'B')
391		) ON CONFLICT (user_id) DO UPDATE SET vector=EXCLUDED.vector
392		"""
393		txn.execute(
394		sql,
395		(
396		user_id,
397		get_localpart_from_id(user_id),
398		get_domain_from_id(user_id),
399		display_name,
400		),
401		)
402		else:
403		# TODO: Remove this code after we've bumped the minimum version
404		# of postgres to always support upserts, so we can get rid of
405		# `new_entry` usage
406		if new_entry is True:
407		sql = """
408		INSERT INTO user_directory_search(user_id, vector)
409		VALUES (?,
410		setweight(to_tsvector('english', ?), 'A')
411		\|\| setweight(to_tsvector('english', ?), 'D')
412		\|\| setweight(to_tsvector('english', COALESCE(?, '')), 'B')
413		)
414		"""
415		txn.execute(
416		sql,
417		(
418		user_id,
419		get_localpart_from_id(user_id),
420		get_domain_from_id(user_id),
421		display_name,
422		),
423		)
424		elif new_entry is False:
425		sql = """
426		UPDATE user_directory_search
427		SET vector = setweight(to_tsvector('english', ?), 'A')
428		\|\| setweight(to_tsvector('english', ?), 'D')
429		\|\| setweight(to_tsvector('english', COALESCE(?, '')), 'B')
430		WHERE user_id = ?
431		"""
432		txn.execute(
433		sql,
434		(
435		get_localpart_from_id(user_id),
436		get_domain_from_id(user_id),
437		display_name,
438		user_id,
439		),
440		)
441		else:
442		raise RuntimeError(
443		"upsert returned None when 'can_native_upsert' is False"
444		)
445		elif isinstance(self.database_engine, Sqlite3Engine):
446		value = "%s %s" % (user_id, display_name) if display_name else user_id
447		self.db.simple_upsert_txn(
448		txn,
449		table="user_directory_search",
450		keyvalues={"user_id": user_id},
451		values={"value": value},
452		lock=False, # We're only inserter
453		)
454		else:
455		# This should be unreachable.
456		raise Exception("Unrecognized database engine")
457
458		txn.call_after(self.get_user_in_directory.invalidate, (user_id,))
459
460		return self.db.runInteraction(
461		"update_profile_in_user_dir", _update_profile_in_user_dir_txn
462		)
463
464		def add_users_who_share_private_room(self, room_id, user_id_tuples):
465		"""Insert entries into the users_who_share_private_rooms table. The first
466		user should be a local user.
467
468		Args:
469		room_id (str)
470		user_id_tuples([(str, str)]): iterable of 2-tuple of user IDs.
471		"""
472
473		def _add_users_who_share_room_txn(txn):
474		self.db.simple_upsert_many_txn(
475		txn,
476		table="users_who_share_private_rooms",
477		key_names=["user_id", "other_user_id", "room_id"],
478		key_values=[
479		(user_id, other_user_id, room_id)
480		for user_id, other_user_id in user_id_tuples
481		],
482		value_names=(),
483		value_values=None,
484		)
485
486		return self.db.runInteraction(
487		"add_users_who_share_room", _add_users_who_share_room_txn
488		)
489
490		def add_users_in_public_rooms(self, room_id, user_ids):
491		"""Insert entries into the users_who_share_private_rooms table. The first
492		user should be a local user.
493
494		Args:
495		room_id (str)
496		user_ids (list[str])
497		"""
498
499		def _add_users_in_public_rooms_txn(txn):
500
501		self.db.simple_upsert_many_txn(
502		txn,
503		table="users_in_public_rooms",
504		key_names=["user_id", "room_id"],
505		key_values=[(user_id, room_id) for user_id in user_ids],
506		value_names=(),
507		value_values=None,
508		)
509
510		return self.db.runInteraction(
511		"add_users_in_public_rooms", _add_users_in_public_rooms_txn
512		)
513
514		def delete_all_from_user_dir(self):
515		"""Delete the entire user directory
516		"""
517
518		def _delete_all_from_user_dir_txn(txn):
519		txn.execute("DELETE FROM user_directory")
520		txn.execute("DELETE FROM user_directory_search")
521		txn.execute("DELETE FROM users_in_public_rooms")
522		txn.execute("DELETE FROM users_who_share_private_rooms")
523		txn.call_after(self.get_user_in_directory.invalidate_all)
524
525		return self.db.runInteraction(
526		"delete_all_from_user_dir", _delete_all_from_user_dir_txn
527		)
528
529		@cached()
530		def get_user_in_directory(self, user_id):
531		return self.db.simple_select_one(
532		table="user_directory",
533		keyvalues={"user_id": user_id},
534		retcols=("display_name", "avatar_url"),
535		allow_none=True,
536		desc="get_user_in_directory",
537		)
538
539		def update_user_directory_stream_pos(self, stream_id):
540		return self.db.simple_update_one(
541		table="user_directory_stream_pos",
542		keyvalues={},
543		updatevalues={"stream_id": stream_id},
544		desc="update_user_directory_stream_pos",
545		)
546
547
548		class UserDirectoryStore(UserDirectoryBackgroundUpdateStore):
549
550		# How many records do we calculate before sending it to
551		# add_users_who_share_private_rooms?
552		SHARE_PRIVATE_WORKING_SET = 500
553
554		def __init__(self, database: Database, db_conn, hs):
555		super(UserDirectoryStore, self).__init__(database, db_conn, hs)
556
557		def remove_from_user_dir(self, user_id):
558		def _remove_from_user_dir_txn(txn):
559		self.db.simple_delete_txn(
560		txn, table="user_directory", keyvalues={"user_id": user_id}
561		)
562		self.db.simple_delete_txn(
563		txn, table="user_directory_search", keyvalues={"user_id": user_id}
564		)
565		self.db.simple_delete_txn(
566		txn, table="users_in_public_rooms", keyvalues={"user_id": user_id}
567		)
568		self.db.simple_delete_txn(
569		txn,
570		table="users_who_share_private_rooms",
571		keyvalues={"user_id": user_id},
572		)
573		self.db.simple_delete_txn(
574		txn,
575		table="users_who_share_private_rooms",
576		keyvalues={"other_user_id": user_id},
577		)
578		txn.call_after(self.get_user_in_directory.invalidate, (user_id,))
579
580		return self.db.runInteraction("remove_from_user_dir", _remove_from_user_dir_txn)
581
582		@defer.inlineCallbacks
583		def get_users_in_dir_due_to_room(self, room_id):
584		"""Get all user_ids that are in the room directory because they're
585		in the given room_id
586		"""
587		user_ids_share_pub = yield self.db.simple_select_onecol(
588		table="users_in_public_rooms",
589		keyvalues={"room_id": room_id},
590		retcol="user_id",
591		desc="get_users_in_dir_due_to_room",
592		)
593
594		user_ids_share_priv = yield self.db.simple_select_onecol(
595		table="users_who_share_private_rooms",
596		keyvalues={"room_id": room_id},
597		retcol="other_user_id",
598		desc="get_users_in_dir_due_to_room",
599		)
600
601		user_ids = set(user_ids_share_pub)
602		user_ids.update(user_ids_share_priv)
603
604		return user_ids
605
606		def remove_user_who_share_room(self, user_id, room_id):
607		"""
608		Deletes entries in the users_who_share_*_rooms table. The first
609		user should be a local user.
610
611		Args:
612		user_id (str)
613		room_id (str)
614		"""
615
616		def _remove_user_who_share_room_txn(txn):
617		self.db.simple_delete_txn(
618		txn,
619		table="users_who_share_private_rooms",
620		keyvalues={"user_id": user_id, "room_id": room_id},
621		)
622		self.db.simple_delete_txn(
623		txn,
624		table="users_who_share_private_rooms",
625		keyvalues={"other_user_id": user_id, "room_id": room_id},
626		)
627		self.db.simple_delete_txn(
628		txn,
629		table="users_in_public_rooms",
630		keyvalues={"user_id": user_id, "room_id": room_id},
631		)
632
633		return self.db.runInteraction(
634		"remove_user_who_share_room", _remove_user_who_share_room_txn
635		)
636
637		@defer.inlineCallbacks
638		def get_user_dir_rooms_user_is_in(self, user_id):
639		"""
640		Returns the rooms that a user is in.
641
642		Args:
643		user_id(str): Must be a local user
644
645		Returns:
646		list: user_id
647		"""
648		rows = yield self.db.simple_select_onecol(
649		table="users_who_share_private_rooms",
650		keyvalues={"user_id": user_id},
651		retcol="room_id",
652		desc="get_rooms_user_is_in",
653		)
654
655		pub_rows = yield self.db.simple_select_onecol(
656		table="users_in_public_rooms",
657		keyvalues={"user_id": user_id},
658		retcol="room_id",
659		desc="get_rooms_user_is_in",
660		)
661
662		users = set(pub_rows)
663		users.update(rows)
664		return list(users)
665
666		@defer.inlineCallbacks
667		def get_rooms_in_common_for_users(self, user_id, other_user_id):
668		"""Given two user_ids find out the list of rooms they share.
669		"""
670		sql = """
671		SELECT room_id FROM (
672		SELECT c.room_id FROM current_state_events AS c
673		INNER JOIN room_memberships AS m USING (event_id)
674		WHERE type = 'm.room.member'
675		AND m.membership = 'join'
676		AND state_key = ?
677		) AS f1 INNER JOIN (
678		SELECT c.room_id FROM current_state_events AS c
679		INNER JOIN room_memberships AS m USING (event_id)
680		WHERE type = 'm.room.member'
681		AND m.membership = 'join'
682		AND state_key = ?
683		) f2 USING (room_id)
684		"""
685
686		rows = yield self.db.execute(
687		"get_rooms_in_common_for_users", None, sql, user_id, other_user_id
688		)
689
690		return [room_id for room_id, in rows]
691
692		def get_user_directory_stream_pos(self):
693		return self.db.simple_select_one_onecol(
694		table="user_directory_stream_pos",
695		keyvalues={},
696		retcol="stream_id",
697		desc="get_user_directory_stream_pos",
698		)
699
700		@defer.inlineCallbacks
701		def search_user_dir(self, user_id, search_term, limit):
702		"""Searches for users in directory
703
704		Returns:
705		dict of the form::
706
707		{
708		"limited": <bool>, # whether there were more results or not
709		"results": [ # Ordered by best match first
710		{
711		"user_id": <user_id>,
712		"display_name": <display_name>,
713		"avatar_url": <avatar_url>
714		}
715		]
716		}
717		"""
718
719		if self.hs.config.user_directory_search_all_users:
720		join_args = (user_id,)
721		where_clause = "user_id != ?"
722		else:
723		join_args = (user_id,)
724		where_clause = """
725		(
726		EXISTS (select 1 from users_in_public_rooms WHERE user_id = t.user_id)
727		OR EXISTS (
728		SELECT 1 FROM users_who_share_private_rooms
729		WHERE user_id = ? AND other_user_id = t.user_id
730		)
731		)
732		"""
733
734		if isinstance(self.database_engine, PostgresEngine):
735		full_query, exact_query, prefix_query = _parse_query_postgres(search_term)
736
737		# We order by rank and then if they have profile info
738		# The ranking algorithm is hand tweaked for "best" results. Broadly
739		# the idea is we give a higher weight to exact matches.
740		# The array of numbers are the weights for the various part of the
741		# search: (domain, _, display name, localpart)
742		sql = """
743		SELECT d.user_id AS user_id, display_name, avatar_url
744		FROM user_directory_search as t
745		INNER JOIN user_directory AS d USING (user_id)
746		WHERE
747		%s
748		AND vector @@ to_tsquery('english', ?)
749		ORDER BY
750		(CASE WHEN d.user_id IS NOT NULL THEN 4.0 ELSE 1.0 END)
751		* (CASE WHEN display_name IS NOT NULL THEN 1.2 ELSE 1.0 END)
752		* (CASE WHEN avatar_url IS NOT NULL THEN 1.2 ELSE 1.0 END)
753		* (
754		3 * ts_rank_cd(
755		'{0.1, 0.1, 0.9, 1.0}',
756		vector,
757		to_tsquery('english', ?),
758		8
759		)
760		+ ts_rank_cd(
761		'{0.1, 0.1, 0.9, 1.0}',
762		vector,
763		to_tsquery('english', ?),
764		8
765		)
766		)
767		DESC,
768		display_name IS NULL,
769		avatar_url IS NULL
770		LIMIT ?
771		""" % (
772		where_clause,
773		)
774		args = join_args + (full_query, exact_query, prefix_query, limit + 1)
775		elif isinstance(self.database_engine, Sqlite3Engine):
776		search_query = _parse_query_sqlite(search_term)
777
778		sql = """
779		SELECT d.user_id AS user_id, display_name, avatar_url
780		FROM user_directory_search as t
781		INNER JOIN user_directory AS d USING (user_id)
782		WHERE
783		%s
784		AND value MATCH ?
785		ORDER BY
786		rank(matchinfo(user_directory_search)) DESC,
787		display_name IS NULL,
788		avatar_url IS NULL
789		LIMIT ?
790		""" % (
791		where_clause,
792		)
793		args = join_args + (search_query, limit + 1)
794		else:
795		# This should be unreachable.
796		raise Exception("Unrecognized database engine")
797
798		results = yield self.db.execute(
799		"search_user_dir", self.db.cursor_to_dict, sql, *args
800		)
801
802		limited = len(results) > limit
803
804		return {"limited": limited, "results": results}
805
806
807		def _parse_query_sqlite(search_term):
808		"""Takes a plain unicode string from the user and converts it into a form
809		that can be passed to database.
810		We use this so that we can add prefix matching, which isn't something
811		that is supported by default.
812
813		We specifically add both a prefix and non prefix matching term so that
814		exact matches get ranked higher.
815		"""
816
817		# Pull out the individual words, discarding any non-word characters.
818		results = re.findall(r"([\w\-]+)", search_term, re.UNICODE)
819		return " & ".join("(%s* OR %s)" % (result, result) for result in results)
820
821
822		def _parse_query_postgres(search_term):
823		"""Takes a plain unicode string from the user and converts it into a form
824		that can be passed to database.
825		We use this so that we can add prefix matching, which isn't something
826		that is supported by default.
827		"""
828
829		# Pull out the individual words, discarding any non-word characters.
830		results = re.findall(r"([\w\-]+)", search_term, re.UNICODE)
831
832		both = " & ".join("(%s:* \| %s)" % (result, result) for result in results)
833		exact = " & ".join("%s" % (result,) for result in results)
834		prefix = " & ".join("%s:*" % (result,) for result in results)
835
836		return both, exact, prefix

+0

-113

~~synapse/storage/data_stores/main/user_erasure_store.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2018 New Vector Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import operator
16
17		from synapse.storage._base import SQLBaseStore
18		from synapse.util.caches.descriptors import cached, cachedList
19
20
21		class UserErasureWorkerStore(SQLBaseStore):
22		@cached()
23		def is_user_erased(self, user_id):
24		"""
25		Check if the given user id has requested erasure
26
27		Args:
28		user_id (str): full user id to check
29
30		Returns:
31		Deferred[bool]: True if the user has requested erasure
32		"""
33		return self.db.simple_select_onecol(
34		table="erased_users",
35		keyvalues={"user_id": user_id},
36		retcol="1",
37		desc="is_user_erased",
38		).addCallback(operator.truth)
39
40		@cachedList(
41		cached_method_name="is_user_erased", list_name="user_ids", inlineCallbacks=True
42		)
43		def are_users_erased(self, user_ids):
44		"""
45		Checks which users in a list have requested erasure
46
47		Args:
48		user_ids (iterable[str]): full user id to check
49
50		Returns:
51		Deferred[dict[str, bool]]:
52		for each user, whether the user has requested erasure.
53		"""
54		# this serves the dual purpose of (a) making sure we can do len and
55		# iterate it multiple times, and (b) avoiding duplicates.
56		user_ids = tuple(set(user_ids))
57
58		rows = yield self.db.simple_select_many_batch(
59		table="erased_users",
60		column="user_id",
61		iterable=user_ids,
62		retcols=("user_id",),
63		desc="are_users_erased",
64		)
65		erased_users = {row["user_id"] for row in rows}
66
67		res = {u: u in erased_users for u in user_ids}
68		return res
69
70
71		class UserErasureStore(UserErasureWorkerStore):
72		def mark_user_erased(self, user_id: str) -> None:
73		"""Indicate that user_id wishes their message history to be erased.
74
75		Args:
76		user_id: full user_id to be erased
77		"""
78
79		def f(txn):
80		# first check if they are already in the list
81		txn.execute("SELECT 1 FROM erased_users WHERE user_id = ?", (user_id,))
82		if txn.fetchone():
83		return
84
85		# they are not already there: do the insert.
86		txn.execute("INSERT INTO erased_users (user_id) VALUES (?)", (user_id,))
87
88		self._invalidate_cache_and_stream(txn, self.is_user_erased, (user_id,))
89
90		return self.db.runInteraction("mark_user_erased", f)
91
92		def mark_user_not_erased(self, user_id: str) -> None:
93		"""Indicate that user_id is no longer erased.
94
95		Args:
96		user_id: full user_id to be un-erased
97		"""
98
99		def f(txn):
100		# first check if they are already in the list
101		txn.execute("SELECT 1 FROM erased_users WHERE user_id = ?", (user_id,))
102		if not txn.fetchone():
103		return
104
105		# They are there, delete them.
106		self.simple_delete_one_txn(
107		txn, "erased_users", keyvalues={"user_id": user_id}
108		)
109
110		self._invalidate_cache_and_stream(txn, self.is_user_erased, (user_id,))
111
112		return self.db.runInteraction("mark_user_not_erased", f)

+0

-16

~~synapse/storage/data_stores/state/__init__.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2019 The Matrix.org Foundation C.I.C.
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		from synapse.storage.data_stores.state.store import StateGroupDataStore # noqa: F401

+0

-372

~~synapse/storage/data_stores/state/bg_updates.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16
17		from twisted.internet import defer
18
19		from synapse.storage._base import SQLBaseStore
20		from synapse.storage.database import Database
21		from synapse.storage.engines import PostgresEngine
22		from synapse.storage.state import StateFilter
23
24		logger = logging.getLogger(__name__)
25
26
27		MAX_STATE_DELTA_HOPS = 100
28
29
30		class StateGroupBackgroundUpdateStore(SQLBaseStore):
31		"""Defines functions related to state groups needed to run the state backgroud
32		updates.
33		"""
34
35		def _count_state_group_hops_txn(self, txn, state_group):
36		"""Given a state group, count how many hops there are in the tree.
37
38		This is used to ensure the delta chains don't get too long.
39		"""
40		if isinstance(self.database_engine, PostgresEngine):
41		sql = """
42		WITH RECURSIVE state(state_group) AS (
43		VALUES(?::bigint)
44		UNION ALL
45		SELECT prev_state_group FROM state_group_edges e, state s
46		WHERE s.state_group = e.state_group
47		)
48		SELECT count(*) FROM state;
49		"""
50
51		txn.execute(sql, (state_group,))
52		row = txn.fetchone()
53		if row and row[0]:
54		return row[0]
55		else:
56		return 0
57		else:
58		# We don't use WITH RECURSIVE on sqlite3 as there are distributions
59		# that ship with an sqlite3 version that doesn't support it (e.g. wheezy)
60		next_group = state_group
61		count = 0
62
63		while next_group:
64		next_group = self.db.simple_select_one_onecol_txn(
65		txn,
66		table="state_group_edges",
67		keyvalues={"state_group": next_group},
68		retcol="prev_state_group",
69		allow_none=True,
70		)
71		if next_group:
72		count += 1
73
74		return count
75
76		def _get_state_groups_from_groups_txn(
77		self, txn, groups, state_filter=StateFilter.all()
78		):
79		results = {group: {} for group in groups}
80
81		where_clause, where_args = state_filter.make_sql_filter_clause()
82
83		# Unless the filter clause is empty, we're going to append it after an
84		# existing where clause
85		if where_clause:
86		where_clause = " AND (%s)" % (where_clause,)
87
88		if isinstance(self.database_engine, PostgresEngine):
89		# Temporarily disable sequential scans in this transaction. This is
90		# a temporary hack until we can add the right indices in
91		txn.execute("SET LOCAL enable_seqscan=off")
92
93		# The below query walks the state_group tree so that the "state"
94		# table includes all state_groups in the tree. It then joins
95		# against `state_groups_state` to fetch the latest state.
96		# It assumes that previous state groups are always numerically
97		# lesser.
98		# The PARTITION is used to get the event_id in the greatest state
99		# group for the given type, state_key.
100		# This may return multiple rows per (type, state_key), but last_value
101		# should be the same.
102		sql = """
103		WITH RECURSIVE state(state_group) AS (
104		VALUES(?::bigint)
105		UNION ALL
106		SELECT prev_state_group FROM state_group_edges e, state s
107		WHERE s.state_group = e.state_group
108		)
109		SELECT DISTINCT ON (type, state_key)
110		type, state_key, event_id
111		FROM state_groups_state
112		WHERE state_group IN (
113		SELECT state_group FROM state
114		) %s
115		ORDER BY type, state_key, state_group DESC
116		"""
117
118		for group in groups:
119		args = [group]
120		args.extend(where_args)
121
122		txn.execute(sql % (where_clause,), args)
123		for row in txn:
124		typ, state_key, event_id = row
125		key = (typ, state_key)
126		results[group][key] = event_id
127		else:
128		max_entries_returned = state_filter.max_entries_returned()
129
130		# We don't use WITH RECURSIVE on sqlite3 as there are distributions
131		# that ship with an sqlite3 version that doesn't support it (e.g. wheezy)
132		for group in groups:
133		next_group = group
134
135		while next_group:
136		# We did this before by getting the list of group ids, and
137		# then passing that list to sqlite to get latest event for
138		# each (type, state_key). However, that was terribly slow
139		# without the right indices (which we can't add until
140		# after we finish deduping state, which requires this func)
141		args = [next_group]
142		args.extend(where_args)
143
144		txn.execute(
145		"SELECT type, state_key, event_id FROM state_groups_state"
146		" WHERE state_group = ? " + where_clause,
147		args,
148		)
149		results[group].update(
150		((typ, state_key), event_id)
151		for typ, state_key, event_id in txn
152		if (typ, state_key) not in results[group]
153		)
154
155		# If the number of entries in the (type,state_key)->event_id dict
156		# matches the number of (type,state_keys) types we were searching
157		# for, then we must have found them all, so no need to go walk
158		# further down the tree... UNLESS our types filter contained
159		# wildcards (i.e. Nones) in which case we have to do an exhaustive
160		# search
161		if (
162		max_entries_returned is not None
163		and len(results[group]) == max_entries_returned
164		):
165		break
166
167		next_group = self.db.simple_select_one_onecol_txn(
168		txn,
169		table="state_group_edges",
170		keyvalues={"state_group": next_group},
171		retcol="prev_state_group",
172		allow_none=True,
173		)
174
175		return results
176
177
178		class StateBackgroundUpdateStore(StateGroupBackgroundUpdateStore):
179
180		STATE_GROUP_DEDUPLICATION_UPDATE_NAME = "state_group_state_deduplication"
181		STATE_GROUP_INDEX_UPDATE_NAME = "state_group_state_type_index"
182		STATE_GROUPS_ROOM_INDEX_UPDATE_NAME = "state_groups_room_id_idx"
183
184		def __init__(self, database: Database, db_conn, hs):
185		super(StateBackgroundUpdateStore, self).__init__(database, db_conn, hs)
186		self.db.updates.register_background_update_handler(
187		self.STATE_GROUP_DEDUPLICATION_UPDATE_NAME,
188		self._background_deduplicate_state,
189		)
190		self.db.updates.register_background_update_handler(
191		self.STATE_GROUP_INDEX_UPDATE_NAME, self._background_index_state
192		)
193		self.db.updates.register_background_index_update(
194		self.STATE_GROUPS_ROOM_INDEX_UPDATE_NAME,
195		index_name="state_groups_room_id_idx",
196		table="state_groups",
197		columns=["room_id"],
198		)
199
200		@defer.inlineCallbacks
201		def _background_deduplicate_state(self, progress, batch_size):
202		"""This background update will slowly deduplicate state by reencoding
203		them as deltas.
204		"""
205		last_state_group = progress.get("last_state_group", 0)
206		rows_inserted = progress.get("rows_inserted", 0)
207		max_group = progress.get("max_group", None)
208
209		BATCH_SIZE_SCALE_FACTOR = 100
210
211		batch_size = max(1, int(batch_size / BATCH_SIZE_SCALE_FACTOR))
212
213		if max_group is None:
214		rows = yield self.db.execute(
215		"_background_deduplicate_state",
216		None,
217		"SELECT coalesce(max(id), 0) FROM state_groups",
218		)
219		max_group = rows[0][0]
220
221		def reindex_txn(txn):
222		new_last_state_group = last_state_group
223		for count in range(batch_size):
224		txn.execute(
225		"SELECT id, room_id FROM state_groups"
226		" WHERE ? < id AND id <= ?"
227		" ORDER BY id ASC"
228		" LIMIT 1",
229		(new_last_state_group, max_group),
230		)
231		row = txn.fetchone()
232		if row:
233		state_group, room_id = row
234
235		if not row or not state_group:
236		return True, count
237
238		txn.execute(
239		"SELECT state_group FROM state_group_edges"
240		" WHERE state_group = ?",
241		(state_group,),
242		)
243
244		# If we reach a point where we've already started inserting
245		# edges we should stop.
246		if txn.fetchall():
247		return True, count
248
249		txn.execute(
250		"SELECT coalesce(max(id), 0) FROM state_groups"
251		" WHERE id < ? AND room_id = ?",
252		(state_group, room_id),
253		)
254		(prev_group,) = txn.fetchone()
255		new_last_state_group = state_group
256
257		if prev_group:
258		potential_hops = self._count_state_group_hops_txn(txn, prev_group)
259		if potential_hops >= MAX_STATE_DELTA_HOPS:
260		# We want to ensure chains are at most this long,#
261		# otherwise read performance degrades.
262		continue
263
264		prev_state = self._get_state_groups_from_groups_txn(
265		txn, [prev_group]
266		)
267		prev_state = prev_state[prev_group]
268
269		curr_state = self._get_state_groups_from_groups_txn(
270		txn, [state_group]
271		)
272		curr_state = curr_state[state_group]
273
274		if not set(prev_state.keys()) - set(curr_state.keys()):
275		# We can only do a delta if the current has a strict super set
276		# of keys
277
278		delta_state = {
279		key: value
280		for key, value in curr_state.items()
281		if prev_state.get(key, None) != value
282		}
283
284		self.db.simple_delete_txn(
285		txn,
286		table="state_group_edges",
287		keyvalues={"state_group": state_group},
288		)
289
290		self.db.simple_insert_txn(
291		txn,
292		table="state_group_edges",
293		values={
294		"state_group": state_group,
295		"prev_state_group": prev_group,
296		},
297		)
298
299		self.db.simple_delete_txn(
300		txn,
301		table="state_groups_state",
302		keyvalues={"state_group": state_group},
303		)
304
305		self.db.simple_insert_many_txn(
306		txn,
307		table="state_groups_state",
308		values=[
309		{
310		"state_group": state_group,
311		"room_id": room_id,
312		"type": key[0],
313		"state_key": key[1],
314		"event_id": state_id,
315		}
316		for key, state_id in delta_state.items()
317		],
318		)
319
320		progress = {
321		"last_state_group": state_group,
322		"rows_inserted": rows_inserted + batch_size,
323		"max_group": max_group,
324		}
325
326		self.db.updates._background_update_progress_txn(
327		txn, self.STATE_GROUP_DEDUPLICATION_UPDATE_NAME, progress
328		)
329
330		return False, batch_size
331
332		finished, result = yield self.db.runInteraction(
333		self.STATE_GROUP_DEDUPLICATION_UPDATE_NAME, reindex_txn
334		)
335
336		if finished:
337		yield self.db.updates._end_background_update(
338		self.STATE_GROUP_DEDUPLICATION_UPDATE_NAME
339		)
340
341		return result * BATCH_SIZE_SCALE_FACTOR
342
343		@defer.inlineCallbacks
344		def _background_index_state(self, progress, batch_size):
345		def reindex_txn(conn):
346		conn.rollback()
347		if isinstance(self.database_engine, PostgresEngine):
348		# postgres insists on autocommit for the index
349		conn.set_session(autocommit=True)
350		try:
351		txn = conn.cursor()
352		txn.execute(
353		"CREATE INDEX CONCURRENTLY state_groups_state_type_idx"
354		" ON state_groups_state(state_group, type, state_key)"
355		)
356		txn.execute("DROP INDEX IF EXISTS state_groups_state_id")
357		finally:
358		conn.set_session(autocommit=False)
359		else:
360		txn = conn.cursor()
361		txn.execute(
362		"CREATE INDEX state_groups_state_type_idx"
363		" ON state_groups_state(state_group, type, state_key)"
364		)
365		txn.execute("DROP INDEX IF EXISTS state_groups_state_id")
366
367		yield self.db.runWithConnection(reindex_txn)
368
369		yield self.db.updates._end_background_update(self.STATE_GROUP_INDEX_UPDATE_NAME)
370
371		return 1

+0

-16

~~synapse/storage/data_stores/state/schema/delta/23/drop_state_index.sql~~ less more

0		/* Copyright 2015, 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		DROP INDEX IF EXISTS state_groups_state_tuple;

+0

-33

~~synapse/storage/data_stores/state/schema/delta/30/state_stream.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		/* We used to create a table called current_state_resets, but this is no
17		* longer used and is removed in delta 54.
18		*/
19
20		/* The outlier events that have aquired a state group typically through
21		* backfill. This is tracked separately to the events table, as assigning a
22		* state group change the position of the existing event in the stream
23		* ordering.
24		* However since a stream_ordering is assigned in persist_event for the
25		* (event, state) pair, we can use that stream_ordering to identify when
26		* the new state was assigned for the event.
27		*/
28		CREATE TABLE IF NOT EXISTS ex_outlier_stream(
29		event_stream_ordering BIGINT PRIMARY KEY NOT NULL,
30		event_id TEXT NOT NULL,
31		state_group BIGINT NOT NULL
32		);

+0

-19

~~synapse/storage/data_stores/state/schema/delta/32/remove_state_indices.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15
16		-- The following indices are redundant, other indices are equivalent or
17		-- supersets
18		DROP INDEX IF EXISTS state_groups_id; -- Duplicate of PRIMARY KEY

+0

-17

~~synapse/storage/data_stores/state/schema/delta/35/add_state_index.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT into background_updates (update_name, progress_json, depends_on)
16		VALUES ('state_group_state_type_index', '{}', 'state_group_state_deduplication');

+0

-22

~~synapse/storage/data_stores/state/schema/delta/35/state.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE state_group_edges(
16		state_group BIGINT NOT NULL,
17		prev_state_group BIGINT NOT NULL
18		);
19
20		CREATE INDEX state_group_edges_idx ON state_group_edges(state_group);
21		CREATE INDEX state_group_edges_prev_idx ON state_group_edges(prev_state_group);

+0

-17

~~synapse/storage/data_stores/state/schema/delta/35/state_dedupe.sql~~ less more

0		/* Copyright 2016 OpenMarket Ltd
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT into background_updates (update_name, progress_json)
16		VALUES ('state_group_state_deduplication', '{}');

+0

-34

~~synapse/storage/data_stores/state/schema/delta/47/state_group_seq.py~~ less more

0		# Copyright 2018 New Vector Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		from synapse.storage.engines import PostgresEngine
15
16
17		def run_create(cur, database_engine, args, *kwargs):
18		if isinstance(database_engine, PostgresEngine):
19		# if we already have some state groups, we want to start making new
20		# ones with a higher id.
21		cur.execute("SELECT max(id) FROM state_groups")
22		row = cur.fetchone()
23
24		if row[0] is None:
25		start_val = 1
26		else:
27		start_val = row[0] + 1
28
29		cur.execute("CREATE SEQUENCE state_group_id_seq START WITH %s", (start_val,))
30
31
32		def run_upgrade(args, *kwargs):
33		pass

+0

-17

~~synapse/storage/data_stores/state/schema/delta/56/state_group_room_idx.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C.
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		INSERT INTO background_updates (update_name, progress_json) VALUES
16		('state_groups_room_id_idx', '{}');

+0

-37

~~synapse/storage/data_stores/state/schema/full_schemas/54/full.sql~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE TABLE state_groups (
16		id BIGINT PRIMARY KEY,
17		room_id TEXT NOT NULL,
18		event_id TEXT NOT NULL
19		);
20
21		CREATE TABLE state_groups_state (
22		state_group BIGINT NOT NULL,
23		room_id TEXT NOT NULL,
24		type TEXT NOT NULL,
25		state_key TEXT NOT NULL,
26		event_id TEXT NOT NULL
27		);
28
29		CREATE TABLE state_group_edges (
30		state_group BIGINT NOT NULL,
31		prev_state_group BIGINT NOT NULL
32		);
33
34		CREATE INDEX state_group_edges_idx ON state_group_edges (state_group);
35		CREATE INDEX state_group_edges_prev_idx ON state_group_edges (prev_state_group);
36		CREATE INDEX state_groups_state_type_idx ON state_groups_state (state_group, type, state_key);

+0

-21

~~synapse/storage/data_stores/state/schema/full_schemas/54/sequence.sql.postgres~~ less more

0		/* Copyright 2019 The Matrix.org Foundation C.I.C
1		*
2		* Licensed under the Apache License, Version 2.0 (the "License");
3		* you may not use this file except in compliance with the License.
4		* You may obtain a copy of the License at
5		*
6		* http://www.apache.org/licenses/LICENSE-2.0
7		*
8		* Unless required by applicable law or agreed to in writing, software
9		* distributed under the License is distributed on an "AS IS" BASIS,
10		* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		* See the License for the specific language governing permissions and
12		* limitations under the License.
13		*/
14
15		CREATE SEQUENCE state_group_id_seq
16		START WITH 1
17		INCREMENT BY 1
18		NO MINVALUE
19		NO MAXVALUE
20		CACHE 1;

+0

-649

~~synapse/storage/data_stores/state/store.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		import logging
16		from collections import namedtuple
17		from typing import Dict, Iterable, List, Set, Tuple
18
19		from twisted.internet import defer
20
21		from synapse.api.constants import EventTypes
22		from synapse.storage._base import SQLBaseStore
23		from synapse.storage.data_stores.state.bg_updates import StateBackgroundUpdateStore
24		from synapse.storage.database import Database
25		from synapse.storage.state import StateFilter
26		from synapse.storage.types import Cursor
27		from synapse.storage.util.sequence import build_sequence_generator
28		from synapse.types import StateMap
29		from synapse.util.caches.descriptors import cached
30		from synapse.util.caches.dictionary_cache import DictionaryCache
31
32		logger = logging.getLogger(__name__)
33
34
35		MAX_STATE_DELTA_HOPS = 100
36
37
38		class _GetStateGroupDelta(
39		namedtuple("_GetStateGroupDelta", ("prev_group", "delta_ids"))
40		):
41		"""Return type of get_state_group_delta that implements __len__, which lets
42		us use the itrable flag when caching
43		"""
44
45		__slots__ = []
46
47		def __len__(self):
48		return len(self.delta_ids) if self.delta_ids else 0
49
50
51		class StateGroupDataStore(StateBackgroundUpdateStore, SQLBaseStore):
52		"""A data store for fetching/storing state groups.
53		"""
54
55		def __init__(self, database: Database, db_conn, hs):
56		super(StateGroupDataStore, self).__init__(database, db_conn, hs)
57
58		# Originally the state store used a single DictionaryCache to cache the
59		# event IDs for the state types in a given state group to avoid hammering
60		# on the state_group* tables.
61		#
62		# The point of using a DictionaryCache is that it can cache a subset
63		# of the state events for a given state group (i.e. a subset of the keys for a
64		# given dict which is an entry in the cache for a given state group ID).
65		#
66		# However, this poses problems when performing complicated queries
67		# on the store - for instance: "give me all the state for this group, but
68		# limit members to this subset of users", as DictionaryCache's API isn't
69		# rich enough to say "please cache any of these fields, apart from this subset".
70		# This is problematic when lazy loading members, which requires this behaviour,
71		# as without it the cache has no choice but to speculatively load all
72		# state events for the group, which negates the efficiency being sought.
73		#
74		# Rather than overcomplicating DictionaryCache's API, we instead split the
75		# state_group_cache into two halves - one for tracking non-member events,
76		# and the other for tracking member_events. This means that lazy loading
77		# queries can be made in a cache-friendly manner by querying both caches
78		# separately and then merging the result. So for the example above, you
79		# would query the members cache for a specific subset of state keys
80		# (which DictionaryCache will handle efficiently and fine) and the non-members
81		# cache for all state (which DictionaryCache will similarly handle fine)
82		# and then just merge the results together.
83		#
84		# We size the non-members cache to be smaller than the members cache as the
85		# vast majority of state in Matrix (today) is member events.
86
87		self._state_group_cache = DictionaryCache(
88		"stateGroupCache",
89		# TODO: this hasn't been tuned yet
90		50000,
91		)
92		self._state_group_members_cache = DictionaryCache(
93		"stateGroupMembersCache", 500000,
94		)
95
96		def get_max_state_group_txn(txn: Cursor):
97		txn.execute("SELECT COALESCE(max(id), 0) FROM state_groups")
98		return txn.fetchone()[0]
99
100		self._state_group_seq_gen = build_sequence_generator(
101		self.database_engine, get_max_state_group_txn, "state_group_id_seq"
102		)
103
104		@cached(max_entries=10000, iterable=True)
105		def get_state_group_delta(self, state_group):
106		"""Given a state group try to return a previous group and a delta between
107		the old and the new.
108
109		Returns:
110		(prev_group, delta_ids), where both may be None.
111		"""
112
113		def _get_state_group_delta_txn(txn):
114		prev_group = self.db.simple_select_one_onecol_txn(
115		txn,
116		table="state_group_edges",
117		keyvalues={"state_group": state_group},
118		retcol="prev_state_group",
119		allow_none=True,
120		)
121
122		if not prev_group:
123		return _GetStateGroupDelta(None, None)
124
125		delta_ids = self.db.simple_select_list_txn(
126		txn,
127		table="state_groups_state",
128		keyvalues={"state_group": state_group},
129		retcols=("type", "state_key", "event_id"),
130		)
131
132		return _GetStateGroupDelta(
133		prev_group,
134		{(row["type"], row["state_key"]): row["event_id"] for row in delta_ids},
135		)
136
137		return self.db.runInteraction(
138		"get_state_group_delta", _get_state_group_delta_txn
139		)
140
141		@defer.inlineCallbacks
142		def _get_state_groups_from_groups(
143		self, groups: List[int], state_filter: StateFilter
144		):
145		"""Returns the state groups for a given set of groups from the
146		database, filtering on types of state events.
147
148		Args:
149		groups: list of state group IDs to query
150		state_filter: The state filter used to fetch state
151		from the database.
152		Returns:
153		Deferred[Dict[int, StateMap[str]]]: Dict of state group to state map.
154		"""
155		results = {}
156
157		chunks = [groups[i : i + 100] for i in range(0, len(groups), 100)]
158		for chunk in chunks:
159		res = yield self.db.runInteraction(
160		"_get_state_groups_from_groups",
161		self._get_state_groups_from_groups_txn,
162		chunk,
163		state_filter,
164		)
165		results.update(res)
166
167		return results
168
169		def _get_state_for_group_using_cache(self, cache, group, state_filter):
170		"""Checks if group is in cache. See `_get_state_for_groups`
171
172		Args:
173		cache(DictionaryCache): the state group cache to use
174		group(int): The state group to lookup
175		state_filter (StateFilter): The state filter used to fetch state
176		from the database.
177
178		Returns 2-tuple (`state_dict`, `got_all`).
179		`got_all` is a bool indicating if we successfully retrieved all
180		requests state from the cache, if False we need to query the DB for the
181		missing state.
182		"""
183		is_all, known_absent, state_dict_ids = cache.get(group)
184
185		if is_all or state_filter.is_full():
186		# Either we have everything or want everything, either way
187		# `is_all` tells us whether we've gotten everything.
188		return state_filter.filter_state(state_dict_ids), is_all
189
190		# tracks whether any of our requested types are missing from the cache
191		missing_types = False
192
193		if state_filter.has_wildcards():
194		# We don't know if we fetched all the state keys for the types in
195		# the filter that are wildcards, so we have to assume that we may
196		# have missed some.
197		missing_types = True
198		else:
199		# There aren't any wild cards, so `concrete_types()` returns the
200		# complete list of event types we're wanting.
201		for key in state_filter.concrete_types():
202		if key not in state_dict_ids and key not in known_absent:
203		missing_types = True
204		break
205
206		return state_filter.filter_state(state_dict_ids), not missing_types
207
208		@defer.inlineCallbacks
209		def _get_state_for_groups(
210		self, groups: Iterable[int], state_filter: StateFilter = StateFilter.all()
211		):
212		"""Gets the state at each of a list of state groups, optionally
213		filtering by type/state_key
214
215		Args:
216		groups: list of state groups for which we want
217		to get the state.
218		state_filter: The state filter used to fetch state
219		from the database.
220		Returns:
221		Deferred[Dict[int, StateMap[str]]]: Dict of state group to state map.
222		"""
223
224		member_filter, non_member_filter = state_filter.get_member_split()
225
226		# Now we look them up in the member and non-member caches
227		(
228		non_member_state,
229		incomplete_groups_nm,
230		) = yield self._get_state_for_groups_using_cache(
231		groups, self._state_group_cache, state_filter=non_member_filter
232		)
233
234		(
235		member_state,
236		incomplete_groups_m,
237		) = yield self._get_state_for_groups_using_cache(
238		groups, self._state_group_members_cache, state_filter=member_filter
239		)
240
241		state = dict(non_member_state)
242		for group in groups:
243		state[group].update(member_state[group])
244
245		# Now fetch any missing groups from the database
246
247		incomplete_groups = incomplete_groups_m \| incomplete_groups_nm
248
249		if not incomplete_groups:
250		return state
251
252		cache_sequence_nm = self._state_group_cache.sequence
253		cache_sequence_m = self._state_group_members_cache.sequence
254
255		# Help the cache hit ratio by expanding the filter a bit
256		db_state_filter = state_filter.return_expanded()
257
258		group_to_state_dict = yield self._get_state_groups_from_groups(
259		list(incomplete_groups), state_filter=db_state_filter
260		)
261
262		# Now lets update the caches
263		self._insert_into_cache(
264		group_to_state_dict,
265		db_state_filter,
266		cache_seq_num_members=cache_sequence_m,
267		cache_seq_num_non_members=cache_sequence_nm,
268		)
269
270		# And finally update the result dict, by filtering out any extra
271		# stuff we pulled out of the database.
272		for group, group_state_dict in group_to_state_dict.items():
273		# We just replace any existing entries, as we will have loaded
274		# everything we need from the database anyway.
275		state[group] = state_filter.filter_state(group_state_dict)
276
277		return state
278
279		def _get_state_for_groups_using_cache(
280		self, groups: Iterable[int], cache: DictionaryCache, state_filter: StateFilter
281		) -> Tuple[Dict[int, StateMap[str]], Set[int]]:
282		"""Gets the state at each of a list of state groups, optionally
283		filtering by type/state_key, querying from a specific cache.
284
285		Args:
286		groups: list of state groups for which we want to get the state.
287		cache: the cache of group ids to state dicts which
288		we will pass through - either the normal state cache or the
289		specific members state cache.
290		state_filter: The state filter used to fetch state from the
291		database.
292
293		Returns:
294		Tuple of dict of state_group_id to state map of entries in the
295		cache, and the state group ids either missing from the cache or
296		incomplete.
297		"""
298		results = {}
299		incomplete_groups = set()
300		for group in set(groups):
301		state_dict_ids, got_all = self._get_state_for_group_using_cache(
302		cache, group, state_filter
303		)
304		results[group] = state_dict_ids
305
306		if not got_all:
307		incomplete_groups.add(group)
308
309		return results, incomplete_groups
310
311		def _insert_into_cache(
312		self,
313		group_to_state_dict,
314		state_filter,
315		cache_seq_num_members,
316		cache_seq_num_non_members,
317		):
318		"""Inserts results from querying the database into the relevant cache.
319
320		Args:
321		group_to_state_dict (dict): The new entries pulled from database.
322		Map from state group to state dict
323		state_filter (StateFilter): The state filter used to fetch state
324		from the database.
325		cache_seq_num_members (int): Sequence number of member cache since
326		last lookup in cache
327		cache_seq_num_non_members (int): Sequence number of member cache since
328		last lookup in cache
329		"""
330
331		# We need to work out which types we've fetched from the DB for the
332		# member vs non-member caches. This should be as accurate as possible,
333		# but can be an underestimate (e.g. when we have wild cards)
334
335		member_filter, non_member_filter = state_filter.get_member_split()
336		if member_filter.is_full():
337		# We fetched all member events
338		member_types = None
339		else:
340		# `concrete_types()` will only return a subset when there are wild
341		# cards in the filter, but that's fine.
342		member_types = member_filter.concrete_types()
343
344		if non_member_filter.is_full():
345		# We fetched all non member events
346		non_member_types = None
347		else:
348		non_member_types = non_member_filter.concrete_types()
349
350		for group, group_state_dict in group_to_state_dict.items():
351		state_dict_members = {}
352		state_dict_non_members = {}
353
354		for k, v in group_state_dict.items():
355		if k[0] == EventTypes.Member:
356		state_dict_members[k] = v
357		else:
358		state_dict_non_members[k] = v
359
360		self._state_group_members_cache.update(
361		cache_seq_num_members,
362		key=group,
363		value=state_dict_members,
364		fetched_keys=member_types,
365		)
366
367		self._state_group_cache.update(
368		cache_seq_num_non_members,
369		key=group,
370		value=state_dict_non_members,
371		fetched_keys=non_member_types,
372		)
373
374		def store_state_group(
375		self, event_id, room_id, prev_group, delta_ids, current_state_ids
376		):
377		"""Store a new set of state, returning a newly assigned state group.
378
379		Args:
380		event_id (str): The event ID for which the state was calculated
381		room_id (str)
382		prev_group (int\|None): A previous state group for the room, optional.
383		delta_ids (dict\|None): The delta between state at `prev_group` and
384		`current_state_ids`, if `prev_group` was given. Same format as
385		`current_state_ids`.
386		current_state_ids (dict): The state to store. Map of (type, state_key)
387		to event_id.
388
389		Returns:
390		Deferred[int]: The state group ID
391		"""
392
393		def _store_state_group_txn(txn):
394		if current_state_ids is None:
395		# AFAIK, this can never happen
396		raise Exception("current_state_ids cannot be None")
397
398		state_group = self._state_group_seq_gen.get_next_id_txn(txn)
399
400		self.db.simple_insert_txn(
401		txn,
402		table="state_groups",
403		values={"id": state_group, "room_id": room_id, "event_id": event_id},
404		)
405
406		# We persist as a delta if we can, while also ensuring the chain
407		# of deltas isn't tooo long, as otherwise read performance degrades.
408		if prev_group:
409		is_in_db = self.db.simple_select_one_onecol_txn(
410		txn,
411		table="state_groups",
412		keyvalues={"id": prev_group},
413		retcol="id",
414		allow_none=True,
415		)
416		if not is_in_db:
417		raise Exception(
418		"Trying to persist state with unpersisted prev_group: %r"
419		% (prev_group,)
420		)
421
422		potential_hops = self._count_state_group_hops_txn(txn, prev_group)
423		if prev_group and potential_hops < MAX_STATE_DELTA_HOPS:
424		self.db.simple_insert_txn(
425		txn,
426		table="state_group_edges",
427		values={"state_group": state_group, "prev_state_group": prev_group},
428		)
429
430		self.db.simple_insert_many_txn(
431		txn,
432		table="state_groups_state",
433		values=[
434		{
435		"state_group": state_group,
436		"room_id": room_id,
437		"type": key[0],
438		"state_key": key[1],
439		"event_id": state_id,
440		}
441		for key, state_id in delta_ids.items()
442		],
443		)
444		else:
445		self.db.simple_insert_many_txn(
446		txn,
447		table="state_groups_state",
448		values=[
449		{
450		"state_group": state_group,
451		"room_id": room_id,
452		"type": key[0],
453		"state_key": key[1],
454		"event_id": state_id,
455		}
456		for key, state_id in current_state_ids.items()
457		],
458		)
459
460		# Prefill the state group caches with this group.
461		# It's fine to use the sequence like this as the state group map
462		# is immutable. (If the map wasn't immutable then this prefill could
463		# race with another update)
464
465		current_member_state_ids = {
466		s: ev
467		for (s, ev) in current_state_ids.items()
468		if s[0] == EventTypes.Member
469		}
470		txn.call_after(
471		self._state_group_members_cache.update,
472		self._state_group_members_cache.sequence,
473		key=state_group,
474		value=dict(current_member_state_ids),
475		)
476
477		current_non_member_state_ids = {
478		s: ev
479		for (s, ev) in current_state_ids.items()
480		if s[0] != EventTypes.Member
481		}
482		txn.call_after(
483		self._state_group_cache.update,
484		self._state_group_cache.sequence,
485		key=state_group,
486		value=dict(current_non_member_state_ids),
487		)
488
489		return state_group
490
491		return self.db.runInteraction("store_state_group", _store_state_group_txn)
492
493		def purge_unreferenced_state_groups(
494		self, room_id: str, state_groups_to_delete
495		) -> defer.Deferred:
496		"""Deletes no longer referenced state groups and de-deltas any state
497		groups that reference them.
498
499		Args:
500		room_id: The room the state groups belong to (must all be in the
501		same room).
502		state_groups_to_delete (Collection[int]): Set of all state groups
503		to delete.
504		"""
505
506		return self.db.runInteraction(
507		"purge_unreferenced_state_groups",
508		self._purge_unreferenced_state_groups,
509		room_id,
510		state_groups_to_delete,
511		)
512
513		def _purge_unreferenced_state_groups(self, txn, room_id, state_groups_to_delete):
514		logger.info(
515		"[purge] found %i state groups to delete", len(state_groups_to_delete)
516		)
517
518		rows = self.db.simple_select_many_txn(
519		txn,
520		table="state_group_edges",
521		column="prev_state_group",
522		iterable=state_groups_to_delete,
523		keyvalues={},
524		retcols=("state_group",),
525		)
526
527		remaining_state_groups = {
528		row["state_group"]
529		for row in rows
530		if row["state_group"] not in state_groups_to_delete
531		}
532
533		logger.info(
534		"[purge] de-delta-ing %i remaining state groups",
535		len(remaining_state_groups),
536		)
537
538		# Now we turn the state groups that reference to-be-deleted state
539		# groups to non delta versions.
540		for sg in remaining_state_groups:
541		logger.info("[purge] de-delta-ing remaining state group %s", sg)
542		curr_state = self._get_state_groups_from_groups_txn(txn, [sg])
543		curr_state = curr_state[sg]
544
545		self.db.simple_delete_txn(
546		txn, table="state_groups_state", keyvalues={"state_group": sg}
547		)
548
549		self.db.simple_delete_txn(
550		txn, table="state_group_edges", keyvalues={"state_group": sg}
551		)
552
553		self.db.simple_insert_many_txn(
554		txn,
555		table="state_groups_state",
556		values=[
557		{
558		"state_group": sg,
559		"room_id": room_id,
560		"type": key[0],
561		"state_key": key[1],
562		"event_id": state_id,
563		}
564		for key, state_id in curr_state.items()
565		],
566		)
567
568		logger.info("[purge] removing redundant state groups")
569		txn.executemany(
570		"DELETE FROM state_groups_state WHERE state_group = ?",
571		((sg,) for sg in state_groups_to_delete),
572		)
573		txn.executemany(
574		"DELETE FROM state_groups WHERE id = ?",
575		((sg,) for sg in state_groups_to_delete),
576		)
577
578		@defer.inlineCallbacks
579		def get_previous_state_groups(self, state_groups):
580		"""Fetch the previous groups of the given state groups.
581
582		Args:
583		state_groups (Iterable[int])
584
585		Returns:
586		Deferred[dict[int, int]]: mapping from state group to previous
587		state group.
588		"""
589
590		rows = yield self.db.simple_select_many_batch(
591		table="state_group_edges",
592		column="prev_state_group",
593		iterable=state_groups,
594		keyvalues={},
595		retcols=("prev_state_group", "state_group"),
596		desc="get_previous_state_groups",
597		)
598
599		return {row["state_group"]: row["prev_state_group"] for row in rows}
600
601		def purge_room_state(self, room_id, state_groups_to_delete):
602		"""Deletes all record of a room from state tables
603
604		Args:
605		room_id (str):
606		state_groups_to_delete (list[int]): State groups to delete
607		"""
608
609		return self.db.runInteraction(
610		"purge_room_state",
611		self._purge_room_state_txn,
612		room_id,
613		state_groups_to_delete,
614		)
615
616		def _purge_room_state_txn(self, txn, room_id, state_groups_to_delete):
617		# first we have to delete the state groups states
618		logger.info("[purge] removing %s from state_groups_state", room_id)
619
620		self.db.simple_delete_many_txn(
621		txn,
622		table="state_groups_state",
623		column="state_group",
624		iterable=state_groups_to_delete,
625		keyvalues={},
626		)
627
628		# ... and the state group edges
629		logger.info("[purge] removing %s from state_group_edges", room_id)
630
631		self.db.simple_delete_many_txn(
632		txn,
633		table="state_group_edges",
634		column="state_group",
635		iterable=state_groups_to_delete,
636		keyvalues={},
637		)
638
639		# ... and the state groups
640		logger.info("[purge] removing %s from state_groups", room_id)
641
642		self.db.simple_delete_many_txn(
643		txn,
644		table="state_groups",
645		column="id",
646		iterable=state_groups_to_delete,
647		keyvalues={},
648		)

+11

-9

synapse/storage/database.py less more

48	48	from synapse.storage.types import Connection, Cursor
49	49	from synapse.types import Collection
50	50
51		logger = logging.getLogger(__name__)
52
53	51	# python 3 does not have a maximum int value
54	52	MAX_TXN_ID = 2 ** 63 - 1
	53
	54	logger = logging.getLogger(__name__)
55	55
56	56	sql_logger = logging.getLogger("synapse.storage.SQL")
57	57	transaction_logger = logging.getLogger("synapse.storage.txn")

232	232	try:
233	233	return func(sql, *args)
234	234	except Exception as e:
235		logger.debug("[SQL FAIL] {%s} %s", self.name, e)
	235	sql_logger.debug("[SQL FAIL] {%s} %s", self.name, e)
236	236	raise
237	237	finally:
238	238	secs = time.time() - start

278	278	return top_n_counters
279	279
280	280
281		class Database(object):
	281	class DatabasePool(object):
282	282	"""Wraps a single physical database and connection pool.
283	283
284	284	A single database may be used by multiple data stores.

418	418	except self.engine.module.OperationalError as e:
419	419	# This can happen if the database disappears mid
420	420	# transaction.
421		logger.warning(
	421	transaction_logger.warning(
422	422	"[TXN OPERROR] {%s} %s %d/%d", name, e, i, N,
423	423	)
424	424	if i < N:

426	426	try:
427	427	conn.rollback()
428	428	except self.engine.module.Error as e1:
429		logger.warning("[TXN EROLL] {%s} %s", name, e1)
	429	transaction_logger.warning("[TXN EROLL] {%s} %s", name, e1)
430	430	continue
431	431	raise
432	432	except self.engine.module.DatabaseError as e:
433	433	if self.engine.is_deadlock(e):
434		logger.warning("[TXN DEADLOCK] {%s} %d/%d", name, i, N)
	434	transaction_logger.warning(
	435	"[TXN DEADLOCK] {%s} %d/%d", name, i, N
	436	)
435	437	if i < N:
436	438	i += 1
437	439	try:
438	440	conn.rollback()
439	441	except self.engine.module.Error as e1:
440		logger.warning(
	442	transaction_logger.warning(
441	443	"[TXN EROLL] {%s} %s", name, e1,
442	444	)
443	445	continue

477	479	# [2]: https://github.com/python/cpython/blob/v3.8.0/Modules/_sqlite/cursor.c#L236
478	480	cursor.close()
479	481	except Exception as e:
480		logger.debug("[TXN FAIL] {%s} %s", name, e)
	482	transaction_logger.debug("[TXN FAIL] {%s} %s", name, e)
481	483	raise
482	484	finally:
483	485	end = monotonic_time()

+101

-0

synapse/storage/databases/__init__.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2019 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16
	17	from synapse.storage.database import DatabasePool, make_conn
	18	from synapse.storage.databases.main.events import PersistEventsStore
	19	from synapse.storage.databases.state import StateGroupDataStore
	20	from synapse.storage.engines import create_engine
	21	from synapse.storage.prepare_database import prepare_database
	22
	23	logger = logging.getLogger(__name__)
	24
	25
	26	class Databases(object):
	27	"""The various databases.
	28
	29	These are low level interfaces to physical databases.
	30
	31	Attributes:
	32	main (DataStore)
	33	"""
	34
	35	def __init__(self, main_store_class, hs):
	36	# Note we pass in the main store class here as workers use a different main
	37	# store.
	38
	39	self.databases = []
	40	main = None
	41	state = None
	42	persist_events = None
	43
	44	for database_config in hs.config.database.databases:
	45	db_name = database_config.name
	46	engine = create_engine(database_config.config)
	47
	48	with make_conn(database_config, engine) as db_conn:
	49	logger.info("Preparing database %r...", db_name)
	50
	51	engine.check_database(db_conn)
	52	prepare_database(
	53	db_conn, engine, hs.config, databases=database_config.databases,
	54	)
	55
	56	database = DatabasePool(hs, database_config, engine)
	57
	58	if "main" in database_config.databases:
	59	logger.info("Starting 'main' data store")
	60
	61	# Sanity check we don't try and configure the main store on
	62	# multiple databases.
	63	if main:
	64	raise Exception("'main' data store already configured")
	65
	66	main = main_store_class(database, db_conn, hs)
	67
	68	# If we're on a process that can persist events also
	69	# instantiate a `PersistEventsStore`
	70	if hs.config.worker.writers.events == hs.get_instance_name():
	71	persist_events = PersistEventsStore(hs, database, main)
	72
	73	if "state" in database_config.databases:
	74	logger.info("Starting 'state' data store")
	75
	76	# Sanity check we don't try and configure the state store on
	77	# multiple databases.
	78	if state:
	79	raise Exception("'state' data store already configured")
	80
	81	state = StateGroupDataStore(database, db_conn, hs)
	82
	83	db_conn.commit()
	84
	85	self.databases.append(database)
	86
	87	logger.info("Database %r prepared", db_name)
	88
	89	# Sanity check that we have actually configured all the required stores.
	90	if not main:
	91	raise Exception("No 'main' data store configured")
	92
	93	if not state:
	94	raise Exception("No 'main' data store configured")
	95
	96	# We use local variables here to ensure that the databases do not have
	97	# optional types.
	98	self.main = main
	99	self.state = state
	100	self.persist_events = persist_events

+596

-0

synapse/storage/databases/main/__init__.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2018 New Vector Ltd
	3	# Copyright 2019 The Matrix.org Foundation C.I.C.
	4	#
	5	# Licensed under the Apache License, Version 2.0 (the "License");
	6	# you may not use this file except in compliance with the License.
	7	# You may obtain a copy of the License at
	8	#
	9	# http://www.apache.org/licenses/LICENSE-2.0
	10	#
	11	# Unless required by applicable law or agreed to in writing, software
	12	# distributed under the License is distributed on an "AS IS" BASIS,
	13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	14	# See the License for the specific language governing permissions and
	15	# limitations under the License.
	16
	17	import calendar
	18	import logging
	19	import time
	20
	21	from synapse.api.constants import PresenceState
	22	from synapse.config.homeserver import HomeServerConfig
	23	from synapse.storage.database import DatabasePool
	24	from synapse.storage.engines import PostgresEngine
	25	from synapse.storage.util.id_generators import (
	26	IdGenerator,
	27	MultiWriterIdGenerator,
	28	StreamIdGenerator,
	29	)
	30	from synapse.util.caches.stream_change_cache import StreamChangeCache
	31
	32	from .account_data import AccountDataStore
	33	from .appservice import ApplicationServiceStore, ApplicationServiceTransactionStore
	34	from .cache import CacheInvalidationWorkerStore
	35	from .censor_events import CensorEventsStore
	36	from .client_ips import ClientIpStore
	37	from .deviceinbox import DeviceInboxStore
	38	from .devices import DeviceStore
	39	from .directory import DirectoryStore
	40	from .e2e_room_keys import EndToEndRoomKeyStore
	41	from .end_to_end_keys import EndToEndKeyStore
	42	from .event_federation import EventFederationStore
	43	from .event_push_actions import EventPushActionsStore
	44	from .events_bg_updates import EventsBackgroundUpdatesStore
	45	from .filtering import FilteringStore
	46	from .group_server import GroupServerStore
	47	from .keys import KeyStore
	48	from .media_repository import MediaRepositoryStore
	49	from .metrics import ServerMetricsStore
	50	from .monthly_active_users import MonthlyActiveUsersStore
	51	from .openid import OpenIdStore
	52	from .presence import PresenceStore, UserPresenceState
	53	from .profile import ProfileStore
	54	from .purge_events import PurgeEventsStore
	55	from .push_rule import PushRuleStore
	56	from .pusher import PusherStore
	57	from .receipts import ReceiptsStore
	58	from .registration import RegistrationStore
	59	from .rejections import RejectionsStore
	60	from .relations import RelationsStore
	61	from .room import RoomStore
	62	from .roommember import RoomMemberStore
	63	from .search import SearchStore
	64	from .signatures import SignatureStore
	65	from .state import StateStore
	66	from .stats import StatsStore
	67	from .stream import StreamStore
	68	from .tags import TagsStore
	69	from .transactions import TransactionStore
	70	from .ui_auth import UIAuthStore
	71	from .user_directory import UserDirectoryStore
	72	from .user_erasure_store import UserErasureStore
	73
	74	logger = logging.getLogger(__name__)
	75
	76
	77	class DataStore(
	78	EventsBackgroundUpdatesStore,
	79	RoomMemberStore,
	80	RoomStore,
	81	RegistrationStore,
	82	StreamStore,
	83	ProfileStore,
	84	PresenceStore,
	85	TransactionStore,
	86	DirectoryStore,
	87	KeyStore,
	88	StateStore,
	89	SignatureStore,
	90	ApplicationServiceStore,
	91	PurgeEventsStore,
	92	EventFederationStore,
	93	MediaRepositoryStore,
	94	RejectionsStore,
	95	FilteringStore,
	96	PusherStore,
	97	PushRuleStore,
	98	ApplicationServiceTransactionStore,
	99	ReceiptsStore,
	100	EndToEndKeyStore,
	101	EndToEndRoomKeyStore,
	102	SearchStore,
	103	TagsStore,
	104	AccountDataStore,
	105	EventPushActionsStore,
	106	OpenIdStore,
	107	ClientIpStore,
	108	DeviceStore,
	109	DeviceInboxStore,
	110	UserDirectoryStore,
	111	GroupServerStore,
	112	UserErasureStore,
	113	MonthlyActiveUsersStore,
	114	StatsStore,
	115	RelationsStore,
	116	CensorEventsStore,
	117	UIAuthStore,
	118	CacheInvalidationWorkerStore,
	119	ServerMetricsStore,
	120	):
	121	def __init__(self, database: DatabasePool, db_conn, hs):
	122	self.hs = hs
	123	self._clock = hs.get_clock()
	124	self.database_engine = database.engine
	125
	126	self._presence_id_gen = StreamIdGenerator(
	127	db_conn, "presence_stream", "stream_id"
	128	)
	129	self._device_inbox_id_gen = StreamIdGenerator(
	130	db_conn, "device_inbox", "stream_id"
	131	)
	132	self._public_room_id_gen = StreamIdGenerator(
	133	db_conn, "public_room_list_stream", "stream_id"
	134	)
	135	self._device_list_id_gen = StreamIdGenerator(
	136	db_conn,
	137	"device_lists_stream",
	138	"stream_id",
	139	extra_tables=[
	140	("user_signature_stream", "stream_id"),
	141	("device_lists_outbound_pokes", "stream_id"),
	142	],
	143	)
	144	self._cross_signing_id_gen = StreamIdGenerator(
	145	db_conn, "e2e_cross_signing_keys", "stream_id"
	146	)
	147
	148	self._access_tokens_id_gen = IdGenerator(db_conn, "access_tokens", "id")
	149	self._event_reports_id_gen = IdGenerator(db_conn, "event_reports", "id")
	150	self._push_rule_id_gen = IdGenerator(db_conn, "push_rules", "id")
	151	self._push_rules_enable_id_gen = IdGenerator(db_conn, "push_rules_enable", "id")
	152	self._pushers_id_gen = StreamIdGenerator(
	153	db_conn, "pushers", "id", extra_tables=[("deleted_pushers", "stream_id")]
	154	)
	155	self._group_updates_id_gen = StreamIdGenerator(
	156	db_conn, "local_group_updates", "stream_id"
	157	)
	158
	159	if isinstance(self.database_engine, PostgresEngine):
	160	self._cache_id_gen = MultiWriterIdGenerator(
	161	db_conn,
	162	database,
	163	instance_name="master",
	164	table="cache_invalidation_stream_by_instance",
	165	instance_column="instance_name",
	166	id_column="stream_id",
	167	sequence_name="cache_invalidation_stream_seq",
	168	)
	169	else:
	170	self._cache_id_gen = None
	171
	172	super(DataStore, self).__init__(database, db_conn, hs)
	173
	174	self._presence_on_startup = self._get_active_presence(db_conn)
	175
	176	presence_cache_prefill, min_presence_val = self.db_pool.get_cache_dict(
	177	db_conn,
	178	"presence_stream",
	179	entity_column="user_id",
	180	stream_column="stream_id",
	181	max_value=self._presence_id_gen.get_current_token(),
	182	)
	183	self.presence_stream_cache = StreamChangeCache(
	184	"PresenceStreamChangeCache",
	185	min_presence_val,
	186	prefilled_cache=presence_cache_prefill,
	187	)
	188
	189	max_device_inbox_id = self._device_inbox_id_gen.get_current_token()
	190	device_inbox_prefill, min_device_inbox_id = self.db_pool.get_cache_dict(
	191	db_conn,
	192	"device_inbox",
	193	entity_column="user_id",
	194	stream_column="stream_id",
	195	max_value=max_device_inbox_id,
	196	limit=1000,
	197	)
	198	self._device_inbox_stream_cache = StreamChangeCache(
	199	"DeviceInboxStreamChangeCache",
	200	min_device_inbox_id,
	201	prefilled_cache=device_inbox_prefill,
	202	)
	203	# The federation outbox and the local device inbox uses the same
	204	# stream_id generator.
	205	device_outbox_prefill, min_device_outbox_id = self.db_pool.get_cache_dict(
	206	db_conn,
	207	"device_federation_outbox",
	208	entity_column="destination",
	209	stream_column="stream_id",
	210	max_value=max_device_inbox_id,
	211	limit=1000,
	212	)
	213	self._device_federation_outbox_stream_cache = StreamChangeCache(
	214	"DeviceFederationOutboxStreamChangeCache",
	215	min_device_outbox_id,
	216	prefilled_cache=device_outbox_prefill,
	217	)
	218
	219	device_list_max = self._device_list_id_gen.get_current_token()
	220	self._device_list_stream_cache = StreamChangeCache(
	221	"DeviceListStreamChangeCache", device_list_max
	222	)
	223	self._user_signature_stream_cache = StreamChangeCache(
	224	"UserSignatureStreamChangeCache", device_list_max
	225	)
	226	self._device_list_federation_stream_cache = StreamChangeCache(
	227	"DeviceListFederationStreamChangeCache", device_list_max
	228	)
	229
	230	events_max = self._stream_id_gen.get_current_token()
	231	curr_state_delta_prefill, min_curr_state_delta_id = self.db_pool.get_cache_dict(
	232	db_conn,
	233	"current_state_delta_stream",
	234	entity_column="room_id",
	235	stream_column="stream_id",
	236	max_value=events_max, # As we share the stream id with events token
	237	limit=1000,
	238	)
	239	self._curr_state_delta_stream_cache = StreamChangeCache(
	240	"_curr_state_delta_stream_cache",
	241	min_curr_state_delta_id,
	242	prefilled_cache=curr_state_delta_prefill,
	243	)
	244
	245	_group_updates_prefill, min_group_updates_id = self.db_pool.get_cache_dict(
	246	db_conn,
	247	"local_group_updates",
	248	entity_column="user_id",
	249	stream_column="stream_id",
	250	max_value=self._group_updates_id_gen.get_current_token(),
	251	limit=1000,
	252	)
	253	self._group_updates_stream_cache = StreamChangeCache(
	254	"_group_updates_stream_cache",
	255	min_group_updates_id,
	256	prefilled_cache=_group_updates_prefill,
	257	)
	258
	259	self._stream_order_on_start = self.get_room_max_stream_ordering()
	260	self._min_stream_order_on_start = self.get_room_min_stream_ordering()
	261
	262	# Used in _generate_user_daily_visits to keep track of progress
	263	self._last_user_visit_update = self._get_start_of_day()
	264
	265	def take_presence_startup_info(self):
	266	active_on_startup = self._presence_on_startup
	267	self._presence_on_startup = None
	268	return active_on_startup
	269
	270	def _get_active_presence(self, db_conn):
	271	"""Fetch non-offline presence from the database so that we can register
	272	the appropriate time outs.
	273	"""
	274
	275	sql = (
	276	"SELECT user_id, state, last_active_ts, last_federation_update_ts,"
	277	" last_user_sync_ts, status_msg, currently_active FROM presence_stream"
	278	" WHERE state != ?"
	279	)
	280	sql = self.database_engine.convert_param_style(sql)
	281
	282	txn = db_conn.cursor()
	283	txn.execute(sql, (PresenceState.OFFLINE,))
	284	rows = self.db_pool.cursor_to_dict(txn)
	285	txn.close()
	286
	287	for row in rows:
	288	row["currently_active"] = bool(row["currently_active"])
	289
	290	return [UserPresenceState(**row) for row in rows]
	291
	292	def count_daily_users(self):
	293	"""
	294	Counts the number of users who used this homeserver in the last 24 hours.
	295	"""
	296	yesterday = int(self._clock.time_msec()) - (1000 * 60 * 60 * 24)
	297	return self.db_pool.runInteraction(
	298	"count_daily_users", self._count_users, yesterday
	299	)
	300
	301	def count_monthly_users(self):
	302	"""
	303	Counts the number of users who used this homeserver in the last 30 days.
	304	Note this method is intended for phonehome metrics only and is different
	305	from the mau figure in synapse.storage.monthly_active_users which,
	306	amongst other things, includes a 3 day grace period before a user counts.
	307	"""
	308	thirty_days_ago = int(self._clock.time_msec()) - (1000 * 60 * 60 * 24 * 30)
	309	return self.db_pool.runInteraction(
	310	"count_monthly_users", self._count_users, thirty_days_ago
	311	)
	312
	313	def _count_users(self, txn, time_from):
	314	"""
	315	Returns number of users seen in the past time_from period
	316	"""
	317	sql = """
	318	SELECT COALESCE(count(*), 0) FROM (
	319	SELECT user_id FROM user_ips
	320	WHERE last_seen > ?
	321	GROUP BY user_id
	322	) u
	323	"""
	324	txn.execute(sql, (time_from,))
	325	(count,) = txn.fetchone()
	326	return count
	327
	328	def count_r30_users(self):
	329	"""
	330	Counts the number of 30 day retained users, defined as:-
	331	* Users who have created their accounts more than 30 days ago
	332	* Where last seen at most 30 days ago
	333	* Where account creation and last_seen are > 30 days apart
	334
	335	Returns counts globaly for a given user as well as breaking
	336	by platform
	337	"""
	338
	339	def _count_r30_users(txn):
	340	thirty_days_in_secs = 86400 * 30
	341	now = int(self._clock.time())
	342	thirty_days_ago_in_secs = now - thirty_days_in_secs
	343
	344	sql = """
	345	SELECT platform, COALESCE(count(*), 0) FROM (
	346	SELECT
	347	users.name, platform, users.creation_ts * 1000,
	348	MAX(uip.last_seen)
	349	FROM users
	350	INNER JOIN (
	351	SELECT
	352	user_id,
	353	last_seen,
	354	CASE
	355	WHEN user_agent LIKE '%%Android%%' THEN 'android'
	356	WHEN user_agent LIKE '%%iOS%%' THEN 'ios'
	357	WHEN user_agent LIKE '%%Electron%%' THEN 'electron'
	358	WHEN user_agent LIKE '%%Mozilla%%' THEN 'web'
	359	WHEN user_agent LIKE '%%Gecko%%' THEN 'web'
	360	ELSE 'unknown'
	361	END
	362	AS platform
	363	FROM user_ips
	364	) uip
	365	ON users.name = uip.user_id
	366	AND users.appservice_id is NULL
	367	AND users.creation_ts < ?
	368	AND uip.last_seen/1000 > ?
	369	AND (uip.last_seen/1000) - users.creation_ts > 86400 * 30
	370	GROUP BY users.name, platform, users.creation_ts
	371	) u GROUP BY platform
	372	"""
	373
	374	results = {}
	375	txn.execute(sql, (thirty_days_ago_in_secs, thirty_days_ago_in_secs))
	376
	377	for row in txn:
	378	if row[0] == "unknown":
	379	pass
	380	results[row[0]] = row[1]
	381
	382	sql = """
	383	SELECT COALESCE(count(*), 0) FROM (
	384	SELECT users.name, users.creation_ts * 1000,
	385	MAX(uip.last_seen)
	386	FROM users
	387	INNER JOIN (
	388	SELECT
	389	user_id,
	390	last_seen
	391	FROM user_ips
	392	) uip
	393	ON users.name = uip.user_id
	394	AND appservice_id is NULL
	395	AND users.creation_ts < ?
	396	AND uip.last_seen/1000 > ?
	397	AND (uip.last_seen/1000) - users.creation_ts > 86400 * 30
	398	GROUP BY users.name, users.creation_ts
	399	) u
	400	"""
	401
	402	txn.execute(sql, (thirty_days_ago_in_secs, thirty_days_ago_in_secs))
	403
	404	(count,) = txn.fetchone()
	405	results["all"] = count
	406
	407	return results
	408
	409	return self.db_pool.runInteraction("count_r30_users", _count_r30_users)
	410
	411	def _get_start_of_day(self):
	412	"""
	413	Returns millisecond unixtime for start of UTC day.
	414	"""
	415	now = time.gmtime()
	416	today_start = calendar.timegm((now.tm_year, now.tm_mon, now.tm_mday, 0, 0, 0))
	417	return today_start * 1000
	418
	419	def generate_user_daily_visits(self):
	420	"""
	421	Generates daily visit data for use in cohort/ retention analysis
	422	"""
	423
	424	def _generate_user_daily_visits(txn):
	425	logger.info("Calling _generate_user_daily_visits")
	426	today_start = self._get_start_of_day()
	427	a_day_in_milliseconds = 24 * 60 * 60 * 1000
	428	now = self.clock.time_msec()
	429
	430	sql = """
	431	INSERT INTO user_daily_visits (user_id, device_id, timestamp)
	432	SELECT u.user_id, u.device_id, ?
	433	FROM user_ips AS u
	434	LEFT JOIN (
	435	SELECT user_id, device_id, timestamp FROM user_daily_visits
	436	WHERE timestamp = ?
	437	) udv
	438	ON u.user_id = udv.user_id AND u.device_id=udv.device_id
	439	INNER JOIN users ON users.name=u.user_id
	440	WHERE last_seen > ? AND last_seen <= ?
	441	AND udv.timestamp IS NULL AND users.is_guest=0
	442	AND users.appservice_id IS NULL
	443	GROUP BY u.user_id, u.device_id
	444	"""
	445
	446	# This means that the day has rolled over but there could still
	447	# be entries from the previous day. There is an edge case
	448	# where if the user logs in at 23:59 and overwrites their
	449	# last_seen at 00:01 then they will not be counted in the
	450	# previous day's stats - it is important that the query is run
	451	# often to minimise this case.
	452	if today_start > self._last_user_visit_update:
	453	yesterday_start = today_start - a_day_in_milliseconds
	454	txn.execute(
	455	sql,
	456	(
	457	yesterday_start,
	458	yesterday_start,
	459	self._last_user_visit_update,
	460	today_start,
	461	),
	462	)
	463	self._last_user_visit_update = today_start
	464
	465	txn.execute(
	466	sql, (today_start, today_start, self._last_user_visit_update, now)
	467	)
	468	# Update _last_user_visit_update to now. The reason to do this
	469	# rather just clamping to the beginning of the day is to limit
	470	# the size of the join - meaning that the query can be run more
	471	# frequently
	472	self._last_user_visit_update = now
	473
	474	return self.db_pool.runInteraction(
	475	"generate_user_daily_visits", _generate_user_daily_visits
	476	)
	477
	478	def get_users(self):
	479	"""Function to retrieve a list of users in users table.
	480
	481	Args:
	482	Returns:
	483	defer.Deferred: resolves to list[dict[str, Any]]
	484	"""
	485	return self.db_pool.simple_select_list(
	486	table="users",
	487	keyvalues={},
	488	retcols=[
	489	"name",
	490	"password_hash",
	491	"is_guest",
	492	"admin",
	493	"user_type",
	494	"deactivated",
	495	],
	496	desc="get_users",
	497	)
	498
	499	def get_users_paginate(
	500	self, start, limit, name=None, guests=True, deactivated=False
	501	):
	502	"""Function to retrieve a paginated list of users from
	503	users list. This will return a json list of users and the
	504	total number of users matching the filter criteria.
	505
	506	Args:
	507	start (int): start number to begin the query from
	508	limit (int): number of rows to retrieve
	509	name (string): filter for user names
	510	guests (bool): whether to in include guest users
	511	deactivated (bool): whether to include deactivated users
	512	Returns:
	513	defer.Deferred: resolves to list[dict[str, Any]], int
	514	"""
	515
	516	def get_users_paginate_txn(txn):
	517	filters = []
	518	args = []
	519
	520	if name:
	521	filters.append("name LIKE ?")
	522	args.append("%" + name + "%")
	523
	524	if not guests:
	525	filters.append("is_guest = 0")
	526
	527	if not deactivated:
	528	filters.append("deactivated = 0")
	529
	530	where_clause = "WHERE " + " AND ".join(filters) if len(filters) > 0 else ""
	531
	532	sql = "SELECT COUNT(*) as total_users FROM users %s" % (where_clause)
	533	txn.execute(sql, args)
	534	count = txn.fetchone()[0]
	535
	536	args = [self.hs.config.server_name] + args + [limit, start]
	537	sql = """
	538	SELECT name, user_type, is_guest, admin, deactivated, displayname, avatar_url
	539	FROM users as u
	540	LEFT JOIN profiles AS p ON u.name = '@' \|\| p.user_id \|\| ':' \|\| ?
	541	{}
	542	ORDER BY u.name LIMIT ? OFFSET ?
	543	""".format(
	544	where_clause
	545	)
	546	txn.execute(sql, args)
	547	users = self.db_pool.cursor_to_dict(txn)
	548	return users, count
	549
	550	return self.db_pool.runInteraction(
	551	"get_users_paginate_txn", get_users_paginate_txn
	552	)
	553
	554	def search_users(self, term):
	555	"""Function to search users list for one or more users with
	556	the matched term.
	557
	558	Args:
	559	term (str): search term
	560	col (str): column to query term should be matched to
	561	Returns:
	562	defer.Deferred: resolves to list[dict[str, Any]]
	563	"""
	564	return self.db_pool.simple_search_list(
	565	table="users",
	566	term=term,
	567	col="name",
	568	retcols=["name", "password_hash", "is_guest", "admin", "user_type"],
	569	desc="search_users",
	570	)
	571
	572
	573	def check_database_before_upgrade(cur, database_engine, config: HomeServerConfig):
	574	"""Called before upgrading an existing database to check that it is broadly sane
	575	compared with the configuration.
	576	"""
	577	domain = config.server_name
	578
	579	sql = database_engine.convert_param_style(
	580	"SELECT COUNT(*) FROM users WHERE name NOT LIKE ?"
	581	)
	582	pat = "%:" + domain
	583	cur.execute(sql, (pat,))
	584	num_not_matching = cur.fetchall()[0][0]
	585	if num_not_matching == 0:
	586	return
	587
	588	raise Exception(
	589	"Found users in database not native to %s!\n"
	590	"You cannot changed a synapse server_name after it's been configured"
	591	% (domain,)
	592	)
	593
	594
	595	__all__ = ["DataStore", "check_database_before_upgrade"]

+438

-0

synapse/storage/databases/main/account_data.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2018 New Vector Ltd
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import abc
	17	import logging
	18	from typing import List, Optional, Tuple
	19
	20	from twisted.internet import defer
	21
	22	from synapse.storage._base import SQLBaseStore, db_to_json
	23	from synapse.storage.database import DatabasePool
	24	from synapse.storage.util.id_generators import StreamIdGenerator
	25	from synapse.types import JsonDict
	26	from synapse.util import json_encoder
	27	from synapse.util.caches.descriptors import _CacheContext, cached
	28	from synapse.util.caches.stream_change_cache import StreamChangeCache
	29
	30	logger = logging.getLogger(__name__)
	31
	32
	33	class AccountDataWorkerStore(SQLBaseStore):
	34	"""This is an abstract base class where subclasses must implement
	35	`get_max_account_data_stream_id` which can be called in the initializer.
	36	"""
	37
	38	# This ABCMeta metaclass ensures that we cannot be instantiated without
	39	# the abstract methods being implemented.
	40	__metaclass__ = abc.ABCMeta
	41
	42	def __init__(self, database: DatabasePool, db_conn, hs):
	43	account_max = self.get_max_account_data_stream_id()
	44	self._account_data_stream_cache = StreamChangeCache(
	45	"AccountDataAndTagsChangeCache", account_max
	46	)
	47
	48	super(AccountDataWorkerStore, self).__init__(database, db_conn, hs)
	49
	50	@abc.abstractmethod
	51	def get_max_account_data_stream_id(self):
	52	"""Get the current max stream ID for account data stream
	53
	54	Returns:
	55	int
	56	"""
	57	raise NotImplementedError()
	58
	59	@cached()
	60	def get_account_data_for_user(self, user_id):
	61	"""Get all the client account_data for a user.
	62
	63	Args:
	64	user_id(str): The user to get the account_data for.
	65	Returns:
	66	A deferred pair of a dict of global account_data and a dict
	67	mapping from room_id string to per room account_data dicts.
	68	"""
	69
	70	def get_account_data_for_user_txn(txn):
	71	rows = self.db_pool.simple_select_list_txn(
	72	txn,
	73	"account_data",
	74	{"user_id": user_id},
	75	["account_data_type", "content"],
	76	)
	77
	78	global_account_data = {
	79	row["account_data_type"]: db_to_json(row["content"]) for row in rows
	80	}
	81
	82	rows = self.db_pool.simple_select_list_txn(
	83	txn,
	84	"room_account_data",
	85	{"user_id": user_id},
	86	["room_id", "account_data_type", "content"],
	87	)
	88
	89	by_room = {}
	90	for row in rows:
	91	room_data = by_room.setdefault(row["room_id"], {})
	92	room_data[row["account_data_type"]] = db_to_json(row["content"])
	93
	94	return global_account_data, by_room
	95
	96	return self.db_pool.runInteraction(
	97	"get_account_data_for_user", get_account_data_for_user_txn
	98	)
	99
	100	@cached(num_args=2, max_entries=5000)
	101	async def get_global_account_data_by_type_for_user(
	102	self, data_type: str, user_id: str
	103	) -> Optional[JsonDict]:
	104	"""
	105	Returns:
	106	The account data.
	107	"""
	108	result = await self.db_pool.simple_select_one_onecol(
	109	table="account_data",
	110	keyvalues={"user_id": user_id, "account_data_type": data_type},
	111	retcol="content",
	112	desc="get_global_account_data_by_type_for_user",
	113	allow_none=True,
	114	)
	115
	116	if result:
	117	return db_to_json(result)
	118	else:
	119	return None
	120
	121	@cached(num_args=2)
	122	def get_account_data_for_room(self, user_id, room_id):
	123	"""Get all the client account_data for a user for a room.
	124
	125	Args:
	126	user_id(str): The user to get the account_data for.
	127	room_id(str): The room to get the account_data for.
	128	Returns:
	129	A deferred dict of the room account_data
	130	"""
	131
	132	def get_account_data_for_room_txn(txn):
	133	rows = self.db_pool.simple_select_list_txn(
	134	txn,
	135	"room_account_data",
	136	{"user_id": user_id, "room_id": room_id},
	137	["account_data_type", "content"],
	138	)
	139
	140	return {
	141	row["account_data_type"]: db_to_json(row["content"]) for row in rows
	142	}
	143
	144	return self.db_pool.runInteraction(
	145	"get_account_data_for_room", get_account_data_for_room_txn
	146	)
	147
	148	@cached(num_args=3, max_entries=5000)
	149	def get_account_data_for_room_and_type(self, user_id, room_id, account_data_type):
	150	"""Get the client account_data of given type for a user for a room.
	151
	152	Args:
	153	user_id(str): The user to get the account_data for.
	154	room_id(str): The room to get the account_data for.
	155	account_data_type (str): The account data type to get.
	156	Returns:
	157	A deferred of the room account_data for that type, or None if
	158	there isn't any set.
	159	"""
	160
	161	def get_account_data_for_room_and_type_txn(txn):
	162	content_json = self.db_pool.simple_select_one_onecol_txn(
	163	txn,
	164	table="room_account_data",
	165	keyvalues={
	166	"user_id": user_id,
	167	"room_id": room_id,
	168	"account_data_type": account_data_type,
	169	},
	170	retcol="content",
	171	allow_none=True,
	172	)
	173
	174	return db_to_json(content_json) if content_json else None
	175
	176	return self.db_pool.runInteraction(
	177	"get_account_data_for_room_and_type", get_account_data_for_room_and_type_txn
	178	)
	179
	180	async def get_updated_global_account_data(
	181	self, last_id: int, current_id: int, limit: int
	182	) -> List[Tuple[int, str, str]]:
	183	"""Get the global account_data that has changed, for the account_data stream
	184
	185	Args:
	186	last_id: the last stream_id from the previous batch.
	187	current_id: the maximum stream_id to return up to
	188	limit: the maximum number of rows to return
	189
	190	Returns:
	191	A list of tuples of stream_id int, user_id string,
	192	and type string.
	193	"""
	194	if last_id == current_id:
	195	return []
	196
	197	def get_updated_global_account_data_txn(txn):
	198	sql = (
	199	"SELECT stream_id, user_id, account_data_type"
	200	" FROM account_data WHERE ? < stream_id AND stream_id <= ?"
	201	" ORDER BY stream_id ASC LIMIT ?"
	202	)
	203	txn.execute(sql, (last_id, current_id, limit))
	204	return txn.fetchall()
	205
	206	return await self.db_pool.runInteraction(
	207	"get_updated_global_account_data", get_updated_global_account_data_txn
	208	)
	209
	210	async def get_updated_room_account_data(
	211	self, last_id: int, current_id: int, limit: int
	212	) -> List[Tuple[int, str, str, str]]:
	213	"""Get the global account_data that has changed, for the account_data stream
	214
	215	Args:
	216	last_id: the last stream_id from the previous batch.
	217	current_id: the maximum stream_id to return up to
	218	limit: the maximum number of rows to return
	219
	220	Returns:
	221	A list of tuples of stream_id int, user_id string,
	222	room_id string and type string.
	223	"""
	224	if last_id == current_id:
	225	return []
	226
	227	def get_updated_room_account_data_txn(txn):
	228	sql = (
	229	"SELECT stream_id, user_id, room_id, account_data_type"
	230	" FROM room_account_data WHERE ? < stream_id AND stream_id <= ?"
	231	" ORDER BY stream_id ASC LIMIT ?"
	232	)
	233	txn.execute(sql, (last_id, current_id, limit))
	234	return txn.fetchall()
	235
	236	return await self.db_pool.runInteraction(
	237	"get_updated_room_account_data", get_updated_room_account_data_txn
	238	)
	239
	240	def get_updated_account_data_for_user(self, user_id, stream_id):
	241	"""Get all the client account_data for a that's changed for a user
	242
	243	Args:
	244	user_id(str): The user to get the account_data for.
	245	stream_id(int): The point in the stream since which to get updates
	246	Returns:
	247	A deferred pair of a dict of global account_data and a dict
	248	mapping from room_id string to per room account_data dicts.
	249	"""
	250
	251	def get_updated_account_data_for_user_txn(txn):
	252	sql = (
	253	"SELECT account_data_type, content FROM account_data"
	254	" WHERE user_id = ? AND stream_id > ?"
	255	)
	256
	257	txn.execute(sql, (user_id, stream_id))
	258
	259	global_account_data = {row[0]: db_to_json(row[1]) for row in txn}
	260
	261	sql = (
	262	"SELECT room_id, account_data_type, content FROM room_account_data"
	263	" WHERE user_id = ? AND stream_id > ?"
	264	)
	265
	266	txn.execute(sql, (user_id, stream_id))
	267
	268	account_data_by_room = {}
	269	for row in txn:
	270	room_account_data = account_data_by_room.setdefault(row[0], {})
	271	room_account_data[row[1]] = db_to_json(row[2])
	272
	273	return global_account_data, account_data_by_room
	274
	275	changed = self._account_data_stream_cache.has_entity_changed(
	276	user_id, int(stream_id)
	277	)
	278	if not changed:
	279	return defer.succeed(({}, {}))
	280
	281	return self.db_pool.runInteraction(
	282	"get_updated_account_data_for_user", get_updated_account_data_for_user_txn
	283	)
	284
	285	@cached(num_args=2, cache_context=True, max_entries=5000)
	286	async def is_ignored_by(
	287	self, ignored_user_id: str, ignorer_user_id: str, cache_context: _CacheContext
	288	) -> bool:
	289	ignored_account_data = await self.get_global_account_data_by_type_for_user(
	290	"m.ignored_user_list",
	291	ignorer_user_id,
	292	on_invalidate=cache_context.invalidate,
	293	)
	294	if not ignored_account_data:
	295	return False
	296
	297	return ignored_user_id in ignored_account_data.get("ignored_users", {})
	298
	299
	300	class AccountDataStore(AccountDataWorkerStore):
	301	def __init__(self, database: DatabasePool, db_conn, hs):
	302	self._account_data_id_gen = StreamIdGenerator(
	303	db_conn,
	304	"account_data_max_stream_id",
	305	"stream_id",
	306	extra_tables=[
	307	("room_account_data", "stream_id"),
	308	("room_tags_revisions", "stream_id"),
	309	],
	310	)
	311
	312	super(AccountDataStore, self).__init__(database, db_conn, hs)
	313
	314	def get_max_account_data_stream_id(self) -> int:
	315	"""Get the current max stream id for the private user data stream
	316
	317	Returns:
	318	The maximum stream ID.
	319	"""
	320	return self._account_data_id_gen.get_current_token()
	321
	322	async def add_account_data_to_room(
	323	self, user_id: str, room_id: str, account_data_type: str, content: JsonDict
	324	) -> int:
	325	"""Add some account_data to a room for a user.
	326
	327	Args:
	328	user_id: The user to add a tag for.
	329	room_id: The room to add a tag for.
	330	account_data_type: The type of account_data to add.
	331	content: A json object to associate with the tag.
	332
	333	Returns:
	334	The maximum stream ID.
	335	"""
	336	content_json = json_encoder.encode(content)
	337
	338	with self._account_data_id_gen.get_next() as next_id:
	339	# no need to lock here as room_account_data has a unique constraint
	340	# on (user_id, room_id, account_data_type) so simple_upsert will
	341	# retry if there is a conflict.
	342	await self.db_pool.simple_upsert(
	343	desc="add_room_account_data",
	344	table="room_account_data",
	345	keyvalues={
	346	"user_id": user_id,
	347	"room_id": room_id,
	348	"account_data_type": account_data_type,
	349	},
	350	values={"stream_id": next_id, "content": content_json},
	351	lock=False,
	352	)
	353
	354	# it's theoretically possible for the above to succeed and the
	355	# below to fail - in which case we might reuse a stream id on
	356	# restart, and the above update might not get propagated. That
	357	# doesn't sound any worse than the whole update getting lost,
	358	# which is what would happen if we combined the two into one
	359	# transaction.
	360	await self._update_max_stream_id(next_id)
	361
	362	self._account_data_stream_cache.entity_has_changed(user_id, next_id)
	363	self.get_account_data_for_user.invalidate((user_id,))
	364	self.get_account_data_for_room.invalidate((user_id, room_id))
	365	self.get_account_data_for_room_and_type.prefill(
	366	(user_id, room_id, account_data_type), content
	367	)
	368
	369	return self._account_data_id_gen.get_current_token()
	370
	371	async def add_account_data_for_user(
	372	self, user_id: str, account_data_type: str, content: JsonDict
	373	) -> int:
	374	"""Add some account_data to a room for a user.
	375
	376	Args:
	377	user_id: The user to add a tag for.
	378	account_data_type: The type of account_data to add.
	379	content: A json object to associate with the tag.
	380
	381	Returns:
	382	The maximum stream ID.
	383	"""
	384	content_json = json_encoder.encode(content)
	385
	386	with self._account_data_id_gen.get_next() as next_id:
	387	# no need to lock here as account_data has a unique constraint on
	388	# (user_id, account_data_type) so simple_upsert will retry if
	389	# there is a conflict.
	390	await self.db_pool.simple_upsert(
	391	desc="add_user_account_data",
	392	table="account_data",
	393	keyvalues={"user_id": user_id, "account_data_type": account_data_type},
	394	values={"stream_id": next_id, "content": content_json},
	395	lock=False,
	396	)
	397
	398	# it's theoretically possible for the above to succeed and the
	399	# below to fail - in which case we might reuse a stream id on
	400	# restart, and the above update might not get propagated. That
	401	# doesn't sound any worse than the whole update getting lost,
	402	# which is what would happen if we combined the two into one
	403	# transaction.
	404	#
	405	# Note: This is only here for backwards compat to allow admins to
	406	# roll back to a previous Synapse version. Next time we update the
	407	# database version we can remove this table.
	408	await self._update_max_stream_id(next_id)
	409
	410	self._account_data_stream_cache.entity_has_changed(user_id, next_id)
	411	self.get_account_data_for_user.invalidate((user_id,))
	412	self.get_global_account_data_by_type_for_user.invalidate(
	413	(account_data_type, user_id)
	414	)
	415
	416	return self._account_data_id_gen.get_current_token()
	417
	418	def _update_max_stream_id(self, next_id: int):
	419	"""Update the max stream_id
	420
	421	Args:
	422	next_id: The the revision to advance to.
	423	"""
	424
	425	# Note: This is only here for backwards compat to allow admins to
	426	# roll back to a previous Synapse version. Next time we update the
	427	# database version we can remove this table.
	428
	429	def _update(txn):
	430	update_max_id_sql = (
	431	"UPDATE account_data_max_stream_id"
	432	" SET stream_id = ?"
	433	" WHERE stream_id < ?"
	434	)
	435	txn.execute(update_max_id_sql, (next_id, next_id))
	436
	437	return self.db_pool.runInteraction("update_account_data_max_stream_id", _update)

+366

-0

synapse/storage/databases/main/appservice.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2015, 2016 OpenMarket Ltd
	2	# Copyright 2018 New Vector Ltd
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15	import logging
	16	import re
	17
	18	from canonicaljson import json
	19
	20	from synapse.appservice import AppServiceTransaction
	21	from synapse.config.appservice import load_appservices
	22	from synapse.storage._base import SQLBaseStore, db_to_json
	23	from synapse.storage.database import DatabasePool
	24	from synapse.storage.databases.main.events_worker import EventsWorkerStore
	25
	26	logger = logging.getLogger(__name__)
	27
	28
	29	def _make_exclusive_regex(services_cache):
	30	# We precompile a regex constructed from all the regexes that the AS's
	31	# have registered for exclusive users.
	32	exclusive_user_regexes = [
	33	regex.pattern
	34	for service in services_cache
	35	for regex in service.get_exclusive_user_regexes()
	36	]
	37	if exclusive_user_regexes:
	38	exclusive_user_regex = "\|".join("(" + r + ")" for r in exclusive_user_regexes)
	39	exclusive_user_regex = re.compile(exclusive_user_regex)
	40	else:
	41	# We handle this case specially otherwise the constructed regex
	42	# will always match
	43	exclusive_user_regex = None
	44
	45	return exclusive_user_regex
	46
	47
	48	class ApplicationServiceWorkerStore(SQLBaseStore):
	49	def __init__(self, database: DatabasePool, db_conn, hs):
	50	self.services_cache = load_appservices(
	51	hs.hostname, hs.config.app_service_config_files
	52	)
	53	self.exclusive_user_regex = _make_exclusive_regex(self.services_cache)
	54
	55	super(ApplicationServiceWorkerStore, self).__init__(database, db_conn, hs)
	56
	57	def get_app_services(self):
	58	return self.services_cache
	59
	60	def get_if_app_services_interested_in_user(self, user_id):
	61	"""Check if the user is one associated with an app service (exclusively)
	62	"""
	63	if self.exclusive_user_regex:
	64	return bool(self.exclusive_user_regex.match(user_id))
	65	else:
	66	return False
	67
	68	def get_app_service_by_user_id(self, user_id):
	69	"""Retrieve an application service from their user ID.
	70
	71	All application services have associated with them a particular user ID.
	72	There is no distinguishing feature on the user ID which indicates it
	73	represents an application service. This function allows you to map from
	74	a user ID to an application service.
	75
	76	Args:
	77	user_id(str): The user ID to see if it is an application service.
	78	Returns:
	79	synapse.appservice.ApplicationService or None.
	80	"""
	81	for service in self.services_cache:
	82	if service.sender == user_id:
	83	return service
	84	return None
	85
	86	def get_app_service_by_token(self, token):
	87	"""Get the application service with the given appservice token.
	88
	89	Args:
	90	token (str): The application service token.
	91	Returns:
	92	synapse.appservice.ApplicationService or None.
	93	"""
	94	for service in self.services_cache:
	95	if service.token == token:
	96	return service
	97	return None
	98
	99	def get_app_service_by_id(self, as_id):
	100	"""Get the application service with the given appservice ID.
	101
	102	Args:
	103	as_id (str): The application service ID.
	104	Returns:
	105	synapse.appservice.ApplicationService or None.
	106	"""
	107	for service in self.services_cache:
	108	if service.id == as_id:
	109	return service
	110	return None
	111
	112
	113	class ApplicationServiceStore(ApplicationServiceWorkerStore):
	114	# This is currently empty due to there not being any AS storage functions
	115	# that can't be run on the workers. Since this may change in future, and
	116	# to keep consistency with the other stores, we keep this empty class for
	117	# now.
	118	pass
	119
	120
	121	class ApplicationServiceTransactionWorkerStore(
	122	ApplicationServiceWorkerStore, EventsWorkerStore
	123	):
	124	async def get_appservices_by_state(self, state):
	125	"""Get a list of application services based on their state.
	126
	127	Args:
	128	state(ApplicationServiceState): The state to filter on.
	129	Returns:
	130	A list of ApplicationServices, which may be empty.
	131	"""
	132	results = await self.db_pool.simple_select_list(
	133	"application_services_state", {"state": state}, ["as_id"]
	134	)
	135	# NB: This assumes this class is linked with ApplicationServiceStore
	136	as_list = self.get_app_services()
	137	services = []
	138
	139	for res in results:
	140	for service in as_list:
	141	if service.id == res["as_id"]:
	142	services.append(service)
	143	return services
	144
	145	async def get_appservice_state(self, service):
	146	"""Get the application service state.
	147
	148	Args:
	149	service(ApplicationService): The service whose state to set.
	150	Returns:
	151	An ApplicationServiceState.
	152	"""
	153	result = await self.db_pool.simple_select_one(
	154	"application_services_state",
	155	{"as_id": service.id},
	156	["state"],
	157	allow_none=True,
	158	desc="get_appservice_state",
	159	)
	160	if result:
	161	return result.get("state")
	162	return None
	163
	164	def set_appservice_state(self, service, state):
	165	"""Set the application service state.
	166
	167	Args:
	168	service(ApplicationService): The service whose state to set.
	169	state(ApplicationServiceState): The connectivity state to apply.
	170	Returns:
	171	A Deferred which resolves when the state was set successfully.
	172	"""
	173	return self.db_pool.simple_upsert(
	174	"application_services_state", {"as_id": service.id}, {"state": state}
	175	)
	176
	177	def create_appservice_txn(self, service, events):
	178	"""Atomically creates a new transaction for this application service
	179	with the given list of events.
	180
	181	Args:
	182	service(ApplicationService): The service who the transaction is for.
	183	events(list<Event>): A list of events to put in the transaction.
	184	Returns:
	185	AppServiceTransaction: A new transaction.
	186	"""
	187
	188	def _create_appservice_txn(txn):
	189	# work out new txn id (highest txn id for this service += 1)
	190	# The highest id may be the last one sent (in which case it is last_txn)
	191	# or it may be the highest in the txns list (which are waiting to be/are
	192	# being sent)
	193	last_txn_id = self._get_last_txn(txn, service.id)
	194
	195	txn.execute(
	196	"SELECT MAX(txn_id) FROM application_services_txns WHERE as_id=?",
	197	(service.id,),
	198	)
	199	highest_txn_id = txn.fetchone()[0]
	200	if highest_txn_id is None:
	201	highest_txn_id = 0
	202
	203	new_txn_id = max(highest_txn_id, last_txn_id) + 1
	204
	205	# Insert new txn into txn table
	206	event_ids = json.dumps([e.event_id for e in events])
	207	txn.execute(
	208	"INSERT INTO application_services_txns(as_id, txn_id, event_ids) "
	209	"VALUES(?,?,?)",
	210	(service.id, new_txn_id, event_ids),
	211	)
	212	return AppServiceTransaction(service=service, id=new_txn_id, events=events)
	213
	214	return self.db_pool.runInteraction(
	215	"create_appservice_txn", _create_appservice_txn
	216	)
	217
	218	def complete_appservice_txn(self, txn_id, service):
	219	"""Completes an application service transaction.
	220
	221	Args:
	222	txn_id(str): The transaction ID being completed.
	223	service(ApplicationService): The application service which was sent
	224	this transaction.
	225	Returns:
	226	A Deferred which resolves if this transaction was stored
	227	successfully.
	228	"""
	229	txn_id = int(txn_id)
	230
	231	def _complete_appservice_txn(txn):
	232	# Debugging query: Make sure the txn being completed is EXACTLY +1 from
	233	# what was there before. If it isn't, we've got problems (e.g. the AS
	234	# has probably missed some events), so whine loudly but still continue,
	235	# since it shouldn't fail completion of the transaction.
	236	last_txn_id = self._get_last_txn(txn, service.id)
	237	if (last_txn_id + 1) != txn_id:
	238	logger.error(
	239	"appservice: Completing a transaction which has an ID > 1 from "
	240	"the last ID sent to this AS. We've either dropped events or "
	241	"sent it to the AS out of order. FIX ME. last_txn=%s "
	242	"completing_txn=%s service_id=%s",
	243	last_txn_id,
	244	txn_id,
	245	service.id,
	246	)
	247
	248	# Set current txn_id for AS to 'txn_id'
	249	self.db_pool.simple_upsert_txn(
	250	txn,
	251	"application_services_state",
	252	{"as_id": service.id},
	253	{"last_txn": txn_id},
	254	)
	255
	256	# Delete txn
	257	self.db_pool.simple_delete_txn(
	258	txn,
	259	"application_services_txns",
	260	{"txn_id": txn_id, "as_id": service.id},
	261	)
	262
	263	return self.db_pool.runInteraction(
	264	"complete_appservice_txn", _complete_appservice_txn
	265	)
	266
	267	async def get_oldest_unsent_txn(self, service):
	268	"""Get the oldest transaction which has not been sent for this
	269	service.
	270
	271	Args:
	272	service(ApplicationService): The app service to get the oldest txn.
	273	Returns:
	274	An AppServiceTransaction or None.
	275	"""
	276
	277	def _get_oldest_unsent_txn(txn):
	278	# Monotonically increasing txn ids, so just select the smallest
	279	# one in the txns table (we delete them when they are sent)
	280	txn.execute(
	281	"SELECT * FROM application_services_txns WHERE as_id=?"
	282	" ORDER BY txn_id ASC LIMIT 1",
	283	(service.id,),
	284	)
	285	rows = self.db_pool.cursor_to_dict(txn)
	286	if not rows:
	287	return None
	288
	289	entry = rows[0]
	290
	291	return entry
	292
	293	entry = await self.db_pool.runInteraction(
	294	"get_oldest_unsent_appservice_txn", _get_oldest_unsent_txn
	295	)
	296
	297	if not entry:
	298	return None
	299
	300	event_ids = db_to_json(entry["event_ids"])
	301
	302	events = await self.get_events_as_list(event_ids)
	303
	304	return AppServiceTransaction(service=service, id=entry["txn_id"], events=events)
	305
	306	def _get_last_txn(self, txn, service_id):
	307	txn.execute(
	308	"SELECT last_txn FROM application_services_state WHERE as_id=?",
	309	(service_id,),
	310	)
	311	last_txn_id = txn.fetchone()
	312	if last_txn_id is None or last_txn_id[0] is None: # no row exists
	313	return 0
	314	else:
	315	return int(last_txn_id[0]) # select 'last_txn' col
	316
	317	def set_appservice_last_pos(self, pos):
	318	def set_appservice_last_pos_txn(txn):
	319	txn.execute(
	320	"UPDATE appservice_stream_position SET stream_ordering = ?", (pos,)
	321	)
	322
	323	return self.db_pool.runInteraction(
	324	"set_appservice_last_pos", set_appservice_last_pos_txn
	325	)
	326
	327	async def get_new_events_for_appservice(self, current_id, limit):
	328	"""Get all new evnets"""
	329
	330	def get_new_events_for_appservice_txn(txn):
	331	sql = (
	332	"SELECT e.stream_ordering, e.event_id"
	333	" FROM events AS e"
	334	" WHERE"
	335	" (SELECT stream_ordering FROM appservice_stream_position)"
	336	" < e.stream_ordering"
	337	" AND e.stream_ordering <= ?"
	338	" ORDER BY e.stream_ordering ASC"
	339	" LIMIT ?"
	340	)
	341
	342	txn.execute(sql, (current_id, limit))
	343	rows = txn.fetchall()
	344
	345	upper_bound = current_id
	346	if len(rows) == limit:
	347	upper_bound = rows[-1][0]
	348
	349	return upper_bound, [row[1] for row in rows]
	350
	351	upper_bound, event_ids = await self.db_pool.runInteraction(
	352	"get_new_events_for_appservice", get_new_events_for_appservice_txn
	353	)
	354
	355	events = await self.get_events_as_list(event_ids)
	356
	357	return upper_bound, events
	358
	359
	360	class ApplicationServiceTransactionStore(ApplicationServiceTransactionWorkerStore):
	361	# This is currently empty due to there not being any AS storage functions
	362	# that can't be run on the workers. Since this may change in future, and
	363	# to keep consistency with the other stores, we keep this empty class for
	364	# now.
	365	pass

+306

-0

synapse/storage/databases/main/cache.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2019 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15
	16	import itertools
	17	import logging
	18	from typing import Any, Iterable, List, Optional, Tuple
	19
	20	from synapse.api.constants import EventTypes
	21	from synapse.replication.tcp.streams import BackfillStream, CachesStream
	22	from synapse.replication.tcp.streams.events import (
	23	EventsStream,
	24	EventsStreamCurrentStateRow,
	25	EventsStreamEventRow,
	26	)
	27	from synapse.storage._base import SQLBaseStore
	28	from synapse.storage.database import DatabasePool
	29	from synapse.storage.engines import PostgresEngine
	30	from synapse.util.iterutils import batch_iter
	31
	32	logger = logging.getLogger(__name__)
	33
	34
	35	# This is a special cache name we use to batch multiple invalidations of caches
	36	# based on the current state when notifying workers over replication.
	37	CURRENT_STATE_CACHE_NAME = "cs_cache_fake"
	38
	39
	40	class CacheInvalidationWorkerStore(SQLBaseStore):
	41	def __init__(self, database: DatabasePool, db_conn, hs):
	42	super().__init__(database, db_conn, hs)
	43
	44	self._instance_name = hs.get_instance_name()
	45
	46	async def get_all_updated_caches(
	47	self, instance_name: str, last_id: int, current_id: int, limit: int
	48	) -> Tuple[List[Tuple[int, tuple]], int, bool]:
	49	"""Get updates for caches replication stream.
	50
	51	Args:
	52	instance_name: The writer we want to fetch updates from. Unused
	53	here since there is only ever one writer.
	54	last_id: The token to fetch updates from. Exclusive.
	55	current_id: The token to fetch updates up to. Inclusive.
	56	limit: The requested limit for the number of rows to return. The
	57	function may return more or fewer rows.
	58
	59	Returns:
	60	A tuple consisting of: the updates, a token to use to fetch
	61	subsequent updates, and whether we returned fewer rows than exists
	62	between the requested tokens due to the limit.
	63
	64	The token returned can be used in a subsequent call to this
	65	function to get further updatees.
	66
	67	The updates are a list of 2-tuples of stream ID and the row data
	68	"""
	69
	70	if last_id == current_id:
	71	return [], current_id, False
	72
	73	def get_all_updated_caches_txn(txn):
	74	# We purposefully don't bound by the current token, as we want to
	75	# send across cache invalidations as quickly as possible. Cache
	76	# invalidations are idempotent, so duplicates are fine.
	77	sql = """
	78	SELECT stream_id, cache_func, keys, invalidation_ts
	79	FROM cache_invalidation_stream_by_instance
	80	WHERE stream_id > ? AND instance_name = ?
	81	ORDER BY stream_id ASC
	82	LIMIT ?
	83	"""
	84	txn.execute(sql, (last_id, instance_name, limit))
	85	updates = [(row[0], row[1:]) for row in txn]
	86	limited = False
	87	upto_token = current_id
	88	if len(updates) >= limit:
	89	upto_token = updates[-1][0]
	90	limited = True
	91
	92	return updates, upto_token, limited
	93
	94	return await self.db_pool.runInteraction(
	95	"get_all_updated_caches", get_all_updated_caches_txn
	96	)
	97
	98	def process_replication_rows(self, stream_name, instance_name, token, rows):
	99	if stream_name == EventsStream.NAME:
	100	for row in rows:
	101	self._process_event_stream_row(token, row)
	102	elif stream_name == BackfillStream.NAME:
	103	for row in rows:
	104	self._invalidate_caches_for_event(
	105	-token,
	106	row.event_id,
	107	row.room_id,
	108	row.type,
	109	row.state_key,
	110	row.redacts,
	111	row.relates_to,
	112	backfilled=True,
	113	)
	114	elif stream_name == CachesStream.NAME:
	115	if self._cache_id_gen:
	116	self._cache_id_gen.advance(instance_name, token)
	117
	118	for row in rows:
	119	if row.cache_func == CURRENT_STATE_CACHE_NAME:
	120	if row.keys is None:
	121	raise Exception(
	122	"Can't send an 'invalidate all' for current state cache"
	123	)
	124
	125	room_id = row.keys[0]
	126	members_changed = set(row.keys[1:])
	127	self._invalidate_state_caches(room_id, members_changed)
	128	else:
	129	self._attempt_to_invalidate_cache(row.cache_func, row.keys)
	130
	131	super().process_replication_rows(stream_name, instance_name, token, rows)
	132
	133	def _process_event_stream_row(self, token, row):
	134	data = row.data
	135
	136	if row.type == EventsStreamEventRow.TypeId:
	137	self._invalidate_caches_for_event(
	138	token,
	139	data.event_id,
	140	data.room_id,
	141	data.type,
	142	data.state_key,
	143	data.redacts,
	144	data.relates_to,
	145	backfilled=False,
	146	)
	147	elif row.type == EventsStreamCurrentStateRow.TypeId:
	148	self._curr_state_delta_stream_cache.entity_has_changed(
	149	row.data.room_id, token
	150	)
	151
	152	if data.type == EventTypes.Member:
	153	self.get_rooms_for_user_with_stream_ordering.invalidate(
	154	(data.state_key,)
	155	)
	156	else:
	157	raise Exception("Unknown events stream row type %s" % (row.type,))
	158
	159	def _invalidate_caches_for_event(
	160	self,
	161	stream_ordering,
	162	event_id,
	163	room_id,
	164	etype,
	165	state_key,
	166	redacts,
	167	relates_to,
	168	backfilled,
	169	):
	170	self._invalidate_get_event_cache(event_id)
	171
	172	self.get_latest_event_ids_in_room.invalidate((room_id,))
	173
	174	self.get_unread_event_push_actions_by_room_for_user.invalidate_many((room_id,))
	175
	176	if not backfilled:
	177	self._events_stream_cache.entity_has_changed(room_id, stream_ordering)
	178
	179	if redacts:
	180	self._invalidate_get_event_cache(redacts)
	181
	182	if etype == EventTypes.Member:
	183	self._membership_stream_cache.entity_has_changed(state_key, stream_ordering)
	184	self.get_invited_rooms_for_local_user.invalidate((state_key,))
	185
	186	if relates_to:
	187	self.get_relations_for_event.invalidate_many((relates_to,))
	188	self.get_aggregation_groups_for_event.invalidate_many((relates_to,))
	189	self.get_applicable_edit.invalidate((relates_to,))
	190
	191	async def invalidate_cache_and_stream(self, cache_name: str, keys: Tuple[Any, ...]):
	192	"""Invalidates the cache and adds it to the cache stream so slaves
	193	will know to invalidate their caches.
	194
	195	This should only be used to invalidate caches where slaves won't
	196	otherwise know from other replication streams that the cache should
	197	be invalidated.
	198	"""
	199	cache_func = getattr(self, cache_name, None)
	200	if not cache_func:
	201	return
	202
	203	cache_func.invalidate(keys)
	204	await self.db_pool.runInteraction(
	205	"invalidate_cache_and_stream",
	206	self._send_invalidation_to_replication,
	207	cache_func.__name__,
	208	keys,
	209	)
	210
	211	def _invalidate_cache_and_stream(self, txn, cache_func, keys):
	212	"""Invalidates the cache and adds it to the cache stream so slaves
	213	will know to invalidate their caches.
	214
	215	This should only be used to invalidate caches where slaves won't
	216	otherwise know from other replication streams that the cache should
	217	be invalidated.
	218	"""
	219	txn.call_after(cache_func.invalidate, keys)
	220	self._send_invalidation_to_replication(txn, cache_func.__name__, keys)
	221
	222	def _invalidate_all_cache_and_stream(self, txn, cache_func):
	223	"""Invalidates the entire cache and adds it to the cache stream so slaves
	224	will know to invalidate their caches.
	225	"""
	226
	227	txn.call_after(cache_func.invalidate_all)
	228	self._send_invalidation_to_replication(txn, cache_func.__name__, None)
	229
	230	def _invalidate_state_caches_and_stream(self, txn, room_id, members_changed):
	231	"""Special case invalidation of caches based on current state.
	232
	233	We special case this so that we can batch the cache invalidations into a
	234	single replication poke.
	235
	236	Args:
	237	txn
	238	room_id (str): Room where state changed
	239	members_changed (iterable[str]): The user_ids of members that have changed
	240	"""
	241	txn.call_after(self._invalidate_state_caches, room_id, members_changed)
	242
	243	if members_changed:
	244	# We need to be careful that the size of the `members_changed` list
	245	# isn't so large that it causes problems sending over replication, so we
	246	# send them in chunks.
	247	# Max line length is 16K, and max user ID length is 255, so 50 should
	248	# be safe.
	249	for chunk in batch_iter(members_changed, 50):
	250	keys = itertools.chain([room_id], chunk)
	251	self._send_invalidation_to_replication(
	252	txn, CURRENT_STATE_CACHE_NAME, keys
	253	)
	254	else:
	255	# if no members changed, we still need to invalidate the other caches.
	256	self._send_invalidation_to_replication(
	257	txn, CURRENT_STATE_CACHE_NAME, [room_id]
	258	)
	259
	260	def _send_invalidation_to_replication(
	261	self, txn, cache_name: str, keys: Optional[Iterable[Any]]
	262	):
	263	"""Notifies replication that given cache has been invalidated.
	264
	265	Note that this does not invalidate the cache locally.
	266
	267	Args:
	268	txn
	269	cache_name
	270	keys: Entry to invalidate. If None will invalidate all.
	271	"""
	272
	273	if cache_name == CURRENT_STATE_CACHE_NAME and keys is None:
	274	raise Exception(
	275	"Can't stream invalidate all with magic current state cache"
	276	)
	277
	278	if isinstance(self.database_engine, PostgresEngine):
	279	# get_next() returns a context manager which is designed to wrap
	280	# the transaction. However, we want to only get an ID when we want
	281	# to use it, here, so we need to call __enter__ manually, and have
	282	# __exit__ called after the transaction finishes.
	283	stream_id = self._cache_id_gen.get_next_txn(txn)
	284	txn.call_after(self.hs.get_notifier().on_new_replication_data)
	285
	286	if keys is not None:
	287	keys = list(keys)
	288
	289	self.db_pool.simple_insert_txn(
	290	txn,
	291	table="cache_invalidation_stream_by_instance",
	292	values={
	293	"stream_id": stream_id,
	294	"instance_name": self._instance_name,
	295	"cache_func": cache_name,
	296	"keys": keys,
	297	"invalidation_ts": self.clock.time_msec(),
	298	},
	299	)
	300
	301	def get_cache_stream_token(self, instance_name):
	302	if self._cache_id_gen:
	303	return self._cache_id_gen.get_current_token(instance_name)
	304	else:
	305	return 0

+207

-0

synapse/storage/databases/main/censor_events.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16	from typing import TYPE_CHECKING
	17
	18	from synapse.events.utils import prune_event_dict
	19	from synapse.metrics.background_process_metrics import run_as_background_process
	20	from synapse.storage._base import SQLBaseStore
	21	from synapse.storage.database import DatabasePool
	22	from synapse.storage.databases.main.cache import CacheInvalidationWorkerStore
	23	from synapse.storage.databases.main.events import encode_json
	24	from synapse.storage.databases.main.events_worker import EventsWorkerStore
	25
	26	if TYPE_CHECKING:
	27	from synapse.server import HomeServer
	28
	29
	30	logger = logging.getLogger(__name__)
	31
	32
	33	class CensorEventsStore(EventsWorkerStore, CacheInvalidationWorkerStore, SQLBaseStore):
	34	def __init__(self, database: DatabasePool, db_conn, hs: "HomeServer"):
	35	super().__init__(database, db_conn, hs)
	36
	37	def _censor_redactions():
	38	return run_as_background_process(
	39	"_censor_redactions", self._censor_redactions
	40	)
	41
	42	if self.hs.config.redaction_retention_period is not None:
	43	hs.get_clock().looping_call(_censor_redactions, 5 * 60 * 1000)
	44
	45	async def _censor_redactions(self):
	46	"""Censors all redactions older than the configured period that haven't
	47	been censored yet.
	48
	49	By censor we mean update the event_json table with the redacted event.
	50	"""
	51
	52	if self.hs.config.redaction_retention_period is None:
	53	return
	54
	55	if not (
	56	await self.db_pool.updates.has_completed_background_update(
	57	"redactions_have_censored_ts_idx"
	58	)
	59	):
	60	# We don't want to run this until the appropriate index has been
	61	# created.
	62	return
	63
	64	before_ts = self._clock.time_msec() - self.hs.config.redaction_retention_period
	65
	66	# We fetch all redactions that:
	67	# 1. point to an event we have,
	68	# 2. has a received_ts from before the cut off, and
	69	# 3. we haven't yet censored.
	70	#
	71	# This is limited to 100 events to ensure that we don't try and do too
	72	# much at once. We'll get called again so this should eventually catch
	73	# up.
	74	sql = """
	75	SELECT redactions.event_id, redacts FROM redactions
	76	LEFT JOIN events AS original_event ON (
	77	redacts = original_event.event_id
	78	)
	79	WHERE NOT have_censored
	80	AND redactions.received_ts <= ?
	81	ORDER BY redactions.received_ts ASC
	82	LIMIT ?
	83	"""
	84
	85	rows = await self.db_pool.execute(
	86	"_censor_redactions_fetch", None, sql, before_ts, 100
	87	)
	88
	89	updates = []
	90
	91	for redaction_id, event_id in rows:
	92	redaction_event = await self.get_event(redaction_id, allow_none=True)
	93	original_event = await self.get_event(
	94	event_id, allow_rejected=True, allow_none=True
	95	)
	96
	97	# The SQL above ensures that we have both the redaction and
	98	# original event, so if the `get_event` calls return None it
	99	# means that the redaction wasn't allowed. Either way we know that
	100	# the result won't change so we mark the fact that we've checked.
	101	if (
	102	redaction_event
	103	and original_event
	104	and original_event.internal_metadata.is_redacted()
	105	):
	106	# Redaction was allowed
	107	pruned_json = encode_json(
	108	prune_event_dict(
	109	original_event.room_version, original_event.get_dict()
	110	)
	111	)
	112	else:
	113	# Redaction wasn't allowed
	114	pruned_json = None
	115
	116	updates.append((redaction_id, event_id, pruned_json))
	117
	118	def _update_censor_txn(txn):
	119	for redaction_id, event_id, pruned_json in updates:
	120	if pruned_json:
	121	self._censor_event_txn(txn, event_id, pruned_json)
	122
	123	self.db_pool.simple_update_one_txn(
	124	txn,
	125	table="redactions",
	126	keyvalues={"event_id": redaction_id},
	127	updatevalues={"have_censored": True},
	128	)
	129
	130	await self.db_pool.runInteraction("_update_censor_txn", _update_censor_txn)
	131
	132	def _censor_event_txn(self, txn, event_id, pruned_json):
	133	"""Censor an event by replacing its JSON in the event_json table with the
	134	provided pruned JSON.
	135
	136	Args:
	137	txn (LoggingTransaction): The database transaction.
	138	event_id (str): The ID of the event to censor.
	139	pruned_json (str): The pruned JSON
	140	"""
	141	self.db_pool.simple_update_one_txn(
	142	txn,
	143	table="event_json",
	144	keyvalues={"event_id": event_id},
	145	updatevalues={"json": pruned_json},
	146	)
	147
	148	async def expire_event(self, event_id: str) -> None:
	149	"""Retrieve and expire an event that has expired, and delete its associated
	150	expiry timestamp. If the event can't be retrieved, delete its associated
	151	timestamp so we don't try to expire it again in the future.
	152
	153	Args:
	154	event_id: The ID of the event to delete.
	155	"""
	156	# Try to retrieve the event's content from the database or the event cache.
	157	event = await self.get_event(event_id)
	158
	159	def delete_expired_event_txn(txn):
	160	# Delete the expiry timestamp associated with this event from the database.
	161	self._delete_event_expiry_txn(txn, event_id)
	162
	163	if not event:
	164	# If we can't find the event, log a warning and delete the expiry date
	165	# from the database so that we don't try to expire it again in the
	166	# future.
	167	logger.warning(
	168	"Can't expire event %s because we don't have it.", event_id
	169	)
	170	return
	171
	172	# Prune the event's dict then convert it to JSON.
	173	pruned_json = encode_json(
	174	prune_event_dict(event.room_version, event.get_dict())
	175	)
	176
	177	# Update the event_json table to replace the event's JSON with the pruned
	178	# JSON.
	179	self._censor_event_txn(txn, event.event_id, pruned_json)
	180
	181	# We need to invalidate the event cache entry for this event because we
	182	# changed its content in the database. We can't call
	183	# self._invalidate_cache_and_stream because self.get_event_cache isn't of the
	184	# right type.
	185	txn.call_after(self._get_event_cache.invalidate, (event.event_id,))
	186	# Send that invalidation to replication so that other workers also invalidate
	187	# the event cache.
	188	self._send_invalidation_to_replication(
	189	txn, "_get_event_cache", (event.event_id,)
	190	)
	191
	192	await self.db_pool.runInteraction(
	193	"delete_expired_event", delete_expired_event_txn
	194	)
	195
	196	def _delete_event_expiry_txn(self, txn, event_id):
	197	"""Delete the expiry timestamp associated with an event ID without deleting the
	198	actual event.
	199
	200	Args:
	201	txn (LoggingTransaction): The transaction to use to perform the deletion.
	202	event_id (str): The event ID to delete the associated expiry timestamp of.
	203	"""
	204	return self.db_pool.simple_delete_txn(
	205	txn=txn, table="event_expiry", keyvalues={"event_id": event_id}
	206	)

+573

-0

synapse/storage/databases/main/client_ips.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16	from typing import Dict, Optional, Tuple
	17
	18	from synapse.metrics.background_process_metrics import wrap_as_background_process
	19	from synapse.storage._base import SQLBaseStore
	20	from synapse.storage.database import DatabasePool, make_tuple_comparison_clause
	21	from synapse.util.caches.descriptors import Cache
	22
	23	logger = logging.getLogger(__name__)
	24
	25	# Number of msec of granularity to store the user IP 'last seen' time. Smaller
	26	# times give more inserts into the database even for readonly API hits
	27	# 120 seconds == 2 minutes
	28	LAST_SEEN_GRANULARITY = 120 * 1000
	29
	30
	31	class ClientIpBackgroundUpdateStore(SQLBaseStore):
	32	def __init__(self, database: DatabasePool, db_conn, hs):
	33	super(ClientIpBackgroundUpdateStore, self).__init__(database, db_conn, hs)
	34
	35	self.db_pool.updates.register_background_index_update(
	36	"user_ips_device_index",
	37	index_name="user_ips_device_id",
	38	table="user_ips",
	39	columns=["user_id", "device_id", "last_seen"],
	40	)
	41
	42	self.db_pool.updates.register_background_index_update(
	43	"user_ips_last_seen_index",
	44	index_name="user_ips_last_seen",
	45	table="user_ips",
	46	columns=["user_id", "last_seen"],
	47	)
	48
	49	self.db_pool.updates.register_background_index_update(
	50	"user_ips_last_seen_only_index",
	51	index_name="user_ips_last_seen_only",
	52	table="user_ips",
	53	columns=["last_seen"],
	54	)
	55
	56	self.db_pool.updates.register_background_update_handler(
	57	"user_ips_analyze", self._analyze_user_ip
	58	)
	59
	60	self.db_pool.updates.register_background_update_handler(
	61	"user_ips_remove_dupes", self._remove_user_ip_dupes
	62	)
	63
	64	# Register a unique index
	65	self.db_pool.updates.register_background_index_update(
	66	"user_ips_device_unique_index",
	67	index_name="user_ips_user_token_ip_unique_index",
	68	table="user_ips",
	69	columns=["user_id", "access_token", "ip"],
	70	unique=True,
	71	)
	72
	73	# Drop the old non-unique index
	74	self.db_pool.updates.register_background_update_handler(
	75	"user_ips_drop_nonunique_index", self._remove_user_ip_nonunique
	76	)
	77
	78	# Update the last seen info in devices.
	79	self.db_pool.updates.register_background_update_handler(
	80	"devices_last_seen", self._devices_last_seen_update
	81	)
	82
	83	async def _remove_user_ip_nonunique(self, progress, batch_size):
	84	def f(conn):
	85	txn = conn.cursor()
	86	txn.execute("DROP INDEX IF EXISTS user_ips_user_ip")
	87	txn.close()
	88
	89	await self.db_pool.runWithConnection(f)
	90	await self.db_pool.updates._end_background_update(
	91	"user_ips_drop_nonunique_index"
	92	)
	93	return 1
	94
	95	async def _analyze_user_ip(self, progress, batch_size):
	96	# Background update to analyze user_ips table before we run the
	97	# deduplication background update. The table may not have been analyzed
	98	# for ages due to the table locks.
	99	#
	100	# This will lock out the naive upserts to user_ips while it happens, but
	101	# the analyze should be quick (28GB table takes ~10s)
	102	def user_ips_analyze(txn):
	103	txn.execute("ANALYZE user_ips")
	104
	105	await self.db_pool.runInteraction("user_ips_analyze", user_ips_analyze)
	106
	107	await self.db_pool.updates._end_background_update("user_ips_analyze")
	108
	109	return 1
	110
	111	async def _remove_user_ip_dupes(self, progress, batch_size):
	112	# This works function works by scanning the user_ips table in batches
	113	# based on `last_seen`. For each row in a batch it searches the rest of
	114	# the table to see if there are any duplicates, if there are then they
	115	# are removed and replaced with a suitable row.
	116
	117	# Fetch the start of the batch
	118	begin_last_seen = progress.get("last_seen", 0)
	119
	120	def get_last_seen(txn):
	121	txn.execute(
	122	"""
	123	SELECT last_seen FROM user_ips
	124	WHERE last_seen > ?
	125	ORDER BY last_seen
	126	LIMIT 1
	127	OFFSET ?
	128	""",
	129	(begin_last_seen, batch_size),
	130	)
	131	row = txn.fetchone()
	132	if row:
	133	return row[0]
	134	else:
	135	return None
	136
	137	# Get a last seen that has roughly `batch_size` since `begin_last_seen`
	138	end_last_seen = await self.db_pool.runInteraction(
	139	"user_ips_dups_get_last_seen", get_last_seen
	140	)
	141
	142	# If it returns None, then we're processing the last batch
	143	last = end_last_seen is None
	144
	145	logger.info(
	146	"Scanning for duplicate 'user_ips' rows in range: %s <= last_seen < %s",
	147	begin_last_seen,
	148	end_last_seen,
	149	)
	150
	151	def remove(txn):
	152	# This works by looking at all entries in the given time span, and
	153	# then for each (user_id, access_token, ip) tuple in that range
	154	# checking for any duplicates in the rest of the table (via a join).
	155	# It then only returns entries which have duplicates, and the max
	156	# last_seen across all duplicates, which can the be used to delete
	157	# all other duplicates.
	158	# It is efficient due to the existence of (user_id, access_token,
	159	# ip) and (last_seen) indices.
	160
	161	# Define the search space, which requires handling the last batch in
	162	# a different way
	163	if last:
	164	clause = "? <= last_seen"
	165	args = (begin_last_seen,)
	166	else:
	167	clause = "? <= last_seen AND last_seen < ?"
	168	args = (begin_last_seen, end_last_seen)
	169
	170	# (Note: The DISTINCT in the inner query is important to ensure that
	171	# the COUNT(*) is accurate, otherwise double counting may happen due
	172	# to the join effectively being a cross product)
	173	txn.execute(
	174	"""
	175	SELECT user_id, access_token, ip,
	176	MAX(device_id), MAX(user_agent), MAX(last_seen),
	177	COUNT(*)
	178	FROM (
	179	SELECT DISTINCT user_id, access_token, ip
	180	FROM user_ips
	181	WHERE {}
	182	) c
	183	INNER JOIN user_ips USING (user_id, access_token, ip)
	184	GROUP BY user_id, access_token, ip
	185	HAVING count(*) > 1
	186	""".format(
	187	clause
	188	),
	189	args,
	190	)
	191	res = txn.fetchall()
	192
	193	# We've got some duplicates
	194	for i in res:
	195	user_id, access_token, ip, device_id, user_agent, last_seen, count = i
	196
	197	# We want to delete the duplicates so we end up with only a
	198	# single row.
	199	#
	200	# The naive way of doing this would be just to delete all rows
	201	# and reinsert a constructed row. However, if there are a lot of
	202	# duplicate rows this can cause the table to grow a lot, which
	203	# can be problematic in two ways:
	204	# 1. If user_ips is already large then this can cause the
	205	# table to rapidly grow, potentially filling the disk.
	206	# 2. Reinserting a lot of rows can confuse the table
	207	# statistics for postgres, causing it to not use the
	208	# correct indices for the query above, resulting in a full
	209	# table scan. This is incredibly slow for large tables and
	210	# can kill database performance. (This seems to mainly
	211	# happen for the last query where the clause is simply `? <
	212	# last_seen`)
	213	#
	214	# So instead we want to delete all but one of the duplicate
	215	# rows. That is hard to do reliably, so we cheat and do a two
	216	# step process:
	217	# 1. Delete all rows with a last_seen strictly less than the
	218	# max last_seen. This hopefully results in deleting all but
	219	# one row the majority of the time, but there may be
	220	# duplicate last_seen
	221	# 2. If multiple rows remain, we fall back to the naive method
	222	# and simply delete all rows and reinsert.
	223	#
	224	# Note that this relies on no new duplicate rows being inserted,
	225	# but if that is happening then this entire process is futile
	226	# anyway.
	227
	228	# Do step 1:
	229
	230	txn.execute(
	231	"""
	232	DELETE FROM user_ips
	233	WHERE user_id = ? AND access_token = ? AND ip = ? AND last_seen < ?
	234	""",
	235	(user_id, access_token, ip, last_seen),
	236	)
	237	if txn.rowcount == count - 1:
	238	# We deleted all but one of the duplicate rows, i.e. there
	239	# is exactly one remaining and so there is nothing left to
	240	# do.
	241	continue
	242	elif txn.rowcount >= count:
	243	raise Exception(
	244	"We deleted more duplicate rows from 'user_ips' than expected"
	245	)
	246
	247	# The previous step didn't delete enough rows, so we fallback to
	248	# step 2:
	249
	250	# Drop all the duplicates
	251	txn.execute(
	252	"""
	253	DELETE FROM user_ips
	254	WHERE user_id = ? AND access_token = ? AND ip = ?
	255	""",
	256	(user_id, access_token, ip),
	257	)
	258
	259	# Add in one to be the last_seen
	260	txn.execute(
	261	"""
	262	INSERT INTO user_ips
	263	(user_id, access_token, ip, device_id, user_agent, last_seen)
	264	VALUES (?, ?, ?, ?, ?, ?)
	265	""",
	266	(user_id, access_token, ip, device_id, user_agent, last_seen),
	267	)
	268
	269	self.db_pool.updates._background_update_progress_txn(
	270	txn, "user_ips_remove_dupes", {"last_seen": end_last_seen}
	271	)
	272
	273	await self.db_pool.runInteraction("user_ips_dups_remove", remove)
	274
	275	if last:
	276	await self.db_pool.updates._end_background_update("user_ips_remove_dupes")
	277
	278	return batch_size
	279
	280	async def _devices_last_seen_update(self, progress, batch_size):
	281	"""Background update to insert last seen info into devices table
	282	"""
	283
	284	last_user_id = progress.get("last_user_id", "")
	285	last_device_id = progress.get("last_device_id", "")
	286
	287	def _devices_last_seen_update_txn(txn):
	288	# This consists of two queries:
	289	#
	290	# 1. The sub-query searches for the next N devices and joins
	291	# against user_ips to find the max last_seen associated with
	292	# that device.
	293	# 2. The outer query then joins again against user_ips on
	294	# user/device/last_seen. This should hopefully only
	295	# return one row, but if it does return more than one then
	296	# we'll just end up updating the same device row multiple
	297	# times, which is fine.
	298
	299	where_clause, where_args = make_tuple_comparison_clause(
	300	self.database_engine,
	301	[("user_id", last_user_id), ("device_id", last_device_id)],
	302	)
	303
	304	sql = """
	305	SELECT
	306	last_seen, ip, user_agent, user_id, device_id
	307	FROM (
	308	SELECT
	309	user_id, device_id, MAX(u.last_seen) AS last_seen
	310	FROM devices
	311	INNER JOIN user_ips AS u USING (user_id, device_id)
	312	WHERE %(where_clause)s
	313	GROUP BY user_id, device_id
	314	ORDER BY user_id ASC, device_id ASC
	315	LIMIT ?
	316	) c
	317	INNER JOIN user_ips AS u USING (user_id, device_id, last_seen)
	318	""" % {
	319	"where_clause": where_clause
	320	}
	321	txn.execute(sql, where_args + [batch_size])
	322
	323	rows = txn.fetchall()
	324	if not rows:
	325	return 0
	326
	327	sql = """
	328	UPDATE devices
	329	SET last_seen = ?, ip = ?, user_agent = ?
	330	WHERE user_id = ? AND device_id = ?
	331	"""
	332	txn.execute_batch(sql, rows)
	333
	334	_, _, _, user_id, device_id = rows[-1]
	335	self.db_pool.updates._background_update_progress_txn(
	336	txn,
	337	"devices_last_seen",
	338	{"last_user_id": user_id, "last_device_id": device_id},
	339	)
	340
	341	return len(rows)
	342
	343	updated = await self.db_pool.runInteraction(
	344	"_devices_last_seen_update", _devices_last_seen_update_txn
	345	)
	346
	347	if not updated:
	348	await self.db_pool.updates._end_background_update("devices_last_seen")
	349
	350	return updated
	351
	352
	353	class ClientIpStore(ClientIpBackgroundUpdateStore):
	354	def __init__(self, database: DatabasePool, db_conn, hs):
	355
	356	self.client_ip_last_seen = Cache(
	357	name="client_ip_last_seen", keylen=4, max_entries=50000
	358	)
	359
	360	super(ClientIpStore, self).__init__(database, db_conn, hs)
	361
	362	self.user_ips_max_age = hs.config.user_ips_max_age
	363
	364	# (user_id, access_token, ip,) -> (user_agent, device_id, last_seen)
	365	self._batch_row_update = {}
	366
	367	self._client_ip_looper = self._clock.looping_call(
	368	self._update_client_ips_batch, 5 * 1000
	369	)
	370	self.hs.get_reactor().addSystemEventTrigger(
	371	"before", "shutdown", self._update_client_ips_batch
	372	)
	373
	374	if self.user_ips_max_age:
	375	self._clock.looping_call(self._prune_old_user_ips, 5 * 1000)
	376
	377	async def insert_client_ip(
	378	self, user_id, access_token, ip, user_agent, device_id, now=None
	379	):
	380	if not now:
	381	now = int(self._clock.time_msec())
	382	key = (user_id, access_token, ip)
	383
	384	try:
	385	last_seen = self.client_ip_last_seen.get(key)
	386	except KeyError:
	387	last_seen = None
	388	await self.populate_monthly_active_users(user_id)
	389	# Rate-limited inserts
	390	if last_seen is not None and (now - last_seen) < LAST_SEEN_GRANULARITY:
	391	return
	392
	393	self.client_ip_last_seen.prefill(key, now)
	394
	395	self._batch_row_update[key] = (user_agent, device_id, now)
	396
	397	@wrap_as_background_process("update_client_ips")
	398	def _update_client_ips_batch(self):
	399
	400	# If the DB pool has already terminated, don't try updating
	401	if not self.db_pool.is_running():
	402	return
	403
	404	to_update = self._batch_row_update
	405	self._batch_row_update = {}
	406
	407	return self.db_pool.runInteraction(
	408	"_update_client_ips_batch", self._update_client_ips_batch_txn, to_update
	409	)
	410
	411	def _update_client_ips_batch_txn(self, txn, to_update):
	412	if "user_ips" in self.db_pool._unsafe_to_upsert_tables or (
	413	not self.database_engine.can_native_upsert
	414	):
	415	self.database_engine.lock_table(txn, "user_ips")
	416
	417	for entry in to_update.items():
	418	(user_id, access_token, ip), (user_agent, device_id, last_seen) = entry
	419
	420	try:
	421	self.db_pool.simple_upsert_txn(
	422	txn,
	423	table="user_ips",
	424	keyvalues={
	425	"user_id": user_id,
	426	"access_token": access_token,
	427	"ip": ip,
	428	},
	429	values={
	430	"user_agent": user_agent,
	431	"device_id": device_id,
	432	"last_seen": last_seen,
	433	},
	434	lock=False,
	435	)
	436
	437	# Technically an access token might not be associated with
	438	# a device so we need to check.
	439	if device_id:
	440	# this is always an update rather than an upsert: the row should
	441	# already exist, and if it doesn't, that may be because it has been
	442	# deleted, and we don't want to re-create it.
	443	self.db_pool.simple_update_txn(
	444	txn,
	445	table="devices",
	446	keyvalues={"user_id": user_id, "device_id": device_id},
	447	updatevalues={
	448	"user_agent": user_agent,
	449	"last_seen": last_seen,
	450	"ip": ip,
	451	},
	452	)
	453	except Exception as e:
	454	# Failed to upsert, log and continue
	455	logger.error("Failed to insert client IP %r: %r", entry, e)
	456
	457	async def get_last_client_ip_by_device(
	458	self, user_id: str, device_id: Optional[str]
	459	) -> Dict[Tuple[str, str], dict]:
	460	"""For each device_id listed, give the user_ip it was last seen on
	461
	462	Args:
	463	user_id: The user to fetch devices for.
	464	device_id: If None fetches all devices for the user
	465
	466	Returns:
	467	A dictionary mapping a tuple of (user_id, device_id) to dicts, with
	468	keys giving the column names from the devices table.
	469	"""
	470
	471	keyvalues = {"user_id": user_id}
	472	if device_id is not None:
	473	keyvalues["device_id"] = device_id
	474
	475	res = await self.db_pool.simple_select_list(
	476	table="devices",
	477	keyvalues=keyvalues,
	478	retcols=("user_id", "ip", "user_agent", "device_id", "last_seen"),
	479	)
	480
	481	ret = {(d["user_id"], d["device_id"]): d for d in res}
	482	for key in self._batch_row_update:
	483	uid, access_token, ip = key
	484	if uid == user_id:
	485	user_agent, did, last_seen = self._batch_row_update[key]
	486	if not device_id or did == device_id:
	487	ret[(user_id, device_id)] = {
	488	"user_id": user_id,
	489	"access_token": access_token,
	490	"ip": ip,
	491	"user_agent": user_agent,
	492	"device_id": did,
	493	"last_seen": last_seen,
	494	}
	495	return ret
	496
	497	async def get_user_ip_and_agents(self, user):
	498	user_id = user.to_string()
	499	results = {}
	500
	501	for key in self._batch_row_update:
	502	uid, access_token, ip, = key
	503	if uid == user_id:
	504	user_agent, _, last_seen = self._batch_row_update[key]
	505	results[(access_token, ip)] = (user_agent, last_seen)
	506
	507	rows = await self.db_pool.simple_select_list(
	508	table="user_ips",
	509	keyvalues={"user_id": user_id},
	510	retcols=["access_token", "ip", "user_agent", "last_seen"],
	511	desc="get_user_ip_and_agents",
	512	)
	513
	514	results.update(
	515	((row["access_token"], row["ip"]), (row["user_agent"], row["last_seen"]))
	516	for row in rows
	517	)
	518	return [
	519	{
	520	"access_token": access_token,
	521	"ip": ip,
	522	"user_agent": user_agent,
	523	"last_seen": last_seen,
	524	}
	525	for (access_token, ip), (user_agent, last_seen) in results.items()
	526	]
	527
	528	@wrap_as_background_process("prune_old_user_ips")
	529	async def _prune_old_user_ips(self):
	530	"""Removes entries in user IPs older than the configured period.
	531	"""
	532
	533	if self.user_ips_max_age is None:
	534	# Nothing to do
	535	return
	536
	537	if not await self.db_pool.updates.has_completed_background_update(
	538	"devices_last_seen"
	539	):
	540	# Only start pruning if we have finished populating the devices
	541	# last seen info.
	542	return
	543
	544	# We do a slightly funky SQL delete to ensure we don't try and delete
	545	# too much at once (as the table may be very large from before we
	546	# started pruning).
	547	#
	548	# This works by finding the max last_seen that is less than the given
	549	# time, but has no more than N rows before it, deleting all rows with
	550	# a lesser last_seen time. (We COALESCE so that the sub-SELECT always
	551	# returns exactly one row).
	552	sql = """
	553	DELETE FROM user_ips
	554	WHERE last_seen <= (
	555	SELECT COALESCE(MAX(last_seen), -1)
	556	FROM (
	557	SELECT last_seen FROM user_ips
	558	WHERE last_seen <= ?
	559	ORDER BY last_seen ASC
	560	LIMIT 5000
	561	) AS u
	562	)
	563	"""
	564
	565	timestamp = self.clock.time_msec() - self.user_ips_max_age
	566
	567	def _prune_old_user_ips_txn(txn):
	568	txn.execute(sql, (timestamp,))
	569
	570	await self.db_pool.runInteraction(
	571	"_prune_old_user_ips", _prune_old_user_ips_txn
	572	)

+479

-0

synapse/storage/databases/main/deviceinbox.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16	from typing import List, Tuple
	17
	18	from synapse.logging.opentracing import log_kv, set_tag, trace
	19	from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
	20	from synapse.storage.database import DatabasePool
	21	from synapse.util import json_encoder
	22	from synapse.util.caches.expiringcache import ExpiringCache
	23
	24	logger = logging.getLogger(__name__)
	25
	26
	27	class DeviceInboxWorkerStore(SQLBaseStore):
	28	def get_to_device_stream_token(self):
	29	return self._device_inbox_id_gen.get_current_token()
	30
	31	async def get_new_messages_for_device(
	32	self,
	33	user_id: str,
	34	device_id: str,
	35	last_stream_id: int,
	36	current_stream_id: int,
	37	limit: int = 100,
	38	) -> Tuple[List[dict], int]:
	39	"""
	40	Args:
	41	user_id: The recipient user_id.
	42	device_id: The recipient device_id.
	43	last_stream_id: The last stream ID checked.
	44	current_stream_id: The current position of the to device
	45	message stream.
	46	limit: The maximum number of messages to retrieve.
	47
	48	Returns:
	49	A list of messages for the device and where in the stream the messages got to.
	50	"""
	51	has_changed = self._device_inbox_stream_cache.has_entity_changed(
	52	user_id, last_stream_id
	53	)
	54	if not has_changed:
	55	return ([], current_stream_id)
	56
	57	def get_new_messages_for_device_txn(txn):
	58	sql = (
	59	"SELECT stream_id, message_json FROM device_inbox"
	60	" WHERE user_id = ? AND device_id = ?"
	61	" AND ? < stream_id AND stream_id <= ?"
	62	" ORDER BY stream_id ASC"
	63	" LIMIT ?"
	64	)
	65	txn.execute(
	66	sql, (user_id, device_id, last_stream_id, current_stream_id, limit)
	67	)
	68	messages = []
	69	for row in txn:
	70	stream_pos = row[0]
	71	messages.append(db_to_json(row[1]))
	72	if len(messages) < limit:
	73	stream_pos = current_stream_id
	74	return messages, stream_pos
	75
	76	return await self.db_pool.runInteraction(
	77	"get_new_messages_for_device", get_new_messages_for_device_txn
	78	)
	79
	80	@trace
	81	async def delete_messages_for_device(
	82	self, user_id: str, device_id: str, up_to_stream_id: int
	83	) -> int:
	84	"""
	85	Args:
	86	user_id: The recipient user_id.
	87	device_id: The recipient device_id.
	88	up_to_stream_id: Where to delete messages up to.
	89
	90	Returns:
	91	The number of messages deleted.
	92	"""
	93	# If we have cached the last stream id we've deleted up to, we can
	94	# check if there is likely to be anything that needs deleting
	95	last_deleted_stream_id = self._last_device_delete_cache.get(
	96	(user_id, device_id), None
	97	)
	98
	99	set_tag("last_deleted_stream_id", last_deleted_stream_id)
	100
	101	if last_deleted_stream_id:
	102	has_changed = self._device_inbox_stream_cache.has_entity_changed(
	103	user_id, last_deleted_stream_id
	104	)
	105	if not has_changed:
	106	log_kv({"message": "No changes in cache since last check"})
	107	return 0
	108
	109	def delete_messages_for_device_txn(txn):
	110	sql = (
	111	"DELETE FROM device_inbox"
	112	" WHERE user_id = ? AND device_id = ?"
	113	" AND stream_id <= ?"
	114	)
	115	txn.execute(sql, (user_id, device_id, up_to_stream_id))
	116	return txn.rowcount
	117
	118	count = await self.db_pool.runInteraction(
	119	"delete_messages_for_device", delete_messages_for_device_txn
	120	)
	121
	122	log_kv(
	123	{"message": "deleted {} messages for device".format(count), "count": count}
	124	)
	125
	126	# Update the cache, ensuring that we only ever increase the value
	127	last_deleted_stream_id = self._last_device_delete_cache.get(
	128	(user_id, device_id), 0
	129	)
	130	self._last_device_delete_cache[(user_id, device_id)] = max(
	131	last_deleted_stream_id, up_to_stream_id
	132	)
	133
	134	return count
	135
	136	@trace
	137	async def get_new_device_msgs_for_remote(
	138	self, destination, last_stream_id, current_stream_id, limit
	139	) -> Tuple[List[dict], int]:
	140	"""
	141	Args:
	142	destination(str): The name of the remote server.
	143	last_stream_id(int\|long): The last position of the device message stream
	144	that the server sent up to.
	145	current_stream_id(int\|long): The current position of the device
	146	message stream.
	147	Returns:
	148	A list of messages for the device and where in the stream the messages got to.
	149	"""
	150
	151	set_tag("destination", destination)
	152	set_tag("last_stream_id", last_stream_id)
	153	set_tag("current_stream_id", current_stream_id)
	154	set_tag("limit", limit)
	155
	156	has_changed = self._device_federation_outbox_stream_cache.has_entity_changed(
	157	destination, last_stream_id
	158	)
	159	if not has_changed or last_stream_id == current_stream_id:
	160	log_kv({"message": "No new messages in stream"})
	161	return ([], current_stream_id)
	162
	163	if limit <= 0:
	164	# This can happen if we run out of room for EDUs in the transaction.
	165	return ([], last_stream_id)
	166
	167	@trace
	168	def get_new_messages_for_remote_destination_txn(txn):
	169	sql = (
	170	"SELECT stream_id, messages_json FROM device_federation_outbox"
	171	" WHERE destination = ?"
	172	" AND ? < stream_id AND stream_id <= ?"
	173	" ORDER BY stream_id ASC"
	174	" LIMIT ?"
	175	)
	176	txn.execute(sql, (destination, last_stream_id, current_stream_id, limit))
	177	messages = []
	178	for row in txn:
	179	stream_pos = row[0]
	180	messages.append(db_to_json(row[1]))
	181	if len(messages) < limit:
	182	log_kv({"message": "Set stream position to current position"})
	183	stream_pos = current_stream_id
	184	return messages, stream_pos
	185
	186	return await self.db_pool.runInteraction(
	187	"get_new_device_msgs_for_remote",
	188	get_new_messages_for_remote_destination_txn,
	189	)
	190
	191	@trace
	192	def delete_device_msgs_for_remote(self, destination, up_to_stream_id):
	193	"""Used to delete messages when the remote destination acknowledges
	194	their receipt.
	195
	196	Args:
	197	destination(str): The destination server_name
	198	up_to_stream_id(int): Where to delete messages up to.
	199	Returns:
	200	A deferred that resolves when the messages have been deleted.
	201	"""
	202
	203	def delete_messages_for_remote_destination_txn(txn):
	204	sql = (
	205	"DELETE FROM device_federation_outbox"
	206	" WHERE destination = ?"
	207	" AND stream_id <= ?"
	208	)
	209	txn.execute(sql, (destination, up_to_stream_id))
	210
	211	return self.db_pool.runInteraction(
	212	"delete_device_msgs_for_remote", delete_messages_for_remote_destination_txn
	213	)
	214
	215	async def get_all_new_device_messages(
	216	self, instance_name: str, last_id: int, current_id: int, limit: int
	217	) -> Tuple[List[Tuple[int, tuple]], int, bool]:
	218	"""Get updates for to device replication stream.
	219
	220	Args:
	221	instance_name: The writer we want to fetch updates from. Unused
	222	here since there is only ever one writer.
	223	last_id: The token to fetch updates from. Exclusive.
	224	current_id: The token to fetch updates up to. Inclusive.
	225	limit: The requested limit for the number of rows to return. The
	226	function may return more or fewer rows.
	227
	228	Returns:
	229	A tuple consisting of: the updates, a token to use to fetch
	230	subsequent updates, and whether we returned fewer rows than exists
	231	between the requested tokens due to the limit.
	232
	233	The token returned can be used in a subsequent call to this
	234	function to get further updatees.
	235
	236	The updates are a list of 2-tuples of stream ID and the row data
	237	"""
	238
	239	if last_id == current_id:
	240	return [], current_id, False
	241
	242	def get_all_new_device_messages_txn(txn):
	243	# We limit like this as we might have multiple rows per stream_id, and
	244	# we want to make sure we always get all entries for any stream_id
	245	# we return.
	246	upper_pos = min(current_id, last_id + limit)
	247	sql = (
	248	"SELECT max(stream_id), user_id"
	249	" FROM device_inbox"
	250	" WHERE ? < stream_id AND stream_id <= ?"
	251	" GROUP BY user_id"
	252	)
	253	txn.execute(sql, (last_id, upper_pos))
	254	updates = [(row[0], row[1:]) for row in txn]
	255
	256	sql = (
	257	"SELECT max(stream_id), destination"
	258	" FROM device_federation_outbox"
	259	" WHERE ? < stream_id AND stream_id <= ?"
	260	" GROUP BY destination"
	261	)
	262	txn.execute(sql, (last_id, upper_pos))
	263	updates.extend((row[0], row[1:]) for row in txn)
	264
	265	# Order by ascending stream ordering
	266	updates.sort()
	267
	268	limited = False
	269	upto_token = current_id
	270	if len(updates) >= limit:
	271	upto_token = updates[-1][0]
	272	limited = True
	273
	274	return updates, upto_token, limited
	275
	276	return await self.db_pool.runInteraction(
	277	"get_all_new_device_messages", get_all_new_device_messages_txn
	278	)
	279
	280
	281	class DeviceInboxBackgroundUpdateStore(SQLBaseStore):
	282	DEVICE_INBOX_STREAM_ID = "device_inbox_stream_drop"
	283
	284	def __init__(self, database: DatabasePool, db_conn, hs):
	285	super(DeviceInboxBackgroundUpdateStore, self).__init__(database, db_conn, hs)
	286
	287	self.db_pool.updates.register_background_index_update(
	288	"device_inbox_stream_index",
	289	index_name="device_inbox_stream_id_user_id",
	290	table="device_inbox",
	291	columns=["stream_id", "user_id"],
	292	)
	293
	294	self.db_pool.updates.register_background_update_handler(
	295	self.DEVICE_INBOX_STREAM_ID, self._background_drop_index_device_inbox
	296	)
	297
	298	async def _background_drop_index_device_inbox(self, progress, batch_size):
	299	def reindex_txn(conn):
	300	txn = conn.cursor()
	301	txn.execute("DROP INDEX IF EXISTS device_inbox_stream_id")
	302	txn.close()
	303
	304	await self.db_pool.runWithConnection(reindex_txn)
	305
	306	await self.db_pool.updates._end_background_update(self.DEVICE_INBOX_STREAM_ID)
	307
	308	return 1
	309
	310
	311	class DeviceInboxStore(DeviceInboxWorkerStore, DeviceInboxBackgroundUpdateStore):
	312	DEVICE_INBOX_STREAM_ID = "device_inbox_stream_drop"
	313
	314	def __init__(self, database: DatabasePool, db_conn, hs):
	315	super(DeviceInboxStore, self).__init__(database, db_conn, hs)
	316
	317	# Map of (user_id, device_id) to the last stream_id that has been
	318	# deleted up to. This is so that we can no op deletions.
	319	self._last_device_delete_cache = ExpiringCache(
	320	cache_name="last_device_delete_cache",
	321	clock=self._clock,
	322	max_len=10000,
	323	expiry_ms=30 * 60 * 1000,
	324	)
	325
	326	@trace
	327	async def add_messages_to_device_inbox(
	328	self,
	329	local_messages_by_user_then_device: dict,
	330	remote_messages_by_destination: dict,
	331	) -> int:
	332	"""Used to send messages from this server.
	333
	334	Args:
	335	local_messages_by_user_and_device:
	336	Dictionary of user_id to device_id to message.
	337	remote_messages_by_destination:
	338	Dictionary of destination server_name to the EDU JSON to send.
	339
	340	Returns:
	341	The new stream_id.
	342	"""
	343
	344	def add_messages_txn(txn, now_ms, stream_id):
	345	# Add the local messages directly to the local inbox.
	346	self._add_messages_to_local_device_inbox_txn(
	347	txn, stream_id, local_messages_by_user_then_device
	348	)
	349
	350	# Add the remote messages to the federation outbox.
	351	# We'll send them to a remote server when we next send a
	352	# federation transaction to that destination.
	353	sql = (
	354	"INSERT INTO device_federation_outbox"
	355	" (destination, stream_id, queued_ts, messages_json)"
	356	" VALUES (?,?,?,?)"
	357	)
	358	rows = []
	359	for destination, edu in remote_messages_by_destination.items():
	360	edu_json = json_encoder.encode(edu)
	361	rows.append((destination, stream_id, now_ms, edu_json))
	362	txn.executemany(sql, rows)
	363
	364	with self._device_inbox_id_gen.get_next() as stream_id:
	365	now_ms = self.clock.time_msec()
	366	await self.db_pool.runInteraction(
	367	"add_messages_to_device_inbox", add_messages_txn, now_ms, stream_id
	368	)
	369	for user_id in local_messages_by_user_then_device.keys():
	370	self._device_inbox_stream_cache.entity_has_changed(user_id, stream_id)
	371	for destination in remote_messages_by_destination.keys():
	372	self._device_federation_outbox_stream_cache.entity_has_changed(
	373	destination, stream_id
	374	)
	375
	376	return self._device_inbox_id_gen.get_current_token()
	377
	378	async def add_messages_from_remote_to_device_inbox(
	379	self, origin: str, message_id: str, local_messages_by_user_then_device: dict
	380	) -> int:
	381	def add_messages_txn(txn, now_ms, stream_id):
	382	# Check if we've already inserted a matching message_id for that
	383	# origin. This can happen if the origin doesn't receive our
	384	# acknowledgement from the first time we received the message.
	385	already_inserted = self.db_pool.simple_select_one_txn(
	386	txn,
	387	table="device_federation_inbox",
	388	keyvalues={"origin": origin, "message_id": message_id},
	389	retcols=("message_id",),
	390	allow_none=True,
	391	)
	392	if already_inserted is not None:
	393	return
	394
	395	# Add an entry for this message_id so that we know we've processed
	396	# it.
	397	self.db_pool.simple_insert_txn(
	398	txn,
	399	table="device_federation_inbox",
	400	values={
	401	"origin": origin,
	402	"message_id": message_id,
	403	"received_ts": now_ms,
	404	},
	405	)
	406
	407	# Add the messages to the approriate local device inboxes so that
	408	# they'll be sent to the devices when they next sync.
	409	self._add_messages_to_local_device_inbox_txn(
	410	txn, stream_id, local_messages_by_user_then_device
	411	)
	412
	413	with self._device_inbox_id_gen.get_next() as stream_id:
	414	now_ms = self.clock.time_msec()
	415	await self.db_pool.runInteraction(
	416	"add_messages_from_remote_to_device_inbox",
	417	add_messages_txn,
	418	now_ms,
	419	stream_id,
	420	)
	421	for user_id in local_messages_by_user_then_device.keys():
	422	self._device_inbox_stream_cache.entity_has_changed(user_id, stream_id)
	423
	424	return stream_id
	425
	426	def _add_messages_to_local_device_inbox_txn(
	427	self, txn, stream_id, messages_by_user_then_device
	428	):
	429	local_by_user_then_device = {}
	430	for user_id, messages_by_device in messages_by_user_then_device.items():
	431	messages_json_for_user = {}
	432	devices = list(messages_by_device.keys())
	433	if len(devices) == 1 and devices[0] == "*":
	434	# Handle wildcard device_ids.
	435	sql = "SELECT device_id FROM devices WHERE user_id = ?"
	436	txn.execute(sql, (user_id,))
	437	message_json = json_encoder.encode(messages_by_device["*"])
	438	for row in txn:
	439	# Add the message for all devices for this user on this
	440	# server.
	441	device = row[0]
	442	messages_json_for_user[device] = message_json
	443	else:
	444	if not devices:
	445	continue
	446
	447	clause, args = make_in_list_sql_clause(
	448	txn.database_engine, "device_id", devices
	449	)
	450	sql = "SELECT device_id FROM devices WHERE user_id = ? AND " + clause
	451
	452	# TODO: Maybe this needs to be done in batches if there are
	453	# too many local devices for a given user.
	454	txn.execute(sql, [user_id] + list(args))
	455	for row in txn:
	456	# Only insert into the local inbox if the device exists on
	457	# this server
	458	device = row[0]
	459	message_json = json_encoder.encode(messages_by_device[device])
	460	messages_json_for_user[device] = message_json
	461
	462	if messages_json_for_user:
	463	local_by_user_then_device[user_id] = messages_json_for_user
	464
	465	if not local_by_user_then_device:
	466	return
	467
	468	sql = (
	469	"INSERT INTO device_inbox"
	470	" (user_id, device_id, stream_id, message_json)"
	471	" VALUES (?,?,?,?)"
	472	)
	473	rows = []
	474	for user_id, messages_by_device in local_by_user_then_device.items():
	475	for device_id, message_json in messages_by_device.items():
	476	rows.append((user_id, device_id, stream_id, message_json))
	477
	478	txn.executemany(sql, rows)

+1339

-0

synapse/storage/databases/main/devices.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2016 OpenMarket Ltd
	2	# Copyright 2019 New Vector Ltd
	3	# Copyright 2019 The Matrix.org Foundation C.I.C.
	4	#
	5	# Licensed under the Apache License, Version 2.0 (the "License");
	6	# you may not use this file except in compliance with the License.
	7	# You may obtain a copy of the License at
	8	#
	9	# http://www.apache.org/licenses/LICENSE-2.0
	10	#
	11	# Unless required by applicable law or agreed to in writing, software
	12	# distributed under the License is distributed on an "AS IS" BASIS,
	13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	14	# See the License for the specific language governing permissions and
	15	# limitations under the License.
	16	import logging
	17	from typing import Dict, Iterable, List, Optional, Set, Tuple
	18
	19	from synapse.api.errors import Codes, StoreError
	20	from synapse.logging.opentracing import (
	21	get_active_span_text_map,
	22	set_tag,
	23	trace,
	24	whitelisted_homeserver,
	25	)
	26	from synapse.metrics.background_process_metrics import run_as_background_process
	27	from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
	28	from synapse.storage.database import (
	29	DatabasePool,
	30	LoggingTransaction,
	31	make_tuple_comparison_clause,
	32	)
	33	from synapse.types import Collection, JsonDict, get_verify_key_from_cross_signing_key
	34	from synapse.util import json_encoder
	35	from synapse.util.caches.descriptors import Cache, cached, cachedList
	36	from synapse.util.iterutils import batch_iter
	37	from synapse.util.stringutils import shortstr
	38
	39	logger = logging.getLogger(__name__)
	40
	41	DROP_DEVICE_LIST_STREAMS_NON_UNIQUE_INDEXES = (
	42	"drop_device_list_streams_non_unique_indexes"
	43	)
	44
	45	BG_UPDATE_REMOVE_DUP_OUTBOUND_POKES = "remove_dup_outbound_pokes"
	46
	47
	48	class DeviceWorkerStore(SQLBaseStore):
	49	def get_device(self, user_id: str, device_id: str):
	50	"""Retrieve a device. Only returns devices that are not marked as
	51	hidden.
	52
	53	Args:
	54	user_id: The ID of the user which owns the device
	55	device_id: The ID of the device to retrieve
	56	Returns:
	57	defer.Deferred for a dict containing the device information
	58	Raises:
	59	StoreError: if the device is not found
	60	"""
	61	return self.db_pool.simple_select_one(
	62	table="devices",
	63	keyvalues={"user_id": user_id, "device_id": device_id, "hidden": False},
	64	retcols=("user_id", "device_id", "display_name"),
	65	desc="get_device",
	66	)
	67
	68	async def get_devices_by_user(self, user_id: str) -> Dict[str, Dict[str, str]]:
	69	"""Retrieve all of a user's registered devices. Only returns devices
	70	that are not marked as hidden.
	71
	72	Args:
	73	user_id:
	74	Returns:
	75	A mapping from device_id to a dict containing "device_id", "user_id"
	76	and "display_name" for each device.
	77	"""
	78	devices = await self.db_pool.simple_select_list(
	79	table="devices",
	80	keyvalues={"user_id": user_id, "hidden": False},
	81	retcols=("user_id", "device_id", "display_name"),
	82	desc="get_devices_by_user",
	83	)
	84
	85	return {d["device_id"]: d for d in devices}
	86
	87	@trace
	88	async def get_device_updates_by_remote(
	89	self, destination: str, from_stream_id: int, limit: int
	90	) -> Tuple[int, List[Tuple[str, dict]]]:
	91	"""Get a stream of device updates to send to the given remote server.
	92
	93	Args:
	94	destination: The host the device updates are intended for
	95	from_stream_id: The minimum stream_id to filter updates by, exclusive
	96	limit: Maximum number of device updates to return
	97
	98	Returns:
	99	A mapping from the current stream id (ie, the stream id of the last
	100	update included in the response), and the list of updates, where
	101	each update is a pair of EDU type and EDU contents.
	102	"""
	103	now_stream_id = self._device_list_id_gen.get_current_token()
	104
	105	has_changed = self._device_list_federation_stream_cache.has_entity_changed(
	106	destination, int(from_stream_id)
	107	)
	108	if not has_changed:
	109	return now_stream_id, []
	110
	111	updates = await self.db_pool.runInteraction(
	112	"get_device_updates_by_remote",
	113	self._get_device_updates_by_remote_txn,
	114	destination,
	115	from_stream_id,
	116	now_stream_id,
	117	limit,
	118	)
	119
	120	# Return an empty list if there are no updates
	121	if not updates:
	122	return now_stream_id, []
	123
	124	# get the cross-signing keys of the users in the list, so that we can
	125	# determine which of the device changes were cross-signing keys
	126	users = {r[0] for r in updates}
	127	master_key_by_user = {}
	128	self_signing_key_by_user = {}
	129	for user in users:
	130	cross_signing_key = await self.get_e2e_cross_signing_key(user, "master")
	131	if cross_signing_key:
	132	key_id, verify_key = get_verify_key_from_cross_signing_key(
	133	cross_signing_key
	134	)
	135	# verify_key is a VerifyKey from signedjson, which uses
	136	# .version to denote the portion of the key ID after the
	137	# algorithm and colon, which is the device ID
	138	master_key_by_user[user] = {
	139	"key_info": cross_signing_key,
	140	"device_id": verify_key.version,
	141	}
	142
	143	cross_signing_key = await self.get_e2e_cross_signing_key(
	144	user, "self_signing"
	145	)
	146	if cross_signing_key:
	147	key_id, verify_key = get_verify_key_from_cross_signing_key(
	148	cross_signing_key
	149	)
	150	self_signing_key_by_user[user] = {
	151	"key_info": cross_signing_key,
	152	"device_id": verify_key.version,
	153	}
	154
	155	# Perform the equivalent of a GROUP BY
	156	#
	157	# Iterate through the updates list and copy non-duplicate
	158	# (user_id, device_id) entries into a map, with the value being
	159	# the max stream_id across each set of duplicate entries
	160	#
	161	# maps (user_id, device_id) -> (stream_id, opentracing_context)
	162	#
	163	# opentracing_context contains the opentracing metadata for the request
	164	# that created the poke
	165	#
	166	# The most recent request's opentracing_context is used as the
	167	# context which created the Edu.
	168
	169	query_map = {}
	170	cross_signing_keys_by_user = {}
	171	for user_id, device_id, update_stream_id, update_context in updates:
	172	if (
	173	user_id in master_key_by_user
	174	and device_id == master_key_by_user[user_id]["device_id"]
	175	):
	176	result = cross_signing_keys_by_user.setdefault(user_id, {})
	177	result["master_key"] = master_key_by_user[user_id]["key_info"]
	178	elif (
	179	user_id in self_signing_key_by_user
	180	and device_id == self_signing_key_by_user[user_id]["device_id"]
	181	):
	182	result = cross_signing_keys_by_user.setdefault(user_id, {})
	183	result["self_signing_key"] = self_signing_key_by_user[user_id][
	184	"key_info"
	185	]
	186	else:
	187	key = (user_id, device_id)
	188
	189	previous_update_stream_id, _ = query_map.get(key, (0, None))
	190
	191	if update_stream_id > previous_update_stream_id:
	192	query_map[key] = (update_stream_id, update_context)
	193
	194	results = await self._get_device_update_edus_by_remote(
	195	destination, from_stream_id, query_map
	196	)
	197
	198	# add the updated cross-signing keys to the results list
	199	for user_id, result in cross_signing_keys_by_user.items():
	200	result["user_id"] = user_id
	201	# FIXME: switch to m.signing_key_update when MSC1756 is merged into the spec
	202	results.append(("org.matrix.signing_key_update", result))
	203
	204	return now_stream_id, results
	205
	206	def _get_device_updates_by_remote_txn(
	207	self,
	208	txn: LoggingTransaction,
	209	destination: str,
	210	from_stream_id: int,
	211	now_stream_id: int,
	212	limit: int,
	213	):
	214	"""Return device update information for a given remote destination
	215
	216	Args:
	217	txn: The transaction to execute
	218	destination: The host the device updates are intended for
	219	from_stream_id: The minimum stream_id to filter updates by, exclusive
	220	now_stream_id: The maximum stream_id to filter updates by, inclusive
	221	limit: Maximum number of device updates to return
	222
	223	Returns:
	224	List: List of device updates
	225	"""
	226	# get the list of device updates that need to be sent
	227	sql = """
	228	SELECT user_id, device_id, stream_id, opentracing_context FROM device_lists_outbound_pokes
	229	WHERE destination = ? AND ? < stream_id AND stream_id <= ?
	230	ORDER BY stream_id
	231	LIMIT ?
	232	"""
	233	txn.execute(sql, (destination, from_stream_id, now_stream_id, limit))
	234
	235	return list(txn)
	236
	237	async def _get_device_update_edus_by_remote(
	238	self,
	239	destination: str,
	240	from_stream_id: int,
	241	query_map: Dict[Tuple[str, str], Tuple[int, Optional[str]]],
	242	) -> List[Tuple[str, dict]]:
	243	"""Returns a list of device update EDUs as well as E2EE keys
	244
	245	Args:
	246	destination: The host the device updates are intended for
	247	from_stream_id: The minimum stream_id to filter updates by, exclusive
	248	query_map (Dict[(str, str): (int, str\|None)]): Dictionary mapping
	249	user_id/device_id to update stream_id and the relevant json-encoded
	250	opentracing context
	251
	252	Returns:
	253	List of objects representing an device update EDU
	254	"""
	255	devices = (
	256	await self.db_pool.runInteraction(
	257	"_get_e2e_device_keys_txn",
	258	self._get_e2e_device_keys_txn,
	259	query_map.keys(),
	260	include_all_devices=True,
	261	include_deleted_devices=True,
	262	)
	263	if query_map
	264	else {}
	265	)
	266
	267	results = []
	268	for user_id, user_devices in devices.items():
	269	# The prev_id for the first row is always the last row before
	270	# `from_stream_id`
	271	prev_id = await self._get_last_device_update_for_remote_user(
	272	destination, user_id, from_stream_id
	273	)
	274
	275	# make sure we go through the devices in stream order
	276	device_ids = sorted(
	277	user_devices.keys(), key=lambda i: query_map[(user_id, i)][0],
	278	)
	279
	280	for device_id in device_ids:
	281	device = user_devices[device_id]
	282	stream_id, opentracing_context = query_map[(user_id, device_id)]
	283	result = {
	284	"user_id": user_id,
	285	"device_id": device_id,
	286	"prev_id": [prev_id] if prev_id else [],
	287	"stream_id": stream_id,
	288	"org.matrix.opentracing_context": opentracing_context,
	289	}
	290
	291	prev_id = stream_id
	292
	293	if device is not None:
	294	key_json = device.get("key_json", None)
	295	if key_json:
	296	result["keys"] = db_to_json(key_json)
	297
	298	if "signatures" in device:
	299	for sig_user_id, sigs in device["signatures"].items():
	300	result["keys"].setdefault("signatures", {}).setdefault(
	301	sig_user_id, {}
	302	).update(sigs)
	303
	304	device_display_name = device.get("device_display_name", None)
	305	if device_display_name:
	306	result["device_display_name"] = device_display_name
	307	else:
	308	result["deleted"] = True
	309
	310	results.append(("m.device_list_update", result))
	311
	312	return results
	313
	314	def _get_last_device_update_for_remote_user(
	315	self, destination: str, user_id: str, from_stream_id: int
	316	):
	317	def f(txn):
	318	prev_sent_id_sql = """
	319	SELECT coalesce(max(stream_id), 0) as stream_id
	320	FROM device_lists_outbound_last_success
	321	WHERE destination = ? AND user_id = ? AND stream_id <= ?
	322	"""
	323	txn.execute(prev_sent_id_sql, (destination, user_id, from_stream_id))
	324	rows = txn.fetchall()
	325	return rows[0][0]
	326
	327	return self.db_pool.runInteraction("get_last_device_update_for_remote_user", f)
	328
	329	def mark_as_sent_devices_by_remote(self, destination: str, stream_id: int):
	330	"""Mark that updates have successfully been sent to the destination.
	331	"""
	332	return self.db_pool.runInteraction(
	333	"mark_as_sent_devices_by_remote",
	334	self._mark_as_sent_devices_by_remote_txn,
	335	destination,
	336	stream_id,
	337	)
	338
	339	def _mark_as_sent_devices_by_remote_txn(
	340	self, txn: LoggingTransaction, destination: str, stream_id: int
	341	) -> None:
	342	# We update the device_lists_outbound_last_success with the successfully
	343	# poked users.
	344	sql = """
	345	SELECT user_id, coalesce(max(o.stream_id), 0)
	346	FROM device_lists_outbound_pokes as o
	347	WHERE destination = ? AND o.stream_id <= ?
	348	GROUP BY user_id
	349	"""
	350	txn.execute(sql, (destination, stream_id))
	351	rows = txn.fetchall()
	352
	353	self.db_pool.simple_upsert_many_txn(
	354	txn=txn,
	355	table="device_lists_outbound_last_success",
	356	key_names=("destination", "user_id"),
	357	key_values=((destination, user_id) for user_id, _ in rows),
	358	value_names=("stream_id",),
	359	value_values=((stream_id,) for _, stream_id in rows),
	360	)
	361
	362	# Delete all sent outbound pokes
	363	sql = """
	364	DELETE FROM device_lists_outbound_pokes
	365	WHERE destination = ? AND stream_id <= ?
	366	"""
	367	txn.execute(sql, (destination, stream_id))
	368
	369	async def add_user_signature_change_to_streams(
	370	self, from_user_id: str, user_ids: List[str]
	371	) -> int:
	372	"""Persist that a user has made new signatures
	373
	374	Args:
	375	from_user_id: the user who made the signatures
	376	user_ids: the users who were signed
	377
	378	Returns:
	379	THe new stream ID.
	380	"""
	381
	382	with self._device_list_id_gen.get_next() as stream_id:
	383	await self.db_pool.runInteraction(
	384	"add_user_sig_change_to_streams",
	385	self._add_user_signature_change_txn,
	386	from_user_id,
	387	user_ids,
	388	stream_id,
	389	)
	390	return stream_id
	391
	392	def _add_user_signature_change_txn(
	393	self,
	394	txn: LoggingTransaction,
	395	from_user_id: str,
	396	user_ids: List[str],
	397	stream_id: int,
	398	) -> None:
	399	txn.call_after(
	400	self._user_signature_stream_cache.entity_has_changed,
	401	from_user_id,
	402	stream_id,
	403	)
	404	self.db_pool.simple_insert_txn(
	405	txn,
	406	"user_signature_stream",
	407	values={
	408	"stream_id": stream_id,
	409	"from_user_id": from_user_id,
	410	"user_ids": json_encoder.encode(user_ids),
	411	},
	412	)
	413
	414	def get_device_stream_token(self) -> int:
	415	return self._device_list_id_gen.get_current_token()
	416
	417	@trace
	418	async def get_user_devices_from_cache(
	419	self, query_list: List[Tuple[str, str]]
	420	) -> Tuple[Set[str], Dict[str, Dict[str, JsonDict]]]:
	421	"""Get the devices (and keys if any) for remote users from the cache.
	422
	423	Args:
	424	query_list: List of (user_id, device_ids), if device_ids is
	425	falsey then return all device ids for that user.
	426
	427	Returns:
	428	A tuple of (user_ids_not_in_cache, results_map), where
	429	user_ids_not_in_cache is a set of user_ids and results_map is a
	430	mapping of user_id -> device_id -> device_info.
	431	"""
	432	user_ids = {user_id for user_id, _ in query_list}
	433	user_map = await self.get_device_list_last_stream_id_for_remotes(list(user_ids))
	434
	435	# We go and check if any of the users need to have their device lists
	436	# resynced. If they do then we remove them from the cached list.
	437	users_needing_resync = await self.get_user_ids_requiring_device_list_resync(
	438	user_ids
	439	)
	440	user_ids_in_cache = {
	441	user_id for user_id, stream_id in user_map.items() if stream_id
	442	} - users_needing_resync
	443	user_ids_not_in_cache = user_ids - user_ids_in_cache
	444
	445	results = {}
	446	for user_id, device_id in query_list:
	447	if user_id not in user_ids_in_cache:
	448	continue
	449
	450	if device_id:
	451	device = await self._get_cached_user_device(user_id, device_id)
	452	results.setdefault(user_id, {})[device_id] = device
	453	else:
	454	results[user_id] = await self.get_cached_devices_for_user(user_id)
	455
	456	set_tag("in_cache", results)
	457	set_tag("not_in_cache", user_ids_not_in_cache)
	458
	459	return user_ids_not_in_cache, results
	460
	461	@cached(num_args=2, tree=True)
	462	async def _get_cached_user_device(self, user_id: str, device_id: str) -> JsonDict:
	463	content = await self.db_pool.simple_select_one_onecol(
	464	table="device_lists_remote_cache",
	465	keyvalues={"user_id": user_id, "device_id": device_id},
	466	retcol="content",
	467	desc="_get_cached_user_device",
	468	)
	469	return db_to_json(content)
	470
	471	@cached()
	472	async def get_cached_devices_for_user(self, user_id: str) -> Dict[str, JsonDict]:
	473	devices = await self.db_pool.simple_select_list(
	474	table="device_lists_remote_cache",
	475	keyvalues={"user_id": user_id},
	476	retcols=("device_id", "content"),
	477	desc="get_cached_devices_for_user",
	478	)
	479	return {
	480	device["device_id"]: db_to_json(device["content"]) for device in devices
	481	}
	482
	483	def get_devices_with_keys_by_user(self, user_id: str):
	484	"""Get all devices (with any device keys) for a user
	485
	486	Returns:
	487	Deferred which resolves to (stream_id, devices)
	488	"""
	489	return self.db_pool.runInteraction(
	490	"get_devices_with_keys_by_user",
	491	self._get_devices_with_keys_by_user_txn,
	492	user_id,
	493	)
	494
	495	def _get_devices_with_keys_by_user_txn(
	496	self, txn: LoggingTransaction, user_id: str
	497	) -> Tuple[int, List[JsonDict]]:
	498	now_stream_id = self._device_list_id_gen.get_current_token()
	499
	500	devices = self._get_e2e_device_keys_txn(
	501	txn, [(user_id, None)], include_all_devices=True
	502	)
	503
	504	if devices:
	505	user_devices = devices[user_id]
	506	results = []
	507	for device_id, device in user_devices.items():
	508	result = {"device_id": device_id}
	509
	510	key_json = device.get("key_json", None)
	511	if key_json:
	512	result["keys"] = db_to_json(key_json)
	513
	514	if "signatures" in device:
	515	for sig_user_id, sigs in device["signatures"].items():
	516	result["keys"].setdefault("signatures", {}).setdefault(
	517	sig_user_id, {}
	518	).update(sigs)
	519
	520	device_display_name = device.get("device_display_name", None)
	521	if device_display_name:
	522	result["device_display_name"] = device_display_name
	523
	524	results.append(result)
	525
	526	return now_stream_id, results
	527
	528	return now_stream_id, []
	529
	530	async def get_users_whose_devices_changed(
	531	self, from_key: str, user_ids: Iterable[str]
	532	) -> Set[str]:
	533	"""Get set of users whose devices have changed since `from_key` that
	534	are in the given list of user_ids.
	535
	536	Args:
	537	from_key: The device lists stream token
	538	user_ids: The user IDs to query for devices.
	539
	540	Returns:
	541	The set of user_ids whose devices have changed since `from_key`
	542	"""
	543	from_key = int(from_key)
	544
	545	# Get set of users who may have changed. Users not in the returned
	546	# list have definitely not changed.
	547	to_check = self._device_list_stream_cache.get_entities_changed(
	548	user_ids, from_key
	549	)
	550
	551	if not to_check:
	552	return set()
	553
	554	def _get_users_whose_devices_changed_txn(txn):
	555	changes = set()
	556
	557	sql = """
	558	SELECT DISTINCT user_id FROM device_lists_stream
	559	WHERE stream_id > ?
	560	AND
	561	"""
	562
	563	for chunk in batch_iter(to_check, 100):
	564	clause, args = make_in_list_sql_clause(
	565	txn.database_engine, "user_id", chunk
	566	)
	567	txn.execute(sql + clause, (from_key,) + tuple(args))
	568	changes.update(user_id for user_id, in txn)
	569
	570	return changes
	571
	572	return await self.db_pool.runInteraction(
	573	"get_users_whose_devices_changed", _get_users_whose_devices_changed_txn
	574	)
	575
	576	async def get_users_whose_signatures_changed(
	577	self, user_id: str, from_key: str
	578	) -> Set[str]:
	579	"""Get the users who have new cross-signing signatures made by `user_id` since
	580	`from_key`.
	581
	582	Args:
	583	user_id: the user who made the signatures
	584	from_key: The device lists stream token
	585
	586	Returns:
	587	A set of user IDs with updated signatures.
	588	"""
	589	from_key = int(from_key)
	590	if self._user_signature_stream_cache.has_entity_changed(user_id, from_key):
	591	sql = """
	592	SELECT DISTINCT user_ids FROM user_signature_stream
	593	WHERE from_user_id = ? AND stream_id > ?
	594	"""
	595	rows = await self.db_pool.execute(
	596	"get_users_whose_signatures_changed", None, sql, user_id, from_key
	597	)
	598	return {user for row in rows for user in db_to_json(row[0])}
	599	else:
	600	return set()
	601
	602	async def get_all_device_list_changes_for_remotes(
	603	self, instance_name: str, last_id: int, current_id: int, limit: int
	604	) -> Tuple[List[Tuple[int, tuple]], int, bool]:
	605	"""Get updates for device lists replication stream.
	606
	607	Args:
	608	instance_name: The writer we want to fetch updates from. Unused
	609	here since there is only ever one writer.
	610	last_id: The token to fetch updates from. Exclusive.
	611	current_id: The token to fetch updates up to. Inclusive.
	612	limit: The requested limit for the number of rows to return. The
	613	function may return more or fewer rows.
	614
	615	Returns:
	616	A tuple consisting of: the updates, a token to use to fetch
	617	subsequent updates, and whether we returned fewer rows than exists
	618	between the requested tokens due to the limit.
	619
	620	The token returned can be used in a subsequent call to this
	621	function to get further updates.
	622
	623	The updates are a list of 2-tuples of stream ID and the row data
	624	"""
	625
	626	if last_id == current_id:
	627	return [], current_id, False
	628
	629	def _get_all_device_list_changes_for_remotes(txn):
	630	# This query Does The Right Thing where it'll correctly apply the
	631	# bounds to the inner queries.
	632	sql = """
	633	SELECT stream_id, entity FROM (
	634	SELECT stream_id, user_id AS entity FROM device_lists_stream
	635	UNION ALL
	636	SELECT stream_id, destination AS entity FROM device_lists_outbound_pokes
	637	) AS e
	638	WHERE ? < stream_id AND stream_id <= ?
	639	LIMIT ?
	640	"""
	641
	642	txn.execute(sql, (last_id, current_id, limit))
	643	updates = [(row[0], row[1:]) for row in txn]
	644	limited = False
	645	upto_token = current_id
	646	if len(updates) >= limit:
	647	upto_token = updates[-1][0]
	648	limited = True
	649
	650	return updates, upto_token, limited
	651
	652	return await self.db_pool.runInteraction(
	653	"get_all_device_list_changes_for_remotes",
	654	_get_all_device_list_changes_for_remotes,
	655	)
	656
	657	@cached(max_entries=10000)
	658	def get_device_list_last_stream_id_for_remote(self, user_id: str):
	659	"""Get the last stream_id we got for a user. May be None if we haven't
	660	got any information for them.
	661	"""
	662	return self.db_pool.simple_select_one_onecol(
	663	table="device_lists_remote_extremeties",
	664	keyvalues={"user_id": user_id},
	665	retcol="stream_id",
	666	desc="get_device_list_last_stream_id_for_remote",
	667	allow_none=True,
	668	)
	669
	670	@cachedList(
	671	cached_method_name="get_device_list_last_stream_id_for_remote",
	672	list_name="user_ids",
	673	inlineCallbacks=True,
	674	)
	675	def get_device_list_last_stream_id_for_remotes(self, user_ids: str):
	676	rows = yield self.db_pool.simple_select_many_batch(
	677	table="device_lists_remote_extremeties",
	678	column="user_id",
	679	iterable=user_ids,
	680	retcols=("user_id", "stream_id"),
	681	desc="get_device_list_last_stream_id_for_remotes",
	682	)
	683
	684	results = {user_id: None for user_id in user_ids}
	685	results.update({row["user_id"]: row["stream_id"] for row in rows})
	686
	687	return results
	688
	689	async def get_user_ids_requiring_device_list_resync(
	690	self, user_ids: Optional[Collection[str]] = None,
	691	) -> Set[str]:
	692	"""Given a list of remote users return the list of users that we
	693	should resync the device lists for. If None is given instead of a list,
	694	return every user that we should resync the device lists for.
	695
	696	Returns:
	697	The IDs of users whose device lists need resync.
	698	"""
	699	if user_ids:
	700	rows = await self.db_pool.simple_select_many_batch(
	701	table="device_lists_remote_resync",
	702	column="user_id",
	703	iterable=user_ids,
	704	retcols=("user_id",),
	705	desc="get_user_ids_requiring_device_list_resync_with_iterable",
	706	)
	707	else:
	708	rows = await self.db_pool.simple_select_list(
	709	table="device_lists_remote_resync",
	710	keyvalues=None,
	711	retcols=("user_id",),
	712	desc="get_user_ids_requiring_device_list_resync",
	713	)
	714
	715	return {row["user_id"] for row in rows}
	716
	717	def mark_remote_user_device_cache_as_stale(self, user_id: str):
	718	"""Records that the server has reason to believe the cache of the devices
	719	for the remote users is out of date.
	720	"""
	721	return self.db_pool.simple_upsert(
	722	table="device_lists_remote_resync",
	723	keyvalues={"user_id": user_id},
	724	values={},
	725	insertion_values={"added_ts": self._clock.time_msec()},
	726	desc="make_remote_user_device_cache_as_stale",
	727	)
	728
	729	def mark_remote_user_device_list_as_unsubscribed(self, user_id: str):
	730	"""Mark that we no longer track device lists for remote user.
	731	"""
	732
	733	def _mark_remote_user_device_list_as_unsubscribed_txn(txn):
	734	self.db_pool.simple_delete_txn(
	735	txn,
	736	table="device_lists_remote_extremeties",
	737	keyvalues={"user_id": user_id},
	738	)
	739	self._invalidate_cache_and_stream(
	740	txn, self.get_device_list_last_stream_id_for_remote, (user_id,)
	741	)
	742
	743	return self.db_pool.runInteraction(
	744	"mark_remote_user_device_list_as_unsubscribed",
	745	_mark_remote_user_device_list_as_unsubscribed_txn,
	746	)
	747
	748
	749	class DeviceBackgroundUpdateStore(SQLBaseStore):
	750	def __init__(self, database: DatabasePool, db_conn, hs):
	751	super(DeviceBackgroundUpdateStore, self).__init__(database, db_conn, hs)
	752
	753	self.db_pool.updates.register_background_index_update(
	754	"device_lists_stream_idx",
	755	index_name="device_lists_stream_user_id",
	756	table="device_lists_stream",
	757	columns=["user_id", "device_id"],
	758	)
	759
	760	# create a unique index on device_lists_remote_cache
	761	self.db_pool.updates.register_background_index_update(
	762	"device_lists_remote_cache_unique_idx",
	763	index_name="device_lists_remote_cache_unique_id",
	764	table="device_lists_remote_cache",
	765	columns=["user_id", "device_id"],
	766	unique=True,
	767	)
	768
	769	# And one on device_lists_remote_extremeties
	770	self.db_pool.updates.register_background_index_update(
	771	"device_lists_remote_extremeties_unique_idx",
	772	index_name="device_lists_remote_extremeties_unique_idx",
	773	table="device_lists_remote_extremeties",
	774	columns=["user_id"],
	775	unique=True,
	776	)
	777
	778	# once they complete, we can remove the old non-unique indexes.
	779	self.db_pool.updates.register_background_update_handler(
	780	DROP_DEVICE_LIST_STREAMS_NON_UNIQUE_INDEXES,
	781	self._drop_device_list_streams_non_unique_indexes,
	782	)
	783
	784	# clear out duplicate device list outbound pokes
	785	self.db_pool.updates.register_background_update_handler(
	786	BG_UPDATE_REMOVE_DUP_OUTBOUND_POKES, self._remove_duplicate_outbound_pokes,
	787	)
	788
	789	# a pair of background updates that were added during the 1.14 release cycle,
	790	# but replaced with 58/06dlols_unique_idx.py
	791	self.db_pool.updates.register_noop_background_update(
	792	"device_lists_outbound_last_success_unique_idx",
	793	)
	794	self.db_pool.updates.register_noop_background_update(
	795	"drop_device_lists_outbound_last_success_non_unique_idx",
	796	)
	797
	798	async def _drop_device_list_streams_non_unique_indexes(self, progress, batch_size):
	799	def f(conn):
	800	txn = conn.cursor()
	801	txn.execute("DROP INDEX IF EXISTS device_lists_remote_cache_id")
	802	txn.execute("DROP INDEX IF EXISTS device_lists_remote_extremeties_id")
	803	txn.close()
	804
	805	await self.db_pool.runWithConnection(f)
	806	await self.db_pool.updates._end_background_update(
	807	DROP_DEVICE_LIST_STREAMS_NON_UNIQUE_INDEXES
	808	)
	809	return 1
	810
	811	async def _remove_duplicate_outbound_pokes(self, progress, batch_size):
	812	# for some reason, we have accumulated duplicate entries in
	813	# device_lists_outbound_pokes, which makes prune_outbound_device_list_pokes less
	814	# efficient.
	815	#
	816	# For each duplicate, we delete all the existing rows and put one back.
	817
	818	KEY_COLS = ["stream_id", "destination", "user_id", "device_id"]
	819	last_row = progress.get(
	820	"last_row",
	821	{"stream_id": 0, "destination": "", "user_id": "", "device_id": ""},
	822	)
	823
	824	def _txn(txn):
	825	clause, args = make_tuple_comparison_clause(
	826	self.db_pool.engine, [(x, last_row[x]) for x in KEY_COLS]
	827	)
	828	sql = """
	829	SELECT stream_id, destination, user_id, device_id, MAX(ts) AS ts
	830	FROM device_lists_outbound_pokes
	831	WHERE %s
	832	GROUP BY %s
	833	HAVING count(*) > 1
	834	ORDER BY %s
	835	LIMIT ?
	836	""" % (
	837	clause, # WHERE
	838	",".join(KEY_COLS), # GROUP BY
	839	",".join(KEY_COLS), # ORDER BY
	840	)
	841	txn.execute(sql, args + [batch_size])
	842	rows = self.db_pool.cursor_to_dict(txn)
	843
	844	row = None
	845	for row in rows:
	846	self.db_pool.simple_delete_txn(
	847	txn, "device_lists_outbound_pokes", {x: row[x] for x in KEY_COLS},
	848	)
	849
	850	row["sent"] = False
	851	self.db_pool.simple_insert_txn(
	852	txn, "device_lists_outbound_pokes", row,
	853	)
	854
	855	if row:
	856	self.db_pool.updates._background_update_progress_txn(
	857	txn, BG_UPDATE_REMOVE_DUP_OUTBOUND_POKES, {"last_row": row},
	858	)
	859
	860	return len(rows)
	861
	862	rows = await self.db_pool.runInteraction(
	863	BG_UPDATE_REMOVE_DUP_OUTBOUND_POKES, _txn
	864	)
	865
	866	if not rows:
	867	await self.db_pool.updates._end_background_update(
	868	BG_UPDATE_REMOVE_DUP_OUTBOUND_POKES
	869	)
	870
	871	return rows
	872
	873
	874	class DeviceStore(DeviceWorkerStore, DeviceBackgroundUpdateStore):
	875	def __init__(self, database: DatabasePool, db_conn, hs):
	876	super(DeviceStore, self).__init__(database, db_conn, hs)
	877
	878	# Map of (user_id, device_id) -> bool. If there is an entry that implies
	879	# the device exists.
	880	self.device_id_exists_cache = Cache(
	881	name="device_id_exists", keylen=2, max_entries=10000
	882	)
	883
	884	self._clock.looping_call(self._prune_old_outbound_device_pokes, 60 * 60 * 1000)
	885
	886	async def store_device(
	887	self, user_id: str, device_id: str, initial_device_display_name: str
	888	) -> bool:
	889	"""Ensure the given device is known; add it to the store if not
	890
	891	Args:
	892	user_id: id of user associated with the device
	893	device_id: id of device
	894	initial_device_display_name: initial displayname of the device.
	895	Ignored if device exists.
	896
	897	Returns:
	898	Whether the device was inserted or an existing device existed with that ID.
	899
	900	Raises:
	901	StoreError: if the device is already in use
	902	"""
	903	key = (user_id, device_id)
	904	if self.device_id_exists_cache.get(key, None):
	905	return False
	906
	907	try:
	908	inserted = await self.db_pool.simple_insert(
	909	"devices",
	910	values={
	911	"user_id": user_id,
	912	"device_id": device_id,
	913	"display_name": initial_device_display_name,
	914	"hidden": False,
	915	},
	916	desc="store_device",
	917	or_ignore=True,
	918	)
	919	if not inserted:
	920	# if the device already exists, check if it's a real device, or
	921	# if the device ID is reserved by something else
	922	hidden = await self.db_pool.simple_select_one_onecol(
	923	"devices",
	924	keyvalues={"user_id": user_id, "device_id": device_id},
	925	retcol="hidden",
	926	)
	927	if hidden:
	928	raise StoreError(400, "The device ID is in use", Codes.FORBIDDEN)
	929	self.device_id_exists_cache.prefill(key, True)
	930	return inserted
	931	except StoreError:
	932	raise
	933	except Exception as e:
	934	logger.error(
	935	"store_device with device_id=%s(%r) user_id=%s(%r)"
	936	" display_name=%s(%r) failed: %s",
	937	type(device_id).__name__,
	938	device_id,
	939	type(user_id).__name__,
	940	user_id,
	941	type(initial_device_display_name).__name__,
	942	initial_device_display_name,
	943	e,
	944	)
	945	raise StoreError(500, "Problem storing device.")
	946
	947	async def delete_device(self, user_id: str, device_id: str) -> None:
	948	"""Delete a device.
	949
	950	Args:
	951	user_id: The ID of the user which owns the device
	952	device_id: The ID of the device to delete
	953	"""
	954	await self.db_pool.simple_delete_one(
	955	table="devices",
	956	keyvalues={"user_id": user_id, "device_id": device_id, "hidden": False},
	957	desc="delete_device",
	958	)
	959
	960	self.device_id_exists_cache.invalidate((user_id, device_id))
	961
	962	async def delete_devices(self, user_id: str, device_ids: List[str]) -> None:
	963	"""Deletes several devices.
	964
	965	Args:
	966	user_id: The ID of the user which owns the devices
	967	device_ids: The IDs of the devices to delete
	968	"""
	969	await self.db_pool.simple_delete_many(
	970	table="devices",
	971	column="device_id",
	972	iterable=device_ids,
	973	keyvalues={"user_id": user_id, "hidden": False},
	974	desc="delete_devices",
	975	)
	976	for device_id in device_ids:
	977	self.device_id_exists_cache.invalidate((user_id, device_id))
	978
	979	async def update_device(
	980	self, user_id: str, device_id: str, new_display_name: Optional[str] = None
	981	) -> None:
	982	"""Update a device. Only updates the device if it is not marked as
	983	hidden.
	984
	985	Args:
	986	user_id: The ID of the user which owns the device
	987	device_id: The ID of the device to update
	988	new_display_name: new displayname for device; None to leave unchanged
	989	Raises:
	990	StoreError: if the device is not found
	991	"""
	992	updates = {}
	993	if new_display_name is not None:
	994	updates["display_name"] = new_display_name
	995	if not updates:
	996	return None
	997	await self.db_pool.simple_update_one(
	998	table="devices",
	999	keyvalues={"user_id": user_id, "device_id": device_id, "hidden": False},
	1000	updatevalues=updates,
	1001	desc="update_device",
	1002	)
	1003
	1004	def update_remote_device_list_cache_entry(
	1005	self, user_id: str, device_id: str, content: JsonDict, stream_id: int
	1006	):
	1007	"""Updates a single device in the cache of a remote user's devicelist.
	1008
	1009	Note: assumes that we are the only thread that can be updating this user's
	1010	device list.
	1011
	1012	Args:
	1013	user_id: User to update device list for
	1014	device_id: ID of decivice being updated
	1015	content: new data on this device
	1016	stream_id: the version of the device list
	1017
	1018	Returns:
	1019	Deferred[None]
	1020	"""
	1021	return self.db_pool.runInteraction(
	1022	"update_remote_device_list_cache_entry",
	1023	self._update_remote_device_list_cache_entry_txn,
	1024	user_id,
	1025	device_id,
	1026	content,
	1027	stream_id,
	1028	)
	1029
	1030	def _update_remote_device_list_cache_entry_txn(
	1031	self,
	1032	txn: LoggingTransaction,
	1033	user_id: str,
	1034	device_id: str,
	1035	content: JsonDict,
	1036	stream_id: int,
	1037	) -> None:
	1038	if content.get("deleted"):
	1039	self.db_pool.simple_delete_txn(
	1040	txn,
	1041	table="device_lists_remote_cache",
	1042	keyvalues={"user_id": user_id, "device_id": device_id},
	1043	)
	1044
	1045	txn.call_after(self.device_id_exists_cache.invalidate, (user_id, device_id))
	1046	else:
	1047	self.db_pool.simple_upsert_txn(
	1048	txn,
	1049	table="device_lists_remote_cache",
	1050	keyvalues={"user_id": user_id, "device_id": device_id},
	1051	values={"content": json_encoder.encode(content)},
	1052	# we don't need to lock, because we assume we are the only thread
	1053	# updating this user's devices.
	1054	lock=False,
	1055	)
	1056
	1057	txn.call_after(self._get_cached_user_device.invalidate, (user_id, device_id))
	1058	txn.call_after(self.get_cached_devices_for_user.invalidate, (user_id,))
	1059	txn.call_after(
	1060	self.get_device_list_last_stream_id_for_remote.invalidate, (user_id,)
	1061	)
	1062
	1063	self.db_pool.simple_upsert_txn(
	1064	txn,
	1065	table="device_lists_remote_extremeties",
	1066	keyvalues={"user_id": user_id},
	1067	values={"stream_id": stream_id},
	1068	# again, we can assume we are the only thread updating this user's
	1069	# extremity.
	1070	lock=False,
	1071	)
	1072
	1073	def update_remote_device_list_cache(
	1074	self, user_id: str, devices: List[dict], stream_id: int
	1075	):
	1076	"""Replace the entire cache of the remote user's devices.
	1077
	1078	Note: assumes that we are the only thread that can be updating this user's
	1079	device list.
	1080
	1081	Args:
	1082	user_id: User to update device list for
	1083	devices: list of device objects supplied over federation
	1084	stream_id: the version of the device list
	1085
	1086	Returns:
	1087	Deferred[None]
	1088	"""
	1089	return self.db_pool.runInteraction(
	1090	"update_remote_device_list_cache",
	1091	self._update_remote_device_list_cache_txn,
	1092	user_id,
	1093	devices,
	1094	stream_id,
	1095	)
	1096
	1097	def _update_remote_device_list_cache_txn(
	1098	self, txn: LoggingTransaction, user_id: str, devices: List[dict], stream_id: int
	1099	):
	1100	self.db_pool.simple_delete_txn(
	1101	txn, table="device_lists_remote_cache", keyvalues={"user_id": user_id}
	1102	)
	1103
	1104	self.db_pool.simple_insert_many_txn(
	1105	txn,
	1106	table="device_lists_remote_cache",
	1107	values=[
	1108	{
	1109	"user_id": user_id,
	1110	"device_id": content["device_id"],
	1111	"content": json_encoder.encode(content),
	1112	}
	1113	for content in devices
	1114	],
	1115	)
	1116
	1117	txn.call_after(self.get_cached_devices_for_user.invalidate, (user_id,))
	1118	txn.call_after(self._get_cached_user_device.invalidate_many, (user_id,))
	1119	txn.call_after(
	1120	self.get_device_list_last_stream_id_for_remote.invalidate, (user_id,)
	1121	)
	1122
	1123	self.db_pool.simple_upsert_txn(
	1124	txn,
	1125	table="device_lists_remote_extremeties",
	1126	keyvalues={"user_id": user_id},
	1127	values={"stream_id": stream_id},
	1128	# we don't need to lock, because we can assume we are the only thread
	1129	# updating this user's extremity.
	1130	lock=False,
	1131	)
	1132
	1133	# If we're replacing the remote user's device list cache presumably
	1134	# we've done a full resync, so we remove the entry that says we need
	1135	# to resync
	1136	self.db_pool.simple_delete_txn(
	1137	txn, table="device_lists_remote_resync", keyvalues={"user_id": user_id},
	1138	)
	1139
	1140	async def add_device_change_to_streams(
	1141	self, user_id: str, device_ids: Collection[str], hosts: List[str]
	1142	):
	1143	"""Persist that a user's devices have been updated, and which hosts
	1144	(if any) should be poked.
	1145	"""
	1146	if not device_ids:
	1147	return
	1148
	1149	with self._device_list_id_gen.get_next_mult(len(device_ids)) as stream_ids:
	1150	await self.db_pool.runInteraction(
	1151	"add_device_change_to_stream",
	1152	self._add_device_change_to_stream_txn,
	1153	user_id,
	1154	device_ids,
	1155	stream_ids,
	1156	)
	1157
	1158	if not hosts:
	1159	return stream_ids[-1]
	1160
	1161	context = get_active_span_text_map()
	1162	with self._device_list_id_gen.get_next_mult(
	1163	len(hosts) * len(device_ids)
	1164	) as stream_ids:
	1165	await self.db_pool.runInteraction(
	1166	"add_device_outbound_poke_to_stream",
	1167	self._add_device_outbound_poke_to_stream_txn,
	1168	user_id,
	1169	device_ids,
	1170	hosts,
	1171	stream_ids,
	1172	context,
	1173	)
	1174
	1175	return stream_ids[-1]
	1176
	1177	def _add_device_change_to_stream_txn(
	1178	self,
	1179	txn: LoggingTransaction,
	1180	user_id: str,
	1181	device_ids: Collection[str],
	1182	stream_ids: List[str],
	1183	):
	1184	txn.call_after(
	1185	self._device_list_stream_cache.entity_has_changed, user_id, stream_ids[-1],
	1186	)
	1187
	1188	min_stream_id = stream_ids[0]
	1189
	1190	# Delete older entries in the table, as we really only care about
	1191	# when the latest change happened.
	1192	txn.executemany(
	1193	"""
	1194	DELETE FROM device_lists_stream
	1195	WHERE user_id = ? AND device_id = ? AND stream_id < ?
	1196	""",
	1197	[(user_id, device_id, min_stream_id) for device_id in device_ids],
	1198	)
	1199
	1200	self.db_pool.simple_insert_many_txn(
	1201	txn,
	1202	table="device_lists_stream",
	1203	values=[
	1204	{"stream_id": stream_id, "user_id": user_id, "device_id": device_id}
	1205	for stream_id, device_id in zip(stream_ids, device_ids)
	1206	],
	1207	)
	1208
	1209	def _add_device_outbound_poke_to_stream_txn(
	1210	self,
	1211	txn: LoggingTransaction,
	1212	user_id: str,
	1213	device_ids: Collection[str],
	1214	hosts: List[str],
	1215	stream_ids: List[str],
	1216	context: Dict[str, str],
	1217	):
	1218	for host in hosts:
	1219	txn.call_after(
	1220	self._device_list_federation_stream_cache.entity_has_changed,
	1221	host,
	1222	stream_ids[-1],
	1223	)
	1224
	1225	now = self._clock.time_msec()
	1226	next_stream_id = iter(stream_ids)
	1227
	1228	self.db_pool.simple_insert_many_txn(
	1229	txn,
	1230	table="device_lists_outbound_pokes",
	1231	values=[
	1232	{
	1233	"destination": destination,
	1234	"stream_id": next(next_stream_id),
	1235	"user_id": user_id,
	1236	"device_id": device_id,
	1237	"sent": False,
	1238	"ts": now,
	1239	"opentracing_context": json_encoder.encode(context)
	1240	if whitelisted_homeserver(destination)
	1241	else "{}",
	1242	}
	1243	for destination in hosts
	1244	for device_id in device_ids
	1245	],
	1246	)
	1247
	1248	def _prune_old_outbound_device_pokes(self, prune_age: int = 24 * 60 * 60 * 1000):
	1249	"""Delete old entries out of the device_lists_outbound_pokes to ensure
	1250	that we don't fill up due to dead servers.
	1251
	1252	Normally, we try to send device updates as a delta since a previous known point:
	1253	this is done by setting the prev_id in the m.device_list_update EDU. However,
	1254	for that to work, we have to have a complete record of each change to
	1255	each device, which can add up to quite a lot of data.
	1256
	1257	An alternative mechanism is that, if the remote server sees that it has missed
	1258	an entry in the stream_id sequence for a given user, it will request a full
	1259	list of that user's devices. Hence, we can reduce the amount of data we have to
	1260	store (and transmit in some future transaction), by clearing almost everything
	1261	for a given destination out of the database, and having the remote server
	1262	resync.
	1263
	1264	All we need to do is make sure we keep at least one row for each
	1265	(user, destination) pair, to remind us to send a m.device_list_update EDU for
	1266	that user when the destination comes back. It doesn't matter which device
	1267	we keep.
	1268	"""
	1269	yesterday = self._clock.time_msec() - prune_age
	1270
	1271	def _prune_txn(txn):
	1272	# look for (user, destination) pairs which have an update older than
	1273	# the cutoff.
	1274	#
	1275	# For each pair, we also need to know the most recent stream_id, and
	1276	# an arbitrary device_id at that stream_id.
	1277	select_sql = """
	1278	SELECT
	1279	dlop1.destination,
	1280	dlop1.user_id,
	1281	MAX(dlop1.stream_id) AS stream_id,
	1282	(SELECT MIN(dlop2.device_id) AS device_id FROM
	1283	device_lists_outbound_pokes dlop2
	1284	WHERE dlop2.destination = dlop1.destination AND
	1285	dlop2.user_id=dlop1.user_id AND
	1286	dlop2.stream_id=MAX(dlop1.stream_id)
	1287	)
	1288	FROM device_lists_outbound_pokes dlop1
	1289	GROUP BY destination, user_id
	1290	HAVING min(ts) < ? AND count(*) > 1
	1291	"""
	1292
	1293	txn.execute(select_sql, (yesterday,))
	1294	rows = txn.fetchall()
	1295
	1296	if not rows:
	1297	return
	1298
	1299	logger.info(
	1300	"Pruning old outbound device list updates for %i users/destinations: %s",
	1301	len(rows),
	1302	shortstr((row[0], row[1]) for row in rows),
	1303	)
	1304
	1305	# we want to keep the update with the highest stream_id for each user.
	1306	#
	1307	# there might be more than one update (with different device_ids) with the
	1308	# same stream_id, so we also delete all but one rows with the max stream id.
	1309	delete_sql = """
	1310	DELETE FROM device_lists_outbound_pokes
	1311	WHERE destination = ? AND user_id = ? AND (
	1312	stream_id < ? OR
	1313	(stream_id = ? AND device_id != ?)
	1314	)
	1315	"""
	1316	count = 0
	1317	for (destination, user_id, stream_id, device_id) in rows:
	1318	txn.execute(
	1319	delete_sql, (destination, user_id, stream_id, stream_id, device_id)
	1320	)
	1321	count += txn.rowcount
	1322
	1323	# Since we've deleted unsent deltas, we need to remove the entry
	1324	# of last successful sent so that the prev_ids are correctly set.
	1325	sql = """
	1326	DELETE FROM device_lists_outbound_last_success
	1327	WHERE destination = ? AND user_id = ?
	1328	"""
	1329	txn.executemany(sql, ((row[0], row[1]) for row in rows))
	1330
	1331	logger.info("Pruned %d device list outbound pokes", count)
	1332
	1333	return run_as_background_process(
	1334	"prune_old_outbound_device_pokes",
	1335	self.db_pool.runInteraction,
	1336	"_prune_old_outbound_device_pokes",
	1337	_prune_txn,
	1338	)

+194

-0

synapse/storage/databases/main/directory.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from collections import namedtuple
	16	from typing import Iterable, Optional
	17
	18	from synapse.api.errors import SynapseError
	19	from synapse.storage._base import SQLBaseStore
	20	from synapse.types import RoomAlias
	21	from synapse.util.caches.descriptors import cached
	22
	23	RoomAliasMapping = namedtuple("RoomAliasMapping", ("room_id", "room_alias", "servers"))
	24
	25
	26	class DirectoryWorkerStore(SQLBaseStore):
	27	async def get_association_from_room_alias(
	28	self, room_alias: RoomAlias
	29	) -> Optional[RoomAliasMapping]:
	30	"""Gets the room_id and server list for a given room_alias
	31
	32	Args:
	33	room_alias: The alias to translate to an ID.
	34
	35	Returns:
	36	The room alias mapping or None if no association can be found.
	37	"""
	38	room_id = await self.db_pool.simple_select_one_onecol(
	39	"room_aliases",
	40	{"room_alias": room_alias.to_string()},
	41	"room_id",
	42	allow_none=True,
	43	desc="get_association_from_room_alias",
	44	)
	45
	46	if not room_id:
	47	return None
	48
	49	servers = await self.db_pool.simple_select_onecol(
	50	"room_alias_servers",
	51	{"room_alias": room_alias.to_string()},
	52	"server",
	53	desc="get_association_from_room_alias",
	54	)
	55
	56	if not servers:
	57	return None
	58
	59	return RoomAliasMapping(room_id, room_alias.to_string(), servers)
	60
	61	def get_room_alias_creator(self, room_alias):
	62	return self.db_pool.simple_select_one_onecol(
	63	table="room_aliases",
	64	keyvalues={"room_alias": room_alias},
	65	retcol="creator",
	66	desc="get_room_alias_creator",
	67	)
	68
	69	@cached(max_entries=5000)
	70	def get_aliases_for_room(self, room_id):
	71	return self.db_pool.simple_select_onecol(
	72	"room_aliases",
	73	{"room_id": room_id},
	74	"room_alias",
	75	desc="get_aliases_for_room",
	76	)
	77
	78
	79	class DirectoryStore(DirectoryWorkerStore):
	80	async def create_room_alias_association(
	81	self,
	82	room_alias: RoomAlias,
	83	room_id: str,
	84	servers: Iterable[str],
	85	creator: Optional[str] = None,
	86	) -> None:
	87	""" Creates an association between a room alias and room_id/servers
	88
	89	Args:
	90	room_alias: The alias to create.
	91	room_id: The target of the alias.
	92	servers: A list of servers through which it may be possible to join the room
	93	creator: Optional user_id of creator.
	94	"""
	95
	96	def alias_txn(txn):
	97	self.db_pool.simple_insert_txn(
	98	txn,
	99	"room_aliases",
	100	{
	101	"room_alias": room_alias.to_string(),
	102	"room_id": room_id,
	103	"creator": creator,
	104	},
	105	)
	106
	107	self.db_pool.simple_insert_many_txn(
	108	txn,
	109	table="room_alias_servers",
	110	values=[
	111	{"room_alias": room_alias.to_string(), "server": server}
	112	for server in servers
	113	],
	114	)
	115
	116	self._invalidate_cache_and_stream(
	117	txn, self.get_aliases_for_room, (room_id,)
	118	)
	119
	120	try:
	121	await self.db_pool.runInteraction(
	122	"create_room_alias_association", alias_txn
	123	)
	124	except self.database_engine.module.IntegrityError:
	125	raise SynapseError(
	126	409, "Room alias %s already exists" % room_alias.to_string()
	127	)
	128
	129	async def delete_room_alias(self, room_alias: RoomAlias) -> str:
	130	room_id = await self.db_pool.runInteraction(
	131	"delete_room_alias", self._delete_room_alias_txn, room_alias
	132	)
	133
	134	return room_id
	135
	136	def _delete_room_alias_txn(self, txn, room_alias: RoomAlias) -> str:
	137	txn.execute(
	138	"SELECT room_id FROM room_aliases WHERE room_alias = ?",
	139	(room_alias.to_string(),),
	140	)
	141
	142	res = txn.fetchone()
	143	if res:
	144	room_id = res[0]
	145	else:
	146	return None
	147
	148	txn.execute(
	149	"DELETE FROM room_aliases WHERE room_alias = ?", (room_alias.to_string(),)
	150	)
	151
	152	txn.execute(
	153	"DELETE FROM room_alias_servers WHERE room_alias = ?",
	154	(room_alias.to_string(),),
	155	)
	156
	157	self._invalidate_cache_and_stream(txn, self.get_aliases_for_room, (room_id,))
	158
	159	return room_id
	160
	161	def update_aliases_for_room(
	162	self, old_room_id: str, new_room_id: str, creator: Optional[str] = None,
	163	):
	164	"""Repoint all of the aliases for a given room, to a different room.
	165
	166	Args:
	167	old_room_id:
	168	new_room_id:
	169	creator: The user to record as the creator of the new mapping.
	170	If None, the creator will be left unchanged.
	171	"""
	172
	173	def _update_aliases_for_room_txn(txn):
	174	update_creator_sql = ""
	175	sql_params = (new_room_id, old_room_id)
	176	if creator:
	177	update_creator_sql = ", creator = ?"
	178	sql_params = (new_room_id, creator, old_room_id)
	179
	180	sql = "UPDATE room_aliases SET room_id = ? %s WHERE room_id = ?" % (
	181	update_creator_sql,
	182	)
	183	txn.execute(sql, sql_params)
	184	self._invalidate_cache_and_stream(
	185	txn, self.get_aliases_for_room, (old_room_id,)
	186	)
	187	self._invalidate_cache_and_stream(
	188	txn, self.get_aliases_for_room, (new_room_id,)
	189	)
	190
	191	return self.db_pool.runInteraction(
	192	"_update_aliases_for_room_txn", _update_aliases_for_room_txn
	193	)

+436

-0

synapse/storage/databases/main/e2e_room_keys.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2017 New Vector Ltd
	2	# Copyright 2019 Matrix.org Foundation C.I.C.
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	from synapse.api.errors import StoreError
	17	from synapse.logging.opentracing import log_kv, trace
	18	from synapse.storage._base import SQLBaseStore, db_to_json
	19	from synapse.util import json_encoder
	20
	21
	22	class EndToEndRoomKeyStore(SQLBaseStore):
	23	async def update_e2e_room_key(
	24	self, user_id, version, room_id, session_id, room_key
	25	):
	26	"""Replaces the encrypted E2E room key for a given session in a given backup
	27
	28	Args:
	29	user_id(str): the user whose backup we're setting
	30	version(str): the version ID of the backup we're updating
	31	room_id(str): the ID of the room whose keys we're setting
	32	session_id(str): the session whose room_key we're setting
	33	room_key(dict): the room_key being set
	34	Raises:
	35	StoreError
	36	"""
	37
	38	await self.db_pool.simple_update_one(
	39	table="e2e_room_keys",
	40	keyvalues={
	41	"user_id": user_id,
	42	"version": version,
	43	"room_id": room_id,
	44	"session_id": session_id,
	45	},
	46	updatevalues={
	47	"first_message_index": room_key["first_message_index"],
	48	"forwarded_count": room_key["forwarded_count"],
	49	"is_verified": room_key["is_verified"],
	50	"session_data": json_encoder.encode(room_key["session_data"]),
	51	},
	52	desc="update_e2e_room_key",
	53	)
	54
	55	async def add_e2e_room_keys(self, user_id, version, room_keys):
	56	"""Bulk add room keys to a given backup.
	57
	58	Args:
	59	user_id (str): the user whose backup we're adding to
	60	version (str): the version ID of the backup for the set of keys we're adding to
	61	room_keys (iterable[(str, str, dict)]): the keys to add, in the form
	62	(roomID, sessionID, keyData)
	63	"""
	64
	65	values = []
	66	for (room_id, session_id, room_key) in room_keys:
	67	values.append(
	68	{
	69	"user_id": user_id,
	70	"version": version,
	71	"room_id": room_id,
	72	"session_id": session_id,
	73	"first_message_index": room_key["first_message_index"],
	74	"forwarded_count": room_key["forwarded_count"],
	75	"is_verified": room_key["is_verified"],
	76	"session_data": json_encoder.encode(room_key["session_data"]),
	77	}
	78	)
	79	log_kv(
	80	{
	81	"message": "Set room key",
	82	"room_id": room_id,
	83	"session_id": session_id,
	84	"room_key": room_key,
	85	}
	86	)
	87
	88	await self.db_pool.simple_insert_many(
	89	table="e2e_room_keys", values=values, desc="add_e2e_room_keys"
	90	)
	91
	92	@trace
	93	async def get_e2e_room_keys(self, user_id, version, room_id=None, session_id=None):
	94	"""Bulk get the E2E room keys for a given backup, optionally filtered to a given
	95	room, or a given session.
	96
	97	Args:
	98	user_id (str): the user whose backup we're querying
	99	version (str): the version ID of the backup for the set of keys we're querying
	100	room_id (str): Optional. the ID of the room whose keys we're querying, if any.
	101	If not specified, we return the keys for all the rooms in the backup.
	102	session_id (str): Optional. the session whose room_key we're querying, if any.
	103	If specified, we also require the room_id to be specified.
	104	If not specified, we return all the keys in this version of
	105	the backup (or for the specified room)
	106
	107	Returns:
	108	A list of dicts giving the session_data and message metadata for
	109	these room keys.
	110	"""
	111
	112	try:
	113	version = int(version)
	114	except ValueError:
	115	return {"rooms": {}}
	116
	117	keyvalues = {"user_id": user_id, "version": version}
	118	if room_id:
	119	keyvalues["room_id"] = room_id
	120	if session_id:
	121	keyvalues["session_id"] = session_id
	122
	123	rows = await self.db_pool.simple_select_list(
	124	table="e2e_room_keys",
	125	keyvalues=keyvalues,
	126	retcols=(
	127	"user_id",
	128	"room_id",
	129	"session_id",
	130	"first_message_index",
	131	"forwarded_count",
	132	"is_verified",
	133	"session_data",
	134	),
	135	desc="get_e2e_room_keys",
	136	)
	137
	138	sessions = {"rooms": {}}
	139	for row in rows:
	140	room_entry = sessions["rooms"].setdefault(row["room_id"], {"sessions": {}})
	141	room_entry["sessions"][row["session_id"]] = {
	142	"first_message_index": row["first_message_index"],
	143	"forwarded_count": row["forwarded_count"],
	144	# is_verified must be returned to the client as a boolean
	145	"is_verified": bool(row["is_verified"]),
	146	"session_data": db_to_json(row["session_data"]),
	147	}
	148
	149	return sessions
	150
	151	def get_e2e_room_keys_multi(self, user_id, version, room_keys):
	152	"""Get multiple room keys at a time. The difference between this function and
	153	get_e2e_room_keys is that this function can be used to retrieve
	154	multiple specific keys at a time, whereas get_e2e_room_keys is used for
	155	getting all the keys in a backup version, all the keys for a room, or a
	156	specific key.
	157
	158	Args:
	159	user_id (str): the user whose backup we're querying
	160	version (str): the version ID of the backup we're querying about
	161	room_keys (dict[str, dict[str, iterable[str]]]): a map from
	162	room ID -> {"session": [session ids]} indicating the session IDs
	163	that we want to query
	164
	165	Returns:
	166	Deferred[dict[str, dict[str, dict]]]: a map of room IDs to session IDs to room key
	167	"""
	168
	169	return self.db_pool.runInteraction(
	170	"get_e2e_room_keys_multi",
	171	self._get_e2e_room_keys_multi_txn,
	172	user_id,
	173	version,
	174	room_keys,
	175	)
	176
	177	@staticmethod
	178	def _get_e2e_room_keys_multi_txn(txn, user_id, version, room_keys):
	179	if not room_keys:
	180	return {}
	181
	182	where_clauses = []
	183	params = [user_id, version]
	184	for room_id, room in room_keys.items():
	185	sessions = list(room["sessions"])
	186	if not sessions:
	187	continue
	188	params.append(room_id)
	189	params.extend(sessions)
	190	where_clauses.append(
	191	"(room_id = ? AND session_id IN (%s))"
	192	% (",".join(["?" for _ in sessions]),)
	193	)
	194
	195	# check if we're actually querying something
	196	if not where_clauses:
	197	return {}
	198
	199	sql = """
	200	SELECT room_id, session_id, first_message_index, forwarded_count,
	201	is_verified, session_data
	202	FROM e2e_room_keys
	203	WHERE user_id = ? AND version = ? AND (%s)
	204	""" % (
	205	" OR ".join(where_clauses)
	206	)
	207
	208	txn.execute(sql, params)
	209
	210	ret = {}
	211
	212	for row in txn:
	213	room_id = row[0]
	214	session_id = row[1]
	215	ret.setdefault(room_id, {})
	216	ret[room_id][session_id] = {
	217	"first_message_index": row[2],
	218	"forwarded_count": row[3],
	219	"is_verified": row[4],
	220	"session_data": db_to_json(row[5]),
	221	}
	222
	223	return ret
	224
	225	def count_e2e_room_keys(self, user_id, version):
	226	"""Get the number of keys in a backup version.
	227
	228	Args:
	229	user_id (str): the user whose backup we're querying
	230	version (str): the version ID of the backup we're querying about
	231	"""
	232
	233	return self.db_pool.simple_select_one_onecol(
	234	table="e2e_room_keys",
	235	keyvalues={"user_id": user_id, "version": version},
	236	retcol="COUNT(*)",
	237	desc="count_e2e_room_keys",
	238	)
	239
	240	@trace
	241	async def delete_e2e_room_keys(
	242	self, user_id, version, room_id=None, session_id=None
	243	):
	244	"""Bulk delete the E2E room keys for a given backup, optionally filtered to a given
	245	room or a given session.
	246
	247	Args:
	248	user_id(str): the user whose backup we're deleting from
	249	version(str): the version ID of the backup for the set of keys we're deleting
	250	room_id(str): Optional. the ID of the room whose keys we're deleting, if any.
	251	If not specified, we delete the keys for all the rooms in the backup.
	252	session_id(str): Optional. the session whose room_key we're querying, if any.
	253	If specified, we also require the room_id to be specified.
	254	If not specified, we delete all the keys in this version of
	255	the backup (or for the specified room)
	256
	257	Returns:
	258	The deletion transaction
	259	"""
	260
	261	keyvalues = {"user_id": user_id, "version": int(version)}
	262	if room_id:
	263	keyvalues["room_id"] = room_id
	264	if session_id:
	265	keyvalues["session_id"] = session_id
	266
	267	await self.db_pool.simple_delete(
	268	table="e2e_room_keys", keyvalues=keyvalues, desc="delete_e2e_room_keys"
	269	)
	270
	271	@staticmethod
	272	def _get_current_version(txn, user_id):
	273	txn.execute(
	274	"SELECT MAX(version) FROM e2e_room_keys_versions "
	275	"WHERE user_id=? AND deleted=0",
	276	(user_id,),
	277	)
	278	row = txn.fetchone()
	279	if not row:
	280	raise StoreError(404, "No current backup version")
	281	return row[0]
	282
	283	def get_e2e_room_keys_version_info(self, user_id, version=None):
	284	"""Get info metadata about a version of our room_keys backup.
	285
	286	Args:
	287	user_id(str): the user whose backup we're querying
	288	version(str): Optional. the version ID of the backup we're querying about
	289	If missing, we return the information about the current version.
	290	Raises:
	291	StoreError: with code 404 if there are no e2e_room_keys_versions present
	292	Returns:
	293	A deferred dict giving the info metadata for this backup version, with
	294	fields including:
	295	version(str)
	296	algorithm(str)
	297	auth_data(object): opaque dict supplied by the client
	298	etag(int): tag of the keys in the backup
	299	"""
	300
	301	def _get_e2e_room_keys_version_info_txn(txn):
	302	if version is None:
	303	this_version = self._get_current_version(txn, user_id)
	304	else:
	305	try:
	306	this_version = int(version)
	307	except ValueError:
	308	# Our versions are all ints so if we can't convert it to an integer,
	309	# it isn't there.
	310	raise StoreError(404, "No row found")
	311
	312	result = self.db_pool.simple_select_one_txn(
	313	txn,
	314	table="e2e_room_keys_versions",
	315	keyvalues={"user_id": user_id, "version": this_version, "deleted": 0},
	316	retcols=("version", "algorithm", "auth_data", "etag"),
	317	)
	318	result["auth_data"] = db_to_json(result["auth_data"])
	319	result["version"] = str(result["version"])
	320	if result["etag"] is None:
	321	result["etag"] = 0
	322	return result
	323
	324	return self.db_pool.runInteraction(
	325	"get_e2e_room_keys_version_info", _get_e2e_room_keys_version_info_txn
	326	)
	327
	328	@trace
	329	def create_e2e_room_keys_version(self, user_id, info):
	330	"""Atomically creates a new version of this user's e2e_room_keys store
	331	with the given version info.
	332
	333	Args:
	334	user_id(str): the user whose backup we're creating a version
	335	info(dict): the info about the backup version to be created
	336
	337	Returns:
	338	A deferred string for the newly created version ID
	339	"""
	340
	341	def _create_e2e_room_keys_version_txn(txn):
	342	txn.execute(
	343	"SELECT MAX(version) FROM e2e_room_keys_versions WHERE user_id=?",
	344	(user_id,),
	345	)
	346	current_version = txn.fetchone()[0]
	347	if current_version is None:
	348	current_version = "0"
	349
	350	new_version = str(int(current_version) + 1)
	351
	352	self.db_pool.simple_insert_txn(
	353	txn,
	354	table="e2e_room_keys_versions",
	355	values={
	356	"user_id": user_id,
	357	"version": new_version,
	358	"algorithm": info["algorithm"],
	359	"auth_data": json_encoder.encode(info["auth_data"]),
	360	},
	361	)
	362
	363	return new_version
	364
	365	return self.db_pool.runInteraction(
	366	"create_e2e_room_keys_version_txn", _create_e2e_room_keys_version_txn
	367	)
	368
	369	@trace
	370	def update_e2e_room_keys_version(
	371	self, user_id, version, info=None, version_etag=None
	372	):
	373	"""Update a given backup version
	374
	375	Args:
	376	user_id(str): the user whose backup version we're updating
	377	version(str): the version ID of the backup version we're updating
	378	info (dict): the new backup version info to store. If None, then
	379	the backup version info is not updated
	380	version_etag (Optional[int]): etag of the keys in the backup. If
	381	None, then the etag is not updated
	382	"""
	383	updatevalues = {}
	384
	385	if info is not None and "auth_data" in info:
	386	updatevalues["auth_data"] = json_encoder.encode(info["auth_data"])
	387	if version_etag is not None:
	388	updatevalues["etag"] = version_etag
	389
	390	if updatevalues:
	391	return self.db_pool.simple_update(
	392	table="e2e_room_keys_versions",
	393	keyvalues={"user_id": user_id, "version": version},
	394	updatevalues=updatevalues,
	395	desc="update_e2e_room_keys_version",
	396	)
	397
	398	@trace
	399	def delete_e2e_room_keys_version(self, user_id, version=None):
	400	"""Delete a given backup version of the user's room keys.
	401	Doesn't delete their actual key data.
	402
	403	Args:
	404	user_id(str): the user whose backup version we're deleting
	405	version(str): Optional. the version ID of the backup version we're deleting
	406	If missing, we delete the current backup version info.
	407	Raises:
	408	StoreError: with code 404 if there are no e2e_room_keys_versions present,
	409	or if the version requested doesn't exist.
	410	"""
	411
	412	def _delete_e2e_room_keys_version_txn(txn):
	413	if version is None:
	414	this_version = self._get_current_version(txn, user_id)
	415	if this_version is None:
	416	raise StoreError(404, "No current backup version")
	417	else:
	418	this_version = version
	419
	420	self.db_pool.simple_delete_txn(
	421	txn,
	422	table="e2e_room_keys",
	423	keyvalues={"user_id": user_id, "version": this_version},
	424	)
	425
	426	return self.db_pool.simple_update_one_txn(
	427	txn,
	428	table="e2e_room_keys_versions",
	429	keyvalues={"user_id": user_id, "version": this_version},
	430	updatevalues={"deleted": 1},
	431	)
	432
	433	return self.db_pool.runInteraction(
	434	"delete_e2e_room_keys_version", _delete_e2e_room_keys_version_txn
	435	)

+750

-0

synapse/storage/databases/main/end_to_end_keys.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2015, 2016 OpenMarket Ltd
	2	# Copyright 2019 New Vector Ltd
	3	# Copyright 2019 The Matrix.org Foundation C.I.C.
	4	#
	5	# Licensed under the Apache License, Version 2.0 (the "License");
	6	# you may not use this file except in compliance with the License.
	7	# You may obtain a copy of the License at
	8	#
	9	# http://www.apache.org/licenses/LICENSE-2.0
	10	#
	11	# Unless required by applicable law or agreed to in writing, software
	12	# distributed under the License is distributed on an "AS IS" BASIS,
	13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	14	# See the License for the specific language governing permissions and
	15	# limitations under the License.
	16	from typing import Dict, Iterable, List, Optional, Tuple
	17
	18	from canonicaljson import encode_canonical_json
	19
	20	from twisted.enterprise.adbapi import Connection
	21
	22	from synapse.logging.opentracing import log_kv, set_tag, trace
	23	from synapse.storage._base import SQLBaseStore, db_to_json
	24	from synapse.storage.database import make_in_list_sql_clause
	25	from synapse.util import json_encoder
	26	from synapse.util.caches.descriptors import cached, cachedList
	27	from synapse.util.iterutils import batch_iter
	28
	29
	30	class EndToEndKeyWorkerStore(SQLBaseStore):
	31	@trace
	32	async def get_e2e_device_keys(
	33	self, query_list, include_all_devices=False, include_deleted_devices=False
	34	):
	35	"""Fetch a list of device keys.
	36	Args:
	37	query_list(list): List of pairs of user_ids and device_ids.
	38	include_all_devices (bool): whether to include entries for devices
	39	that don't have device keys
	40	include_deleted_devices (bool): whether to include null entries for
	41	devices which no longer exist (but were in the query_list).
	42	This option only takes effect if include_all_devices is true.
	43	Returns:
	44	Dict mapping from user-id to dict mapping from device_id to
	45	key data. The key data will be a dict in the same format as the
	46	DeviceKeys type returned by POST /_matrix/client/r0/keys/query.
	47	"""
	48	set_tag("query_list", query_list)
	49	if not query_list:
	50	return {}
	51
	52	results = await self.db_pool.runInteraction(
	53	"get_e2e_device_keys",
	54	self._get_e2e_device_keys_txn,
	55	query_list,
	56	include_all_devices,
	57	include_deleted_devices,
	58	)
	59
	60	# Build the result structure, un-jsonify the results, and add the
	61	# "unsigned" section
	62	rv = {}
	63	for user_id, device_keys in results.items():
	64	rv[user_id] = {}
	65	for device_id, device_info in device_keys.items():
	66	r = db_to_json(device_info.pop("key_json"))
	67	r["unsigned"] = {}
	68	display_name = device_info["device_display_name"]
	69	if display_name is not None:
	70	r["unsigned"]["device_display_name"] = display_name
	71	if "signatures" in device_info:
	72	for sig_user_id, sigs in device_info["signatures"].items():
	73	r.setdefault("signatures", {}).setdefault(
	74	sig_user_id, {}
	75	).update(sigs)
	76	rv[user_id][device_id] = r
	77
	78	return rv
	79
	80	@trace
	81	def _get_e2e_device_keys_txn(
	82	self, txn, query_list, include_all_devices=False, include_deleted_devices=False
	83	):
	84	set_tag("include_all_devices", include_all_devices)
	85	set_tag("include_deleted_devices", include_deleted_devices)
	86
	87	query_clauses = []
	88	query_params = []
	89	signature_query_clauses = []
	90	signature_query_params = []
	91
	92	if include_all_devices is False:
	93	include_deleted_devices = False
	94
	95	if include_deleted_devices:
	96	deleted_devices = set(query_list)
	97
	98	for (user_id, device_id) in query_list:
	99	query_clause = "user_id = ?"
	100	query_params.append(user_id)
	101	signature_query_clause = "target_user_id = ?"
	102	signature_query_params.append(user_id)
	103
	104	if device_id is not None:
	105	query_clause += " AND device_id = ?"
	106	query_params.append(device_id)
	107	signature_query_clause += " AND target_device_id = ?"
	108	signature_query_params.append(device_id)
	109
	110	signature_query_clause += " AND user_id = ?"
	111	signature_query_params.append(user_id)
	112
	113	query_clauses.append(query_clause)
	114	signature_query_clauses.append(signature_query_clause)
	115
	116	sql = (
	117	"SELECT user_id, device_id, "
	118	" d.display_name AS device_display_name, "
	119	" k.key_json"
	120	" FROM devices d"
	121	" %s JOIN e2e_device_keys_json k USING (user_id, device_id)"
	122	" WHERE %s AND NOT d.hidden"
	123	) % (
	124	"LEFT" if include_all_devices else "INNER",
	125	" OR ".join("(" + q + ")" for q in query_clauses),
	126	)
	127
	128	txn.execute(sql, query_params)
	129	rows = self.db_pool.cursor_to_dict(txn)
	130
	131	result = {}
	132	for row in rows:
	133	if include_deleted_devices:
	134	deleted_devices.remove((row["user_id"], row["device_id"]))
	135	result.setdefault(row["user_id"], {})[row["device_id"]] = row
	136
	137	if include_deleted_devices:
	138	for user_id, device_id in deleted_devices:
	139	result.setdefault(user_id, {})[device_id] = None
	140
	141	# get signatures on the device
	142	signature_sql = ("SELECT * FROM e2e_cross_signing_signatures WHERE %s") % (
	143	" OR ".join("(" + q + ")" for q in signature_query_clauses)
	144	)
	145
	146	txn.execute(signature_sql, signature_query_params)
	147	rows = self.db_pool.cursor_to_dict(txn)
	148
	149	# add each cross-signing signature to the correct device in the result dict.
	150	for row in rows:
	151	signing_user_id = row["user_id"]
	152	signing_key_id = row["key_id"]
	153	target_user_id = row["target_user_id"]
	154	target_device_id = row["target_device_id"]
	155	signature = row["signature"]
	156
	157	target_user_result = result.get(target_user_id)
	158	if not target_user_result:
	159	continue
	160
	161	target_device_result = target_user_result.get(target_device_id)
	162	if not target_device_result:
	163	# note that target_device_result will be None for deleted devices.
	164	continue
	165
	166	target_device_signatures = target_device_result.setdefault("signatures", {})
	167	signing_user_signatures = target_device_signatures.setdefault(
	168	signing_user_id, {}
	169	)
	170	signing_user_signatures[signing_key_id] = signature
	171
	172	log_kv(result)
	173	return result
	174
	175	async def get_e2e_one_time_keys(
	176	self, user_id: str, device_id: str, key_ids: List[str]
	177	) -> Dict[Tuple[str, str], str]:
	178	"""Retrieve a number of one-time keys for a user
	179
	180	Args:
	181	user_id(str): id of user to get keys for
	182	device_id(str): id of device to get keys for
	183	key_ids(list[str]): list of key ids (excluding algorithm) to
	184	retrieve
	185
	186	Returns:
	187	A map from (algorithm, key_id) to json string for key
	188	"""
	189
	190	rows = await self.db_pool.simple_select_many_batch(
	191	table="e2e_one_time_keys_json",
	192	column="key_id",
	193	iterable=key_ids,
	194	retcols=("algorithm", "key_id", "key_json"),
	195	keyvalues={"user_id": user_id, "device_id": device_id},
	196	desc="add_e2e_one_time_keys_check",
	197	)
	198	result = {(row["algorithm"], row["key_id"]): row["key_json"] for row in rows}
	199	log_kv({"message": "Fetched one time keys for user", "one_time_keys": result})
	200	return result
	201
	202	async def add_e2e_one_time_keys(
	203	self,
	204	user_id: str,
	205	device_id: str,
	206	time_now: int,
	207	new_keys: Iterable[Tuple[str, str, str]],
	208	) -> None:
	209	"""Insert some new one time keys for a device. Errors if any of the
	210	keys already exist.
	211
	212	Args:
	213	user_id: id of user to get keys for
	214	device_id: id of device to get keys for
	215	time_now: insertion time to record (ms since epoch)
	216	new_keys: keys to add - each a tuple of (algorithm, key_id, key json)
	217	"""
	218
	219	def _add_e2e_one_time_keys(txn):
	220	set_tag("user_id", user_id)
	221	set_tag("device_id", device_id)
	222	set_tag("new_keys", new_keys)
	223	# We are protected from race between lookup and insertion due to
	224	# a unique constraint. If there is a race of two calls to
	225	# `add_e2e_one_time_keys` then they'll conflict and we will only
	226	# insert one set.
	227	self.db_pool.simple_insert_many_txn(
	228	txn,
	229	table="e2e_one_time_keys_json",
	230	values=[
	231	{
	232	"user_id": user_id,
	233	"device_id": device_id,
	234	"algorithm": algorithm,
	235	"key_id": key_id,
	236	"ts_added_ms": time_now,
	237	"key_json": json_bytes,
	238	}
	239	for algorithm, key_id, json_bytes in new_keys
	240	],
	241	)
	242	self._invalidate_cache_and_stream(
	243	txn, self.count_e2e_one_time_keys, (user_id, device_id)
	244	)
	245
	246	await self.db_pool.runInteraction(
	247	"add_e2e_one_time_keys_insert", _add_e2e_one_time_keys
	248	)
	249
	250	@cached(max_entries=10000)
	251	def count_e2e_one_time_keys(self, user_id, device_id):
	252	""" Count the number of one time keys the server has for a device
	253	Returns:
	254	Dict mapping from algorithm to number of keys for that algorithm.
	255	"""
	256
	257	def _count_e2e_one_time_keys(txn):
	258	sql = (
	259	"SELECT algorithm, COUNT(key_id) FROM e2e_one_time_keys_json"
	260	" WHERE user_id = ? AND device_id = ?"
	261	" GROUP BY algorithm"
	262	)
	263	txn.execute(sql, (user_id, device_id))
	264	result = {}
	265	for algorithm, key_count in txn:
	266	result[algorithm] = key_count
	267	return result
	268
	269	return self.db_pool.runInteraction(
	270	"count_e2e_one_time_keys", _count_e2e_one_time_keys
	271	)
	272
	273	async def get_e2e_cross_signing_key(
	274	self, user_id: str, key_type: str, from_user_id: Optional[str] = None
	275	) -> Optional[dict]:
	276	"""Returns a user's cross-signing key.
	277
	278	Args:
	279	user_id: the user whose key is being requested
	280	key_type: the type of key that is being requested: either 'master'
	281	for a master key, 'self_signing' for a self-signing key, or
	282	'user_signing' for a user-signing key
	283	from_user_id: if specified, signatures made by this user on
	284	the self-signing key will be included in the result
	285
	286	Returns:
	287	dict of the key data or None if not found
	288	"""
	289	res = await self.get_e2e_cross_signing_keys_bulk([user_id], from_user_id)
	290	user_keys = res.get(user_id)
	291	if not user_keys:
	292	return None
	293	return user_keys.get(key_type)
	294
	295	@cached(num_args=1)
	296	def _get_bare_e2e_cross_signing_keys(self, user_id):
	297	"""Dummy function. Only used to make a cache for
	298	_get_bare_e2e_cross_signing_keys_bulk.
	299	"""
	300	raise NotImplementedError()
	301
	302	@cachedList(
	303	cached_method_name="_get_bare_e2e_cross_signing_keys",
	304	list_name="user_ids",
	305	num_args=1,
	306	)
	307	def _get_bare_e2e_cross_signing_keys_bulk(
	308	self, user_ids: List[str]
	309	) -> Dict[str, Dict[str, dict]]:
	310	"""Returns the cross-signing keys for a set of users. The output of this
	311	function should be passed to _get_e2e_cross_signing_signatures_txn if
	312	the signatures for the calling user need to be fetched.
	313
	314	Args:
	315	user_ids (list[str]): the users whose keys are being requested
	316
	317	Returns:
	318	dict[str, dict[str, dict]]: mapping from user ID to key type to key
	319	data. If a user's cross-signing keys were not found, either
	320	their user ID will not be in the dict, or their user ID will map
	321	to None.
	322
	323	"""
	324	return self.db_pool.runInteraction(
	325	"get_bare_e2e_cross_signing_keys_bulk",
	326	self._get_bare_e2e_cross_signing_keys_bulk_txn,
	327	user_ids,
	328	)
	329
	330	def _get_bare_e2e_cross_signing_keys_bulk_txn(
	331	self, txn: Connection, user_ids: List[str],
	332	) -> Dict[str, Dict[str, dict]]:
	333	"""Returns the cross-signing keys for a set of users. The output of this
	334	function should be passed to _get_e2e_cross_signing_signatures_txn if
	335	the signatures for the calling user need to be fetched.
	336
	337	Args:
	338	txn (twisted.enterprise.adbapi.Connection): db connection
	339	user_ids (list[str]): the users whose keys are being requested
	340
	341	Returns:
	342	dict[str, dict[str, dict]]: mapping from user ID to key type to key
	343	data. If a user's cross-signing keys were not found, their user
	344	ID will not be in the dict.
	345
	346	"""
	347	result = {}
	348
	349	for user_chunk in batch_iter(user_ids, 100):
	350	clause, params = make_in_list_sql_clause(
	351	txn.database_engine, "k.user_id", user_chunk
	352	)
	353	sql = (
	354	"""
	355	SELECT k.user_id, k.keytype, k.keydata, k.stream_id
	356	FROM e2e_cross_signing_keys k
	357	INNER JOIN (SELECT user_id, keytype, MAX(stream_id) AS stream_id
	358	FROM e2e_cross_signing_keys
	359	GROUP BY user_id, keytype) s
	360	USING (user_id, stream_id, keytype)
	361	WHERE
	362	"""
	363	+ clause
	364	)
	365
	366	txn.execute(sql, params)
	367	rows = self.db_pool.cursor_to_dict(txn)
	368
	369	for row in rows:
	370	user_id = row["user_id"]
	371	key_type = row["keytype"]
	372	key = db_to_json(row["keydata"])
	373	user_info = result.setdefault(user_id, {})
	374	user_info[key_type] = key
	375
	376	return result
	377
	378	def _get_e2e_cross_signing_signatures_txn(
	379	self, txn: Connection, keys: Dict[str, Dict[str, dict]], from_user_id: str,
	380	) -> Dict[str, Dict[str, dict]]:
	381	"""Returns the cross-signing signatures made by a user on a set of keys.
	382
	383	Args:
	384	txn (twisted.enterprise.adbapi.Connection): db connection
	385	keys (dict[str, dict[str, dict]]): a map of user ID to key type to
	386	key data. This dict will be modified to add signatures.
	387	from_user_id (str): fetch the signatures made by this user
	388
	389	Returns:
	390	dict[str, dict[str, dict]]: mapping from user ID to key type to key
	391	data. The return value will be the same as the keys argument,
	392	with the modifications included.
	393	"""
	394
	395	# find out what cross-signing keys (a.k.a. devices) we need to get
	396	# signatures for. This is a map of (user_id, device_id) to key type
	397	# (device_id is the key's public part).
	398	devices = {}
	399
	400	for user_id, user_info in keys.items():
	401	if user_info is None:
	402	continue
	403	for key_type, key in user_info.items():
	404	device_id = None
	405	for k in key["keys"].values():
	406	device_id = k
	407	devices[(user_id, device_id)] = key_type
	408
	409	for batch in batch_iter(devices.keys(), size=100):
	410	sql = """
	411	SELECT target_user_id, target_device_id, key_id, signature
	412	FROM e2e_cross_signing_signatures
	413	WHERE user_id = ?
	414	AND (%s)
	415	""" % (
	416	" OR ".join(
	417	"(target_user_id = ? AND target_device_id = ?)" for _ in batch
	418	)
	419	)
	420	query_params = [from_user_id]
	421	for item in batch:
	422	# item is a (user_id, device_id) tuple
	423	query_params.extend(item)
	424
	425	txn.execute(sql, query_params)
	426	rows = self.db_pool.cursor_to_dict(txn)
	427
	428	# and add the signatures to the appropriate keys
	429	for row in rows:
	430	key_id = row["key_id"]
	431	target_user_id = row["target_user_id"]
	432	target_device_id = row["target_device_id"]
	433	key_type = devices[(target_user_id, target_device_id)]
	434	# We need to copy everything, because the result may have come
	435	# from the cache. dict.copy only does a shallow copy, so we
	436	# need to recursively copy the dicts that will be modified.
	437	user_info = keys[target_user_id] = keys[target_user_id].copy()
	438	target_user_key = user_info[key_type] = user_info[key_type].copy()
	439	if "signatures" in target_user_key:
	440	signatures = target_user_key["signatures"] = target_user_key[
	441	"signatures"
	442	].copy()
	443	if from_user_id in signatures:
	444	user_sigs = signatures[from_user_id] = signatures[from_user_id]
	445	user_sigs[key_id] = row["signature"]
	446	else:
	447	signatures[from_user_id] = {key_id: row["signature"]}
	448	else:
	449	target_user_key["signatures"] = {
	450	from_user_id: {key_id: row["signature"]}
	451	}
	452
	453	return keys
	454
	455	async def get_e2e_cross_signing_keys_bulk(
	456	self, user_ids: List[str], from_user_id: Optional[str] = None
	457	) -> Dict[str, Dict[str, dict]]:
	458	"""Returns the cross-signing keys for a set of users.
	459
	460	Args:
	461	user_ids: the users whose keys are being requested
	462	from_user_id: if specified, signatures made by this user on
	463	the self-signing keys will be included in the result
	464
	465	Returns:
	466	A map of user ID to key type to key data. If a user's cross-signing
	467	keys were not found, either their user ID will not be in the dict,
	468	or their user ID will map to None.
	469	"""
	470
	471	result = await self._get_bare_e2e_cross_signing_keys_bulk(user_ids)
	472
	473	if from_user_id:
	474	result = await self.db_pool.runInteraction(
	475	"get_e2e_cross_signing_signatures",
	476	self._get_e2e_cross_signing_signatures_txn,
	477	result,
	478	from_user_id,
	479	)
	480
	481	return result
	482
	483	async def get_all_user_signature_changes_for_remotes(
	484	self, instance_name: str, last_id: int, current_id: int, limit: int
	485	) -> Tuple[List[Tuple[int, tuple]], int, bool]:
	486	"""Get updates for groups replication stream.
	487
	488	Note that the user signature stream represents when a user signs their
	489	device with their user-signing key, which is not published to other
	490	users or servers, so no `destination` is needed in the returned
	491	list. However, this is needed to poke workers.
	492
	493	Args:
	494	instance_name: The writer we want to fetch updates from. Unused
	495	here since there is only ever one writer.
	496	last_id: The token to fetch updates from. Exclusive.
	497	current_id: The token to fetch updates up to. Inclusive.
	498	limit: The requested limit for the number of rows to return. The
	499	function may return more or fewer rows.
	500
	501	Returns:
	502	A tuple consisting of: the updates, a token to use to fetch
	503	subsequent updates, and whether we returned fewer rows than exists
	504	between the requested tokens due to the limit.
	505
	506	The token returned can be used in a subsequent call to this
	507	function to get further updatees.
	508
	509	The updates are a list of 2-tuples of stream ID and the row data
	510	"""
	511
	512	if last_id == current_id:
	513	return [], current_id, False
	514
	515	def _get_all_user_signature_changes_for_remotes_txn(txn):
	516	sql = """
	517	SELECT stream_id, from_user_id AS user_id
	518	FROM user_signature_stream
	519	WHERE ? < stream_id AND stream_id <= ?
	520	ORDER BY stream_id ASC
	521	LIMIT ?
	522	"""
	523	txn.execute(sql, (last_id, current_id, limit))
	524
	525	updates = [(row[0], (row[1:])) for row in txn]
	526
	527	limited = False
	528	upto_token = current_id
	529	if len(updates) >= limit:
	530	upto_token = updates[-1][0]
	531	limited = True
	532
	533	return updates, upto_token, limited
	534
	535	return await self.db_pool.runInteraction(
	536	"get_all_user_signature_changes_for_remotes",
	537	_get_all_user_signature_changes_for_remotes_txn,
	538	)
	539
	540
	541	class EndToEndKeyStore(EndToEndKeyWorkerStore, SQLBaseStore):
	542	def set_e2e_device_keys(self, user_id, device_id, time_now, device_keys):
	543	"""Stores device keys for a device. Returns whether there was a change
	544	or the keys were already in the database.
	545	"""
	546
	547	def _set_e2e_device_keys_txn(txn):
	548	set_tag("user_id", user_id)
	549	set_tag("device_id", device_id)
	550	set_tag("time_now", time_now)
	551	set_tag("device_keys", device_keys)
	552
	553	old_key_json = self.db_pool.simple_select_one_onecol_txn(
	554	txn,
	555	table="e2e_device_keys_json",
	556	keyvalues={"user_id": user_id, "device_id": device_id},
	557	retcol="key_json",
	558	allow_none=True,
	559	)
	560
	561	# In py3 we need old_key_json to match new_key_json type. The DB
	562	# returns unicode while encode_canonical_json returns bytes.
	563	new_key_json = encode_canonical_json(device_keys).decode("utf-8")
	564
	565	if old_key_json == new_key_json:
	566	log_kv({"Message": "Device key already stored."})
	567	return False
	568
	569	self.db_pool.simple_upsert_txn(
	570	txn,
	571	table="e2e_device_keys_json",
	572	keyvalues={"user_id": user_id, "device_id": device_id},
	573	values={"ts_added_ms": time_now, "key_json": new_key_json},
	574	)
	575	log_kv({"message": "Device keys stored."})
	576	return True
	577
	578	return self.db_pool.runInteraction(
	579	"set_e2e_device_keys", _set_e2e_device_keys_txn
	580	)
	581
	582	def claim_e2e_one_time_keys(self, query_list):
	583	"""Take a list of one time keys out of the database"""
	584
	585	@trace
	586	def _claim_e2e_one_time_keys(txn):
	587	sql = (
	588	"SELECT key_id, key_json FROM e2e_one_time_keys_json"
	589	" WHERE user_id = ? AND device_id = ? AND algorithm = ?"
	590	" LIMIT 1"
	591	)
	592	result = {}
	593	delete = []
	594	for user_id, device_id, algorithm in query_list:
	595	user_result = result.setdefault(user_id, {})
	596	device_result = user_result.setdefault(device_id, {})
	597	txn.execute(sql, (user_id, device_id, algorithm))
	598	for key_id, key_json in txn:
	599	device_result[algorithm + ":" + key_id] = key_json
	600	delete.append((user_id, device_id, algorithm, key_id))
	601	sql = (
	602	"DELETE FROM e2e_one_time_keys_json"
	603	" WHERE user_id = ? AND device_id = ? AND algorithm = ?"
	604	" AND key_id = ?"
	605	)
	606	for user_id, device_id, algorithm, key_id in delete:
	607	log_kv(
	608	{
	609	"message": "Executing claim e2e_one_time_keys transaction on database."
	610	}
	611	)
	612	txn.execute(sql, (user_id, device_id, algorithm, key_id))
	613	log_kv({"message": "finished executing and invalidating cache"})
	614	self._invalidate_cache_and_stream(
	615	txn, self.count_e2e_one_time_keys, (user_id, device_id)
	616	)
	617	return result
	618
	619	return self.db_pool.runInteraction(
	620	"claim_e2e_one_time_keys", _claim_e2e_one_time_keys
	621	)
	622
	623	def delete_e2e_keys_by_device(self, user_id, device_id):
	624	def delete_e2e_keys_by_device_txn(txn):
	625	log_kv(
	626	{
	627	"message": "Deleting keys for device",
	628	"device_id": device_id,
	629	"user_id": user_id,
	630	}
	631	)
	632	self.db_pool.simple_delete_txn(
	633	txn,
	634	table="e2e_device_keys_json",
	635	keyvalues={"user_id": user_id, "device_id": device_id},
	636	)
	637	self.db_pool.simple_delete_txn(
	638	txn,
	639	table="e2e_one_time_keys_json",
	640	keyvalues={"user_id": user_id, "device_id": device_id},
	641	)
	642	self._invalidate_cache_and_stream(
	643	txn, self.count_e2e_one_time_keys, (user_id, device_id)
	644	)
	645
	646	return self.db_pool.runInteraction(
	647	"delete_e2e_keys_by_device", delete_e2e_keys_by_device_txn
	648	)
	649
	650	def _set_e2e_cross_signing_key_txn(self, txn, user_id, key_type, key):
	651	"""Set a user's cross-signing key.
	652
	653	Args:
	654	txn (twisted.enterprise.adbapi.Connection): db connection
	655	user_id (str): the user to set the signing key for
	656	key_type (str): the type of key that is being set: either 'master'
	657	for a master key, 'self_signing' for a self-signing key, or
	658	'user_signing' for a user-signing key
	659	key (dict): the key data
	660	"""
	661	# the 'key' dict will look something like:
	662	# {
	663	# "user_id": "@alice:example.com",
	664	# "usage": ["self_signing"],
	665	# "keys": {
	666	# "ed25519:base64+self+signing+public+key": "base64+self+signing+public+key",
	667	# },
	668	# "signatures": {
	669	# "@alice:example.com": {
	670	# "ed25519:base64+master+public+key": "base64+signature"
	671	# }
	672	# }
	673	# }
	674	# The "keys" property must only have one entry, which will be the public
	675	# key, so we just grab the first value in there
	676	pubkey = next(iter(key["keys"].values()))
	677
	678	# The cross-signing keys need to occupy the same namespace as devices,
	679	# since signatures are identified by device ID. So add an entry to the
	680	# device table to make sure that we don't have a collision with device
	681	# IDs.
	682	# We only need to do this for local users, since remote servers should be
	683	# responsible for checking this for their own users.
	684	if self.hs.is_mine_id(user_id):
	685	self.db_pool.simple_insert_txn(
	686	txn,
	687	"devices",
	688	values={
	689	"user_id": user_id,
	690	"device_id": pubkey,
	691	"display_name": key_type + " signing key",
	692	"hidden": True,
	693	},
	694	)
	695
	696	# and finally, store the key itself
	697	with self._cross_signing_id_gen.get_next() as stream_id:
	698	self.db_pool.simple_insert_txn(
	699	txn,
	700	"e2e_cross_signing_keys",
	701	values={
	702	"user_id": user_id,
	703	"keytype": key_type,
	704	"keydata": json_encoder.encode(key),
	705	"stream_id": stream_id,
	706	},
	707	)
	708
	709	self._invalidate_cache_and_stream(
	710	txn, self._get_bare_e2e_cross_signing_keys, (user_id,)
	711	)
	712
	713	def set_e2e_cross_signing_key(self, user_id, key_type, key):
	714	"""Set a user's cross-signing key.
	715
	716	Args:
	717	user_id (str): the user to set the user-signing key for
	718	key_type (str): the type of cross-signing key to set
	719	key (dict): the key data
	720	"""
	721	return self.db_pool.runInteraction(
	722	"add_e2e_cross_signing_key",
	723	self._set_e2e_cross_signing_key_txn,
	724	user_id,
	725	key_type,
	726	key,
	727	)
	728
	729	def store_e2e_cross_signing_signatures(self, user_id, signatures):
	730	"""Stores cross-signing signatures.
	731
	732	Args:
	733	user_id (str): the user who made the signatures
	734	signatures (iterable[SignatureListItem]): signatures to add
	735	"""
	736	return self.db_pool.simple_insert_many(
	737	"e2e_cross_signing_signatures",
	738	[
	739	{
	740	"user_id": user_id,
	741	"key_id": item.signing_key_id,
	742	"target_user_id": item.target_user_id,
	743	"target_device_id": item.target_device_id,
	744	"signature": item.signature,
	745	}
	746	for item in signatures
	747	],
	748	"add_e2e_signing_key",
	749	)

+714

-0

synapse/storage/databases/main/event_federation.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	import itertools
	15	import logging
	16	from queue import Empty, PriorityQueue
	17	from typing import Dict, Iterable, List, Optional, Set, Tuple
	18
	19	from synapse.api.errors import StoreError
	20	from synapse.metrics.background_process_metrics import run_as_background_process
	21	from synapse.storage._base import SQLBaseStore, make_in_list_sql_clause
	22	from synapse.storage.database import DatabasePool
	23	from synapse.storage.databases.main.events_worker import EventsWorkerStore
	24	from synapse.storage.databases.main.signatures import SignatureWorkerStore
	25	from synapse.util.caches.descriptors import cached
	26	from synapse.util.iterutils import batch_iter
	27
	28	logger = logging.getLogger(__name__)
	29
	30
	31	class EventFederationWorkerStore(EventsWorkerStore, SignatureWorkerStore, SQLBaseStore):
	32	def get_auth_chain(self, event_ids, include_given=False):
	33	"""Get auth events for given event_ids. The events must be state events.
	34
	35	Args:
	36	event_ids (list): state events
	37	include_given (bool): include the given events in result
	38
	39	Returns:
	40	list of events
	41	"""
	42	return self.get_auth_chain_ids(
	43	event_ids, include_given=include_given
	44	).addCallback(self.get_events_as_list)
	45
	46	def get_auth_chain_ids(
	47	self,
	48	event_ids: List[str],
	49	include_given: bool = False,
	50	ignore_events: Optional[Set[str]] = None,
	51	):
	52	"""Get auth events for given event_ids. The events must be state events.
	53
	54	Args:
	55	event_ids: state events
	56	include_given: include the given events in result
	57	ignore_events: Set of events to exclude from the returned auth
	58	chain. This is useful if the caller will just discard the
	59	given events anyway, and saves us from figuring out their auth
	60	chains if not required.
	61
	62	Returns:
	63	list of event_ids
	64	"""
	65	return self.db_pool.runInteraction(
	66	"get_auth_chain_ids",
	67	self._get_auth_chain_ids_txn,
	68	event_ids,
	69	include_given,
	70	ignore_events,
	71	)
	72
	73	def _get_auth_chain_ids_txn(self, txn, event_ids, include_given, ignore_events):
	74	if ignore_events is None:
	75	ignore_events = set()
	76
	77	if include_given:
	78	results = set(event_ids)
	79	else:
	80	results = set()
	81
	82	base_sql = "SELECT auth_id FROM event_auth WHERE "
	83
	84	front = set(event_ids)
	85	while front:
	86	new_front = set()
	87	for chunk in batch_iter(front, 100):
	88	clause, args = make_in_list_sql_clause(
	89	txn.database_engine, "event_id", chunk
	90	)
	91	txn.execute(base_sql + clause, args)
	92	new_front.update(r[0] for r in txn)
	93
	94	new_front -= ignore_events
	95	new_front -= results
	96
	97	front = new_front
	98	results.update(front)
	99
	100	return list(results)
	101
	102	def get_auth_chain_difference(self, state_sets: List[Set[str]]):
	103	"""Given sets of state events figure out the auth chain difference (as
	104	per state res v2 algorithm).
	105
	106	This equivalent to fetching the full auth chain for each set of state
	107	and returning the events that don't appear in each and every auth
	108	chain.
	109
	110	Returns:
	111	Deferred[Set[str]]
	112	"""
	113
	114	return self.db_pool.runInteraction(
	115	"get_auth_chain_difference",
	116	self._get_auth_chain_difference_txn,
	117	state_sets,
	118	)
	119
	120	def _get_auth_chain_difference_txn(
	121	self, txn, state_sets: List[Set[str]]
	122	) -> Set[str]:
	123
	124	# Algorithm Description
	125	# ~~~~~~~~~~~~~~~~~~~~~
	126	#
	127	# The idea here is to basically walk the auth graph of each state set in
	128	# tandem, keeping track of which auth events are reachable by each state
	129	# set. If we reach an auth event we've already visited (via a different
	130	# state set) then we mark that auth event and all ancestors as reachable
	131	# by the state set. This requires that we keep track of the auth chains
	132	# in memory.
	133	#
	134	# Doing it in a such a way means that we can stop early if all auth
	135	# events we're currently walking are reachable by all state sets.
	136	#
	137	# Note: We can't stop walking an event's auth chain if it is reachable
	138	# by all state sets. This is because other auth chains we're walking
	139	# might be reachable only via the original auth chain. For example,
	140	# given the following auth chain:
	141	#
	142	# A -> C -> D -> E
	143	# / /
	144	# B -´---------´
	145	#
	146	# and state sets {A} and {B} then walking the auth chains of A and B
	147	# would immediately show that C is reachable by both. However, if we
	148	# stopped at C then we'd only reach E via the auth chain of B and so E
	149	# would errornously get included in the returned difference.
	150	#
	151	# The other thing that we do is limit the number of auth chains we walk
	152	# at once, due to practical limits (i.e. we can only query the database
	153	# with a limited set of parameters). We pick the auth chains we walk
	154	# each iteration based on their depth, in the hope that events with a
	155	# lower depth are likely reachable by those with higher depths.
	156	#
	157	# We could use any ordering that we believe would give a rough
	158	# topological ordering, e.g. origin server timestamp. If the ordering
	159	# chosen is not topological then the algorithm still produces the right
	160	# result, but perhaps a bit more inefficiently. This is why it is safe
	161	# to use "depth" here.
	162
	163	initial_events = set(state_sets[0]).union(*state_sets[1:])
	164
	165	# Dict from events in auth chains to which sets cannot reach them.
	166	# I.e. if the set is empty then all sets can reach the event.
	167	event_to_missing_sets = {
	168	event_id: {i for i, a in enumerate(state_sets) if event_id not in a}
	169	for event_id in initial_events
	170	}
	171
	172	# The sorted list of events whose auth chains we should walk.
	173	search = [] # type: List[Tuple[int, str]]
	174
	175	# We need to get the depth of the initial events for sorting purposes.
	176	sql = """
	177	SELECT depth, event_id FROM events
	178	WHERE %s
	179	"""
	180	# the list can be huge, so let's avoid looking them all up in one massive
	181	# query.
	182	for batch in batch_iter(initial_events, 1000):
	183	clause, args = make_in_list_sql_clause(
	184	txn.database_engine, "event_id", batch
	185	)
	186	txn.execute(sql % (clause,), args)
	187
	188	# I think building a temporary list with fetchall is more efficient than
	189	# just `search.extend(txn)`, but this is unconfirmed
	190	search.extend(txn.fetchall())
	191
	192	# sort by depth
	193	search.sort()
	194
	195	# Map from event to its auth events
	196	event_to_auth_events = {} # type: Dict[str, Set[str]]
	197
	198	base_sql = """
	199	SELECT a.event_id, auth_id, depth
	200	FROM event_auth AS a
	201	INNER JOIN events AS e ON (e.event_id = a.auth_id)
	202	WHERE
	203	"""
	204
	205	while search:
	206	# Check whether all our current walks are reachable by all state
	207	# sets. If so we can bail.
	208	if all(not event_to_missing_sets[eid] for _, eid in search):
	209	break
	210
	211	# Fetch the auth events and their depths of the N last events we're
	212	# currently walking
	213	search, chunk = search[:-100], search[-100:]
	214	clause, args = make_in_list_sql_clause(
	215	txn.database_engine, "a.event_id", [e_id for _, e_id in chunk]
	216	)
	217	txn.execute(base_sql + clause, args)
	218
	219	for event_id, auth_event_id, auth_event_depth in txn:
	220	event_to_auth_events.setdefault(event_id, set()).add(auth_event_id)
	221
	222	sets = event_to_missing_sets.get(auth_event_id)
	223	if sets is None:
	224	# First time we're seeing this event, so we add it to the
	225	# queue of things to fetch.
	226	search.append((auth_event_depth, auth_event_id))
	227
	228	# Assume that this event is unreachable from any of the
	229	# state sets until proven otherwise
	230	sets = event_to_missing_sets[auth_event_id] = set(
	231	range(len(state_sets))
	232	)
	233	else:
	234	# We've previously seen this event, so look up its auth
	235	# events and recursively mark all ancestors as reachable
	236	# by the current event's state set.
	237	a_ids = event_to_auth_events.get(auth_event_id)
	238	while a_ids:
	239	new_aids = set()
	240	for a_id in a_ids:
	241	event_to_missing_sets[a_id].intersection_update(
	242	event_to_missing_sets[event_id]
	243	)
	244
	245	b = event_to_auth_events.get(a_id)
	246	if b:
	247	new_aids.update(b)
	248
	249	a_ids = new_aids
	250
	251	# Mark that the auth event is reachable by the approriate sets.
	252	sets.intersection_update(event_to_missing_sets[event_id])
	253
	254	search.sort()
	255
	256	# Return all events where not all sets can reach them.
	257	return {eid for eid, n in event_to_missing_sets.items() if n}
	258
	259	def get_oldest_events_in_room(self, room_id):
	260	return self.db_pool.runInteraction(
	261	"get_oldest_events_in_room", self._get_oldest_events_in_room_txn, room_id
	262	)
	263
	264	def get_oldest_events_with_depth_in_room(self, room_id):
	265	return self.db_pool.runInteraction(
	266	"get_oldest_events_with_depth_in_room",
	267	self.get_oldest_events_with_depth_in_room_txn,
	268	room_id,
	269	)
	270
	271	def get_oldest_events_with_depth_in_room_txn(self, txn, room_id):
	272	sql = (
	273	"SELECT b.event_id, MAX(e.depth) FROM events as e"
	274	" INNER JOIN event_edges as g"
	275	" ON g.event_id = e.event_id"
	276	" INNER JOIN event_backward_extremities as b"
	277	" ON g.prev_event_id = b.event_id"
	278	" WHERE b.room_id = ? AND g.is_state is ?"
	279	" GROUP BY b.event_id"
	280	)
	281
	282	txn.execute(sql, (room_id, False))
	283
	284	return dict(txn)
	285
	286	async def get_max_depth_of(self, event_ids: List[str]) -> int:
	287	"""Returns the max depth of a set of event IDs
	288
	289	Args:
	290	event_ids: The event IDs to calculate the max depth of.
	291	"""
	292	rows = await self.db_pool.simple_select_many_batch(
	293	table="events",
	294	column="event_id",
	295	iterable=event_ids,
	296	retcols=("depth",),
	297	desc="get_max_depth_of",
	298	)
	299
	300	if not rows:
	301	return 0
	302	else:
	303	return max(row["depth"] for row in rows)
	304
	305	def _get_oldest_events_in_room_txn(self, txn, room_id):
	306	return self.db_pool.simple_select_onecol_txn(
	307	txn,
	308	table="event_backward_extremities",
	309	keyvalues={"room_id": room_id},
	310	retcol="event_id",
	311	)
	312
	313	def get_prev_events_for_room(self, room_id: str):
	314	"""
	315	Gets a subset of the current forward extremities in the given room.
	316
	317	Limits the result to 10 extremities, so that we can avoid creating
	318	events which refer to hundreds of prev_events.
	319
	320	Args:
	321	room_id (str): room_id
	322
	323	Returns:
	324	Deferred[List[str]]: the event ids of the forward extremites
	325
	326	"""
	327
	328	return self.db_pool.runInteraction(
	329	"get_prev_events_for_room", self._get_prev_events_for_room_txn, room_id
	330	)
	331
	332	def _get_prev_events_for_room_txn(self, txn, room_id: str):
	333	# we just use the 10 newest events. Older events will become
	334	# prev_events of future events.
	335
	336	sql = """
	337	SELECT e.event_id FROM event_forward_extremities AS f
	338	INNER JOIN events AS e USING (event_id)
	339	WHERE f.room_id = ?
	340	ORDER BY e.depth DESC
	341	LIMIT 10
	342	"""
	343
	344	txn.execute(sql, (room_id,))
	345
	346	return [row[0] for row in txn]
	347
	348	def get_rooms_with_many_extremities(self, min_count, limit, room_id_filter):
	349	"""Get the top rooms with at least N extremities.
	350
	351	Args:
	352	min_count (int): The minimum number of extremities
	353	limit (int): The maximum number of rooms to return.
	354	room_id_filter (iterable[str]): room_ids to exclude from the results
	355
	356	Returns:
	357	Deferred[list]: At most `limit` room IDs that have at least
	358	`min_count` extremities, sorted by extremity count.
	359	"""
	360
	361	def _get_rooms_with_many_extremities_txn(txn):
	362	where_clause = "1=1"
	363	if room_id_filter:
	364	where_clause = "room_id NOT IN (%s)" % (
	365	",".join("?" for _ in room_id_filter),
	366	)
	367
	368	sql = """
	369	SELECT room_id FROM event_forward_extremities
	370	WHERE %s
	371	GROUP BY room_id
	372	HAVING count(*) > ?
	373	ORDER BY count(*) DESC
	374	LIMIT ?
	375	""" % (
	376	where_clause,
	377	)
	378
	379	query_args = list(itertools.chain(room_id_filter, [min_count, limit]))
	380	txn.execute(sql, query_args)
	381	return [room_id for room_id, in txn]
	382
	383	return self.db_pool.runInteraction(
	384	"get_rooms_with_many_extremities", _get_rooms_with_many_extremities_txn
	385	)
	386
	387	@cached(max_entries=5000, iterable=True)
	388	def get_latest_event_ids_in_room(self, room_id):
	389	return self.db_pool.simple_select_onecol(
	390	table="event_forward_extremities",
	391	keyvalues={"room_id": room_id},
	392	retcol="event_id",
	393	desc="get_latest_event_ids_in_room",
	394	)
	395
	396	def get_min_depth(self, room_id):
	397	""" For hte given room, get the minimum depth we have seen for it.
	398	"""
	399	return self.db_pool.runInteraction(
	400	"get_min_depth", self._get_min_depth_interaction, room_id
	401	)
	402
	403	def _get_min_depth_interaction(self, txn, room_id):
	404	min_depth = self.db_pool.simple_select_one_onecol_txn(
	405	txn,
	406	table="room_depth",
	407	keyvalues={"room_id": room_id},
	408	retcol="min_depth",
	409	allow_none=True,
	410	)
	411
	412	return int(min_depth) if min_depth is not None else None
	413
	414	def get_forward_extremeties_for_room(self, room_id, stream_ordering):
	415	"""For a given room_id and stream_ordering, return the forward
	416	extremeties of the room at that point in "time".
	417
	418	Throws a StoreError if we have since purged the index for
	419	stream_orderings from that point.
	420
	421	Args:
	422	room_id (str):
	423	stream_ordering (int):
	424
	425	Returns:
	426	deferred, which resolves to a list of event_ids
	427	"""
	428	# We want to make the cache more effective, so we clamp to the last
	429	# change before the given ordering.
	430	last_change = self._events_stream_cache.get_max_pos_of_last_change(room_id)
	431
	432	# We don't always have a full stream_to_exterm_id table, e.g. after
	433	# the upgrade that introduced it, so we make sure we never ask for a
	434	# stream_ordering from before a restart
	435	last_change = max(self._stream_order_on_start, last_change)
	436
	437	# provided the last_change is recent enough, we now clamp the requested
	438	# stream_ordering to it.
	439	if last_change > self.stream_ordering_month_ago:
	440	stream_ordering = min(last_change, stream_ordering)
	441
	442	return self._get_forward_extremeties_for_room(room_id, stream_ordering)
	443
	444	@cached(max_entries=5000, num_args=2)
	445	def _get_forward_extremeties_for_room(self, room_id, stream_ordering):
	446	"""For a given room_id and stream_ordering, return the forward
	447	extremeties of the room at that point in "time".
	448
	449	Throws a StoreError if we have since purged the index for
	450	stream_orderings from that point.
	451	"""
	452
	453	if stream_ordering <= self.stream_ordering_month_ago:
	454	raise StoreError(400, "stream_ordering too old")
	455
	456	sql = """
	457	SELECT event_id FROM stream_ordering_to_exterm
	458	INNER JOIN (
	459	SELECT room_id, MAX(stream_ordering) AS stream_ordering
	460	FROM stream_ordering_to_exterm
	461	WHERE stream_ordering <= ? GROUP BY room_id
	462	) AS rms USING (room_id, stream_ordering)
	463	WHERE room_id = ?
	464	"""
	465
	466	def get_forward_extremeties_for_room_txn(txn):
	467	txn.execute(sql, (stream_ordering, room_id))
	468	return [event_id for event_id, in txn]
	469
	470	return self.db_pool.runInteraction(
	471	"get_forward_extremeties_for_room", get_forward_extremeties_for_room_txn
	472	)
	473
	474	def get_backfill_events(self, room_id, event_list, limit):
	475	"""Get a list of Events for a given topic that occurred before (and
	476	including) the events in event_list. Return a list of max size `limit`
	477
	478	Args:
	479	txn
	480	room_id (str)
	481	event_list (list)
	482	limit (int)
	483	"""
	484	return (
	485	self.db_pool.runInteraction(
	486	"get_backfill_events",
	487	self._get_backfill_events,
	488	room_id,
	489	event_list,
	490	limit,
	491	)
	492	.addCallback(self.get_events_as_list)
	493	.addCallback(lambda l: sorted(l, key=lambda e: -e.depth))
	494	)
	495
	496	def _get_backfill_events(self, txn, room_id, event_list, limit):
	497	logger.debug("_get_backfill_events: %s, %r, %s", room_id, event_list, limit)
	498
	499	event_results = set()
	500
	501	# We want to make sure that we do a breadth-first, "depth" ordered
	502	# search.
	503
	504	query = (
	505	"SELECT depth, prev_event_id FROM event_edges"
	506	" INNER JOIN events"
	507	" ON prev_event_id = events.event_id"
	508	" WHERE event_edges.event_id = ?"
	509	" AND event_edges.is_state = ?"
	510	" LIMIT ?"
	511	)
	512
	513	queue = PriorityQueue()
	514
	515	for event_id in event_list:
	516	depth = self.db_pool.simple_select_one_onecol_txn(
	517	txn,
	518	table="events",
	519	keyvalues={"event_id": event_id, "room_id": room_id},
	520	retcol="depth",
	521	allow_none=True,
	522	)
	523
	524	if depth:
	525	queue.put((-depth, event_id))
	526
	527	while not queue.empty() and len(event_results) < limit:
	528	try:
	529	_, event_id = queue.get_nowait()
	530	except Empty:
	531	break
	532
	533	if event_id in event_results:
	534	continue
	535
	536	event_results.add(event_id)
	537
	538	txn.execute(query, (event_id, False, limit - len(event_results)))
	539
	540	for row in txn:
	541	if row[1] not in event_results:
	542	queue.put((-row[0], row[1]))
	543
	544	return event_results
	545
	546	async def get_missing_events(self, room_id, earliest_events, latest_events, limit):
	547	ids = await self.db_pool.runInteraction(
	548	"get_missing_events",
	549	self._get_missing_events,
	550	room_id,
	551	earliest_events,
	552	latest_events,
	553	limit,
	554	)
	555	events = await self.get_events_as_list(ids)
	556	return events
	557
	558	def _get_missing_events(self, txn, room_id, earliest_events, latest_events, limit):
	559
	560	seen_events = set(earliest_events)
	561	front = set(latest_events) - seen_events
	562	event_results = []
	563
	564	query = (
	565	"SELECT prev_event_id FROM event_edges "
	566	"WHERE room_id = ? AND event_id = ? AND is_state = ? "
	567	"LIMIT ?"
	568	)
	569
	570	while front and len(event_results) < limit:
	571	new_front = set()
	572	for event_id in front:
	573	txn.execute(
	574	query, (room_id, event_id, False, limit - len(event_results))
	575	)
	576
	577	new_results = {t[0] for t in txn} - seen_events
	578
	579	new_front \|= new_results
	580	seen_events \|= new_results
	581	event_results.extend(new_results)
	582
	583	front = new_front
	584
	585	# we built the list working backwards from latest_events; we now need to
	586	# reverse it so that the events are approximately chronological.
	587	event_results.reverse()
	588	return event_results
	589
	590	async def get_successor_events(self, event_ids: Iterable[str]) -> List[str]:
	591	"""Fetch all events that have the given events as a prev event
	592
	593	Args:
	594	event_ids: The events to use as the previous events.
	595	"""
	596	rows = await self.db_pool.simple_select_many_batch(
	597	table="event_edges",
	598	column="prev_event_id",
	599	iterable=event_ids,
	600	retcols=("event_id",),
	601	desc="get_successor_events",
	602	)
	603
	604	return [row["event_id"] for row in rows]
	605
	606
	607	class EventFederationStore(EventFederationWorkerStore):
	608	""" Responsible for storing and serving up the various graphs associated
	609	with an event. Including the main event graph and the auth chains for an
	610	event.
	611
	612	Also has methods for getting the front (latest) and back (oldest) edges
	613	of the event graphs. These are used to generate the parents for new events
	614	and backfilling from another server respectively.
	615	"""
	616
	617	EVENT_AUTH_STATE_ONLY = "event_auth_state_only"
	618
	619	def __init__(self, database: DatabasePool, db_conn, hs):
	620	super(EventFederationStore, self).__init__(database, db_conn, hs)
	621
	622	self.db_pool.updates.register_background_update_handler(
	623	self.EVENT_AUTH_STATE_ONLY, self._background_delete_non_state_event_auth
	624	)
	625
	626	hs.get_clock().looping_call(
	627	self._delete_old_forward_extrem_cache, 60 * 60 * 1000
	628	)
	629
	630	def _delete_old_forward_extrem_cache(self):
	631	def _delete_old_forward_extrem_cache_txn(txn):
	632	# Delete entries older than a month, while making sure we don't delete
	633	# the only entries for a room.
	634	sql = """
	635	DELETE FROM stream_ordering_to_exterm
	636	WHERE
	637	room_id IN (
	638	SELECT room_id
	639	FROM stream_ordering_to_exterm
	640	WHERE stream_ordering > ?
	641	) AND stream_ordering < ?
	642	"""
	643	txn.execute(
	644	sql, (self.stream_ordering_month_ago, self.stream_ordering_month_ago)
	645	)
	646
	647	return run_as_background_process(
	648	"delete_old_forward_extrem_cache",
	649	self.db_pool.runInteraction,
	650	"_delete_old_forward_extrem_cache",
	651	_delete_old_forward_extrem_cache_txn,
	652	)
	653
	654	def clean_room_for_join(self, room_id):
	655	return self.db_pool.runInteraction(
	656	"clean_room_for_join", self._clean_room_for_join_txn, room_id
	657	)
	658
	659	def _clean_room_for_join_txn(self, txn, room_id):
	660	query = "DELETE FROM event_forward_extremities WHERE room_id = ?"
	661
	662	txn.execute(query, (room_id,))
	663	txn.call_after(self.get_latest_event_ids_in_room.invalidate, (room_id,))
	664
	665	async def _background_delete_non_state_event_auth(self, progress, batch_size):
	666	def delete_event_auth(txn):
	667	target_min_stream_id = progress.get("target_min_stream_id_inclusive")
	668	max_stream_id = progress.get("max_stream_id_exclusive")
	669
	670	if not target_min_stream_id or not max_stream_id:
	671	txn.execute("SELECT COALESCE(MIN(stream_ordering), 0) FROM events")
	672	rows = txn.fetchall()
	673	target_min_stream_id = rows[0][0]
	674
	675	txn.execute("SELECT COALESCE(MAX(stream_ordering), 0) FROM events")
	676	rows = txn.fetchall()
	677	max_stream_id = rows[0][0]
	678
	679	min_stream_id = max_stream_id - batch_size
	680
	681	sql = """
	682	DELETE FROM event_auth
	683	WHERE event_id IN (
	684	SELECT event_id FROM events
	685	LEFT JOIN state_events USING (room_id, event_id)
	686	WHERE ? <= stream_ordering AND stream_ordering < ?
	687	AND state_key IS null
	688	)
	689	"""
	690
	691	txn.execute(sql, (min_stream_id, max_stream_id))
	692
	693	new_progress = {
	694	"target_min_stream_id_inclusive": target_min_stream_id,
	695	"max_stream_id_exclusive": min_stream_id,
	696	}
	697
	698	self.db_pool.updates._background_update_progress_txn(
	699	txn, self.EVENT_AUTH_STATE_ONLY, new_progress
	700	)
	701
	702	return min_stream_id >= target_min_stream_id
	703
	704	result = await self.db_pool.runInteraction(
	705	self.EVENT_AUTH_STATE_ONLY, delete_event_auth
	706	)
	707
	708	if not result:
	709	await self.db_pool.updates._end_background_update(
	710	self.EVENT_AUTH_STATE_ONLY
	711	)
	712
	713	return batch_size

+884

-0

synapse/storage/databases/main/event_push_actions.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2015 OpenMarket Ltd
	2	# Copyright 2018 New Vector Ltd
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import logging
	17	from typing import List
	18
	19	from synapse.metrics.background_process_metrics import run_as_background_process
	20	from synapse.storage._base import LoggingTransaction, SQLBaseStore, db_to_json
	21	from synapse.storage.database import DatabasePool
	22	from synapse.util import json_encoder
	23	from synapse.util.caches.descriptors import cachedInlineCallbacks
	24
	25	logger = logging.getLogger(__name__)
	26
	27
	28	DEFAULT_NOTIF_ACTION = ["notify", {"set_tweak": "highlight", "value": False}]
	29	DEFAULT_HIGHLIGHT_ACTION = [
	30	"notify",
	31	{"set_tweak": "sound", "value": "default"},
	32	{"set_tweak": "highlight"},
	33	]
	34
	35
	36	def _serialize_action(actions, is_highlight):
	37	"""Custom serializer for actions. This allows us to "compress" common actions.
	38
	39	We use the fact that most users have the same actions for notifs (and for
	40	highlights).
	41	We store these default actions as the empty string rather than the full JSON.
	42	Since the empty string isn't valid JSON there is no risk of this clashing with
	43	any real JSON actions
	44	"""
	45	if is_highlight:
	46	if actions == DEFAULT_HIGHLIGHT_ACTION:
	47	return "" # We use empty string as the column is non-NULL
	48	else:
	49	if actions == DEFAULT_NOTIF_ACTION:
	50	return ""
	51	return json_encoder.encode(actions)
	52
	53
	54	def _deserialize_action(actions, is_highlight):
	55	"""Custom deserializer for actions. This allows us to "compress" common actions
	56	"""
	57	if actions:
	58	return db_to_json(actions)
	59
	60	if is_highlight:
	61	return DEFAULT_HIGHLIGHT_ACTION
	62	else:
	63	return DEFAULT_NOTIF_ACTION
	64
	65
	66	class EventPushActionsWorkerStore(SQLBaseStore):
	67	def __init__(self, database: DatabasePool, db_conn, hs):
	68	super(EventPushActionsWorkerStore, self).__init__(database, db_conn, hs)
	69
	70	# These get correctly set by _find_stream_orderings_for_times_txn
	71	self.stream_ordering_month_ago = None
	72	self.stream_ordering_day_ago = None
	73
	74	cur = LoggingTransaction(
	75	db_conn.cursor(),
	76	name="_find_stream_orderings_for_times_txn",
	77	database_engine=self.database_engine,
	78	)
	79	self._find_stream_orderings_for_times_txn(cur)
	80	cur.close()
	81
	82	self.find_stream_orderings_looping_call = self._clock.looping_call(
	83	self._find_stream_orderings_for_times, 10 * 60 * 1000
	84	)
	85	self._rotate_delay = 3
	86	self._rotate_count = 10000
	87
	88	@cachedInlineCallbacks(num_args=3, tree=True, max_entries=5000)
	89	def get_unread_event_push_actions_by_room_for_user(
	90	self, room_id, user_id, last_read_event_id
	91	):
	92	ret = yield self.db_pool.runInteraction(
	93	"get_unread_event_push_actions_by_room",
	94	self._get_unread_counts_by_receipt_txn,
	95	room_id,
	96	user_id,
	97	last_read_event_id,
	98	)
	99	return ret
	100
	101	def _get_unread_counts_by_receipt_txn(
	102	self, txn, room_id, user_id, last_read_event_id
	103	):
	104	sql = (
	105	"SELECT stream_ordering"
	106	" FROM events"
	107	" WHERE room_id = ? AND event_id = ?"
	108	)
	109	txn.execute(sql, (room_id, last_read_event_id))
	110	results = txn.fetchall()
	111	if len(results) == 0:
	112	return {"notify_count": 0, "highlight_count": 0}
	113
	114	stream_ordering = results[0][0]
	115
	116	return self._get_unread_counts_by_pos_txn(
	117	txn, room_id, user_id, stream_ordering
	118	)
	119
	120	def _get_unread_counts_by_pos_txn(self, txn, room_id, user_id, stream_ordering):
	121
	122	# First get number of notifications.
	123	# We don't need to put a notif=1 clause as all rows always have
	124	# notif=1
	125	sql = (
	126	"SELECT count(*)"
	127	" FROM event_push_actions ea"
	128	" WHERE"
	129	" user_id = ?"
	130	" AND room_id = ?"
	131	" AND stream_ordering > ?"
	132	)
	133
	134	txn.execute(sql, (user_id, room_id, stream_ordering))
	135	row = txn.fetchone()
	136	notify_count = row[0] if row else 0
	137
	138	txn.execute(
	139	"""
	140	SELECT notif_count FROM event_push_summary
	141	WHERE room_id = ? AND user_id = ? AND stream_ordering > ?
	142	""",
	143	(room_id, user_id, stream_ordering),
	144	)
	145	rows = txn.fetchall()
	146	if rows:
	147	notify_count += rows[0][0]
	148
	149	# Now get the number of highlights
	150	sql = (
	151	"SELECT count(*)"
	152	" FROM event_push_actions ea"
	153	" WHERE"
	154	" highlight = 1"
	155	" AND user_id = ?"
	156	" AND room_id = ?"
	157	" AND stream_ordering > ?"
	158	)
	159
	160	txn.execute(sql, (user_id, room_id, stream_ordering))
	161	row = txn.fetchone()
	162	highlight_count = row[0] if row else 0
	163
	164	return {"notify_count": notify_count, "highlight_count": highlight_count}
	165
	166	async def get_push_action_users_in_range(
	167	self, min_stream_ordering, max_stream_ordering
	168	):
	169	def f(txn):
	170	sql = (
	171	"SELECT DISTINCT(user_id) FROM event_push_actions WHERE"
	172	" stream_ordering >= ? AND stream_ordering <= ?"
	173	)
	174	txn.execute(sql, (min_stream_ordering, max_stream_ordering))
	175	return [r[0] for r in txn]
	176
	177	ret = await self.db_pool.runInteraction("get_push_action_users_in_range", f)
	178	return ret
	179
	180	async def get_unread_push_actions_for_user_in_range_for_http(
	181	self,
	182	user_id: str,
	183	min_stream_ordering: int,
	184	max_stream_ordering: int,
	185	limit: int = 20,
	186	) -> List[dict]:
	187	"""Get a list of the most recent unread push actions for a given user,
	188	within the given stream ordering range. Called by the httppusher.
	189
	190	Args:
	191	user_id: The user to fetch push actions for.
	192	min_stream_ordering: The exclusive lower bound on the
	193	stream ordering of event push actions to fetch.
	194	max_stream_ordering: The inclusive upper bound on the
	195	stream ordering of event push actions to fetch.
	196	limit: The maximum number of rows to return.
	197	Returns:
	198	A list of dicts with the keys "event_id", "room_id", "stream_ordering", "actions".
	199	The list will be ordered by ascending stream_ordering.
	200	The list will have between 0~limit entries.
	201	"""
	202	# find rooms that have a read receipt in them and return the next
	203	# push actions
	204	def get_after_receipt(txn):
	205	# find rooms that have a read receipt in them and return the next
	206	# push actions
	207	sql = (
	208	"SELECT ep.event_id, ep.room_id, ep.stream_ordering, ep.actions,"
	209	" ep.highlight "
	210	" FROM ("
	211	" SELECT room_id,"
	212	" MAX(stream_ordering) as stream_ordering"
	213	" FROM events"
	214	" INNER JOIN receipts_linearized USING (room_id, event_id)"
	215	" WHERE receipt_type = 'm.read' AND user_id = ?"
	216	" GROUP BY room_id"
	217	") AS rl,"
	218	" event_push_actions AS ep"
	219	" WHERE"
	220	" ep.room_id = rl.room_id"
	221	" AND ep.stream_ordering > rl.stream_ordering"
	222	" AND ep.user_id = ?"
	223	" AND ep.stream_ordering > ?"
	224	" AND ep.stream_ordering <= ?"
	225	" ORDER BY ep.stream_ordering ASC LIMIT ?"
	226	)
	227	args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]
	228	txn.execute(sql, args)
	229	return txn.fetchall()
	230
	231	after_read_receipt = await self.db_pool.runInteraction(
	232	"get_unread_push_actions_for_user_in_range_http_arr", get_after_receipt
	233	)
	234
	235	# There are rooms with push actions in them but you don't have a read receipt in
	236	# them e.g. rooms you've been invited to, so get push actions for rooms which do
	237	# not have read receipts in them too.
	238	def get_no_receipt(txn):
	239	sql = (
	240	"SELECT ep.event_id, ep.room_id, ep.stream_ordering, ep.actions,"
	241	" ep.highlight "
	242	" FROM event_push_actions AS ep"
	243	" INNER JOIN events AS e USING (room_id, event_id)"
	244	" WHERE"
	245	" ep.room_id NOT IN ("
	246	" SELECT room_id FROM receipts_linearized"
	247	" WHERE receipt_type = 'm.read' AND user_id = ?"
	248	" GROUP BY room_id"
	249	" )"
	250	" AND ep.user_id = ?"
	251	" AND ep.stream_ordering > ?"
	252	" AND ep.stream_ordering <= ?"
	253	" ORDER BY ep.stream_ordering ASC LIMIT ?"
	254	)
	255	args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]
	256	txn.execute(sql, args)
	257	return txn.fetchall()
	258
	259	no_read_receipt = await self.db_pool.runInteraction(
	260	"get_unread_push_actions_for_user_in_range_http_nrr", get_no_receipt
	261	)
	262
	263	notifs = [
	264	{
	265	"event_id": row[0],
	266	"room_id": row[1],
	267	"stream_ordering": row[2],
	268	"actions": _deserialize_action(row[3], row[4]),
	269	}
	270	for row in after_read_receipt + no_read_receipt
	271	]
	272
	273	# Now sort it so it's ordered correctly, since currently it will
	274	# contain results from the first query, correctly ordered, followed
	275	# by results from the second query, but we want them all ordered
	276	# by stream_ordering, oldest first.
	277	notifs.sort(key=lambda r: r["stream_ordering"])
	278
	279	# Take only up to the limit. We have to stop at the limit because
	280	# one of the subqueries may have hit the limit.
	281	return notifs[:limit]
	282
	283	async def get_unread_push_actions_for_user_in_range_for_email(
	284	self,
	285	user_id: str,
	286	min_stream_ordering: int,
	287	max_stream_ordering: int,
	288	limit: int = 20,
	289	) -> List[dict]:
	290	"""Get a list of the most recent unread push actions for a given user,
	291	within the given stream ordering range. Called by the emailpusher
	292
	293	Args:
	294	user_id: The user to fetch push actions for.
	295	min_stream_ordering: The exclusive lower bound on the
	296	stream ordering of event push actions to fetch.
	297	max_stream_ordering: The inclusive upper bound on the
	298	stream ordering of event push actions to fetch.
	299	limit: The maximum number of rows to return.
	300	Returns:
	301	A list of dicts with the keys "event_id", "room_id", "stream_ordering", "actions", "received_ts".
	302	The list will be ordered by descending received_ts.
	303	The list will have between 0~limit entries.
	304	"""
	305	# find rooms that have a read receipt in them and return the most recent
	306	# push actions
	307	def get_after_receipt(txn):
	308	sql = (
	309	"SELECT ep.event_id, ep.room_id, ep.stream_ordering, ep.actions,"
	310	" ep.highlight, e.received_ts"
	311	" FROM ("
	312	" SELECT room_id,"
	313	" MAX(stream_ordering) as stream_ordering"
	314	" FROM events"
	315	" INNER JOIN receipts_linearized USING (room_id, event_id)"
	316	" WHERE receipt_type = 'm.read' AND user_id = ?"
	317	" GROUP BY room_id"
	318	") AS rl,"
	319	" event_push_actions AS ep"
	320	" INNER JOIN events AS e USING (room_id, event_id)"
	321	" WHERE"
	322	" ep.room_id = rl.room_id"
	323	" AND ep.stream_ordering > rl.stream_ordering"
	324	" AND ep.user_id = ?"
	325	" AND ep.stream_ordering > ?"
	326	" AND ep.stream_ordering <= ?"
	327	" ORDER BY ep.stream_ordering DESC LIMIT ?"
	328	)
	329	args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]
	330	txn.execute(sql, args)
	331	return txn.fetchall()
	332
	333	after_read_receipt = await self.db_pool.runInteraction(
	334	"get_unread_push_actions_for_user_in_range_email_arr", get_after_receipt
	335	)
	336
	337	# There are rooms with push actions in them but you don't have a read receipt in
	338	# them e.g. rooms you've been invited to, so get push actions for rooms which do
	339	# not have read receipts in them too.
	340	def get_no_receipt(txn):
	341	sql = (
	342	"SELECT ep.event_id, ep.room_id, ep.stream_ordering, ep.actions,"
	343	" ep.highlight, e.received_ts"
	344	" FROM event_push_actions AS ep"
	345	" INNER JOIN events AS e USING (room_id, event_id)"
	346	" WHERE"
	347	" ep.room_id NOT IN ("
	348	" SELECT room_id FROM receipts_linearized"
	349	" WHERE receipt_type = 'm.read' AND user_id = ?"
	350	" GROUP BY room_id"
	351	" )"
	352	" AND ep.user_id = ?"
	353	" AND ep.stream_ordering > ?"
	354	" AND ep.stream_ordering <= ?"
	355	" ORDER BY ep.stream_ordering DESC LIMIT ?"
	356	)
	357	args = [user_id, user_id, min_stream_ordering, max_stream_ordering, limit]
	358	txn.execute(sql, args)
	359	return txn.fetchall()
	360
	361	no_read_receipt = await self.db_pool.runInteraction(
	362	"get_unread_push_actions_for_user_in_range_email_nrr", get_no_receipt
	363	)
	364
	365	# Make a list of dicts from the two sets of results.
	366	notifs = [
	367	{
	368	"event_id": row[0],
	369	"room_id": row[1],
	370	"stream_ordering": row[2],
	371	"actions": _deserialize_action(row[3], row[4]),
	372	"received_ts": row[5],
	373	}
	374	for row in after_read_receipt + no_read_receipt
	375	]
	376
	377	# Now sort it so it's ordered correctly, since currently it will
	378	# contain results from the first query, correctly ordered, followed
	379	# by results from the second query, but we want them all ordered
	380	# by received_ts (most recent first)
	381	notifs.sort(key=lambda r: -(r["received_ts"] or 0))
	382
	383	# Now return the first `limit`
	384	return notifs[:limit]
	385
	386	def get_if_maybe_push_in_range_for_user(self, user_id, min_stream_ordering):
	387	"""A fast check to see if there might be something to push for the
	388	user since the given stream ordering. May return false positives.
	389
	390	Useful to know whether to bother starting a pusher on start up or not.
	391
	392	Args:
	393	user_id (str)
	394	min_stream_ordering (int)
	395
	396	Returns:
	397	Deferred[bool]: True if there may be push to process, False if
	398	there definitely isn't.
	399	"""
	400
	401	def _get_if_maybe_push_in_range_for_user_txn(txn):
	402	sql = """
	403	SELECT 1 FROM event_push_actions
	404	WHERE user_id = ? AND stream_ordering > ?
	405	LIMIT 1
	406	"""
	407
	408	txn.execute(sql, (user_id, min_stream_ordering))
	409	return bool(txn.fetchone())
	410
	411	return self.db_pool.runInteraction(
	412	"get_if_maybe_push_in_range_for_user",
	413	_get_if_maybe_push_in_range_for_user_txn,
	414	)
	415
	416	async def add_push_actions_to_staging(self, event_id, user_id_actions):
	417	"""Add the push actions for the event to the push action staging area.
	418
	419	Args:
	420	event_id (str)
	421	user_id_actions (dict[str, list[dict\|str])]): A dictionary mapping
	422	user_id to list of push actions, where an action can either be
	423	a string or dict.
	424
	425	Returns:
	426	Deferred
	427	"""
	428
	429	if not user_id_actions:
	430	return
	431
	432	# This is a helper function for generating the necessary tuple that
	433	# can be used to inert into the `event_push_actions_staging` table.
	434	def _gen_entry(user_id, actions):
	435	is_highlight = 1 if _action_has_highlight(actions) else 0
	436	return (
	437	event_id, # event_id column
	438	user_id, # user_id column
	439	_serialize_action(actions, is_highlight), # actions column
	440	1, # notif column
	441	is_highlight, # highlight column
	442	)
	443
	444	def _add_push_actions_to_staging_txn(txn):
	445	# We don't use simple_insert_many here to avoid the overhead
	446	# of generating lists of dicts.
	447
	448	sql = """
	449	INSERT INTO event_push_actions_staging
	450	(event_id, user_id, actions, notif, highlight)
	451	VALUES (?, ?, ?, ?, ?)
	452	"""
	453
	454	txn.executemany(
	455	sql,
	456	(
	457	_gen_entry(user_id, actions)
	458	for user_id, actions in user_id_actions.items()
	459	),
	460	)
	461
	462	return await self.db_pool.runInteraction(
	463	"add_push_actions_to_staging", _add_push_actions_to_staging_txn
	464	)
	465
	466	async def remove_push_actions_from_staging(self, event_id: str) -> None:
	467	"""Called if we failed to persist the event to ensure that stale push
	468	actions don't build up in the DB
	469	"""
	470
	471	try:
	472	res = await self.db_pool.simple_delete(
	473	table="event_push_actions_staging",
	474	keyvalues={"event_id": event_id},
	475	desc="remove_push_actions_from_staging",
	476	)
	477	return res
	478	except Exception:
	479	# this method is called from an exception handler, so propagating
	480	# another exception here really isn't helpful - there's nothing
	481	# the caller can do about it. Just log the exception and move on.
	482	logger.exception(
	483	"Error removing push actions after event persistence failure"
	484	)
	485
	486	def _find_stream_orderings_for_times(self):
	487	return run_as_background_process(
	488	"event_push_action_stream_orderings",
	489	self.db_pool.runInteraction,
	490	"_find_stream_orderings_for_times",
	491	self._find_stream_orderings_for_times_txn,
	492	)
	493
	494	def _find_stream_orderings_for_times_txn(self, txn):
	495	logger.info("Searching for stream ordering 1 month ago")
	496	self.stream_ordering_month_ago = self._find_first_stream_ordering_after_ts_txn(
	497	txn, self._clock.time_msec() - 30 * 24 * 60 * 60 * 1000
	498	)
	499	logger.info(
	500	"Found stream ordering 1 month ago: it's %d", self.stream_ordering_month_ago
	501	)
	502	logger.info("Searching for stream ordering 1 day ago")
	503	self.stream_ordering_day_ago = self._find_first_stream_ordering_after_ts_txn(
	504	txn, self._clock.time_msec() - 24 * 60 * 60 * 1000
	505	)
	506	logger.info(
	507	"Found stream ordering 1 day ago: it's %d", self.stream_ordering_day_ago
	508	)
	509
	510	def find_first_stream_ordering_after_ts(self, ts):
	511	"""Gets the stream ordering corresponding to a given timestamp.
	512
	513	Specifically, finds the stream_ordering of the first event that was
	514	received on or after the timestamp. This is done by a binary search on
	515	the events table, since there is no index on received_ts, so is
	516	relatively slow.
	517
	518	Args:
	519	ts (int): timestamp in millis
	520
	521	Returns:
	522	Deferred[int]: stream ordering of the first event received on/after
	523	the timestamp
	524	"""
	525	return self.db_pool.runInteraction(
	526	"_find_first_stream_ordering_after_ts_txn",
	527	self._find_first_stream_ordering_after_ts_txn,
	528	ts,
	529	)
	530
	531	@staticmethod
	532	def _find_first_stream_ordering_after_ts_txn(txn, ts):
	533	"""
	534	Find the stream_ordering of the first event that was received on or
	535	after a given timestamp. This is relatively slow as there is no index
	536	on received_ts but we can then use this to delete push actions before
	537	this.
	538
	539	received_ts must necessarily be in the same order as stream_ordering
	540	and stream_ordering is indexed, so we manually binary search using
	541	stream_ordering
	542
	543	Args:
	544	txn (twisted.enterprise.adbapi.Transaction):
	545	ts (int): timestamp to search for
	546
	547	Returns:
	548	int: stream ordering
	549	"""
	550	txn.execute("SELECT MAX(stream_ordering) FROM events")
	551	max_stream_ordering = txn.fetchone()[0]
	552
	553	if max_stream_ordering is None:
	554	return 0
	555
	556	# We want the first stream_ordering in which received_ts is greater
	557	# than or equal to ts. Call this point X.
	558	#
	559	# We maintain the invariants:
	560	#
	561	# range_start <= X <= range_end
	562	#
	563	range_start = 0
	564	range_end = max_stream_ordering + 1
	565
	566	# Given a stream_ordering, look up the timestamp at that
	567	# stream_ordering.
	568	#
	569	# The array may be sparse (we may be missing some stream_orderings).
	570	# We treat the gaps as the same as having the same value as the
	571	# preceding entry, because we will pick the lowest stream_ordering
	572	# which satisfies our requirement of received_ts >= ts.
	573	#
	574	# For example, if our array of events indexed by stream_ordering is
	575	# [10, <none>, 20], we should treat this as being equivalent to
	576	# [10, 10, 20].
	577	#
	578	sql = (
	579	"SELECT received_ts FROM events"
	580	" WHERE stream_ordering <= ?"
	581	" ORDER BY stream_ordering DESC"
	582	" LIMIT 1"
	583	)
	584
	585	while range_end - range_start > 0:
	586	middle = (range_end + range_start) // 2
	587	txn.execute(sql, (middle,))
	588	row = txn.fetchone()
	589	if row is None:
	590	# no rows with stream_ordering<=middle
	591	range_start = middle + 1
	592	continue
	593
	594	middle_ts = row[0]
	595	if ts > middle_ts:
	596	# we got a timestamp lower than the one we were looking for.
	597	# definitely need to look higher: X > middle.
	598	range_start = middle + 1
	599	else:
	600	# we got a timestamp higher than (or the same as) the one we
	601	# were looking for. We aren't yet sure about the point we
	602	# looked up, but we can be sure that X <= middle.
	603	range_end = middle
	604
	605	return range_end
	606
	607	async def get_time_of_last_push_action_before(self, stream_ordering):
	608	def f(txn):
	609	sql = (
	610	"SELECT e.received_ts"
	611	" FROM event_push_actions AS ep"
	612	" JOIN events e ON ep.room_id = e.room_id AND ep.event_id = e.event_id"
	613	" WHERE ep.stream_ordering > ?"
	614	" ORDER BY ep.stream_ordering ASC"
	615	" LIMIT 1"
	616	)
	617	txn.execute(sql, (stream_ordering,))
	618	return txn.fetchone()
	619
	620	result = await self.db_pool.runInteraction(
	621	"get_time_of_last_push_action_before", f
	622	)
	623	return result[0] if result else None
	624
	625
	626	class EventPushActionsStore(EventPushActionsWorkerStore):
	627	EPA_HIGHLIGHT_INDEX = "epa_highlight_index"
	628
	629	def __init__(self, database: DatabasePool, db_conn, hs):
	630	super(EventPushActionsStore, self).__init__(database, db_conn, hs)
	631
	632	self.db_pool.updates.register_background_index_update(
	633	self.EPA_HIGHLIGHT_INDEX,
	634	index_name="event_push_actions_u_highlight",
	635	table="event_push_actions",
	636	columns=["user_id", "stream_ordering"],
	637	)
	638
	639	self.db_pool.updates.register_background_index_update(
	640	"event_push_actions_highlights_index",
	641	index_name="event_push_actions_highlights_index",
	642	table="event_push_actions",
	643	columns=["user_id", "room_id", "topological_ordering", "stream_ordering"],
	644	where_clause="highlight=1",
	645	)
	646
	647	self._doing_notif_rotation = False
	648	self._rotate_notif_loop = self._clock.looping_call(
	649	self._start_rotate_notifs, 30 * 60 * 1000
	650	)
	651
	652	async def get_push_actions_for_user(
	653	self, user_id, before=None, limit=50, only_highlight=False
	654	):
	655	def f(txn):
	656	before_clause = ""
	657	if before:
	658	before_clause = "AND epa.stream_ordering < ?"
	659	args = [user_id, before, limit]
	660	else:
	661	args = [user_id, limit]
	662
	663	if only_highlight:
	664	if len(before_clause) > 0:
	665	before_clause += " "
	666	before_clause += "AND epa.highlight = 1"
	667
	668	# NB. This assumes event_ids are globally unique since
	669	# it makes the query easier to index
	670	sql = (
	671	"SELECT epa.event_id, epa.room_id,"
	672	" epa.stream_ordering, epa.topological_ordering,"
	673	" epa.actions, epa.highlight, epa.profile_tag, e.received_ts"
	674	" FROM event_push_actions epa, events e"
	675	" WHERE epa.event_id = e.event_id"
	676	" AND epa.user_id = ? %s"
	677	" ORDER BY epa.stream_ordering DESC"
	678	" LIMIT ?" % (before_clause,)
	679	)
	680	txn.execute(sql, args)
	681	return self.db_pool.cursor_to_dict(txn)
	682
	683	push_actions = await self.db_pool.runInteraction("get_push_actions_for_user", f)
	684	for pa in push_actions:
	685	pa["actions"] = _deserialize_action(pa["actions"], pa["highlight"])
	686	return push_actions
	687
	688	async def get_latest_push_action_stream_ordering(self):
	689	def f(txn):
	690	txn.execute("SELECT MAX(stream_ordering) FROM event_push_actions")
	691	return txn.fetchone()
	692
	693	result = await self.db_pool.runInteraction(
	694	"get_latest_push_action_stream_ordering", f
	695	)
	696	return result[0] or 0
	697
	698	def _remove_old_push_actions_before_txn(
	699	self, txn, room_id, user_id, stream_ordering
	700	):
	701	"""
	702	Purges old push actions for a user and room before a given
	703	stream_ordering.
	704
	705	We however keep a months worth of highlighted notifications, so that
	706	users can still get a list of recent highlights.
	707
	708	Args:
	709	txn: The transcation
	710	room_id: Room ID to delete from
	711	user_id: user ID to delete for
	712	stream_ordering: The lowest stream ordering which will
	713	not be deleted.
	714	"""
	715	txn.call_after(
	716	self.get_unread_event_push_actions_by_room_for_user.invalidate_many,
	717	(room_id, user_id),
	718	)
	719
	720	# We need to join on the events table to get the received_ts for
	721	# event_push_actions and sqlite won't let us use a join in a delete so
	722	# we can't just delete where received_ts < x. Furthermore we can
	723	# only identify event_push_actions by a tuple of room_id, event_id
	724	# we we can't use a subquery.
	725	# Instead, we look up the stream ordering for the last event in that
	726	# room received before the threshold time and delete event_push_actions
	727	# in the room with a stream_odering before that.
	728	txn.execute(
	729	"DELETE FROM event_push_actions "
	730	" WHERE user_id = ? AND room_id = ? AND "
	731	" stream_ordering <= ?"
	732	" AND ((stream_ordering < ? AND highlight = 1) or highlight = 0)",
	733	(user_id, room_id, stream_ordering, self.stream_ordering_month_ago),
	734	)
	735
	736	txn.execute(
	737	"""
	738	DELETE FROM event_push_summary
	739	WHERE room_id = ? AND user_id = ? AND stream_ordering <= ?
	740	""",
	741	(room_id, user_id, stream_ordering),
	742	)
	743
	744	def _start_rotate_notifs(self):
	745	return run_as_background_process("rotate_notifs", self._rotate_notifs)
	746
	747	async def _rotate_notifs(self):
	748	if self._doing_notif_rotation or self.stream_ordering_day_ago is None:
	749	return
	750	self._doing_notif_rotation = True
	751
	752	try:
	753	while True:
	754	logger.info("Rotating notifications")
	755
	756	caught_up = await self.db_pool.runInteraction(
	757	"_rotate_notifs", self._rotate_notifs_txn
	758	)
	759	if caught_up:
	760	break
	761	await self.hs.get_clock().sleep(self._rotate_delay)
	762	finally:
	763	self._doing_notif_rotation = False
	764
	765	def _rotate_notifs_txn(self, txn):
	766	"""Archives older notifications into event_push_summary. Returns whether
	767	the archiving process has caught up or not.
	768	"""
	769
	770	old_rotate_stream_ordering = self.db_pool.simple_select_one_onecol_txn(
	771	txn,
	772	table="event_push_summary_stream_ordering",
	773	keyvalues={},
	774	retcol="stream_ordering",
	775	)
	776
	777	# We don't to try and rotate millions of rows at once, so we cap the
	778	# maximum stream ordering we'll rotate before.
	779	txn.execute(
	780	"""
	781	SELECT stream_ordering FROM event_push_actions
	782	WHERE stream_ordering > ?
	783	ORDER BY stream_ordering ASC LIMIT 1 OFFSET ?
	784	""",
	785	(old_rotate_stream_ordering, self._rotate_count),
	786	)
	787	stream_row = txn.fetchone()
	788	if stream_row:
	789	(offset_stream_ordering,) = stream_row
	790	rotate_to_stream_ordering = min(
	791	self.stream_ordering_day_ago, offset_stream_ordering
	792	)
	793	caught_up = offset_stream_ordering >= self.stream_ordering_day_ago
	794	else:
	795	rotate_to_stream_ordering = self.stream_ordering_day_ago
	796	caught_up = True
	797
	798	logger.info("Rotating notifications up to: %s", rotate_to_stream_ordering)
	799
	800	self._rotate_notifs_before_txn(txn, rotate_to_stream_ordering)
	801
	802	# We have caught up iff we were limited by `stream_ordering_day_ago`
	803	return caught_up
	804
	805	def _rotate_notifs_before_txn(self, txn, rotate_to_stream_ordering):
	806	old_rotate_stream_ordering = self.db_pool.simple_select_one_onecol_txn(
	807	txn,
	808	table="event_push_summary_stream_ordering",
	809	keyvalues={},
	810	retcol="stream_ordering",
	811	)
	812
	813	# Calculate the new counts that should be upserted into event_push_summary
	814	sql = """
	815	SELECT user_id, room_id,
	816	coalesce(old.notif_count, 0) + upd.notif_count,
	817	upd.stream_ordering,
	818	old.user_id
	819	FROM (
	820	SELECT user_id, room_id, count(*) as notif_count,
	821	max(stream_ordering) as stream_ordering
	822	FROM event_push_actions
	823	WHERE ? <= stream_ordering AND stream_ordering < ?
	824	AND highlight = 0
	825	GROUP BY user_id, room_id
	826	) AS upd
	827	LEFT JOIN event_push_summary AS old USING (user_id, room_id)
	828	"""
	829
	830	txn.execute(sql, (old_rotate_stream_ordering, rotate_to_stream_ordering))
	831	rows = txn.fetchall()
	832
	833	logger.info("Rotating notifications, handling %d rows", len(rows))
	834
	835	# If the `old.user_id` above is NULL then we know there isn't already an
	836	# entry in the table, so we simply insert it. Otherwise we update the
	837	# existing table.
	838	self.db_pool.simple_insert_many_txn(
	839	txn,
	840	table="event_push_summary",
	841	values=[
	842	{
	843	"user_id": row[0],
	844	"room_id": row[1],
	845	"notif_count": row[2],
	846	"stream_ordering": row[3],
	847	}
	848	for row in rows
	849	if row[4] is None
	850	],
	851	)
	852
	853	txn.executemany(
	854	"""
	855	UPDATE event_push_summary SET notif_count = ?, stream_ordering = ?
	856	WHERE user_id = ? AND room_id = ?
	857	""",
	858	((row[2], row[3], row[0], row[1]) for row in rows if row[4] is not None),
	859	)
	860
	861	txn.execute(
	862	"DELETE FROM event_push_actions"
	863	" WHERE ? <= stream_ordering AND stream_ordering < ? AND highlight = 0",
	864	(old_rotate_stream_ordering, rotate_to_stream_ordering),
	865	)
	866
	867	logger.info("Rotating notifications, deleted %s push actions", txn.rowcount)
	868
	869	txn.execute(
	870	"UPDATE event_push_summary_stream_ordering SET stream_ordering = ?",
	871	(rotate_to_stream_ordering,),
	872	)
	873
	874
	875	def _action_has_highlight(actions):
	876	for action in actions:
	877	try:
	878	if action.get("set_tweak", None) == "highlight":
	879	return action.get("value", True)
	880	except AttributeError:
	881	pass
	882
	883	return False

+1481

-0

synapse/storage/databases/main/events.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2018-2019 New Vector Ltd
	3	# Copyright 2019 The Matrix.org Foundation C.I.C.
	4	#
	5	# Licensed under the Apache License, Version 2.0 (the "License");
	6	# you may not use this file except in compliance with the License.
	7	# You may obtain a copy of the License at
	8	#
	9	# http://www.apache.org/licenses/LICENSE-2.0
	10	#
	11	# Unless required by applicable law or agreed to in writing, software
	12	# distributed under the License is distributed on an "AS IS" BASIS,
	13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	14	# See the License for the specific language governing permissions and
	15	# limitations under the License.
	16	import itertools
	17	import logging
	18	from collections import OrderedDict, namedtuple
	19	from typing import TYPE_CHECKING, Dict, Iterable, List, Tuple
	20
	21	import attr
	22	from prometheus_client import Counter
	23
	24	from twisted.internet import defer
	25
	26	import synapse.metrics
	27	from synapse.api.constants import EventContentFields, EventTypes, RelationTypes
	28	from synapse.api.room_versions import RoomVersions
	29	from synapse.crypto.event_signing import compute_event_reference_hash
	30	from synapse.events import EventBase # noqa: F401
	31	from synapse.events.snapshot import EventContext # noqa: F401
	32	from synapse.logging.utils import log_function
	33	from synapse.storage._base import db_to_json, make_in_list_sql_clause
	34	from synapse.storage.database import DatabasePool, LoggingTransaction
	35	from synapse.storage.databases.main.search import SearchEntry
	36	from synapse.storage.util.id_generators import StreamIdGenerator
	37	from synapse.types import StateMap, get_domain_from_id
	38	from synapse.util.frozenutils import frozendict_json_encoder
	39	from synapse.util.iterutils import batch_iter
	40
	41	if TYPE_CHECKING:
	42	from synapse.server import HomeServer
	43	from synapse.storage.databases.main import DataStore
	44
	45
	46	logger = logging.getLogger(__name__)
	47
	48	persist_event_counter = Counter("synapse_storage_events_persisted_events", "")
	49	event_counter = Counter(
	50	"synapse_storage_events_persisted_events_sep",
	51	"",
	52	["type", "origin_type", "origin_entity"],
	53	)
	54
	55
	56	def encode_json(json_object):
	57	"""
	58	Encode a Python object as JSON and return it in a Unicode string.
	59	"""
	60	out = frozendict_json_encoder.encode(json_object)
	61	if isinstance(out, bytes):
	62	out = out.decode("utf8")
	63	return out
	64
	65
	66	_EventCacheEntry = namedtuple("_EventCacheEntry", ("event", "redacted_event"))
	67
	68
	69	@attr.s(slots=True)
	70	class DeltaState:
	71	"""Deltas to use to update the `current_state_events` table.
	72
	73	Attributes:
	74	to_delete: List of type/state_keys to delete from current state
	75	to_insert: Map of state to upsert into current state
	76	no_longer_in_room: The server is not longer in the room, so the room
	77	should e.g. be removed from `current_state_events` table.
	78	"""
	79
	80	to_delete = attr.ib(type=List[Tuple[str, str]])
	81	to_insert = attr.ib(type=StateMap[str])
	82	no_longer_in_room = attr.ib(type=bool, default=False)
	83
	84
	85	class PersistEventsStore:
	86	"""Contains all the functions for writing events to the database.
	87
	88	Should only be instantiated on one process (when using a worker mode setup).
	89
	90	Note: This is not part of the `DataStore` mixin.
	91	"""
	92
	93	def __init__(
	94	self, hs: "HomeServer", db: DatabasePool, main_data_store: "DataStore"
	95	):
	96	self.hs = hs
	97	self.db_pool = db
	98	self.store = main_data_store
	99	self.database_engine = db.engine
	100	self._clock = hs.get_clock()
	101
	102	self._ephemeral_messages_enabled = hs.config.enable_ephemeral_messages
	103	self.is_mine_id = hs.is_mine_id
	104
	105	# Ideally we'd move these ID gens here, unfortunately some other ID
	106	# generators are chained off them so doing so is a bit of a PITA.
	107	self._backfill_id_gen = self.store._backfill_id_gen # type: StreamIdGenerator
	108	self._stream_id_gen = self.store._stream_id_gen # type: StreamIdGenerator
	109
	110	# This should only exist on instances that are configured to write
	111	assert (
	112	hs.config.worker.writers.events == hs.get_instance_name()
	113	), "Can only instantiate EventsStore on master"
	114
	115	@defer.inlineCallbacks
	116	def _persist_events_and_state_updates(
	117	self,
	118	events_and_contexts: List[Tuple[EventBase, EventContext]],
	119	current_state_for_room: Dict[str, StateMap[str]],
	120	state_delta_for_room: Dict[str, DeltaState],
	121	new_forward_extremeties: Dict[str, List[str]],
	122	backfilled: bool = False,
	123	):
	124	"""Persist a set of events alongside updates to the current state and
	125	forward extremities tables.
	126
	127	Args:
	128	events_and_contexts:
	129	current_state_for_room: Map from room_id to the current state of
	130	the room based on forward extremities
	131	state_delta_for_room: Map from room_id to the delta to apply to
	132	room state
	133	new_forward_extremities: Map from room_id to list of event IDs
	134	that are the new forward extremities of the room.
	135	backfilled
	136
	137	Returns:
	138	Deferred: resolves when the events have been persisted
	139	"""
	140
	141	# We want to calculate the stream orderings as late as possible, as
	142	# we only notify after all events with a lesser stream ordering have
	143	# been persisted. I.e. if we spend 10s inside the with block then
	144	# that will delay all subsequent events from being notified about.
	145	# Hence why we do it down here rather than wrapping the entire
	146	# function.
	147	#
	148	# Its safe to do this after calculating the state deltas etc as we
	149	# only need to protect the persistence of the events. This is to
	150	# ensure that queries of the form "fetch events since X" don't
	151	# return events and stream positions after events that are still in
	152	# flight, as otherwise subsequent requests "fetch event since Y"
	153	# will not return those events.
	154	#
	155	# Note: Multiple instances of this function cannot be in flight at
	156	# the same time for the same room.
	157	if backfilled:
	158	stream_ordering_manager = self._backfill_id_gen.get_next_mult(
	159	len(events_and_contexts)
	160	)
	161	else:
	162	stream_ordering_manager = self._stream_id_gen.get_next_mult(
	163	len(events_and_contexts)
	164	)
	165
	166	with stream_ordering_manager as stream_orderings:
	167	for (event, context), stream in zip(events_and_contexts, stream_orderings):
	168	event.internal_metadata.stream_ordering = stream
	169
	170	yield self.db_pool.runInteraction(
	171	"persist_events",
	172	self._persist_events_txn,
	173	events_and_contexts=events_and_contexts,
	174	backfilled=backfilled,
	175	state_delta_for_room=state_delta_for_room,
	176	new_forward_extremeties=new_forward_extremeties,
	177	)
	178	persist_event_counter.inc(len(events_and_contexts))
	179
	180	if not backfilled:
	181	# backfilled events have negative stream orderings, so we don't
	182	# want to set the event_persisted_position to that.
	183	synapse.metrics.event_persisted_position.set(
	184	events_and_contexts[-1][0].internal_metadata.stream_ordering
	185	)
	186
	187	for event, context in events_and_contexts:
	188	if context.app_service:
	189	origin_type = "local"
	190	origin_entity = context.app_service.id
	191	elif self.hs.is_mine_id(event.sender):
	192	origin_type = "local"
	193	origin_entity = "client"
	194	else:
	195	origin_type = "remote"
	196	origin_entity = get_domain_from_id(event.sender)
	197
	198	event_counter.labels(event.type, origin_type, origin_entity).inc()
	199
	200	for room_id, new_state in current_state_for_room.items():
	201	self.store.get_current_state_ids.prefill((room_id,), new_state)
	202
	203	for room_id, latest_event_ids in new_forward_extremeties.items():
	204	self.store.get_latest_event_ids_in_room.prefill(
	205	(room_id,), list(latest_event_ids)
	206	)
	207
	208	@defer.inlineCallbacks
	209	def _get_events_which_are_prevs(self, event_ids):
	210	"""Filter the supplied list of event_ids to get those which are prev_events of
	211	existing (non-outlier/rejected) events.
	212
	213	Args:
	214	event_ids (Iterable[str]): event ids to filter
	215
	216	Returns:
	217	Deferred[List[str]]: filtered event ids
	218	"""
	219	results = []
	220
	221	def _get_events_which_are_prevs_txn(txn, batch):
	222	sql = """
	223	SELECT prev_event_id, internal_metadata
	224	FROM event_edges
	225	INNER JOIN events USING (event_id)
	226	LEFT JOIN rejections USING (event_id)
	227	LEFT JOIN event_json USING (event_id)
	228	WHERE
	229	NOT events.outlier
	230	AND rejections.event_id IS NULL
	231	AND
	232	"""
	233
	234	clause, args = make_in_list_sql_clause(
	235	self.database_engine, "prev_event_id", batch
	236	)
	237
	238	txn.execute(sql + clause, args)
	239	results.extend(r[0] for r in txn if not db_to_json(r[1]).get("soft_failed"))
	240
	241	for chunk in batch_iter(event_ids, 100):
	242	yield self.db_pool.runInteraction(
	243	"_get_events_which_are_prevs", _get_events_which_are_prevs_txn, chunk
	244	)
	245
	246	return results
	247
	248	@defer.inlineCallbacks
	249	def _get_prevs_before_rejected(self, event_ids):
	250	"""Get soft-failed ancestors to remove from the extremities.
	251
	252	Given a set of events, find all those that have been soft-failed or
	253	rejected. Returns those soft failed/rejected events and their prev
	254	events (whether soft-failed/rejected or not), and recurses up the
	255	prev-event graph until it finds no more soft-failed/rejected events.
	256
	257	This is used to find extremities that are ancestors of new events, but
	258	are separated by soft failed events.
	259
	260	Args:
	261	event_ids (Iterable[str]): Events to find prev events for. Note
	262	that these must have already been persisted.
	263
	264	Returns:
	265	Deferred[set[str]]
	266	"""
	267
	268	# The set of event_ids to return. This includes all soft-failed events
	269	# and their prev events.
	270	existing_prevs = set()
	271
	272	def _get_prevs_before_rejected_txn(txn, batch):
	273	to_recursively_check = batch
	274
	275	while to_recursively_check:
	276	sql = """
	277	SELECT
	278	event_id, prev_event_id, internal_metadata,
	279	rejections.event_id IS NOT NULL
	280	FROM event_edges
	281	INNER JOIN events USING (event_id)
	282	LEFT JOIN rejections USING (event_id)
	283	LEFT JOIN event_json USING (event_id)
	284	WHERE
	285	NOT events.outlier
	286	AND
	287	"""
	288
	289	clause, args = make_in_list_sql_clause(
	290	self.database_engine, "event_id", to_recursively_check
	291	)
	292
	293	txn.execute(sql + clause, args)
	294	to_recursively_check = []
	295
	296	for event_id, prev_event_id, metadata, rejected in txn:
	297	if prev_event_id in existing_prevs:
	298	continue
	299
	300	soft_failed = db_to_json(metadata).get("soft_failed")
	301	if soft_failed or rejected:
	302	to_recursively_check.append(prev_event_id)
	303	existing_prevs.add(prev_event_id)
	304
	305	for chunk in batch_iter(event_ids, 100):
	306	yield self.db_pool.runInteraction(
	307	"_get_prevs_before_rejected", _get_prevs_before_rejected_txn, chunk
	308	)
	309
	310	return existing_prevs
	311
	312	@log_function
	313	def _persist_events_txn(
	314	self,
	315	txn: LoggingTransaction,
	316	events_and_contexts: List[Tuple[EventBase, EventContext]],
	317	backfilled: bool,
	318	state_delta_for_room: Dict[str, DeltaState] = {},
	319	new_forward_extremeties: Dict[str, List[str]] = {},
	320	):
	321	"""Insert some number of room events into the necessary database tables.
	322
	323	Rejected events are only inserted into the events table, the events_json table,
	324	and the rejections table. Things reading from those table will need to check
	325	whether the event was rejected.
	326
	327	Args:
	328	txn
	329	events_and_contexts: events to persist
	330	backfilled: True if the events were backfilled
	331	delete_existing True to purge existing table rows for the events
	332	from the database. This is useful when retrying due to
	333	IntegrityError.
	334	state_delta_for_room: The current-state delta for each room.
	335	new_forward_extremetie: The new forward extremities for each room.
	336	For each room, a list of the event ids which are the forward
	337	extremities.
	338
	339	"""
	340	all_events_and_contexts = events_and_contexts
	341
	342	min_stream_order = events_and_contexts[0][0].internal_metadata.stream_ordering
	343	max_stream_order = events_and_contexts[-1][0].internal_metadata.stream_ordering
	344
	345	self._update_forward_extremities_txn(
	346	txn,
	347	new_forward_extremities=new_forward_extremeties,
	348	max_stream_order=max_stream_order,
	349	)
	350
	351	# Ensure that we don't have the same event twice.
	352	events_and_contexts = self._filter_events_and_contexts_for_duplicates(
	353	events_and_contexts
	354	)
	355
	356	self._update_room_depths_txn(
	357	txn, events_and_contexts=events_and_contexts, backfilled=backfilled
	358	)
	359
	360	# _update_outliers_txn filters out any events which have already been
	361	# persisted, and returns the filtered list.
	362	events_and_contexts = self._update_outliers_txn(
	363	txn, events_and_contexts=events_and_contexts
	364	)
	365
	366	# From this point onwards the events are only events that we haven't
	367	# seen before.
	368
	369	self._store_event_txn(txn, events_and_contexts=events_and_contexts)
	370
	371	# Insert into event_to_state_groups.
	372	self._store_event_state_mappings_txn(txn, events_and_contexts)
	373
	374	# We want to store event_auth mappings for rejected events, as they're
	375	# used in state res v2.
	376	# This is only necessary if the rejected event appears in an accepted
	377	# event's auth chain, but its easier for now just to store them (and
	378	# it doesn't take much storage compared to storing the entire event
	379	# anyway).
	380	self.db_pool.simple_insert_many_txn(
	381	txn,
	382	table="event_auth",
	383	values=[
	384	{
	385	"event_id": event.event_id,
	386	"room_id": event.room_id,
	387	"auth_id": auth_id,
	388	}
	389	for event, _ in events_and_contexts
	390	for auth_id in event.auth_event_ids()
	391	if event.is_state()
	392	],
	393	)
	394
	395	# _store_rejected_events_txn filters out any events which were
	396	# rejected, and returns the filtered list.
	397	events_and_contexts = self._store_rejected_events_txn(
	398	txn, events_and_contexts=events_and_contexts
	399	)
	400
	401	# From this point onwards the events are only ones that weren't
	402	# rejected.
	403
	404	self._update_metadata_tables_txn(
	405	txn,
	406	events_and_contexts=events_and_contexts,
	407	all_events_and_contexts=all_events_and_contexts,
	408	backfilled=backfilled,
	409	)
	410
	411	# We call this last as it assumes we've inserted the events into
	412	# room_memberships, where applicable.
	413	self._update_current_state_txn(txn, state_delta_for_room, min_stream_order)
	414
	415	def _update_current_state_txn(
	416	self,
	417	txn: LoggingTransaction,
	418	state_delta_by_room: Dict[str, DeltaState],
	419	stream_id: int,
	420	):
	421	for room_id, delta_state in state_delta_by_room.items():
	422	to_delete = delta_state.to_delete
	423	to_insert = delta_state.to_insert
	424
	425	if delta_state.no_longer_in_room:
	426	# Server is no longer in the room so we delete the room from
	427	# current_state_events, being careful we've already updated the
	428	# rooms.room_version column (which gets populated in a
	429	# background task).
	430	self._upsert_room_version_txn(txn, room_id)
	431
	432	# Before deleting we populate the current_state_delta_stream
	433	# so that async background tasks get told what happened.
	434	sql = """
	435	INSERT INTO current_state_delta_stream
	436	(stream_id, room_id, type, state_key, event_id, prev_event_id)
	437	SELECT ?, room_id, type, state_key, null, event_id
	438	FROM current_state_events
	439	WHERE room_id = ?
	440	"""
	441	txn.execute(sql, (stream_id, room_id))
	442
	443	self.db_pool.simple_delete_txn(
	444	txn, table="current_state_events", keyvalues={"room_id": room_id},
	445	)
	446	else:
	447	# We're still in the room, so we update the current state as normal.
	448
	449	# First we add entries to the current_state_delta_stream. We
	450	# do this before updating the current_state_events table so
	451	# that we can use it to calculate the `prev_event_id`. (This
	452	# allows us to not have to pull out the existing state
	453	# unnecessarily).
	454	#
	455	# The stream_id for the update is chosen to be the minimum of the stream_ids
	456	# for the batch of the events that we are persisting; that means we do not
	457	# end up in a situation where workers see events before the
	458	# current_state_delta updates.
	459	#
	460	sql = """
	461	INSERT INTO current_state_delta_stream
	462	(stream_id, room_id, type, state_key, event_id, prev_event_id)
	463	SELECT ?, ?, ?, ?, ?, (
	464	SELECT event_id FROM current_state_events
	465	WHERE room_id = ? AND type = ? AND state_key = ?
	466	)
	467	"""
	468	txn.executemany(
	469	sql,
	470	(
	471	(
	472	stream_id,
	473	room_id,
	474	etype,
	475	state_key,
	476	to_insert.get((etype, state_key)),
	477	room_id,
	478	etype,
	479	state_key,
	480	)
	481	for etype, state_key in itertools.chain(to_delete, to_insert)
	482	),
	483	)
	484	# Now we actually update the current_state_events table
	485
	486	txn.executemany(
	487	"DELETE FROM current_state_events"
	488	" WHERE room_id = ? AND type = ? AND state_key = ?",
	489	(
	490	(room_id, etype, state_key)
	491	for etype, state_key in itertools.chain(to_delete, to_insert)
	492	),
	493	)
	494
	495	# We include the membership in the current state table, hence we do
	496	# a lookup when we insert. This assumes that all events have already
	497	# been inserted into room_memberships.
	498	txn.executemany(
	499	"""INSERT INTO current_state_events
	500	(room_id, type, state_key, event_id, membership)
	501	VALUES (?, ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?))
	502	""",
	503	[
	504	(room_id, key[0], key[1], ev_id, ev_id)
	505	for key, ev_id in to_insert.items()
	506	],
	507	)
	508
	509	# We now update `local_current_membership`. We do this regardless
	510	# of whether we're still in the room or not to handle the case where
	511	# e.g. we just got banned (where we need to record that fact here).
	512
	513	# Note: Do we really want to delete rows here (that we do not
	514	# subsequently reinsert below)? While technically correct it means
	515	# we have no record of the fact the user was a member of the
	516	# room but got, say, state reset out of it.
	517	if to_delete or to_insert:
	518	txn.executemany(
	519	"DELETE FROM local_current_membership"
	520	" WHERE room_id = ? AND user_id = ?",
	521	(
	522	(room_id, state_key)
	523	for etype, state_key in itertools.chain(to_delete, to_insert)
	524	if etype == EventTypes.Member and self.is_mine_id(state_key)
	525	),
	526	)
	527
	528	if to_insert:
	529	txn.executemany(
	530	"""INSERT INTO local_current_membership
	531	(room_id, user_id, event_id, membership)
	532	VALUES (?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?))
	533	""",
	534	[
	535	(room_id, key[1], ev_id, ev_id)
	536	for key, ev_id in to_insert.items()
	537	if key[0] == EventTypes.Member and self.is_mine_id(key[1])
	538	],
	539	)
	540
	541	txn.call_after(
	542	self.store._curr_state_delta_stream_cache.entity_has_changed,
	543	room_id,
	544	stream_id,
	545	)
	546
	547	# Invalidate the various caches
	548
	549	# Figure out the changes of membership to invalidate the
	550	# `get_rooms_for_user` cache.
	551	# We find out which membership events we may have deleted
	552	# and which we have added, then we invlidate the caches for all
	553	# those users.
	554	members_changed = {
	555	state_key
	556	for ev_type, state_key in itertools.chain(to_delete, to_insert)
	557	if ev_type == EventTypes.Member
	558	}
	559
	560	for member in members_changed:
	561	txn.call_after(
	562	self.store.get_rooms_for_user_with_stream_ordering.invalidate,
	563	(member,),
	564	)
	565
	566	self.store._invalidate_state_caches_and_stream(
	567	txn, room_id, members_changed
	568	)
	569
	570	def _upsert_room_version_txn(self, txn: LoggingTransaction, room_id: str):
	571	"""Update the room version in the database based off current state
	572	events.
	573
	574	This is used when we're about to delete current state and we want to
	575	ensure that the `rooms.room_version` column is up to date.
	576	"""
	577
	578	sql = """
	579	SELECT json FROM event_json
	580	INNER JOIN current_state_events USING (room_id, event_id)
	581	WHERE room_id = ? AND type = ? AND state_key = ?
	582	"""
	583	txn.execute(sql, (room_id, EventTypes.Create, ""))
	584	row = txn.fetchone()
	585	if row:
	586	event_json = db_to_json(row[0])
	587	content = event_json.get("content", {})
	588	creator = content.get("creator")
	589	room_version_id = content.get("room_version", RoomVersions.V1.identifier)
	590
	591	self.db_pool.simple_upsert_txn(
	592	txn,
	593	table="rooms",
	594	keyvalues={"room_id": room_id},
	595	values={"room_version": room_version_id},
	596	insertion_values={"is_public": False, "creator": creator},
	597	)
	598
	599	def _update_forward_extremities_txn(
	600	self, txn, new_forward_extremities, max_stream_order
	601	):
	602	for room_id, new_extrem in new_forward_extremities.items():
	603	self.db_pool.simple_delete_txn(
	604	txn, table="event_forward_extremities", keyvalues={"room_id": room_id}
	605	)
	606	txn.call_after(
	607	self.store.get_latest_event_ids_in_room.invalidate, (room_id,)
	608	)
	609
	610	self.db_pool.simple_insert_many_txn(
	611	txn,
	612	table="event_forward_extremities",
	613	values=[
	614	{"event_id": ev_id, "room_id": room_id}
	615	for room_id, new_extrem in new_forward_extremities.items()
	616	for ev_id in new_extrem
	617	],
	618	)
	619	# We now insert into stream_ordering_to_exterm a mapping from room_id,
	620	# new stream_ordering to new forward extremeties in the room.
	621	# This allows us to later efficiently look up the forward extremeties
	622	# for a room before a given stream_ordering
	623	self.db_pool.simple_insert_many_txn(
	624	txn,
	625	table="stream_ordering_to_exterm",
	626	values=[
	627	{
	628	"room_id": room_id,
	629	"event_id": event_id,
	630	"stream_ordering": max_stream_order,
	631	}
	632	for room_id, new_extrem in new_forward_extremities.items()
	633	for event_id in new_extrem
	634	],
	635	)
	636
	637	@classmethod
	638	def _filter_events_and_contexts_for_duplicates(cls, events_and_contexts):
	639	"""Ensure that we don't have the same event twice.
	640
	641	Pick the earliest non-outlier if there is one, else the earliest one.
	642
	643	Args:
	644	events_and_contexts (list[(EventBase, EventContext)]):
	645	Returns:
	646	list[(EventBase, EventContext)]: filtered list
	647	"""
	648	new_events_and_contexts = OrderedDict()
	649	for event, context in events_and_contexts:
	650	prev_event_context = new_events_and_contexts.get(event.event_id)
	651	if prev_event_context:
	652	if not event.internal_metadata.is_outlier():
	653	if prev_event_context[0].internal_metadata.is_outlier():
	654	# To ensure correct ordering we pop, as OrderedDict is
	655	# ordered by first insertion.
	656	new_events_and_contexts.pop(event.event_id, None)
	657	new_events_and_contexts[event.event_id] = (event, context)
	658	else:
	659	new_events_and_contexts[event.event_id] = (event, context)
	660	return list(new_events_and_contexts.values())
	661
	662	def _update_room_depths_txn(self, txn, events_and_contexts, backfilled):
	663	"""Update min_depth for each room
	664
	665	Args:
	666	txn (twisted.enterprise.adbapi.Connection): db connection
	667	events_and_contexts (list[(EventBase, EventContext)]): events
	668	we are persisting
	669	backfilled (bool): True if the events were backfilled
	670	"""
	671	depth_updates = {}
	672	for event, context in events_and_contexts:
	673	# Remove the any existing cache entries for the event_ids
	674	txn.call_after(self.store._invalidate_get_event_cache, event.event_id)
	675	if not backfilled:
	676	txn.call_after(
	677	self.store._events_stream_cache.entity_has_changed,
	678	event.room_id,
	679	event.internal_metadata.stream_ordering,
	680	)
	681
	682	if not event.internal_metadata.is_outlier() and not context.rejected:
	683	depth_updates[event.room_id] = max(
	684	event.depth, depth_updates.get(event.room_id, event.depth)
	685	)
	686
	687	for room_id, depth in depth_updates.items():
	688	self._update_min_depth_for_room_txn(txn, room_id, depth)
	689
	690	def _update_outliers_txn(self, txn, events_and_contexts):
	691	"""Update any outliers with new event info.
	692
	693	This turns outliers into ex-outliers (unless the new event was
	694	rejected).
	695
	696	Args:
	697	txn (twisted.enterprise.adbapi.Connection): db connection
	698	events_and_contexts (list[(EventBase, EventContext)]): events
	699	we are persisting
	700
	701	Returns:
	702	list[(EventBase, EventContext)] new list, without events which
	703	are already in the events table.
	704	"""
	705	txn.execute(
	706	"SELECT event_id, outlier FROM events WHERE event_id in (%s)"
	707	% (",".join(["?"] * len(events_and_contexts)),),
	708	[event.event_id for event, _ in events_and_contexts],
	709	)
	710
	711	have_persisted = {event_id: outlier for event_id, outlier in txn}
	712
	713	to_remove = set()
	714	for event, context in events_and_contexts:
	715	if event.event_id not in have_persisted:
	716	continue
	717
	718	to_remove.add(event)
	719
	720	if context.rejected:
	721	# If the event is rejected then we don't care if the event
	722	# was an outlier or not.
	723	continue
	724
	725	outlier_persisted = have_persisted[event.event_id]
	726	if not event.internal_metadata.is_outlier() and outlier_persisted:
	727	# We received a copy of an event that we had already stored as
	728	# an outlier in the database. We now have some state at that
	729	# so we need to update the state_groups table with that state.
	730
	731	# insert into event_to_state_groups.
	732	try:
	733	self._store_event_state_mappings_txn(txn, ((event, context),))
	734	except Exception:
	735	logger.exception("")
	736	raise
	737
	738	metadata_json = encode_json(event.internal_metadata.get_dict())
	739
	740	sql = "UPDATE event_json SET internal_metadata = ? WHERE event_id = ?"
	741	txn.execute(sql, (metadata_json, event.event_id))
	742
	743	# Add an entry to the ex_outlier_stream table to replicate the
	744	# change in outlier status to our workers.
	745	stream_order = event.internal_metadata.stream_ordering
	746	state_group_id = context.state_group
	747	self.db_pool.simple_insert_txn(
	748	txn,
	749	table="ex_outlier_stream",
	750	values={
	751	"event_stream_ordering": stream_order,
	752	"event_id": event.event_id,
	753	"state_group": state_group_id,
	754	},
	755	)
	756
	757	sql = "UPDATE events SET outlier = ? WHERE event_id = ?"
	758	txn.execute(sql, (False, event.event_id))
	759
	760	# Update the event_backward_extremities table now that this
	761	# event isn't an outlier any more.
	762	self._update_backward_extremeties(txn, [event])
	763
	764	return [ec for ec in events_and_contexts if ec[0] not in to_remove]
	765
	766	def _store_event_txn(self, txn, events_and_contexts):
	767	"""Insert new events into the event and event_json tables
	768
	769	Args:
	770	txn (twisted.enterprise.adbapi.Connection): db connection
	771	events_and_contexts (list[(EventBase, EventContext)]): events
	772	we are persisting
	773	"""
	774
	775	if not events_and_contexts:
	776	# nothing to do here
	777	return
	778
	779	def event_dict(event):
	780	d = event.get_dict()
	781	d.pop("redacted", None)
	782	d.pop("redacted_because", None)
	783	return d
	784
	785	self.db_pool.simple_insert_many_txn(
	786	txn,
	787	table="event_json",
	788	values=[
	789	{
	790	"event_id": event.event_id,
	791	"room_id": event.room_id,
	792	"internal_metadata": encode_json(
	793	event.internal_metadata.get_dict()
	794	),
	795	"json": encode_json(event_dict(event)),
	796	"format_version": event.format_version,
	797	}
	798	for event, _ in events_and_contexts
	799	],
	800	)
	801
	802	self.db_pool.simple_insert_many_txn(
	803	txn,
	804	table="events",
	805	values=[
	806	{
	807	"stream_ordering": event.internal_metadata.stream_ordering,
	808	"topological_ordering": event.depth,
	809	"depth": event.depth,
	810	"event_id": event.event_id,
	811	"room_id": event.room_id,
	812	"type": event.type,
	813	"processed": True,
	814	"outlier": event.internal_metadata.is_outlier(),
	815	"origin_server_ts": int(event.origin_server_ts),
	816	"received_ts": self._clock.time_msec(),
	817	"sender": event.sender,
	818	"contains_url": (
	819	"url" in event.content and isinstance(event.content["url"], str)
	820	),
	821	}
	822	for event, _ in events_and_contexts
	823	],
	824	)
	825
	826	for event, _ in events_and_contexts:
	827	if not event.internal_metadata.is_redacted():
	828	# If we're persisting an unredacted event we go and ensure
	829	# that we mark any redactions that reference this event as
	830	# requiring censoring.
	831	self.db_pool.simple_update_txn(
	832	txn,
	833	table="redactions",
	834	keyvalues={"redacts": event.event_id},
	835	updatevalues={"have_censored": False},
	836	)
	837
	838	def _store_rejected_events_txn(self, txn, events_and_contexts):
	839	"""Add rows to the 'rejections' table for received events which were
	840	rejected
	841
	842	Args:
	843	txn (twisted.enterprise.adbapi.Connection): db connection
	844	events_and_contexts (list[(EventBase, EventContext)]): events
	845	we are persisting
	846
	847	Returns:
	848	list[(EventBase, EventContext)] new list, without the rejected
	849	events.
	850	"""
	851	# Remove the rejected events from the list now that we've added them
	852	# to the events table and the events_json table.
	853	to_remove = set()
	854	for event, context in events_and_contexts:
	855	if context.rejected:
	856	# Insert the event_id into the rejections table
	857	self._store_rejections_txn(txn, event.event_id, context.rejected)
	858	to_remove.add(event)
	859
	860	return [ec for ec in events_and_contexts if ec[0] not in to_remove]
	861
	862	def _update_metadata_tables_txn(
	863	self, txn, events_and_contexts, all_events_and_contexts, backfilled
	864	):
	865	"""Update all the miscellaneous tables for new events
	866
	867	Args:
	868	txn (twisted.enterprise.adbapi.Connection): db connection
	869	events_and_contexts (list[(EventBase, EventContext)]): events
	870	we are persisting
	871	all_events_and_contexts (list[(EventBase, EventContext)]): all
	872	events that we were going to persist. This includes events
	873	we've already persisted, etc, that wouldn't appear in
	874	events_and_context.
	875	backfilled (bool): True if the events were backfilled
	876	"""
	877
	878	# Insert all the push actions into the event_push_actions table.
	879	self._set_push_actions_for_event_and_users_txn(
	880	txn,
	881	events_and_contexts=events_and_contexts,
	882	all_events_and_contexts=all_events_and_contexts,
	883	)
	884
	885	if not events_and_contexts:
	886	# nothing to do here
	887	return
	888
	889	for event, context in events_and_contexts:
	890	if event.type == EventTypes.Redaction and event.redacts is not None:
	891	# Remove the entries in the event_push_actions table for the
	892	# redacted event.
	893	self._remove_push_actions_for_event_id_txn(
	894	txn, event.room_id, event.redacts
	895	)
	896
	897	# Remove from relations table.
	898	self._handle_redaction(txn, event.redacts)
	899
	900	# Update the event_forward_extremities, event_backward_extremities and
	901	# event_edges tables.
	902	self._handle_mult_prev_events(
	903	txn, events=[event for event, _ in events_and_contexts]
	904	)
	905
	906	for event, _ in events_and_contexts:
	907	if event.type == EventTypes.Name:
	908	# Insert into the event_search table.
	909	self._store_room_name_txn(txn, event)
	910	elif event.type == EventTypes.Topic:
	911	# Insert into the event_search table.
	912	self._store_room_topic_txn(txn, event)
	913	elif event.type == EventTypes.Message:
	914	# Insert into the event_search table.
	915	self._store_room_message_txn(txn, event)
	916	elif event.type == EventTypes.Redaction and event.redacts is not None:
	917	# Insert into the redactions table.
	918	self._store_redaction(txn, event)
	919	elif event.type == EventTypes.Retention:
	920	# Update the room_retention table.
	921	self._store_retention_policy_for_room_txn(txn, event)
	922
	923	self._handle_event_relations(txn, event)
	924
	925	# Store the labels for this event.
	926	labels = event.content.get(EventContentFields.LABELS)
	927	if labels:
	928	self.insert_labels_for_event_txn(
	929	txn, event.event_id, labels, event.room_id, event.depth
	930	)
	931
	932	if self._ephemeral_messages_enabled:
	933	# If there's an expiry timestamp on the event, store it.
	934	expiry_ts = event.content.get(EventContentFields.SELF_DESTRUCT_AFTER)
	935	if isinstance(expiry_ts, int) and not event.is_state():
	936	self._insert_event_expiry_txn(txn, event.event_id, expiry_ts)
	937
	938	# Insert into the room_memberships table.
	939	self._store_room_members_txn(
	940	txn,
	941	[
	942	event
	943	for event, _ in events_and_contexts
	944	if event.type == EventTypes.Member
	945	],
	946	backfilled=backfilled,
	947	)
	948
	949	# Insert event_reference_hashes table.
	950	self._store_event_reference_hashes_txn(
	951	txn, [event for event, _ in events_and_contexts]
	952	)
	953
	954	state_events_and_contexts = [
	955	ec for ec in events_and_contexts if ec[0].is_state()
	956	]
	957
	958	state_values = []
	959	for event, context in state_events_and_contexts:
	960	vals = {
	961	"event_id": event.event_id,
	962	"room_id": event.room_id,
	963	"type": event.type,
	964	"state_key": event.state_key,
	965	}
	966
	967	# TODO: How does this work with backfilling?
	968	if hasattr(event, "replaces_state"):
	969	vals["prev_state"] = event.replaces_state
	970
	971	state_values.append(vals)
	972
	973	self.db_pool.simple_insert_many_txn(
	974	txn, table="state_events", values=state_values
	975	)
	976
	977	# Prefill the event cache
	978	self._add_to_cache(txn, events_and_contexts)
	979
	980	def _add_to_cache(self, txn, events_and_contexts):
	981	to_prefill = []
	982
	983	rows = []
	984	N = 200
	985	for i in range(0, len(events_and_contexts), N):
	986	ev_map = {e[0].event_id: e[0] for e in events_and_contexts[i : i + N]}
	987	if not ev_map:
	988	break
	989
	990	sql = (
	991	"SELECT "
	992	" e.event_id as event_id, "
	993	" r.redacts as redacts,"
	994	" rej.event_id as rejects "
	995	" FROM events as e"
	996	" LEFT JOIN rejections as rej USING (event_id)"
	997	" LEFT JOIN redactions as r ON e.event_id = r.redacts"
	998	" WHERE "
	999	)
	1000
	1001	clause, args = make_in_list_sql_clause(
	1002	self.database_engine, "e.event_id", list(ev_map)
	1003	)
	1004
	1005	txn.execute(sql + clause, args)
	1006	rows = self.db_pool.cursor_to_dict(txn)
	1007	for row in rows:
	1008	event = ev_map[row["event_id"]]
	1009	if not row["rejects"] and not row["redacts"]:
	1010	to_prefill.append(
	1011	_EventCacheEntry(event=event, redacted_event=None)
	1012	)
	1013
	1014	def prefill():
	1015	for cache_entry in to_prefill:
	1016	self.store._get_event_cache.prefill(
	1017	(cache_entry[0].event_id,), cache_entry
	1018	)
	1019
	1020	txn.call_after(prefill)
	1021
	1022	def _store_redaction(self, txn, event):
	1023	# invalidate the cache for the redacted event
	1024	txn.call_after(self.store._invalidate_get_event_cache, event.redacts)
	1025
	1026	self.db_pool.simple_insert_txn(
	1027	txn,
	1028	table="redactions",
	1029	values={
	1030	"event_id": event.event_id,
	1031	"redacts": event.redacts,
	1032	"received_ts": self._clock.time_msec(),
	1033	},
	1034	)
	1035
	1036	def insert_labels_for_event_txn(
	1037	self, txn, event_id, labels, room_id, topological_ordering
	1038	):
	1039	"""Store the mapping between an event's ID and its labels, with one row per
	1040	(event_id, label) tuple.
	1041
	1042	Args:
	1043	txn (LoggingTransaction): The transaction to execute.
	1044	event_id (str): The event's ID.
	1045	labels (list[str]): A list of text labels.
	1046	room_id (str): The ID of the room the event was sent to.
	1047	topological_ordering (int): The position of the event in the room's topology.
	1048	"""
	1049	return self.db_pool.simple_insert_many_txn(
	1050	txn=txn,
	1051	table="event_labels",
	1052	values=[
	1053	{
	1054	"event_id": event_id,
	1055	"label": label,
	1056	"room_id": room_id,
	1057	"topological_ordering": topological_ordering,
	1058	}
	1059	for label in labels
	1060	],
	1061	)
	1062
	1063	def _insert_event_expiry_txn(self, txn, event_id, expiry_ts):
	1064	"""Save the expiry timestamp associated with a given event ID.
	1065
	1066	Args:
	1067	txn (LoggingTransaction): The database transaction to use.
	1068	event_id (str): The event ID the expiry timestamp is associated with.
	1069	expiry_ts (int): The timestamp at which to expire (delete) the event.
	1070	"""
	1071	return self.db_pool.simple_insert_txn(
	1072	txn=txn,
	1073	table="event_expiry",
	1074	values={"event_id": event_id, "expiry_ts": expiry_ts},
	1075	)
	1076
	1077	def _store_event_reference_hashes_txn(self, txn, events):
	1078	"""Store a hash for a PDU
	1079	Args:
	1080	txn (cursor):
	1081	events (list): list of Events.
	1082	"""
	1083
	1084	vals = []
	1085	for event in events:
	1086	ref_alg, ref_hash_bytes = compute_event_reference_hash(event)
	1087	vals.append(
	1088	{
	1089	"event_id": event.event_id,
	1090	"algorithm": ref_alg,
	1091	"hash": memoryview(ref_hash_bytes),
	1092	}
	1093	)
	1094
	1095	self.db_pool.simple_insert_many_txn(
	1096	txn, table="event_reference_hashes", values=vals
	1097	)
	1098
	1099	def _store_room_members_txn(self, txn, events, backfilled):
	1100	"""Store a room member in the database.
	1101	"""
	1102	self.db_pool.simple_insert_many_txn(
	1103	txn,
	1104	table="room_memberships",
	1105	values=[
	1106	{
	1107	"event_id": event.event_id,
	1108	"user_id": event.state_key,
	1109	"sender": event.user_id,
	1110	"room_id": event.room_id,
	1111	"membership": event.membership,
	1112	"display_name": event.content.get("displayname", None),
	1113	"avatar_url": event.content.get("avatar_url", None),
	1114	}
	1115	for event in events
	1116	],
	1117	)
	1118
	1119	for event in events:
	1120	txn.call_after(
	1121	self.store._membership_stream_cache.entity_has_changed,
	1122	event.state_key,
	1123	event.internal_metadata.stream_ordering,
	1124	)
	1125	txn.call_after(
	1126	self.store.get_invited_rooms_for_local_user.invalidate,
	1127	(event.state_key,),
	1128	)
	1129
	1130	# We update the local_current_membership table only if the event is
	1131	# "current", i.e., its something that has just happened.
	1132	#
	1133	# This will usually get updated by the `current_state_events` handling,
	1134	# unless its an outlier, and an outlier is only "current" if it's an "out of
	1135	# band membership", like a remote invite or a rejection of a remote invite.
	1136	if (
	1137	self.is_mine_id(event.state_key)
	1138	and not backfilled
	1139	and event.internal_metadata.is_outlier()
	1140	and event.internal_metadata.is_out_of_band_membership()
	1141	):
	1142	self.db_pool.simple_upsert_txn(
	1143	txn,
	1144	table="local_current_membership",
	1145	keyvalues={"room_id": event.room_id, "user_id": event.state_key},
	1146	values={
	1147	"event_id": event.event_id,
	1148	"membership": event.membership,
	1149	},
	1150	)
	1151
	1152	def _handle_event_relations(self, txn, event):
	1153	"""Handles inserting relation data during peristence of events
	1154
	1155	Args:
	1156	txn
	1157	event (EventBase)
	1158	"""
	1159	relation = event.content.get("m.relates_to")
	1160	if not relation:
	1161	# No relations
	1162	return
	1163
	1164	rel_type = relation.get("rel_type")
	1165	if rel_type not in (
	1166	RelationTypes.ANNOTATION,
	1167	RelationTypes.REFERENCE,
	1168	RelationTypes.REPLACE,
	1169	):
	1170	# Unknown relation type
	1171	return
	1172
	1173	parent_id = relation.get("event_id")
	1174	if not parent_id:
	1175	# Invalid relation
	1176	return
	1177
	1178	aggregation_key = relation.get("key")
	1179
	1180	self.db_pool.simple_insert_txn(
	1181	txn,
	1182	table="event_relations",
	1183	values={
	1184	"event_id": event.event_id,
	1185	"relates_to_id": parent_id,
	1186	"relation_type": rel_type,
	1187	"aggregation_key": aggregation_key,
	1188	},
	1189	)
	1190
	1191	txn.call_after(self.store.get_relations_for_event.invalidate_many, (parent_id,))
	1192	txn.call_after(
	1193	self.store.get_aggregation_groups_for_event.invalidate_many, (parent_id,)
	1194	)
	1195
	1196	if rel_type == RelationTypes.REPLACE:
	1197	txn.call_after(self.store.get_applicable_edit.invalidate, (parent_id,))
	1198
	1199	def _handle_redaction(self, txn, redacted_event_id):
	1200	"""Handles receiving a redaction and checking whether we need to remove
	1201	any redacted relations from the database.
	1202
	1203	Args:
	1204	txn
	1205	redacted_event_id (str): The event that was redacted.
	1206	"""
	1207
	1208	self.db_pool.simple_delete_txn(
	1209	txn, table="event_relations", keyvalues={"event_id": redacted_event_id}
	1210	)
	1211
	1212	def _store_room_topic_txn(self, txn, event):
	1213	if hasattr(event, "content") and "topic" in event.content:
	1214	self.store_event_search_txn(
	1215	txn, event, "content.topic", event.content["topic"]
	1216	)
	1217
	1218	def _store_room_name_txn(self, txn, event):
	1219	if hasattr(event, "content") and "name" in event.content:
	1220	self.store_event_search_txn(
	1221	txn, event, "content.name", event.content["name"]
	1222	)
	1223
	1224	def _store_room_message_txn(self, txn, event):
	1225	if hasattr(event, "content") and "body" in event.content:
	1226	self.store_event_search_txn(
	1227	txn, event, "content.body", event.content["body"]
	1228	)
	1229
	1230	def _store_retention_policy_for_room_txn(self, txn, event):
	1231	if hasattr(event, "content") and (
	1232	"min_lifetime" in event.content or "max_lifetime" in event.content
	1233	):
	1234	if (
	1235	"min_lifetime" in event.content
	1236	and not isinstance(event.content.get("min_lifetime"), int)
	1237	) or (
	1238	"max_lifetime" in event.content
	1239	and not isinstance(event.content.get("max_lifetime"), int)
	1240	):
	1241	# Ignore the event if one of the value isn't an integer.
	1242	return
	1243
	1244	self.db_pool.simple_insert_txn(
	1245	txn=txn,
	1246	table="room_retention",
	1247	values={
	1248	"room_id": event.room_id,
	1249	"event_id": event.event_id,
	1250	"min_lifetime": event.content.get("min_lifetime"),
	1251	"max_lifetime": event.content.get("max_lifetime"),
	1252	},
	1253	)
	1254
	1255	self.store._invalidate_cache_and_stream(
	1256	txn, self.store.get_retention_policy_for_room, (event.room_id,)
	1257	)
	1258
	1259	def store_event_search_txn(self, txn, event, key, value):
	1260	"""Add event to the search table
	1261
	1262	Args:
	1263	txn (cursor):
	1264	event (EventBase):
	1265	key (str):
	1266	value (str):
	1267	"""
	1268	self.store.store_search_entries_txn(
	1269	txn,
	1270	(
	1271	SearchEntry(
	1272	key=key,
	1273	value=value,
	1274	event_id=event.event_id,
	1275	room_id=event.room_id,
	1276	stream_ordering=event.internal_metadata.stream_ordering,
	1277	origin_server_ts=event.origin_server_ts,
	1278	),
	1279	),
	1280	)
	1281
	1282	def _set_push_actions_for_event_and_users_txn(
	1283	self, txn, events_and_contexts, all_events_and_contexts
	1284	):
	1285	"""Handles moving push actions from staging table to main
	1286	event_push_actions table for all events in `events_and_contexts`.
	1287
	1288	Also ensures that all events in `all_events_and_contexts` are removed
	1289	from the push action staging area.
	1290
	1291	Args:
	1292	events_and_contexts (list[(EventBase, EventContext)]): events
	1293	we are persisting
	1294	all_events_and_contexts (list[(EventBase, EventContext)]): all
	1295	events that we were going to persist. This includes events
	1296	we've already persisted, etc, that wouldn't appear in
	1297	events_and_context.
	1298	"""
	1299
	1300	sql = """
	1301	INSERT INTO event_push_actions (
	1302	room_id, event_id, user_id, actions, stream_ordering,
	1303	topological_ordering, notif, highlight
	1304	)
	1305	SELECT ?, event_id, user_id, actions, ?, ?, notif, highlight
	1306	FROM event_push_actions_staging
	1307	WHERE event_id = ?
	1308	"""
	1309
	1310	if events_and_contexts:
	1311	txn.executemany(
	1312	sql,
	1313	(
	1314	(
	1315	event.room_id,
	1316	event.internal_metadata.stream_ordering,
	1317	event.depth,
	1318	event.event_id,
	1319	)
	1320	for event, _ in events_and_contexts
	1321	),
	1322	)
	1323
	1324	for event, _ in events_and_contexts:
	1325	user_ids = self.db_pool.simple_select_onecol_txn(
	1326	txn,
	1327	table="event_push_actions_staging",
	1328	keyvalues={"event_id": event.event_id},
	1329	retcol="user_id",
	1330	)
	1331
	1332	for uid in user_ids:
	1333	txn.call_after(
	1334	self.store.get_unread_event_push_actions_by_room_for_user.invalidate_many,
	1335	(event.room_id, uid),
	1336	)
	1337
	1338	# Now we delete the staging area for all events that were being
	1339	# persisted.
	1340	txn.executemany(
	1341	"DELETE FROM event_push_actions_staging WHERE event_id = ?",
	1342	((event.event_id,) for event, _ in all_events_and_contexts),
	1343	)
	1344
	1345	def _remove_push_actions_for_event_id_txn(self, txn, room_id, event_id):
	1346	# Sad that we have to blow away the cache for the whole room here
	1347	txn.call_after(
	1348	self.store.get_unread_event_push_actions_by_room_for_user.invalidate_many,
	1349	(room_id,),
	1350	)
	1351	txn.execute(
	1352	"DELETE FROM event_push_actions WHERE room_id = ? AND event_id = ?",
	1353	(room_id, event_id),
	1354	)
	1355
	1356	def _store_rejections_txn(self, txn, event_id, reason):
	1357	self.db_pool.simple_insert_txn(
	1358	txn,
	1359	table="rejections",
	1360	values={
	1361	"event_id": event_id,
	1362	"reason": reason,
	1363	"last_check": self._clock.time_msec(),
	1364	},
	1365	)
	1366
	1367	def _store_event_state_mappings_txn(
	1368	self, txn, events_and_contexts: Iterable[Tuple[EventBase, EventContext]]
	1369	):
	1370	state_groups = {}
	1371	for event, context in events_and_contexts:
	1372	if event.internal_metadata.is_outlier():
	1373	continue
	1374
	1375	# if the event was rejected, just give it the same state as its
	1376	# predecessor.
	1377	if context.rejected:
	1378	state_groups[event.event_id] = context.state_group_before_event
	1379	continue
	1380
	1381	state_groups[event.event_id] = context.state_group
	1382
	1383	self.db_pool.simple_insert_many_txn(
	1384	txn,
	1385	table="event_to_state_groups",
	1386	values=[
	1387	{"state_group": state_group_id, "event_id": event_id}
	1388	for event_id, state_group_id in state_groups.items()
	1389	],
	1390	)
	1391
	1392	for event_id, state_group_id in state_groups.items():
	1393	txn.call_after(
	1394	self.store._get_state_group_for_event.prefill,
	1395	(event_id,),
	1396	state_group_id,
	1397	)
	1398
	1399	def _update_min_depth_for_room_txn(self, txn, room_id, depth):
	1400	min_depth = self.store._get_min_depth_interaction(txn, room_id)
	1401
	1402	if min_depth is not None and depth >= min_depth:
	1403	return
	1404
	1405	self.db_pool.simple_upsert_txn(
	1406	txn,
	1407	table="room_depth",
	1408	keyvalues={"room_id": room_id},
	1409	values={"min_depth": depth},
	1410	)
	1411
	1412	def _handle_mult_prev_events(self, txn, events):
	1413	"""
	1414	For the given event, update the event edges table and forward and
	1415	backward extremities tables.
	1416	"""
	1417	self.db_pool.simple_insert_many_txn(
	1418	txn,
	1419	table="event_edges",
	1420	values=[
	1421	{
	1422	"event_id": ev.event_id,
	1423	"prev_event_id": e_id,
	1424	"room_id": ev.room_id,
	1425	"is_state": False,
	1426	}
	1427	for ev in events
	1428	for e_id in ev.prev_event_ids()
	1429	],
	1430	)
	1431
	1432	self._update_backward_extremeties(txn, events)
	1433
	1434	def _update_backward_extremeties(self, txn, events):
	1435	"""Updates the event_backward_extremities tables based on the new/updated
	1436	events being persisted.
	1437
	1438	This is called for new events and for events that were outliers, but
	1439	are now being persisted as non-outliers.
	1440
	1441	Forward extremities are handled when we first start persisting the events.
	1442	"""
	1443	events_by_room = {}
	1444	for ev in events:
	1445	events_by_room.setdefault(ev.room_id, []).append(ev)
	1446
	1447	query = (
	1448	"INSERT INTO event_backward_extremities (event_id, room_id)"
	1449	" SELECT ?, ? WHERE NOT EXISTS ("
	1450	" SELECT 1 FROM event_backward_extremities"
	1451	" WHERE event_id = ? AND room_id = ?"
	1452	" )"
	1453	" AND NOT EXISTS ("
	1454	" SELECT 1 FROM events WHERE event_id = ? AND room_id = ? "
	1455	" AND outlier = ?"
	1456	" )"
	1457	)
	1458
	1459	txn.executemany(
	1460	query,
	1461	[
	1462	(e_id, ev.room_id, e_id, ev.room_id, e_id, ev.room_id, False)
	1463	for ev in events
	1464	for e_id in ev.prev_event_ids()
	1465	if not ev.internal_metadata.is_outlier()
	1466	],
	1467	)
	1468
	1469	query = (
	1470	"DELETE FROM event_backward_extremities"
	1471	" WHERE event_id = ? AND room_id = ?"
	1472	)
	1473	txn.executemany(
	1474	query,
	1475	[
	1476	(ev.event_id, ev.room_id)
	1477	for ev in events
	1478	if not ev.internal_metadata.is_outlier()
	1479	],
	1480	)

+585

-0

synapse/storage/databases/main/events_bg_updates.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2019 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16
	17	from twisted.internet import defer
	18
	19	from synapse.api.constants import EventContentFields
	20	from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
	21	from synapse.storage.database import DatabasePool
	22
	23	logger = logging.getLogger(__name__)
	24
	25
	26	class EventsBackgroundUpdatesStore(SQLBaseStore):
	27
	28	EVENT_ORIGIN_SERVER_TS_NAME = "event_origin_server_ts"
	29	EVENT_FIELDS_SENDER_URL_UPDATE_NAME = "event_fields_sender_url"
	30	DELETE_SOFT_FAILED_EXTREMITIES = "delete_soft_failed_extremities"
	31
	32	def __init__(self, database: DatabasePool, db_conn, hs):
	33	super(EventsBackgroundUpdatesStore, self).__init__(database, db_conn, hs)
	34
	35	self.db_pool.updates.register_background_update_handler(
	36	self.EVENT_ORIGIN_SERVER_TS_NAME, self._background_reindex_origin_server_ts
	37	)
	38	self.db_pool.updates.register_background_update_handler(
	39	self.EVENT_FIELDS_SENDER_URL_UPDATE_NAME,
	40	self._background_reindex_fields_sender,
	41	)
	42
	43	self.db_pool.updates.register_background_index_update(
	44	"event_contains_url_index",
	45	index_name="event_contains_url_index",
	46	table="events",
	47	columns=["room_id", "topological_ordering", "stream_ordering"],
	48	where_clause="contains_url = true AND outlier = false",
	49	)
	50
	51	# an event_id index on event_search is useful for the purge_history
	52	# api. Plus it means we get to enforce some integrity with a UNIQUE
	53	# clause
	54	self.db_pool.updates.register_background_index_update(
	55	"event_search_event_id_idx",
	56	index_name="event_search_event_id_idx",
	57	table="event_search",
	58	columns=["event_id"],
	59	unique=True,
	60	psql_only=True,
	61	)
	62
	63	self.db_pool.updates.register_background_update_handler(
	64	self.DELETE_SOFT_FAILED_EXTREMITIES, self._cleanup_extremities_bg_update
	65	)
	66
	67	self.db_pool.updates.register_background_update_handler(
	68	"redactions_received_ts", self._redactions_received_ts
	69	)
	70
	71	# This index gets deleted in `event_fix_redactions_bytes` update
	72	self.db_pool.updates.register_background_index_update(
	73	"event_fix_redactions_bytes_create_index",
	74	index_name="redactions_censored_redacts",
	75	table="redactions",
	76	columns=["redacts"],
	77	where_clause="have_censored",
	78	)
	79
	80	self.db_pool.updates.register_background_update_handler(
	81	"event_fix_redactions_bytes", self._event_fix_redactions_bytes
	82	)
	83
	84	self.db_pool.updates.register_background_update_handler(
	85	"event_store_labels", self._event_store_labels
	86	)
	87
	88	self.db_pool.updates.register_background_index_update(
	89	"redactions_have_censored_ts_idx",
	90	index_name="redactions_have_censored_ts",
	91	table="redactions",
	92	columns=["received_ts"],
	93	where_clause="NOT have_censored",
	94	)
	95
	96	@defer.inlineCallbacks
	97	def _background_reindex_fields_sender(self, progress, batch_size):
	98	target_min_stream_id = progress["target_min_stream_id_inclusive"]
	99	max_stream_id = progress["max_stream_id_exclusive"]
	100	rows_inserted = progress.get("rows_inserted", 0)
	101
	102	INSERT_CLUMP_SIZE = 1000
	103
	104	def reindex_txn(txn):
	105	sql = (
	106	"SELECT stream_ordering, event_id, json FROM events"
	107	" INNER JOIN event_json USING (event_id)"
	108	" WHERE ? <= stream_ordering AND stream_ordering < ?"
	109	" ORDER BY stream_ordering DESC"
	110	" LIMIT ?"
	111	)
	112
	113	txn.execute(sql, (target_min_stream_id, max_stream_id, batch_size))
	114
	115	rows = txn.fetchall()
	116	if not rows:
	117	return 0
	118
	119	min_stream_id = rows[-1][0]
	120
	121	update_rows = []
	122	for row in rows:
	123	try:
	124	event_id = row[1]
	125	event_json = db_to_json(row[2])
	126	sender = event_json["sender"]
	127	content = event_json["content"]
	128
	129	contains_url = "url" in content
	130	if contains_url:
	131	contains_url &= isinstance(content["url"], str)
	132	except (KeyError, AttributeError):
	133	# If the event is missing a necessary field then
	134	# skip over it.
	135	continue
	136
	137	update_rows.append((sender, contains_url, event_id))
	138
	139	sql = "UPDATE events SET sender = ?, contains_url = ? WHERE event_id = ?"
	140
	141	for index in range(0, len(update_rows), INSERT_CLUMP_SIZE):
	142	clump = update_rows[index : index + INSERT_CLUMP_SIZE]
	143	txn.executemany(sql, clump)
	144
	145	progress = {
	146	"target_min_stream_id_inclusive": target_min_stream_id,
	147	"max_stream_id_exclusive": min_stream_id,
	148	"rows_inserted": rows_inserted + len(rows),
	149	}
	150
	151	self.db_pool.updates._background_update_progress_txn(
	152	txn, self.EVENT_FIELDS_SENDER_URL_UPDATE_NAME, progress
	153	)
	154
	155	return len(rows)
	156
	157	result = yield self.db_pool.runInteraction(
	158	self.EVENT_FIELDS_SENDER_URL_UPDATE_NAME, reindex_txn
	159	)
	160
	161	if not result:
	162	yield self.db_pool.updates._end_background_update(
	163	self.EVENT_FIELDS_SENDER_URL_UPDATE_NAME
	164	)
	165
	166	return result
	167
	168	@defer.inlineCallbacks
	169	def _background_reindex_origin_server_ts(self, progress, batch_size):
	170	target_min_stream_id = progress["target_min_stream_id_inclusive"]
	171	max_stream_id = progress["max_stream_id_exclusive"]
	172	rows_inserted = progress.get("rows_inserted", 0)
	173
	174	INSERT_CLUMP_SIZE = 1000
	175
	176	def reindex_search_txn(txn):
	177	sql = (
	178	"SELECT stream_ordering, event_id FROM events"
	179	" WHERE ? <= stream_ordering AND stream_ordering < ?"
	180	" ORDER BY stream_ordering DESC"
	181	" LIMIT ?"
	182	)
	183
	184	txn.execute(sql, (target_min_stream_id, max_stream_id, batch_size))
	185
	186	rows = txn.fetchall()
	187	if not rows:
	188	return 0
	189
	190	min_stream_id = rows[-1][0]
	191	event_ids = [row[1] for row in rows]
	192
	193	rows_to_update = []
	194
	195	chunks = [event_ids[i : i + 100] for i in range(0, len(event_ids), 100)]
	196	for chunk in chunks:
	197	ev_rows = self.db_pool.simple_select_many_txn(
	198	txn,
	199	table="event_json",
	200	column="event_id",
	201	iterable=chunk,
	202	retcols=["event_id", "json"],
	203	keyvalues={},
	204	)
	205
	206	for row in ev_rows:
	207	event_id = row["event_id"]
	208	event_json = db_to_json(row["json"])
	209	try:
	210	origin_server_ts = event_json["origin_server_ts"]
	211	except (KeyError, AttributeError):
	212	# If the event is missing a necessary field then
	213	# skip over it.
	214	continue
	215
	216	rows_to_update.append((origin_server_ts, event_id))
	217
	218	sql = "UPDATE events SET origin_server_ts = ? WHERE event_id = ?"
	219
	220	for index in range(0, len(rows_to_update), INSERT_CLUMP_SIZE):
	221	clump = rows_to_update[index : index + INSERT_CLUMP_SIZE]
	222	txn.executemany(sql, clump)
	223
	224	progress = {
	225	"target_min_stream_id_inclusive": target_min_stream_id,
	226	"max_stream_id_exclusive": min_stream_id,
	227	"rows_inserted": rows_inserted + len(rows_to_update),
	228	}
	229
	230	self.db_pool.updates._background_update_progress_txn(
	231	txn, self.EVENT_ORIGIN_SERVER_TS_NAME, progress
	232	)
	233
	234	return len(rows_to_update)
	235
	236	result = yield self.db_pool.runInteraction(
	237	self.EVENT_ORIGIN_SERVER_TS_NAME, reindex_search_txn
	238	)
	239
	240	if not result:
	241	yield self.db_pool.updates._end_background_update(
	242	self.EVENT_ORIGIN_SERVER_TS_NAME
	243	)
	244
	245	return result
	246
	247	@defer.inlineCallbacks
	248	def _cleanup_extremities_bg_update(self, progress, batch_size):
	249	"""Background update to clean out extremities that should have been
	250	deleted previously.
	251
	252	Mainly used to deal with the aftermath of #5269.
	253	"""
	254
	255	# This works by first copying all existing forward extremities into the
	256	# `_extremities_to_check` table at start up, and then checking each
	257	# event in that table whether we have any descendants that are not
	258	# soft-failed/rejected. If that is the case then we delete that event
	259	# from the forward extremities table.
	260	#
	261	# For efficiency, we do this in batches by recursively pulling out all
	262	# descendants of a batch until we find the non soft-failed/rejected
	263	# events, i.e. the set of descendants whose chain of prev events back
	264	# to the batch of extremities are all soft-failed or rejected.
	265	# Typically, we won't find any such events as extremities will rarely
	266	# have any descendants, but if they do then we should delete those
	267	# extremities.
	268
	269	def _cleanup_extremities_bg_update_txn(txn):
	270	# The set of extremity event IDs that we're checking this round
	271	original_set = set()
	272
	273	# A dict[str, set[str]] of event ID to their prev events.
	274	graph = {}
	275
	276	# The set of descendants of the original set that are not rejected
	277	# nor soft-failed. Ancestors of these events should be removed
	278	# from the forward extremities table.
	279	non_rejected_leaves = set()
	280
	281	# Set of event IDs that have been soft failed, and for which we
	282	# should check if they have descendants which haven't been soft
	283	# failed.
	284	soft_failed_events_to_lookup = set()
	285
	286	# First, we get `batch_size` events from the table, pulling out
	287	# their successor events, if any, and the successor events'
	288	# rejection status.
	289	txn.execute(
	290	"""SELECT prev_event_id, event_id, internal_metadata,
	291	rejections.event_id IS NOT NULL, events.outlier
	292	FROM (
	293	SELECT event_id AS prev_event_id
	294	FROM _extremities_to_check
	295	LIMIT ?
	296	) AS f
	297	LEFT JOIN event_edges USING (prev_event_id)
	298	LEFT JOIN events USING (event_id)
	299	LEFT JOIN event_json USING (event_id)
	300	LEFT JOIN rejections USING (event_id)
	301	""",
	302	(batch_size,),
	303	)
	304
	305	for prev_event_id, event_id, metadata, rejected, outlier in txn:
	306	original_set.add(prev_event_id)
	307
	308	if not event_id or outlier:
	309	# Common case where the forward extremity doesn't have any
	310	# descendants.
	311	continue
	312
	313	graph.setdefault(event_id, set()).add(prev_event_id)
	314
	315	soft_failed = False
	316	if metadata:
	317	soft_failed = db_to_json(metadata).get("soft_failed")
	318
	319	if soft_failed or rejected:
	320	soft_failed_events_to_lookup.add(event_id)
	321	else:
	322	non_rejected_leaves.add(event_id)
	323
	324	# Now we recursively check all the soft-failed descendants we
	325	# found above in the same way, until we have nothing left to
	326	# check.
	327	while soft_failed_events_to_lookup:
	328	# We only want to do 100 at a time, so we split given list
	329	# into two.
	330	batch = list(soft_failed_events_to_lookup)
	331	to_check, to_defer = batch[:100], batch[100:]
	332	soft_failed_events_to_lookup = set(to_defer)
	333
	334	sql = """SELECT prev_event_id, event_id, internal_metadata,
	335	rejections.event_id IS NOT NULL
	336	FROM event_edges
	337	INNER JOIN events USING (event_id)
	338	INNER JOIN event_json USING (event_id)
	339	LEFT JOIN rejections USING (event_id)
	340	WHERE
	341	NOT events.outlier
	342	AND
	343	"""
	344	clause, args = make_in_list_sql_clause(
	345	self.database_engine, "prev_event_id", to_check
	346	)
	347	txn.execute(sql + clause, list(args))
	348
	349	for prev_event_id, event_id, metadata, rejected in txn:
	350	if event_id in graph:
	351	# Already handled this event previously, but we still
	352	# want to record the edge.
	353	graph[event_id].add(prev_event_id)
	354	continue
	355
	356	graph[event_id] = {prev_event_id}
	357
	358	soft_failed = db_to_json(metadata).get("soft_failed")
	359	if soft_failed or rejected:
	360	soft_failed_events_to_lookup.add(event_id)
	361	else:
	362	non_rejected_leaves.add(event_id)
	363
	364	# We have a set of non-soft-failed descendants, so we recurse up
	365	# the graph to find all ancestors and add them to the set of event
	366	# IDs that we can delete from forward extremities table.
	367	to_delete = set()
	368	while non_rejected_leaves:
	369	event_id = non_rejected_leaves.pop()
	370	prev_event_ids = graph.get(event_id, set())
	371	non_rejected_leaves.update(prev_event_ids)
	372	to_delete.update(prev_event_ids)
	373
	374	to_delete.intersection_update(original_set)
	375
	376	deleted = self.db_pool.simple_delete_many_txn(
	377	txn=txn,
	378	table="event_forward_extremities",
	379	column="event_id",
	380	iterable=to_delete,
	381	keyvalues={},
	382	)
	383
	384	logger.info(
	385	"Deleted %d forward extremities of %d checked, to clean up #5269",
	386	deleted,
	387	len(original_set),
	388	)
	389
	390	if deleted:
	391	# We now need to invalidate the caches of these rooms
	392	rows = self.db_pool.simple_select_many_txn(
	393	txn,
	394	table="events",
	395	column="event_id",
	396	iterable=to_delete,
	397	keyvalues={},
	398	retcols=("room_id",),
	399	)
	400	room_ids = {row["room_id"] for row in rows}
	401	for room_id in room_ids:
	402	txn.call_after(
	403	self.get_latest_event_ids_in_room.invalidate, (room_id,)
	404	)
	405
	406	self.db_pool.simple_delete_many_txn(
	407	txn=txn,
	408	table="_extremities_to_check",
	409	column="event_id",
	410	iterable=original_set,
	411	keyvalues={},
	412	)
	413
	414	return len(original_set)
	415
	416	num_handled = yield self.db_pool.runInteraction(
	417	"_cleanup_extremities_bg_update", _cleanup_extremities_bg_update_txn
	418	)
	419
	420	if not num_handled:
	421	yield self.db_pool.updates._end_background_update(
	422	self.DELETE_SOFT_FAILED_EXTREMITIES
	423	)
	424
	425	def _drop_table_txn(txn):
	426	txn.execute("DROP TABLE _extremities_to_check")
	427
	428	yield self.db_pool.runInteraction(
	429	"_cleanup_extremities_bg_update_drop_table", _drop_table_txn
	430	)
	431
	432	return num_handled
	433
	434	@defer.inlineCallbacks
	435	def _redactions_received_ts(self, progress, batch_size):
	436	"""Handles filling out the `received_ts` column in redactions.
	437	"""
	438	last_event_id = progress.get("last_event_id", "")
	439
	440	def _redactions_received_ts_txn(txn):
	441	# Fetch the set of event IDs that we want to update
	442	sql = """
	443	SELECT event_id FROM redactions
	444	WHERE event_id > ?
	445	ORDER BY event_id ASC
	446	LIMIT ?
	447	"""
	448
	449	txn.execute(sql, (last_event_id, batch_size))
	450
	451	rows = txn.fetchall()
	452	if not rows:
	453	return 0
	454
	455	(upper_event_id,) = rows[-1]
	456
	457	# Update the redactions with the received_ts.
	458	#
	459	# Note: Not all events have an associated received_ts, so we
	460	# fallback to using origin_server_ts. If we for some reason don't
	461	# have an origin_server_ts, lets just use the current timestamp.
	462	#
	463	# We don't want to leave it null, as then we'll never try and
	464	# censor those redactions.
	465	sql = """
	466	UPDATE redactions
	467	SET received_ts = (
	468	SELECT COALESCE(received_ts, origin_server_ts, ?) FROM events
	469	WHERE events.event_id = redactions.event_id
	470	)
	471	WHERE ? <= event_id AND event_id <= ?
	472	"""
	473
	474	txn.execute(sql, (self._clock.time_msec(), last_event_id, upper_event_id))
	475
	476	self.db_pool.updates._background_update_progress_txn(
	477	txn, "redactions_received_ts", {"last_event_id": upper_event_id}
	478	)
	479
	480	return len(rows)
	481
	482	count = yield self.db_pool.runInteraction(
	483	"_redactions_received_ts", _redactions_received_ts_txn
	484	)
	485
	486	if not count:
	487	yield self.db_pool.updates._end_background_update("redactions_received_ts")
	488
	489	return count
	490
	491	@defer.inlineCallbacks
	492	def _event_fix_redactions_bytes(self, progress, batch_size):
	493	"""Undoes hex encoded censored redacted event JSON.
	494	"""
	495
	496	def _event_fix_redactions_bytes_txn(txn):
	497	# This update is quite fast due to new index.
	498	txn.execute(
	499	"""
	500	UPDATE event_json
	501	SET
	502	json = convert_from(json::bytea, 'utf8')
	503	FROM redactions
	504	WHERE
	505	redactions.have_censored
	506	AND event_json.event_id = redactions.redacts
	507	AND json NOT LIKE '{%';
	508	"""
	509	)
	510
	511	txn.execute("DROP INDEX redactions_censored_redacts")
	512
	513	yield self.db_pool.runInteraction(
	514	"_event_fix_redactions_bytes", _event_fix_redactions_bytes_txn
	515	)
	516
	517	yield self.db_pool.updates._end_background_update("event_fix_redactions_bytes")
	518
	519	return 1
	520
	521	@defer.inlineCallbacks
	522	def _event_store_labels(self, progress, batch_size):
	523	"""Background update handler which will store labels for existing events."""
	524	last_event_id = progress.get("last_event_id", "")
	525
	526	def _event_store_labels_txn(txn):
	527	txn.execute(
	528	"""
	529	SELECT event_id, json FROM event_json
	530	LEFT JOIN event_labels USING (event_id)
	531	WHERE event_id > ? AND label IS NULL
	532	ORDER BY event_id LIMIT ?
	533	""",
	534	(last_event_id, batch_size),
	535	)
	536
	537	results = list(txn)
	538
	539	nbrows = 0
	540	last_row_event_id = ""
	541	for (event_id, event_json_raw) in results:
	542	try:
	543	event_json = db_to_json(event_json_raw)
	544
	545	self.db_pool.simple_insert_many_txn(
	546	txn=txn,
	547	table="event_labels",
	548	values=[
	549	{
	550	"event_id": event_id,
	551	"label": label,
	552	"room_id": event_json["room_id"],
	553	"topological_ordering": event_json["depth"],
	554	}
	555	for label in event_json["content"].get(
	556	EventContentFields.LABELS, []
	557	)
	558	if isinstance(label, str)
	559	],
	560	)
	561	except Exception as e:
	562	logger.warning(
	563	"Unable to load event %s (no labels will be imported): %s",
	564	event_id,
	565	e,
	566	)
	567
	568	nbrows += 1
	569	last_row_event_id = event_id
	570
	571	self.db_pool.updates._background_update_progress_txn(
	572	txn, "event_store_labels", {"last_event_id": last_row_event_id}
	573	)
	574
	575	return nbrows
	576
	577	num_rows = yield self.db_pool.runInteraction(
	578	desc="event_store_labels", func=_event_store_labels_txn
	579	)
	580
	581	if not num_rows:
	582	yield self.db_pool.updates._end_background_update("event_store_labels")
	583
	584	return num_rows

+1370

-0

synapse/storage/databases/main/events_worker.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2018 New Vector Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from __future__ import division
	16
	17	import itertools
	18	import logging
	19	import threading
	20	from collections import namedtuple
	21	from typing import List, Optional, Tuple
	22
	23	from constantly import NamedConstant, Names
	24
	25	from twisted.internet import defer
	26
	27	from synapse.api.constants import EventTypes
	28	from synapse.api.errors import NotFoundError, SynapseError
	29	from synapse.api.room_versions import (
	30	KNOWN_ROOM_VERSIONS,
	31	EventFormatVersions,
	32	RoomVersions,
	33	)
	34	from synapse.events import make_event_from_dict
	35	from synapse.events.utils import prune_event
	36	from synapse.logging.context import PreserveLoggingContext, current_context
	37	from synapse.metrics.background_process_metrics import run_as_background_process
	38	from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
	39	from synapse.replication.tcp.streams import BackfillStream
	40	from synapse.replication.tcp.streams.events import EventsStream
	41	from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
	42	from synapse.storage.database import DatabasePool
	43	from synapse.storage.util.id_generators import StreamIdGenerator
	44	from synapse.types import get_domain_from_id
	45	from synapse.util.caches.descriptors import Cache, cached, cachedInlineCallbacks
	46	from synapse.util.iterutils import batch_iter
	47	from synapse.util.metrics import Measure
	48
	49	logger = logging.getLogger(__name__)
	50
	51
	52	# These values are used in the `enqueus_event` and `_do_fetch` methods to
	53	# control how we batch/bulk fetch events from the database.
	54	# The values are plucked out of thing air to make initial sync run faster
	55	# on jki.re
	56	# TODO: Make these configurable.
	57	EVENT_QUEUE_THREADS = 3 # Max number of threads that will fetch events
	58	EVENT_QUEUE_ITERATIONS = 3 # No. times we block waiting for requests for events
	59	EVENT_QUEUE_TIMEOUT_S = 0.1 # Timeout when waiting for requests for events
	60
	61
	62	_EventCacheEntry = namedtuple("_EventCacheEntry", ("event", "redacted_event"))
	63
	64
	65	class EventRedactBehaviour(Names):
	66	"""
	67	What to do when retrieving a redacted event from the database.
	68	"""
	69
	70	AS_IS = NamedConstant()
	71	REDACT = NamedConstant()
	72	BLOCK = NamedConstant()
	73
	74
	75	class EventsWorkerStore(SQLBaseStore):
	76	def __init__(self, database: DatabasePool, db_conn, hs):
	77	super(EventsWorkerStore, self).__init__(database, db_conn, hs)
	78
	79	if hs.config.worker.writers.events == hs.get_instance_name():
	80	# We are the process in charge of generating stream ids for events,
	81	# so instantiate ID generators based on the database
	82	self._stream_id_gen = StreamIdGenerator(
	83	db_conn, "events", "stream_ordering",
	84	)
	85	self._backfill_id_gen = StreamIdGenerator(
	86	db_conn,
	87	"events",
	88	"stream_ordering",
	89	step=-1,
	90	extra_tables=[("ex_outlier_stream", "event_stream_ordering")],
	91	)
	92	else:
	93	# Another process is in charge of persisting events and generating
	94	# stream IDs: rely on the replication streams to let us know which
	95	# IDs we can process.
	96	self._stream_id_gen = SlavedIdTracker(db_conn, "events", "stream_ordering")
	97	self._backfill_id_gen = SlavedIdTracker(
	98	db_conn, "events", "stream_ordering", step=-1
	99	)
	100
	101	self._get_event_cache = Cache(
	102	"getEvent",
	103	keylen=3,
	104	max_entries=hs.config.caches.event_cache_size,
	105	apply_cache_factor_from_config=False,
	106	)
	107
	108	self._event_fetch_lock = threading.Condition()
	109	self._event_fetch_list = []
	110	self._event_fetch_ongoing = 0
	111
	112	def process_replication_rows(self, stream_name, instance_name, token, rows):
	113	if stream_name == EventsStream.NAME:
	114	self._stream_id_gen.advance(token)
	115	elif stream_name == BackfillStream.NAME:
	116	self._backfill_id_gen.advance(-token)
	117
	118	super().process_replication_rows(stream_name, instance_name, token, rows)
	119
	120	def get_received_ts(self, event_id):
	121	"""Get received_ts (when it was persisted) for the event.
	122
	123	Raises an exception for unknown events.
	124
	125	Args:
	126	event_id (str)
	127
	128	Returns:
	129	Deferred[int\|None]: Timestamp in milliseconds, or None for events
	130	that were persisted before received_ts was implemented.
	131	"""
	132	return self.db_pool.simple_select_one_onecol(
	133	table="events",
	134	keyvalues={"event_id": event_id},
	135	retcol="received_ts",
	136	desc="get_received_ts",
	137	)
	138
	139	def get_received_ts_by_stream_pos(self, stream_ordering):
	140	"""Given a stream ordering get an approximate timestamp of when it
	141	happened.
	142
	143	This is done by simply taking the received ts of the first event that
	144	has a stream ordering greater than or equal to the given stream pos.
	145	If none exists returns the current time, on the assumption that it must
	146	have happened recently.
	147
	148	Args:
	149	stream_ordering (int)
	150
	151	Returns:
	152	Deferred[int]
	153	"""
	154
	155	def _get_approximate_received_ts_txn(txn):
	156	sql = """
	157	SELECT received_ts FROM events
	158	WHERE stream_ordering >= ?
	159	LIMIT 1
	160	"""
	161
	162	txn.execute(sql, (stream_ordering,))
	163	row = txn.fetchone()
	164	if row and row[0]:
	165	ts = row[0]
	166	else:
	167	ts = self.clock.time_msec()
	168
	169	return ts
	170
	171	return self.db_pool.runInteraction(
	172	"get_approximate_received_ts", _get_approximate_received_ts_txn
	173	)
	174
	175	@defer.inlineCallbacks
	176	def get_event(
	177	self,
	178	event_id: str,
	179	redact_behaviour: EventRedactBehaviour = EventRedactBehaviour.REDACT,
	180	get_prev_content: bool = False,
	181	allow_rejected: bool = False,
	182	allow_none: bool = False,
	183	check_room_id: Optional[str] = None,
	184	):
	185	"""Get an event from the database by event_id.
	186
	187	Args:
	188	event_id: The event_id of the event to fetch
	189
	190	redact_behaviour: Determine what to do with a redacted event. Possible values:
	191	* AS_IS - Return the full event body with no redacted content
	192	* REDACT - Return the event but with a redacted body
	193	* DISALLOW - Do not return redacted events (behave as per allow_none
	194	if the event is redacted)
	195
	196	get_prev_content: If True and event is a state event,
	197	include the previous states content in the unsigned field.
	198
	199	allow_rejected: If True, return rejected events. Otherwise,
	200	behave as per allow_none.
	201
	202	allow_none: If True, return None if no event found, if
	203	False throw a NotFoundError
	204
	205	check_room_id: if not None, check the room of the found event.
	206	If there is a mismatch, behave as per allow_none.
	207
	208	Returns:
	209	Deferred[EventBase\|None]
	210	"""
	211	if not isinstance(event_id, str):
	212	raise TypeError("Invalid event event_id %r" % (event_id,))
	213
	214	events = yield self.get_events_as_list(
	215	[event_id],
	216	redact_behaviour=redact_behaviour,
	217	get_prev_content=get_prev_content,
	218	allow_rejected=allow_rejected,
	219	)
	220
	221	event = events[0] if events else None
	222
	223	if event is not None and check_room_id is not None:
	224	if event.room_id != check_room_id:
	225	event = None
	226
	227	if event is None and not allow_none:
	228	raise NotFoundError("Could not find event %s" % (event_id,))
	229
	230	return event
	231
	232	@defer.inlineCallbacks
	233	def get_events(
	234	self,
	235	event_ids: List[str],
	236	redact_behaviour: EventRedactBehaviour = EventRedactBehaviour.REDACT,
	237	get_prev_content: bool = False,
	238	allow_rejected: bool = False,
	239	):
	240	"""Get events from the database
	241
	242	Args:
	243	event_ids: The event_ids of the events to fetch
	244
	245	redact_behaviour: Determine what to do with a redacted event. Possible
	246	values:
	247	* AS_IS - Return the full event body with no redacted content
	248	* REDACT - Return the event but with a redacted body
	249	* DISALLOW - Do not return redacted events (omit them from the response)
	250
	251	get_prev_content: If True and event is a state event,
	252	include the previous states content in the unsigned field.
	253
	254	allow_rejected: If True, return rejected events. Otherwise,
	255	omits rejeted events from the response.
	256
	257	Returns:
	258	Deferred : Dict from event_id to event.
	259	"""
	260	events = yield self.get_events_as_list(
	261	event_ids,
	262	redact_behaviour=redact_behaviour,
	263	get_prev_content=get_prev_content,
	264	allow_rejected=allow_rejected,
	265	)
	266
	267	return {e.event_id: e for e in events}
	268
	269	@defer.inlineCallbacks
	270	def get_events_as_list(
	271	self,
	272	event_ids: List[str],
	273	redact_behaviour: EventRedactBehaviour = EventRedactBehaviour.REDACT,
	274	get_prev_content: bool = False,
	275	allow_rejected: bool = False,
	276	):
	277	"""Get events from the database and return in a list in the same order
	278	as given by `event_ids` arg.
	279
	280	Unknown events will be omitted from the response.
	281
	282	Args:
	283	event_ids: The event_ids of the events to fetch
	284
	285	redact_behaviour: Determine what to do with a redacted event. Possible values:
	286	* AS_IS - Return the full event body with no redacted content
	287	* REDACT - Return the event but with a redacted body
	288	* DISALLOW - Do not return redacted events (omit them from the response)
	289
	290	get_prev_content: If True and event is a state event,
	291	include the previous states content in the unsigned field.
	292
	293	allow_rejected: If True, return rejected events. Otherwise,
	294	omits rejected events from the response.
	295
	296	Returns:
	297	Deferred[list[EventBase]]: List of events fetched from the database. The
	298	events are in the same order as `event_ids` arg.
	299
	300	Note that the returned list may be smaller than the list of event
	301	IDs if not all events could be fetched.
	302	"""
	303
	304	if not event_ids:
	305	return []
	306
	307	# there may be duplicates so we cast the list to a set
	308	event_entry_map = yield self._get_events_from_cache_or_db(
	309	set(event_ids), allow_rejected=allow_rejected
	310	)
	311
	312	events = []
	313	for event_id in event_ids:
	314	entry = event_entry_map.get(event_id, None)
	315	if not entry:
	316	continue
	317
	318	if not allow_rejected:
	319	assert not entry.event.rejected_reason, (
	320	"rejected event returned from _get_events_from_cache_or_db despite "
	321	"allow_rejected=False"
	322	)
	323
	324	# We may not have had the original event when we received a redaction, so
	325	# we have to recheck auth now.
	326
	327	if not allow_rejected and entry.event.type == EventTypes.Redaction:
	328	if entry.event.redacts is None:
	329	# A redacted redaction doesn't have a `redacts` key, in
	330	# which case lets just withhold the event.
	331	#
	332	# Note: Most of the time if the redactions has been
	333	# redacted we still have the un-redacted event in the DB
	334	# and so we'll still see the `redacts` key. However, this
	335	# isn't always true e.g. if we have censored the event.
	336	logger.debug(
	337	"Withholding redaction event %s as we don't have redacts key",
	338	event_id,
	339	)
	340	continue
	341
	342	redacted_event_id = entry.event.redacts
	343	event_map = yield self._get_events_from_cache_or_db([redacted_event_id])
	344	original_event_entry = event_map.get(redacted_event_id)
	345	if not original_event_entry:
	346	# we don't have the redacted event (or it was rejected).
	347	#
	348	# We assume that the redaction isn't authorized for now; if the
	349	# redacted event later turns up, the redaction will be re-checked,
	350	# and if it is found valid, the original will get redacted before it
	351	# is served to the client.
	352	logger.debug(
	353	"Withholding redaction event %s since we don't (yet) have the "
	354	"original %s",
	355	event_id,
	356	redacted_event_id,
	357	)
	358	continue
	359
	360	original_event = original_event_entry.event
	361	if original_event.type == EventTypes.Create:
	362	# we never serve redactions of Creates to clients.
	363	logger.info(
	364	"Withholding redaction %s of create event %s",
	365	event_id,
	366	redacted_event_id,
	367	)
	368	continue
	369
	370	if original_event.room_id != entry.event.room_id:
	371	logger.info(
	372	"Withholding redaction %s of event %s from a different room",
	373	event_id,
	374	redacted_event_id,
	375	)
	376	continue
	377
	378	if entry.event.internal_metadata.need_to_check_redaction():
	379	original_domain = get_domain_from_id(original_event.sender)
	380	redaction_domain = get_domain_from_id(entry.event.sender)
	381	if original_domain != redaction_domain:
	382	# the senders don't match, so this is forbidden
	383	logger.info(
	384	"Withholding redaction %s whose sender domain %s doesn't "
	385	"match that of redacted event %s %s",
	386	event_id,
	387	redaction_domain,
	388	redacted_event_id,
	389	original_domain,
	390	)
	391	continue
	392
	393	# Update the cache to save doing the checks again.
	394	entry.event.internal_metadata.recheck_redaction = False
	395
	396	event = entry.event
	397
	398	if entry.redacted_event:
	399	if redact_behaviour == EventRedactBehaviour.BLOCK:
	400	# Skip this event
	401	continue
	402	elif redact_behaviour == EventRedactBehaviour.REDACT:
	403	event = entry.redacted_event
	404
	405	events.append(event)
	406
	407	if get_prev_content:
	408	if "replaces_state" in event.unsigned:
	409	prev = yield self.get_event(
	410	event.unsigned["replaces_state"],
	411	get_prev_content=False,
	412	allow_none=True,
	413	)
	414	if prev:
	415	event.unsigned = dict(event.unsigned)
	416	event.unsigned["prev_content"] = prev.content
	417	event.unsigned["prev_sender"] = prev.sender
	418
	419	return events
	420
	421	@defer.inlineCallbacks
	422	def _get_events_from_cache_or_db(self, event_ids, allow_rejected=False):
	423	"""Fetch a bunch of events from the cache or the database.
	424
	425	If events are pulled from the database, they will be cached for future lookups.
	426
	427	Unknown events are omitted from the response.
	428
	429	Args:
	430
	431	event_ids (Iterable[str]): The event_ids of the events to fetch
	432
	433	allow_rejected (bool): Whether to include rejected events. If False,
	434	rejected events are omitted from the response.
	435
	436	Returns:
	437	Deferred[Dict[str, _EventCacheEntry]]:
	438	map from event id to result
	439	"""
	440	event_entry_map = self._get_events_from_cache(
	441	event_ids, allow_rejected=allow_rejected
	442	)
	443
	444	missing_events_ids = [e for e in event_ids if e not in event_entry_map]
	445
	446	if missing_events_ids:
	447	log_ctx = current_context()
	448	log_ctx.record_event_fetch(len(missing_events_ids))
	449
	450	# Note that _get_events_from_db is also responsible for turning db rows
	451	# into FrozenEvents (via _get_event_from_row), which involves seeing if
	452	# the events have been redacted, and if so pulling the redaction event out
	453	# of the database to check it.
	454	#
	455	missing_events = yield self._get_events_from_db(
	456	missing_events_ids, allow_rejected=allow_rejected
	457	)
	458
	459	event_entry_map.update(missing_events)
	460
	461	return event_entry_map
	462
	463	def _invalidate_get_event_cache(self, event_id):
	464	self._get_event_cache.invalidate((event_id,))
	465
	466	def _get_events_from_cache(self, events, allow_rejected, update_metrics=True):
	467	"""Fetch events from the caches
	468
	469	Args:
	470	events (Iterable[str]): list of event_ids to fetch
	471	allow_rejected (bool): Whether to return events that were rejected
	472	update_metrics (bool): Whether to update the cache hit ratio metrics
	473
	474	Returns:
	475	dict of event_id -> _EventCacheEntry for each event_id in cache. If
	476	allow_rejected is `False` then there will still be an entry but it
	477	will be `None`
	478	"""
	479	event_map = {}
	480
	481	for event_id in events:
	482	ret = self._get_event_cache.get(
	483	(event_id,), None, update_metrics=update_metrics
	484	)
	485	if not ret:
	486	continue
	487
	488	if allow_rejected or not ret.event.rejected_reason:
	489	event_map[event_id] = ret
	490	else:
	491	event_map[event_id] = None
	492
	493	return event_map
	494
	495	def _do_fetch(self, conn):
	496	"""Takes a database connection and waits for requests for events from
	497	the _event_fetch_list queue.
	498	"""
	499	i = 0
	500	while True:
	501	with self._event_fetch_lock:
	502	event_list = self._event_fetch_list
	503	self._event_fetch_list = []
	504
	505	if not event_list:
	506	single_threaded = self.database_engine.single_threaded
	507	if single_threaded or i > EVENT_QUEUE_ITERATIONS:
	508	self._event_fetch_ongoing -= 1
	509	return
	510	else:
	511	self._event_fetch_lock.wait(EVENT_QUEUE_TIMEOUT_S)
	512	i += 1
	513	continue
	514	i = 0
	515
	516	self._fetch_event_list(conn, event_list)
	517
	518	def _fetch_event_list(self, conn, event_list):
	519	"""Handle a load of requests from the _event_fetch_list queue
	520
	521	Args:
	522	conn (twisted.enterprise.adbapi.Connection): database connection
	523
	524	event_list (list[Tuple[list[str], Deferred]]):
	525	The fetch requests. Each entry consists of a list of event
	526	ids to be fetched, and a deferred to be completed once the
	527	events have been fetched.
	528
	529	The deferreds are callbacked with a dictionary mapping from event id
	530	to event row. Note that it may well contain additional events that
	531	were not part of this request.
	532	"""
	533	with Measure(self._clock, "_fetch_event_list"):
	534	try:
	535	events_to_fetch = {
	536	event_id for events, _ in event_list for event_id in events
	537	}
	538
	539	row_dict = self.db_pool.new_transaction(
	540	conn, "do_fetch", [], [], self._fetch_event_rows, events_to_fetch
	541	)
	542
	543	# We only want to resolve deferreds from the main thread
	544	def fire():
	545	for _, d in event_list:
	546	d.callback(row_dict)
	547
	548	with PreserveLoggingContext():
	549	self.hs.get_reactor().callFromThread(fire)
	550	except Exception as e:
	551	logger.exception("do_fetch")
	552
	553	# We only want to resolve deferreds from the main thread
	554	def fire(evs, exc):
	555	for _, d in evs:
	556	if not d.called:
	557	with PreserveLoggingContext():
	558	d.errback(exc)
	559
	560	with PreserveLoggingContext():
	561	self.hs.get_reactor().callFromThread(fire, event_list, e)
	562
	563	@defer.inlineCallbacks
	564	def _get_events_from_db(self, event_ids, allow_rejected=False):
	565	"""Fetch a bunch of events from the database.
	566
	567	Returned events will be added to the cache for future lookups.
	568
	569	Unknown events are omitted from the response.
	570
	571	Args:
	572	event_ids (Iterable[str]): The event_ids of the events to fetch
	573
	574	allow_rejected (bool): Whether to include rejected events. If False,
	575	rejected events are omitted from the response.
	576
	577	Returns:
	578	Deferred[Dict[str, _EventCacheEntry]]:
	579	map from event id to result. May return extra events which
	580	weren't asked for.
	581	"""
	582	fetched_events = {}
	583	events_to_fetch = event_ids
	584
	585	while events_to_fetch:
	586	row_map = yield self._enqueue_events(events_to_fetch)
	587
	588	# we need to recursively fetch any redactions of those events
	589	redaction_ids = set()
	590	for event_id in events_to_fetch:
	591	row = row_map.get(event_id)
	592	fetched_events[event_id] = row
	593	if row:
	594	redaction_ids.update(row["redactions"])
	595
	596	events_to_fetch = redaction_ids.difference(fetched_events.keys())
	597	if events_to_fetch:
	598	logger.debug("Also fetching redaction events %s", events_to_fetch)
	599
	600	# build a map from event_id to EventBase
	601	event_map = {}
	602	for event_id, row in fetched_events.items():
	603	if not row:
	604	continue
	605	assert row["event_id"] == event_id
	606
	607	rejected_reason = row["rejected_reason"]
	608
	609	if not allow_rejected and rejected_reason:
	610	continue
	611
	612	d = db_to_json(row["json"])
	613	internal_metadata = db_to_json(row["internal_metadata"])
	614
	615	format_version = row["format_version"]
	616	if format_version is None:
	617	# This means that we stored the event before we had the concept
	618	# of a event format version, so it must be a V1 event.
	619	format_version = EventFormatVersions.V1
	620
	621	room_version_id = row["room_version_id"]
	622
	623	if not room_version_id:
	624	# this should only happen for out-of-band membership events
	625	if not internal_metadata.get("out_of_band_membership"):
	626	logger.warning(
	627	"Room %s for event %s is unknown", d["room_id"], event_id
	628	)
	629	continue
	630
	631	# take a wild stab at the room version based on the event format
	632	if format_version == EventFormatVersions.V1:
	633	room_version = RoomVersions.V1
	634	elif format_version == EventFormatVersions.V2:
	635	room_version = RoomVersions.V3
	636	else:
	637	room_version = RoomVersions.V5
	638	else:
	639	room_version = KNOWN_ROOM_VERSIONS.get(room_version_id)
	640	if not room_version:
	641	logger.warning(
	642	"Event %s in room %s has unknown room version %s",
	643	event_id,
	644	d["room_id"],
	645	room_version_id,
	646	)
	647	continue
	648
	649	if room_version.event_format != format_version:
	650	logger.error(
	651	"Event %s in room %s with version %s has wrong format: "
	652	"expected %s, was %s",
	653	event_id,
	654	d["room_id"],
	655	room_version_id,
	656	room_version.event_format,
	657	format_version,
	658	)
	659	continue
	660
	661	original_ev = make_event_from_dict(
	662	event_dict=d,
	663	room_version=room_version,
	664	internal_metadata_dict=internal_metadata,
	665	rejected_reason=rejected_reason,
	666	)
	667
	668	event_map[event_id] = original_ev
	669
	670	# finally, we can decide whether each one needs redacting, and build
	671	# the cache entries.
	672	result_map = {}
	673	for event_id, original_ev in event_map.items():
	674	redactions = fetched_events[event_id]["redactions"]
	675	redacted_event = self._maybe_redact_event_row(
	676	original_ev, redactions, event_map
	677	)
	678
	679	cache_entry = _EventCacheEntry(
	680	event=original_ev, redacted_event=redacted_event
	681	)
	682
	683	self._get_event_cache.prefill((event_id,), cache_entry)
	684	result_map[event_id] = cache_entry
	685
	686	return result_map
	687
	688	@defer.inlineCallbacks
	689	def _enqueue_events(self, events):
	690	"""Fetches events from the database using the _event_fetch_list. This
	691	allows batch and bulk fetching of events - it allows us to fetch events
	692	without having to create a new transaction for each request for events.
	693
	694	Args:
	695	events (Iterable[str]): events to be fetched.
	696
	697	Returns:
	698	Deferred[Dict[str, Dict]]: map from event id to row data from the database.
	699	May contain events that weren't requested.
	700	"""
	701
	702	events_d = defer.Deferred()
	703	with self._event_fetch_lock:
	704	self._event_fetch_list.append((events, events_d))
	705
	706	self._event_fetch_lock.notify()
	707
	708	if self._event_fetch_ongoing < EVENT_QUEUE_THREADS:
	709	self._event_fetch_ongoing += 1
	710	should_start = True
	711	else:
	712	should_start = False
	713
	714	if should_start:
	715	run_as_background_process(
	716	"fetch_events", self.db_pool.runWithConnection, self._do_fetch
	717	)
	718
	719	logger.debug("Loading %d events: %s", len(events), events)
	720	with PreserveLoggingContext():
	721	row_map = yield events_d
	722	logger.debug("Loaded %d events (%d rows)", len(events), len(row_map))
	723
	724	return row_map
	725
	726	def _fetch_event_rows(self, txn, event_ids):
	727	"""Fetch event rows from the database
	728
	729	Events which are not found are omitted from the result.
	730
	731	The returned per-event dicts contain the following keys:
	732
	733	* event_id (str)
	734
	735	* json (str): json-encoded event structure
	736
	737	* internal_metadata (str): json-encoded internal metadata dict
	738
	739	* format_version (int\|None): The format of the event. Hopefully one
	740	of EventFormatVersions. 'None' means the event predates
	741	EventFormatVersions (so the event is format V1).
	742
	743	* room_version_id (str\|None): The version of the room which contains the event.
	744	Hopefully one of RoomVersions.
	745
	746	Due to historical reasons, there may be a few events in the database which
	747	do not have an associated room; in this case None will be returned here.
	748
	749	* rejected_reason (str\|None): if the event was rejected, the reason
	750	why.
	751
	752	* redactions (List[str]): a list of event-ids which (claim to) redact
	753	this event.
	754
	755	Args:
	756	txn (twisted.enterprise.adbapi.Connection):
	757	event_ids (Iterable[str]): event IDs to fetch
	758
	759	Returns:
	760	Dict[str, Dict]: a map from event id to event info.
	761	"""
	762	event_dict = {}
	763	for evs in batch_iter(event_ids, 200):
	764	sql = """\
	765	SELECT
	766	e.event_id,
	767	e.internal_metadata,
	768	e.json,
	769	e.format_version,
	770	r.room_version,
	771	rej.reason
	772	FROM event_json as e
	773	LEFT JOIN rooms r USING (room_id)
	774	LEFT JOIN rejections as rej USING (event_id)
	775	WHERE """
	776
	777	clause, args = make_in_list_sql_clause(
	778	txn.database_engine, "e.event_id", evs
	779	)
	780
	781	txn.execute(sql + clause, args)
	782
	783	for row in txn:
	784	event_id = row[0]
	785	event_dict[event_id] = {
	786	"event_id": event_id,
	787	"internal_metadata": row[1],
	788	"json": row[2],
	789	"format_version": row[3],
	790	"room_version_id": row[4],
	791	"rejected_reason": row[5],
	792	"redactions": [],
	793	}
	794
	795	# check for redactions
	796	redactions_sql = "SELECT event_id, redacts FROM redactions WHERE "
	797
	798	clause, args = make_in_list_sql_clause(txn.database_engine, "redacts", evs)
	799
	800	txn.execute(redactions_sql + clause, args)
	801
	802	for (redacter, redacted) in txn:
	803	d = event_dict.get(redacted)
	804	if d:
	805	d["redactions"].append(redacter)
	806
	807	return event_dict
	808
	809	def _maybe_redact_event_row(self, original_ev, redactions, event_map):
	810	"""Given an event object and a list of possible redacting event ids,
	811	determine whether to honour any of those redactions and if so return a redacted
	812	event.
	813
	814	Args:
	815	original_ev (EventBase):
	816	redactions (iterable[str]): list of event ids of potential redaction events
	817	event_map (dict[str, EventBase]): other events which have been fetched, in
	818	which we can look up the redaaction events. Map from event id to event.
	819
	820	Returns:
	821	Deferred[EventBase\|None]: if the event should be redacted, a pruned
	822	event object. Otherwise, None.
	823	"""
	824	if original_ev.type == "m.room.create":
	825	# we choose to ignore redactions of m.room.create events.
	826	return None
	827
	828	for redaction_id in redactions:
	829	redaction_event = event_map.get(redaction_id)
	830	if not redaction_event or redaction_event.rejected_reason:
	831	# we don't have the redaction event, or the redaction event was not
	832	# authorized.
	833	logger.debug(
	834	"%s was redacted by %s but redaction not found/authed",
	835	original_ev.event_id,
	836	redaction_id,
	837	)
	838	continue
	839
	840	if redaction_event.room_id != original_ev.room_id:
	841	logger.debug(
	842	"%s was redacted by %s but redaction was in a different room!",
	843	original_ev.event_id,
	844	redaction_id,
	845	)
	846	continue
	847
	848	# Starting in room version v3, some redactions need to be
	849	# rechecked if we didn't have the redacted event at the
	850	# time, so we recheck on read instead.
	851	if redaction_event.internal_metadata.need_to_check_redaction():
	852	expected_domain = get_domain_from_id(original_ev.sender)
	853	if get_domain_from_id(redaction_event.sender) == expected_domain:
	854	# This redaction event is allowed. Mark as not needing a recheck.
	855	redaction_event.internal_metadata.recheck_redaction = False
	856	else:
	857	# Senders don't match, so the event isn't actually redacted
	858	logger.debug(
	859	"%s was redacted by %s but the senders don't match",
	860	original_ev.event_id,
	861	redaction_id,
	862	)
	863	continue
	864
	865	logger.debug("Redacting %s due to %s", original_ev.event_id, redaction_id)
	866
	867	# we found a good redaction event. Redact!
	868	redacted_event = prune_event(original_ev)
	869	redacted_event.unsigned["redacted_by"] = redaction_id
	870
	871	# It's fine to add the event directly, since get_pdu_json
	872	# will serialise this field correctly
	873	redacted_event.unsigned["redacted_because"] = redaction_event
	874
	875	return redacted_event
	876
	877	# no valid redaction found for this event
	878	return None
	879
	880	@defer.inlineCallbacks
	881	def have_events_in_timeline(self, event_ids):
	882	"""Given a list of event ids, check if we have already processed and
	883	stored them as non outliers.
	884	"""
	885	rows = yield self.db_pool.simple_select_many_batch(
	886	table="events",
	887	retcols=("event_id",),
	888	column="event_id",
	889	iterable=list(event_ids),
	890	keyvalues={"outlier": False},
	891	desc="have_events_in_timeline",
	892	)
	893
	894	return {r["event_id"] for r in rows}
	895
	896	@defer.inlineCallbacks
	897	def have_seen_events(self, event_ids):
	898	"""Given a list of event ids, check if we have already processed them.
	899
	900	Args:
	901	event_ids (iterable[str]):
	902
	903	Returns:
	904	Deferred[set[str]]: The events we have already seen.
	905	"""
	906	results = set()
	907
	908	def have_seen_events_txn(txn, chunk):
	909	sql = "SELECT event_id FROM events as e WHERE "
	910	clause, args = make_in_list_sql_clause(
	911	txn.database_engine, "e.event_id", chunk
	912	)
	913	txn.execute(sql + clause, args)
	914	for (event_id,) in txn:
	915	results.add(event_id)
	916
	917	# break the input up into chunks of 100
	918	input_iterator = iter(event_ids)
	919	for chunk in iter(lambda: list(itertools.islice(input_iterator, 100)), []):
	920	yield self.db_pool.runInteraction(
	921	"have_seen_events", have_seen_events_txn, chunk
	922	)
	923	return results
	924
	925	def _get_total_state_event_counts_txn(self, txn, room_id):
	926	"""
	927	See get_total_state_event_counts.
	928	"""
	929	# We join against the events table as that has an index on room_id
	930	sql = """
	931	SELECT COUNT(*) FROM state_events
	932	INNER JOIN events USING (room_id, event_id)
	933	WHERE room_id=?
	934	"""
	935	txn.execute(sql, (room_id,))
	936	row = txn.fetchone()
	937	return row[0] if row else 0
	938
	939	def get_total_state_event_counts(self, room_id):
	940	"""
	941	Gets the total number of state events in a room.
	942
	943	Args:
	944	room_id (str)
	945
	946	Returns:
	947	Deferred[int]
	948	"""
	949	return self.db_pool.runInteraction(
	950	"get_total_state_event_counts",
	951	self._get_total_state_event_counts_txn,
	952	room_id,
	953	)
	954
	955	def _get_current_state_event_counts_txn(self, txn, room_id):
	956	"""
	957	See get_current_state_event_counts.
	958	"""
	959	sql = "SELECT COUNT(*) FROM current_state_events WHERE room_id=?"
	960	txn.execute(sql, (room_id,))
	961	row = txn.fetchone()
	962	return row[0] if row else 0
	963
	964	def get_current_state_event_counts(self, room_id):
	965	"""
	966	Gets the current number of state events in a room.
	967
	968	Args:
	969	room_id (str)
	970
	971	Returns:
	972	Deferred[int]
	973	"""
	974	return self.db_pool.runInteraction(
	975	"get_current_state_event_counts",
	976	self._get_current_state_event_counts_txn,
	977	room_id,
	978	)
	979
	980	@defer.inlineCallbacks
	981	def get_room_complexity(self, room_id):
	982	"""
	983	Get a rough approximation of the complexity of the room. This is used by
	984	remote servers to decide whether they wish to join the room or not.
	985	Higher complexity value indicates that being in the room will consume
	986	more resources.
	987
	988	Args:
	989	room_id (str)
	990
	991	Returns:
	992	Deferred[dict[str:int]] of complexity version to complexity.
	993	"""
	994	state_events = yield self.get_current_state_event_counts(room_id)
	995
	996	# Call this one "v1", so we can introduce new ones as we want to develop
	997	# it.
	998	complexity_v1 = round(state_events / 500, 2)
	999
	1000	return {"v1": complexity_v1}
	1001
	1002	def get_current_backfill_token(self):
	1003	"""The current minimum token that backfilled events have reached"""
	1004	return -self._backfill_id_gen.get_current_token()
	1005
	1006	def get_current_events_token(self):
	1007	"""The current maximum token that events have reached"""
	1008	return self._stream_id_gen.get_current_token()
	1009
	1010	def get_all_new_forward_event_rows(self, last_id, current_id, limit):
	1011	"""Returns new events, for the Events replication stream
	1012
	1013	Args:
	1014	last_id: the last stream_id from the previous batch.
	1015	current_id: the maximum stream_id to return up to
	1016	limit: the maximum number of rows to return
	1017
	1018	Returns: Deferred[List[Tuple]]
	1019	a list of events stream rows. Each tuple consists of a stream id as
	1020	the first element, followed by fields suitable for casting into an
	1021	EventsStreamRow.
	1022	"""
	1023
	1024	def get_all_new_forward_event_rows(txn):
	1025	sql = (
	1026	"SELECT e.stream_ordering, e.event_id, e.room_id, e.type,"
	1027	" state_key, redacts, relates_to_id"
	1028	" FROM events AS e"
	1029	" LEFT JOIN redactions USING (event_id)"
	1030	" LEFT JOIN state_events USING (event_id)"
	1031	" LEFT JOIN event_relations USING (event_id)"
	1032	" WHERE ? < stream_ordering AND stream_ordering <= ?"
	1033	" ORDER BY stream_ordering ASC"
	1034	" LIMIT ?"
	1035	)
	1036	txn.execute(sql, (last_id, current_id, limit))
	1037	return txn.fetchall()
	1038
	1039	return self.db_pool.runInteraction(
	1040	"get_all_new_forward_event_rows", get_all_new_forward_event_rows
	1041	)
	1042
	1043	def get_ex_outlier_stream_rows(self, last_id, current_id):
	1044	"""Returns de-outliered events, for the Events replication stream
	1045
	1046	Args:
	1047	last_id: the last stream_id from the previous batch.
	1048	current_id: the maximum stream_id to return up to
	1049
	1050	Returns: Deferred[List[Tuple]]
	1051	a list of events stream rows. Each tuple consists of a stream id as
	1052	the first element, followed by fields suitable for casting into an
	1053	EventsStreamRow.
	1054	"""
	1055
	1056	def get_ex_outlier_stream_rows_txn(txn):
	1057	sql = (
	1058	"SELECT event_stream_ordering, e.event_id, e.room_id, e.type,"
	1059	" state_key, redacts, relates_to_id"
	1060	" FROM events AS e"
	1061	" INNER JOIN ex_outlier_stream USING (event_id)"
	1062	" LEFT JOIN redactions USING (event_id)"
	1063	" LEFT JOIN state_events USING (event_id)"
	1064	" LEFT JOIN event_relations USING (event_id)"
	1065	" WHERE ? < event_stream_ordering"
	1066	" AND event_stream_ordering <= ?"
	1067	" ORDER BY event_stream_ordering ASC"
	1068	)
	1069
	1070	txn.execute(sql, (last_id, current_id))
	1071	return txn.fetchall()
	1072
	1073	return self.db_pool.runInteraction(
	1074	"get_ex_outlier_stream_rows", get_ex_outlier_stream_rows_txn
	1075	)
	1076
	1077	async def get_all_new_backfill_event_rows(
	1078	self, instance_name: str, last_id: int, current_id: int, limit: int
	1079	) -> Tuple[List[Tuple[int, list]], int, bool]:
	1080	"""Get updates for backfill replication stream, including all new
	1081	backfilled events and events that have gone from being outliers to not.
	1082
	1083	Args:
	1084	instance_name: The writer we want to fetch updates from. Unused
	1085	here since there is only ever one writer.
	1086	last_id: The token to fetch updates from. Exclusive.
	1087	current_id: The token to fetch updates up to. Inclusive.
	1088	limit: The requested limit for the number of rows to return. The
	1089	function may return more or fewer rows.
	1090
	1091	Returns:
	1092	A tuple consisting of: the updates, a token to use to fetch
	1093	subsequent updates, and whether we returned fewer rows than exists
	1094	between the requested tokens due to the limit.
	1095
	1096	The token returned can be used in a subsequent call to this
	1097	function to get further updatees.
	1098
	1099	The updates are a list of 2-tuples of stream ID and the row data
	1100	"""
	1101	if last_id == current_id:
	1102	return [], current_id, False
	1103
	1104	def get_all_new_backfill_event_rows(txn):
	1105	sql = (
	1106	"SELECT -e.stream_ordering, e.event_id, e.room_id, e.type,"
	1107	" state_key, redacts, relates_to_id"
	1108	" FROM events AS e"
	1109	" LEFT JOIN redactions USING (event_id)"
	1110	" LEFT JOIN state_events USING (event_id)"
	1111	" LEFT JOIN event_relations USING (event_id)"
	1112	" WHERE ? > stream_ordering AND stream_ordering >= ?"
	1113	" ORDER BY stream_ordering ASC"
	1114	" LIMIT ?"
	1115	)
	1116	txn.execute(sql, (-last_id, -current_id, limit))
	1117	new_event_updates = [(row[0], row[1:]) for row in txn]
	1118
	1119	limited = False
	1120	if len(new_event_updates) == limit:
	1121	upper_bound = new_event_updates[-1][0]
	1122	limited = True
	1123	else:
	1124	upper_bound = current_id
	1125
	1126	sql = (
	1127	"SELECT -event_stream_ordering, e.event_id, e.room_id, e.type,"
	1128	" state_key, redacts, relates_to_id"
	1129	" FROM events AS e"
	1130	" INNER JOIN ex_outlier_stream USING (event_id)"
	1131	" LEFT JOIN redactions USING (event_id)"
	1132	" LEFT JOIN state_events USING (event_id)"
	1133	" LEFT JOIN event_relations USING (event_id)"
	1134	" WHERE ? > event_stream_ordering"
	1135	" AND event_stream_ordering >= ?"
	1136	" ORDER BY event_stream_ordering DESC"
	1137	)
	1138	txn.execute(sql, (-last_id, -upper_bound))
	1139	new_event_updates.extend((row[0], row[1:]) for row in txn)
	1140
	1141	if len(new_event_updates) >= limit:
	1142	upper_bound = new_event_updates[-1][0]
	1143	limited = True
	1144
	1145	return new_event_updates, upper_bound, limited
	1146
	1147	return await self.db_pool.runInteraction(
	1148	"get_all_new_backfill_event_rows", get_all_new_backfill_event_rows
	1149	)
	1150
	1151	async def get_all_updated_current_state_deltas(
	1152	self, from_token: int, to_token: int, target_row_count: int
	1153	) -> Tuple[List[Tuple], int, bool]:
	1154	"""Fetch updates from current_state_delta_stream
	1155
	1156	Args:
	1157	from_token: The previous stream token. Updates from this stream id will
	1158	be excluded.
	1159
	1160	to_token: The current stream token (ie the upper limit). Updates up to this
	1161	stream id will be included (modulo the 'limit' param)
	1162
	1163	target_row_count: The number of rows to try to return. If more rows are
	1164	available, we will set 'limited' in the result. In the event of a large
	1165	batch, we may return more rows than this.
	1166	Returns:
	1167	A triplet `(updates, new_last_token, limited)`, where:
	1168	* `updates` is a list of database tuples.
	1169	* `new_last_token` is the new position in stream.
	1170	* `limited` is whether there are more updates to fetch.
	1171	"""
	1172
	1173	def get_all_updated_current_state_deltas_txn(txn):
	1174	sql = """
	1175	SELECT stream_id, room_id, type, state_key, event_id
	1176	FROM current_state_delta_stream
	1177	WHERE ? < stream_id AND stream_id <= ?
	1178	ORDER BY stream_id ASC LIMIT ?
	1179	"""
	1180	txn.execute(sql, (from_token, to_token, target_row_count))
	1181	return txn.fetchall()
	1182
	1183	def get_deltas_for_stream_id_txn(txn, stream_id):
	1184	sql = """
	1185	SELECT stream_id, room_id, type, state_key, event_id
	1186	FROM current_state_delta_stream
	1187	WHERE stream_id = ?
	1188	"""
	1189	txn.execute(sql, [stream_id])
	1190	return txn.fetchall()
	1191
	1192	# we need to make sure that, for every stream id in the results, we get all
	1193	# the rows with that stream id.
	1194
	1195	rows = await self.db_pool.runInteraction(
	1196	"get_all_updated_current_state_deltas",
	1197	get_all_updated_current_state_deltas_txn,
	1198	) # type: List[Tuple]
	1199
	1200	# if we've got fewer rows than the limit, we're good
	1201	if len(rows) < target_row_count:
	1202	return rows, to_token, False
	1203
	1204	# we hit the limit, so reduce the upper limit so that we exclude the stream id
	1205	# of the last row in the result.
	1206	assert rows[-1][0] <= to_token
	1207	to_token = rows[-1][0] - 1
	1208
	1209	# search backwards through the list for the point to truncate
	1210	for idx in range(len(rows) - 1, 0, -1):
	1211	if rows[idx - 1][0] <= to_token:
	1212	return rows[:idx], to_token, True
	1213
	1214	# bother. We didn't get a full set of changes for even a single
	1215	# stream id. let's run the query again, without a row limit, but for
	1216	# just one stream id.
	1217	to_token += 1
	1218	rows = await self.db_pool.runInteraction(
	1219	"get_deltas_for_stream_id", get_deltas_for_stream_id_txn, to_token
	1220	)
	1221
	1222	return rows, to_token, True
	1223
	1224	@cached(num_args=5, max_entries=10)
	1225	def get_all_new_events(
	1226	self,
	1227	last_backfill_id,
	1228	last_forward_id,
	1229	current_backfill_id,
	1230	current_forward_id,
	1231	limit,
	1232	):
	1233	"""Get all the new events that have arrived at the server either as
	1234	new events or as backfilled events"""
	1235	have_backfill_events = last_backfill_id != current_backfill_id
	1236	have_forward_events = last_forward_id != current_forward_id
	1237
	1238	if not have_backfill_events and not have_forward_events:
	1239	return defer.succeed(AllNewEventsResult([], [], [], [], []))
	1240
	1241	def get_all_new_events_txn(txn):
	1242	sql = (
	1243	"SELECT e.stream_ordering, e.event_id, e.room_id, e.type,"
	1244	" state_key, redacts"
	1245	" FROM events AS e"
	1246	" LEFT JOIN redactions USING (event_id)"
	1247	" LEFT JOIN state_events USING (event_id)"
	1248	" WHERE ? < stream_ordering AND stream_ordering <= ?"
	1249	" ORDER BY stream_ordering ASC"
	1250	" LIMIT ?"
	1251	)
	1252	if have_forward_events:
	1253	txn.execute(sql, (last_forward_id, current_forward_id, limit))
	1254	new_forward_events = txn.fetchall()
	1255
	1256	if len(new_forward_events) == limit:
	1257	upper_bound = new_forward_events[-1][0]
	1258	else:
	1259	upper_bound = current_forward_id
	1260
	1261	sql = (
	1262	"SELECT event_stream_ordering, event_id, state_group"
	1263	" FROM ex_outlier_stream"
	1264	" WHERE ? > event_stream_ordering"
	1265	" AND event_stream_ordering >= ?"
	1266	" ORDER BY event_stream_ordering DESC"
	1267	)
	1268	txn.execute(sql, (last_forward_id, upper_bound))
	1269	forward_ex_outliers = txn.fetchall()
	1270	else:
	1271	new_forward_events = []
	1272	forward_ex_outliers = []
	1273
	1274	sql = (
	1275	"SELECT -e.stream_ordering, e.event_id, e.room_id, e.type,"
	1276	" state_key, redacts"
	1277	" FROM events AS e"
	1278	" LEFT JOIN redactions USING (event_id)"
	1279	" LEFT JOIN state_events USING (event_id)"
	1280	" WHERE ? > stream_ordering AND stream_ordering >= ?"
	1281	" ORDER BY stream_ordering DESC"
	1282	" LIMIT ?"
	1283	)
	1284	if have_backfill_events:
	1285	txn.execute(sql, (-last_backfill_id, -current_backfill_id, limit))
	1286	new_backfill_events = txn.fetchall()
	1287
	1288	if len(new_backfill_events) == limit:
	1289	upper_bound = new_backfill_events[-1][0]
	1290	else:
	1291	upper_bound = current_backfill_id
	1292
	1293	sql = (
	1294	"SELECT -event_stream_ordering, event_id, state_group"
	1295	" FROM ex_outlier_stream"
	1296	" WHERE ? > event_stream_ordering"
	1297	" AND event_stream_ordering >= ?"
	1298	" ORDER BY event_stream_ordering DESC"
	1299	)
	1300	txn.execute(sql, (-last_backfill_id, -upper_bound))
	1301	backward_ex_outliers = txn.fetchall()
	1302	else:
	1303	new_backfill_events = []
	1304	backward_ex_outliers = []
	1305
	1306	return AllNewEventsResult(
	1307	new_forward_events,
	1308	new_backfill_events,
	1309	forward_ex_outliers,
	1310	backward_ex_outliers,
	1311	)
	1312
	1313	return self.db_pool.runInteraction("get_all_new_events", get_all_new_events_txn)
	1314
	1315	async def is_event_after(self, event_id1, event_id2):
	1316	"""Returns True if event_id1 is after event_id2 in the stream
	1317	"""
	1318	to_1, so_1 = await self.get_event_ordering(event_id1)
	1319	to_2, so_2 = await self.get_event_ordering(event_id2)
	1320	return (to_1, so_1) > (to_2, so_2)
	1321
	1322	@cachedInlineCallbacks(max_entries=5000)
	1323	def get_event_ordering(self, event_id):
	1324	res = yield self.db_pool.simple_select_one(
	1325	table="events",
	1326	retcols=["topological_ordering", "stream_ordering"],
	1327	keyvalues={"event_id": event_id},
	1328	allow_none=True,
	1329	)
	1330
	1331	if not res:
	1332	raise SynapseError(404, "Could not find event %s" % (event_id,))
	1333
	1334	return (int(res["topological_ordering"]), int(res["stream_ordering"]))
	1335
	1336	def get_next_event_to_expire(self):
	1337	"""Retrieve the entry with the lowest expiry timestamp in the event_expiry
	1338	table, or None if there's no more event to expire.
	1339
	1340	Returns: Deferred[Optional[Tuple[str, int]]]
	1341	A tuple containing the event ID as its first element and an expiry timestamp
	1342	as its second one, if there's at least one row in the event_expiry table.
	1343	None otherwise.
	1344	"""
	1345
	1346	def get_next_event_to_expire_txn(txn):
	1347	txn.execute(
	1348	"""
	1349	SELECT event_id, expiry_ts FROM event_expiry
	1350	ORDER BY expiry_ts ASC LIMIT 1
	1351	"""
	1352	)
	1353
	1354	return txn.fetchone()
	1355
	1356	return self.db_pool.runInteraction(
	1357	desc="get_next_event_to_expire", func=get_next_event_to_expire_txn
	1358	)
	1359
	1360
	1361	AllNewEventsResult = namedtuple(
	1362	"AllNewEventsResult",
	1363	[
	1364	"new_forward_events",
	1365	"new_backfill_events",
	1366	"forward_ex_outliers",
	1367	"backward_ex_outliers",
	1368	],
	1369	)

+74

-0

synapse/storage/databases/main/filtering.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2015, 2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from canonicaljson import encode_canonical_json
	16
	17	from synapse.api.errors import Codes, SynapseError
	18	from synapse.storage._base import SQLBaseStore, db_to_json
	19	from synapse.util.caches.descriptors import cached
	20
	21
	22	class FilteringStore(SQLBaseStore):
	23	@cached(num_args=2)
	24	async def get_user_filter(self, user_localpart, filter_id):
	25	# filter_id is BIGINT UNSIGNED, so if it isn't a number, fail
	26	# with a coherent error message rather than 500 M_UNKNOWN.
	27	try:
	28	int(filter_id)
	29	except ValueError:
	30	raise SynapseError(400, "Invalid filter ID", Codes.INVALID_PARAM)
	31
	32	def_json = await self.db_pool.simple_select_one_onecol(
	33	table="user_filters",
	34	keyvalues={"user_id": user_localpart, "filter_id": filter_id},
	35	retcol="filter_json",
	36	allow_none=False,
	37	desc="get_user_filter",
	38	)
	39
	40	return db_to_json(def_json)
	41
	42	def add_user_filter(self, user_localpart, user_filter):
	43	def_json = encode_canonical_json(user_filter)
	44
	45	# Need an atomic transaction to SELECT the maximal ID so far then
	46	# INSERT a new one
	47	def _do_txn(txn):
	48	sql = (
	49	"SELECT filter_id FROM user_filters "
	50	"WHERE user_id = ? AND filter_json = ?"
	51	)
	52	txn.execute(sql, (user_localpart, bytearray(def_json)))
	53	filter_id_response = txn.fetchone()
	54	if filter_id_response is not None:
	55	return filter_id_response[0]
	56
	57	sql = "SELECT MAX(filter_id) FROM user_filters WHERE user_id = ?"
	58	txn.execute(sql, (user_localpart,))
	59	max_id = txn.fetchone()[0]
	60	if max_id is None:
	61	filter_id = 0
	62	else:
	63	filter_id = max_id + 1
	64
	65	sql = (
	66	"INSERT INTO user_filters (user_id, filter_id, filter_json)"
	67	"VALUES(?, ?, ?)"
	68	)
	69	txn.execute(sql, (user_localpart, filter_id, bytearray(def_json)))
	70
	71	return filter_id
	72
	73	return self.db_pool.runInteraction("add_user_filter", _do_txn)

+1288

-0

synapse/storage/databases/main/group_server.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2017 Vector Creations Ltd
	2	# Copyright 2018 New Vector Ltd
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	from typing import List, Optional, Tuple
	17
	18	from synapse.api.errors import SynapseError
	19	from synapse.storage._base import SQLBaseStore, db_to_json
	20	from synapse.types import JsonDict
	21	from synapse.util import json_encoder
	22
	23	# The category ID for the "default" category. We don't store as null in the
	24	# database to avoid the fun of null != null
	25	_DEFAULT_CATEGORY_ID = ""
	26	_DEFAULT_ROLE_ID = ""
	27
	28
	29	class GroupServerWorkerStore(SQLBaseStore):
	30	def get_group(self, group_id):
	31	return self.db_pool.simple_select_one(
	32	table="groups",
	33	keyvalues={"group_id": group_id},
	34	retcols=(
	35	"name",
	36	"short_description",
	37	"long_description",
	38	"avatar_url",
	39	"is_public",
	40	"join_policy",
	41	),
	42	allow_none=True,
	43	desc="get_group",
	44	)
	45
	46	def get_users_in_group(self, group_id, include_private=False):
	47	# TODO: Pagination
	48
	49	keyvalues = {"group_id": group_id}
	50	if not include_private:
	51	keyvalues["is_public"] = True
	52
	53	return self.db_pool.simple_select_list(
	54	table="group_users",
	55	keyvalues=keyvalues,
	56	retcols=("user_id", "is_public", "is_admin"),
	57	desc="get_users_in_group",
	58	)
	59
	60	def get_invited_users_in_group(self, group_id):
	61	# TODO: Pagination
	62
	63	return self.db_pool.simple_select_onecol(
	64	table="group_invites",
	65	keyvalues={"group_id": group_id},
	66	retcol="user_id",
	67	desc="get_invited_users_in_group",
	68	)
	69
	70	def get_rooms_in_group(self, group_id: str, include_private: bool = False):
	71	"""Retrieve the rooms that belong to a given group. Does not return rooms that
	72	lack members.
	73
	74	Args:
	75	group_id: The ID of the group to query for rooms
	76	include_private: Whether to return private rooms in results
	77
	78	Returns:
	79	Deferred[List[Dict[str, str\|bool]]]: A list of dictionaries, each in the
	80	form of:
	81
	82	{
	83	"room_id": "!a_room_id:example.com", # The ID of the room
	84	"is_public": False # Whether this is a public room or not
	85	}
	86	"""
	87	# TODO: Pagination
	88
	89	def _get_rooms_in_group_txn(txn):
	90	sql = """
	91	SELECT room_id, is_public FROM group_rooms
	92	WHERE group_id = ?
	93	AND room_id IN (
	94	SELECT group_rooms.room_id FROM group_rooms
	95	LEFT JOIN room_stats_current ON
	96	group_rooms.room_id = room_stats_current.room_id
	97	AND joined_members > 0
	98	AND local_users_in_room > 0
	99	LEFT JOIN rooms ON
	100	group_rooms.room_id = rooms.room_id
	101	AND (room_version <> '') = ?
	102	)
	103	"""
	104	args = [group_id, False]
	105
	106	if not include_private:
	107	sql += " AND is_public = ?"
	108	args += [True]
	109
	110	txn.execute(sql, args)
	111
	112	return [
	113	{"room_id": room_id, "is_public": is_public}
	114	for room_id, is_public in txn
	115	]
	116
	117	return self.db_pool.runInteraction(
	118	"get_rooms_in_group", _get_rooms_in_group_txn
	119	)
	120
	121	def get_rooms_for_summary_by_category(
	122	self, group_id: str, include_private: bool = False,
	123	):
	124	"""Get the rooms and categories that should be included in a summary request
	125
	126	Args:
	127	group_id: The ID of the group to query the summary for
	128	include_private: Whether to return private rooms in results
	129
	130	Returns:
	131	Deferred[Tuple[List, Dict]]: A tuple containing:
	132
	133	* A list of dictionaries with the keys:
	134	* "room_id": str, the room ID
	135	* "is_public": bool, whether the room is public
	136	* "category_id": str\|None, the category ID if set, else None
	137	* "order": int, the sort order of rooms
	138
	139	* A dictionary with the key:
	140	* category_id (str): a dictionary with the keys:
	141	* "is_public": bool, whether the category is public
	142	* "profile": str, the category profile
	143	* "order": int, the sort order of rooms in this category
	144	"""
	145
	146	def _get_rooms_for_summary_txn(txn):
	147	keyvalues = {"group_id": group_id}
	148	if not include_private:
	149	keyvalues["is_public"] = True
	150
	151	sql = """
	152	SELECT room_id, is_public, category_id, room_order
	153	FROM group_summary_rooms
	154	WHERE group_id = ?
	155	AND room_id IN (
	156	SELECT group_rooms.room_id FROM group_rooms
	157	LEFT JOIN room_stats_current ON
	158	group_rooms.room_id = room_stats_current.room_id
	159	AND joined_members > 0
	160	AND local_users_in_room > 0
	161	LEFT JOIN rooms ON
	162	group_rooms.room_id = rooms.room_id
	163	AND (room_version <> '') = ?
	164	)
	165	"""
	166
	167	if not include_private:
	168	sql += " AND is_public = ?"
	169	txn.execute(sql, (group_id, False, True))
	170	else:
	171	txn.execute(sql, (group_id, False))
	172
	173	rooms = [
	174	{
	175	"room_id": row[0],
	176	"is_public": row[1],
	177	"category_id": row[2] if row[2] != _DEFAULT_CATEGORY_ID else None,
	178	"order": row[3],
	179	}
	180	for row in txn
	181	]
	182
	183	sql = """
	184	SELECT category_id, is_public, profile, cat_order
	185	FROM group_summary_room_categories
	186	INNER JOIN group_room_categories USING (group_id, category_id)
	187	WHERE group_id = ?
	188	"""
	189
	190	if not include_private:
	191	sql += " AND is_public = ?"
	192	txn.execute(sql, (group_id, True))
	193	else:
	194	txn.execute(sql, (group_id,))
	195
	196	categories = {
	197	row[0]: {
	198	"is_public": row[1],
	199	"profile": db_to_json(row[2]),
	200	"order": row[3],
	201	}
	202	for row in txn
	203	}
	204
	205	return rooms, categories
	206
	207	return self.db_pool.runInteraction(
	208	"get_rooms_for_summary", _get_rooms_for_summary_txn
	209	)
	210
	211	async def get_group_categories(self, group_id):
	212	rows = await self.db_pool.simple_select_list(
	213	table="group_room_categories",
	214	keyvalues={"group_id": group_id},
	215	retcols=("category_id", "is_public", "profile"),
	216	desc="get_group_categories",
	217	)
	218
	219	return {
	220	row["category_id"]: {
	221	"is_public": row["is_public"],
	222	"profile": db_to_json(row["profile"]),
	223	}
	224	for row in rows
	225	}
	226
	227	async def get_group_category(self, group_id, category_id):
	228	category = await self.db_pool.simple_select_one(
	229	table="group_room_categories",
	230	keyvalues={"group_id": group_id, "category_id": category_id},
	231	retcols=("is_public", "profile"),
	232	desc="get_group_category",
	233	)
	234
	235	category["profile"] = db_to_json(category["profile"])
	236
	237	return category
	238
	239	async def get_group_roles(self, group_id):
	240	rows = await self.db_pool.simple_select_list(
	241	table="group_roles",
	242	keyvalues={"group_id": group_id},
	243	retcols=("role_id", "is_public", "profile"),
	244	desc="get_group_roles",
	245	)
	246
	247	return {
	248	row["role_id"]: {
	249	"is_public": row["is_public"],
	250	"profile": db_to_json(row["profile"]),
	251	}
	252	for row in rows
	253	}
	254
	255	async def get_group_role(self, group_id, role_id):
	256	role = await self.db_pool.simple_select_one(
	257	table="group_roles",
	258	keyvalues={"group_id": group_id, "role_id": role_id},
	259	retcols=("is_public", "profile"),
	260	desc="get_group_role",
	261	)
	262
	263	role["profile"] = db_to_json(role["profile"])
	264
	265	return role
	266
	267	def get_local_groups_for_room(self, room_id):
	268	"""Get all of the local group that contain a given room
	269	Args:
	270	room_id (str): The ID of a room
	271	Returns:
	272	Deferred[list[str]]: A twisted.Deferred containing a list of group ids
	273	containing this room
	274	"""
	275	return self.db_pool.simple_select_onecol(
	276	table="group_rooms",
	277	keyvalues={"room_id": room_id},
	278	retcol="group_id",
	279	desc="get_local_groups_for_room",
	280	)
	281
	282	def get_users_for_summary_by_role(self, group_id, include_private=False):
	283	"""Get the users and roles that should be included in a summary request
	284
	285	Returns ([users], [roles])
	286	"""
	287
	288	def _get_users_for_summary_txn(txn):
	289	keyvalues = {"group_id": group_id}
	290	if not include_private:
	291	keyvalues["is_public"] = True
	292
	293	sql = """
	294	SELECT user_id, is_public, role_id, user_order
	295	FROM group_summary_users
	296	WHERE group_id = ?
	297	"""
	298
	299	if not include_private:
	300	sql += " AND is_public = ?"
	301	txn.execute(sql, (group_id, True))
	302	else:
	303	txn.execute(sql, (group_id,))
	304
	305	users = [
	306	{
	307	"user_id": row[0],
	308	"is_public": row[1],
	309	"role_id": row[2] if row[2] != _DEFAULT_ROLE_ID else None,
	310	"order": row[3],
	311	}
	312	for row in txn
	313	]
	314
	315	sql = """
	316	SELECT role_id, is_public, profile, role_order
	317	FROM group_summary_roles
	318	INNER JOIN group_roles USING (group_id, role_id)
	319	WHERE group_id = ?
	320	"""
	321
	322	if not include_private:
	323	sql += " AND is_public = ?"
	324	txn.execute(sql, (group_id, True))
	325	else:
	326	txn.execute(sql, (group_id,))
	327
	328	roles = {
	329	row[0]: {
	330	"is_public": row[1],
	331	"profile": db_to_json(row[2]),
	332	"order": row[3],
	333	}
	334	for row in txn
	335	}
	336
	337	return users, roles
	338
	339	return self.db_pool.runInteraction(
	340	"get_users_for_summary_by_role", _get_users_for_summary_txn
	341	)
	342
	343	def is_user_in_group(self, user_id, group_id):
	344	return self.db_pool.simple_select_one_onecol(
	345	table="group_users",
	346	keyvalues={"group_id": group_id, "user_id": user_id},
	347	retcol="user_id",
	348	allow_none=True,
	349	desc="is_user_in_group",
	350	).addCallback(lambda r: bool(r))
	351
	352	def is_user_admin_in_group(self, group_id, user_id):
	353	return self.db_pool.simple_select_one_onecol(
	354	table="group_users",
	355	keyvalues={"group_id": group_id, "user_id": user_id},
	356	retcol="is_admin",
	357	allow_none=True,
	358	desc="is_user_admin_in_group",
	359	)
	360
	361	def is_user_invited_to_local_group(self, group_id, user_id):
	362	"""Has the group server invited a user?
	363	"""
	364	return self.db_pool.simple_select_one_onecol(
	365	table="group_invites",
	366	keyvalues={"group_id": group_id, "user_id": user_id},
	367	retcol="user_id",
	368	desc="is_user_invited_to_local_group",
	369	allow_none=True,
	370	)
	371
	372	def get_users_membership_info_in_group(self, group_id, user_id):
	373	"""Get a dict describing the membership of a user in a group.
	374
	375	Example if joined:
	376
	377	{
	378	"membership": "join",
	379	"is_public": True,
	380	"is_privileged": False,
	381	}
	382
	383	Returns an empty dict if the user is not join/invite/etc
	384	"""
	385
	386	def _get_users_membership_in_group_txn(txn):
	387	row = self.db_pool.simple_select_one_txn(
	388	txn,
	389	table="group_users",
	390	keyvalues={"group_id": group_id, "user_id": user_id},
	391	retcols=("is_admin", "is_public"),
	392	allow_none=True,
	393	)
	394
	395	if row:
	396	return {
	397	"membership": "join",
	398	"is_public": row["is_public"],
	399	"is_privileged": row["is_admin"],
	400	}
	401
	402	row = self.db_pool.simple_select_one_onecol_txn(
	403	txn,
	404	table="group_invites",
	405	keyvalues={"group_id": group_id, "user_id": user_id},
	406	retcol="user_id",
	407	allow_none=True,
	408	)
	409
	410	if row:
	411	return {"membership": "invite"}
	412
	413	return {}
	414
	415	return self.db_pool.runInteraction(
	416	"get_users_membership_info_in_group", _get_users_membership_in_group_txn
	417	)
	418
	419	def get_publicised_groups_for_user(self, user_id):
	420	"""Get all groups a user is publicising
	421	"""
	422	return self.db_pool.simple_select_onecol(
	423	table="local_group_membership",
	424	keyvalues={"user_id": user_id, "membership": "join", "is_publicised": True},
	425	retcol="group_id",
	426	desc="get_publicised_groups_for_user",
	427	)
	428
	429	def get_attestations_need_renewals(self, valid_until_ms):
	430	"""Get all attestations that need to be renewed until givent time
	431	"""
	432
	433	def _get_attestations_need_renewals_txn(txn):
	434	sql = """
	435	SELECT group_id, user_id FROM group_attestations_renewals
	436	WHERE valid_until_ms <= ?
	437	"""
	438	txn.execute(sql, (valid_until_ms,))
	439	return self.db_pool.cursor_to_dict(txn)
	440
	441	return self.db_pool.runInteraction(
	442	"get_attestations_need_renewals", _get_attestations_need_renewals_txn
	443	)
	444
	445	async def get_remote_attestation(self, group_id, user_id):
	446	"""Get the attestation that proves the remote agrees that the user is
	447	in the group.
	448	"""
	449	row = await self.db_pool.simple_select_one(
	450	table="group_attestations_remote",
	451	keyvalues={"group_id": group_id, "user_id": user_id},
	452	retcols=("valid_until_ms", "attestation_json"),
	453	desc="get_remote_attestation",
	454	allow_none=True,
	455	)
	456
	457	now = int(self._clock.time_msec())
	458	if row and now < row["valid_until_ms"]:
	459	return db_to_json(row["attestation_json"])
	460
	461	return None
	462
	463	def get_joined_groups(self, user_id):
	464	return self.db_pool.simple_select_onecol(
	465	table="local_group_membership",
	466	keyvalues={"user_id": user_id, "membership": "join"},
	467	retcol="group_id",
	468	desc="get_joined_groups",
	469	)
	470
	471	def get_all_groups_for_user(self, user_id, now_token):
	472	def _get_all_groups_for_user_txn(txn):
	473	sql = """
	474	SELECT group_id, type, membership, u.content
	475	FROM local_group_updates AS u
	476	INNER JOIN local_group_membership USING (group_id, user_id)
	477	WHERE user_id = ? AND membership != 'leave'
	478	AND stream_id <= ?
	479	"""
	480	txn.execute(sql, (user_id, now_token))
	481	return [
	482	{
	483	"group_id": row[0],
	484	"type": row[1],
	485	"membership": row[2],
	486	"content": db_to_json(row[3]),
	487	}
	488	for row in txn
	489	]
	490
	491	return self.db_pool.runInteraction(
	492	"get_all_groups_for_user", _get_all_groups_for_user_txn
	493	)
	494
	495	async def get_groups_changes_for_user(self, user_id, from_token, to_token):
	496	from_token = int(from_token)
	497	has_changed = self._group_updates_stream_cache.has_entity_changed(
	498	user_id, from_token
	499	)
	500	if not has_changed:
	501	return []
	502
	503	def _get_groups_changes_for_user_txn(txn):
	504	sql = """
	505	SELECT group_id, membership, type, u.content
	506	FROM local_group_updates AS u
	507	INNER JOIN local_group_membership USING (group_id, user_id)
	508	WHERE user_id = ? AND ? < stream_id AND stream_id <= ?
	509	"""
	510	txn.execute(sql, (user_id, from_token, to_token))
	511	return [
	512	{
	513	"group_id": group_id,
	514	"membership": membership,
	515	"type": gtype,
	516	"content": db_to_json(content_json),
	517	}
	518	for group_id, membership, gtype, content_json in txn
	519	]
	520
	521	return await self.db_pool.runInteraction(
	522	"get_groups_changes_for_user", _get_groups_changes_for_user_txn
	523	)
	524
	525	async def get_all_groups_changes(
	526	self, instance_name: str, last_id: int, current_id: int, limit: int
	527	) -> Tuple[List[Tuple[int, tuple]], int, bool]:
	528	"""Get updates for groups replication stream.
	529
	530	Args:
	531	instance_name: The writer we want to fetch updates from. Unused
	532	here since there is only ever one writer.
	533	last_id: The token to fetch updates from. Exclusive.
	534	current_id: The token to fetch updates up to. Inclusive.
	535	limit: The requested limit for the number of rows to return. The
	536	function may return more or fewer rows.
	537
	538	Returns:
	539	A tuple consisting of: the updates, a token to use to fetch
	540	subsequent updates, and whether we returned fewer rows than exists
	541	between the requested tokens due to the limit.
	542
	543	The token returned can be used in a subsequent call to this
	544	function to get further updatees.
	545
	546	The updates are a list of 2-tuples of stream ID and the row data
	547	"""
	548
	549	last_id = int(last_id)
	550	has_changed = self._group_updates_stream_cache.has_any_entity_changed(last_id)
	551
	552	if not has_changed:
	553	return [], current_id, False
	554
	555	def _get_all_groups_changes_txn(txn):
	556	sql = """
	557	SELECT stream_id, group_id, user_id, type, content
	558	FROM local_group_updates
	559	WHERE ? < stream_id AND stream_id <= ?
	560	LIMIT ?
	561	"""
	562	txn.execute(sql, (last_id, current_id, limit))
	563	updates = [
	564	(stream_id, (group_id, user_id, gtype, db_to_json(content_json)))
	565	for stream_id, group_id, user_id, gtype, content_json in txn
	566	]
	567
	568	limited = False
	569	upto_token = current_id
	570	if len(updates) >= limit:
	571	upto_token = updates[-1][0]
	572	limited = True
	573
	574	return updates, upto_token, limited
	575
	576	return await self.db_pool.runInteraction(
	577	"get_all_groups_changes", _get_all_groups_changes_txn
	578	)
	579
	580
	581	class GroupServerStore(GroupServerWorkerStore):
	582	def set_group_join_policy(self, group_id, join_policy):
	583	"""Set the join policy of a group.
	584
	585	join_policy can be one of:
	586	* "invite"
	587	* "open"
	588	"""
	589	return self.db_pool.simple_update_one(
	590	table="groups",
	591	keyvalues={"group_id": group_id},
	592	updatevalues={"join_policy": join_policy},
	593	desc="set_group_join_policy",
	594	)
	595
	596	def add_room_to_summary(self, group_id, room_id, category_id, order, is_public):
	597	return self.db_pool.runInteraction(
	598	"add_room_to_summary",
	599	self._add_room_to_summary_txn,
	600	group_id,
	601	room_id,
	602	category_id,
	603	order,
	604	is_public,
	605	)
	606
	607	def _add_room_to_summary_txn(
	608	self, txn, group_id, room_id, category_id, order, is_public
	609	):
	610	"""Add (or update) room's entry in summary.
	611
	612	Args:
	613	group_id (str)
	614	room_id (str)
	615	category_id (str): If not None then adds the category to the end of
	616	the summary if its not already there. [Optional]
	617	order (int): If not None inserts the room at that position, e.g.
	618	an order of 1 will put the room first. Otherwise, the room gets
	619	added to the end.
	620	"""
	621	room_in_group = self.db_pool.simple_select_one_onecol_txn(
	622	txn,
	623	table="group_rooms",
	624	keyvalues={"group_id": group_id, "room_id": room_id},
	625	retcol="room_id",
	626	allow_none=True,
	627	)
	628	if not room_in_group:
	629	raise SynapseError(400, "room not in group")
	630
	631	if category_id is None:
	632	category_id = _DEFAULT_CATEGORY_ID
	633	else:
	634	cat_exists = self.db_pool.simple_select_one_onecol_txn(
	635	txn,
	636	table="group_room_categories",
	637	keyvalues={"group_id": group_id, "category_id": category_id},
	638	retcol="group_id",
	639	allow_none=True,
	640	)
	641	if not cat_exists:
	642	raise SynapseError(400, "Category doesn't exist")
	643
	644	# TODO: Check category is part of summary already
	645	cat_exists = self.db_pool.simple_select_one_onecol_txn(
	646	txn,
	647	table="group_summary_room_categories",
	648	keyvalues={"group_id": group_id, "category_id": category_id},
	649	retcol="group_id",
	650	allow_none=True,
	651	)
	652	if not cat_exists:
	653	# If not, add it with an order larger than all others
	654	txn.execute(
	655	"""
	656	INSERT INTO group_summary_room_categories
	657	(group_id, category_id, cat_order)
	658	SELECT ?, ?, COALESCE(MAX(cat_order), 0) + 1
	659	FROM group_summary_room_categories
	660	WHERE group_id = ? AND category_id = ?
	661	""",
	662	(group_id, category_id, group_id, category_id),
	663	)
	664
	665	existing = self.db_pool.simple_select_one_txn(
	666	txn,
	667	table="group_summary_rooms",
	668	keyvalues={
	669	"group_id": group_id,
	670	"room_id": room_id,
	671	"category_id": category_id,
	672	},
	673	retcols=("room_order", "is_public"),
	674	allow_none=True,
	675	)
	676
	677	if order is not None:
	678	# Shuffle other room orders that come after the given order
	679	sql = """
	680	UPDATE group_summary_rooms SET room_order = room_order + 1
	681	WHERE group_id = ? AND category_id = ? AND room_order >= ?
	682	"""
	683	txn.execute(sql, (group_id, category_id, order))
	684	elif not existing:
	685	sql = """
	686	SELECT COALESCE(MAX(room_order), 0) + 1 FROM group_summary_rooms
	687	WHERE group_id = ? AND category_id = ?
	688	"""
	689	txn.execute(sql, (group_id, category_id))
	690	(order,) = txn.fetchone()
	691
	692	if existing:
	693	to_update = {}
	694	if order is not None:
	695	to_update["room_order"] = order
	696	if is_public is not None:
	697	to_update["is_public"] = is_public
	698	self.db_pool.simple_update_txn(
	699	txn,
	700	table="group_summary_rooms",
	701	keyvalues={
	702	"group_id": group_id,
	703	"category_id": category_id,
	704	"room_id": room_id,
	705	},
	706	values=to_update,
	707	)
	708	else:
	709	if is_public is None:
	710	is_public = True
	711
	712	self.db_pool.simple_insert_txn(
	713	txn,
	714	table="group_summary_rooms",
	715	values={
	716	"group_id": group_id,
	717	"category_id": category_id,
	718	"room_id": room_id,
	719	"room_order": order,
	720	"is_public": is_public,
	721	},
	722	)
	723
	724	def remove_room_from_summary(self, group_id, room_id, category_id):
	725	if category_id is None:
	726	category_id = _DEFAULT_CATEGORY_ID
	727
	728	return self.db_pool.simple_delete(
	729	table="group_summary_rooms",
	730	keyvalues={
	731	"group_id": group_id,
	732	"category_id": category_id,
	733	"room_id": room_id,
	734	},
	735	desc="remove_room_from_summary",
	736	)
	737
	738	def upsert_group_category(self, group_id, category_id, profile, is_public):
	739	"""Add/update room category for group
	740	"""
	741	insertion_values = {}
	742	update_values = {"category_id": category_id} # This cannot be empty
	743
	744	if profile is None:
	745	insertion_values["profile"] = "{}"
	746	else:
	747	update_values["profile"] = json_encoder.encode(profile)
	748
	749	if is_public is None:
	750	insertion_values["is_public"] = True
	751	else:
	752	update_values["is_public"] = is_public
	753
	754	return self.db_pool.simple_upsert(
	755	table="group_room_categories",
	756	keyvalues={"group_id": group_id, "category_id": category_id},
	757	values=update_values,
	758	insertion_values=insertion_values,
	759	desc="upsert_group_category",
	760	)
	761
	762	def remove_group_category(self, group_id, category_id):
	763	return self.db_pool.simple_delete(
	764	table="group_room_categories",
	765	keyvalues={"group_id": group_id, "category_id": category_id},
	766	desc="remove_group_category",
	767	)
	768
	769	def upsert_group_role(self, group_id, role_id, profile, is_public):
	770	"""Add/remove user role
	771	"""
	772	insertion_values = {}
	773	update_values = {"role_id": role_id} # This cannot be empty
	774
	775	if profile is None:
	776	insertion_values["profile"] = "{}"
	777	else:
	778	update_values["profile"] = json_encoder.encode(profile)
	779
	780	if is_public is None:
	781	insertion_values["is_public"] = True
	782	else:
	783	update_values["is_public"] = is_public
	784
	785	return self.db_pool.simple_upsert(
	786	table="group_roles",
	787	keyvalues={"group_id": group_id, "role_id": role_id},
	788	values=update_values,
	789	insertion_values=insertion_values,
	790	desc="upsert_group_role",
	791	)
	792
	793	def remove_group_role(self, group_id, role_id):
	794	return self.db_pool.simple_delete(
	795	table="group_roles",
	796	keyvalues={"group_id": group_id, "role_id": role_id},
	797	desc="remove_group_role",
	798	)
	799
	800	def add_user_to_summary(self, group_id, user_id, role_id, order, is_public):
	801	return self.db_pool.runInteraction(
	802	"add_user_to_summary",
	803	self._add_user_to_summary_txn,
	804	group_id,
	805	user_id,
	806	role_id,
	807	order,
	808	is_public,
	809	)
	810
	811	def _add_user_to_summary_txn(
	812	self, txn, group_id, user_id, role_id, order, is_public
	813	):
	814	"""Add (or update) user's entry in summary.
	815
	816	Args:
	817	group_id (str)
	818	user_id (str)
	819	role_id (str): If not None then adds the role to the end of
	820	the summary if its not already there. [Optional]
	821	order (int): If not None inserts the user at that position, e.g.
	822	an order of 1 will put the user first. Otherwise, the user gets
	823	added to the end.
	824	"""
	825	user_in_group = self.db_pool.simple_select_one_onecol_txn(
	826	txn,
	827	table="group_users",
	828	keyvalues={"group_id": group_id, "user_id": user_id},
	829	retcol="user_id",
	830	allow_none=True,
	831	)
	832	if not user_in_group:
	833	raise SynapseError(400, "user not in group")
	834
	835	if role_id is None:
	836	role_id = _DEFAULT_ROLE_ID
	837	else:
	838	role_exists = self.db_pool.simple_select_one_onecol_txn(
	839	txn,
	840	table="group_roles",
	841	keyvalues={"group_id": group_id, "role_id": role_id},
	842	retcol="group_id",
	843	allow_none=True,
	844	)
	845	if not role_exists:
	846	raise SynapseError(400, "Role doesn't exist")
	847
	848	# TODO: Check role is part of the summary already
	849	role_exists = self.db_pool.simple_select_one_onecol_txn(
	850	txn,
	851	table="group_summary_roles",
	852	keyvalues={"group_id": group_id, "role_id": role_id},
	853	retcol="group_id",
	854	allow_none=True,
	855	)
	856	if not role_exists:
	857	# If not, add it with an order larger than all others
	858	txn.execute(
	859	"""
	860	INSERT INTO group_summary_roles
	861	(group_id, role_id, role_order)
	862	SELECT ?, ?, COALESCE(MAX(role_order), 0) + 1
	863	FROM group_summary_roles
	864	WHERE group_id = ? AND role_id = ?
	865	""",
	866	(group_id, role_id, group_id, role_id),
	867	)
	868
	869	existing = self.db_pool.simple_select_one_txn(
	870	txn,
	871	table="group_summary_users",
	872	keyvalues={"group_id": group_id, "user_id": user_id, "role_id": role_id},
	873	retcols=("user_order", "is_public"),
	874	allow_none=True,
	875	)
	876
	877	if order is not None:
	878	# Shuffle other users orders that come after the given order
	879	sql = """
	880	UPDATE group_summary_users SET user_order = user_order + 1
	881	WHERE group_id = ? AND role_id = ? AND user_order >= ?
	882	"""
	883	txn.execute(sql, (group_id, role_id, order))
	884	elif not existing:
	885	sql = """
	886	SELECT COALESCE(MAX(user_order), 0) + 1 FROM group_summary_users
	887	WHERE group_id = ? AND role_id = ?
	888	"""
	889	txn.execute(sql, (group_id, role_id))
	890	(order,) = txn.fetchone()
	891
	892	if existing:
	893	to_update = {}
	894	if order is not None:
	895	to_update["user_order"] = order
	896	if is_public is not None:
	897	to_update["is_public"] = is_public
	898	self.db_pool.simple_update_txn(
	899	txn,
	900	table="group_summary_users",
	901	keyvalues={
	902	"group_id": group_id,
	903	"role_id": role_id,
	904	"user_id": user_id,
	905	},
	906	values=to_update,
	907	)
	908	else:
	909	if is_public is None:
	910	is_public = True
	911
	912	self.db_pool.simple_insert_txn(
	913	txn,
	914	table="group_summary_users",
	915	values={
	916	"group_id": group_id,
	917	"role_id": role_id,
	918	"user_id": user_id,
	919	"user_order": order,
	920	"is_public": is_public,
	921	},
	922	)
	923
	924	def remove_user_from_summary(self, group_id, user_id, role_id):
	925	if role_id is None:
	926	role_id = _DEFAULT_ROLE_ID
	927
	928	return self.db_pool.simple_delete(
	929	table="group_summary_users",
	930	keyvalues={"group_id": group_id, "role_id": role_id, "user_id": user_id},
	931	desc="remove_user_from_summary",
	932	)
	933
	934	def add_group_invite(self, group_id, user_id):
	935	"""Record that the group server has invited a user
	936	"""
	937	return self.db_pool.simple_insert(
	938	table="group_invites",
	939	values={"group_id": group_id, "user_id": user_id},
	940	desc="add_group_invite",
	941	)
	942
	943	def add_user_to_group(
	944	self,
	945	group_id,
	946	user_id,
	947	is_admin=False,
	948	is_public=True,
	949	local_attestation=None,
	950	remote_attestation=None,
	951	):
	952	"""Add a user to the group server.
	953
	954	Args:
	955	group_id (str)
	956	user_id (str)
	957	is_admin (bool)
	958	is_public (bool)
	959	local_attestation (dict): The attestation the GS created to give
	960	to the remote server. Optional if the user and group are on the
	961	same server
	962	remote_attestation (dict): The attestation given to GS by remote
	963	server. Optional if the user and group are on the same server
	964	"""
	965
	966	def _add_user_to_group_txn(txn):
	967	self.db_pool.simple_insert_txn(
	968	txn,
	969	table="group_users",
	970	values={
	971	"group_id": group_id,
	972	"user_id": user_id,
	973	"is_admin": is_admin,
	974	"is_public": is_public,
	975	},
	976	)
	977
	978	self.db_pool.simple_delete_txn(
	979	txn,
	980	table="group_invites",
	981	keyvalues={"group_id": group_id, "user_id": user_id},
	982	)
	983
	984	if local_attestation:
	985	self.db_pool.simple_insert_txn(
	986	txn,
	987	table="group_attestations_renewals",
	988	values={
	989	"group_id": group_id,
	990	"user_id": user_id,
	991	"valid_until_ms": local_attestation["valid_until_ms"],
	992	},
	993	)
	994	if remote_attestation:
	995	self.db_pool.simple_insert_txn(
	996	txn,
	997	table="group_attestations_remote",
	998	values={
	999	"group_id": group_id,
	1000	"user_id": user_id,
	1001	"valid_until_ms": remote_attestation["valid_until_ms"],
	1002	"attestation_json": json_encoder.encode(remote_attestation),
	1003	},
	1004	)
	1005
	1006	return self.db_pool.runInteraction("add_user_to_group", _add_user_to_group_txn)
	1007
	1008	def remove_user_from_group(self, group_id, user_id):
	1009	def _remove_user_from_group_txn(txn):
	1010	self.db_pool.simple_delete_txn(
	1011	txn,
	1012	table="group_users",
	1013	keyvalues={"group_id": group_id, "user_id": user_id},
	1014	)
	1015	self.db_pool.simple_delete_txn(
	1016	txn,
	1017	table="group_invites",
	1018	keyvalues={"group_id": group_id, "user_id": user_id},
	1019	)
	1020	self.db_pool.simple_delete_txn(
	1021	txn,
	1022	table="group_attestations_renewals",
	1023	keyvalues={"group_id": group_id, "user_id": user_id},
	1024	)
	1025	self.db_pool.simple_delete_txn(
	1026	txn,
	1027	table="group_attestations_remote",
	1028	keyvalues={"group_id": group_id, "user_id": user_id},
	1029	)
	1030	self.db_pool.simple_delete_txn(
	1031	txn,
	1032	table="group_summary_users",
	1033	keyvalues={"group_id": group_id, "user_id": user_id},
	1034	)
	1035
	1036	return self.db_pool.runInteraction(
	1037	"remove_user_from_group", _remove_user_from_group_txn
	1038	)
	1039
	1040	def add_room_to_group(self, group_id, room_id, is_public):
	1041	return self.db_pool.simple_insert(
	1042	table="group_rooms",
	1043	values={"group_id": group_id, "room_id": room_id, "is_public": is_public},
	1044	desc="add_room_to_group",
	1045	)
	1046
	1047	def update_room_in_group_visibility(self, group_id, room_id, is_public):
	1048	return self.db_pool.simple_update(
	1049	table="group_rooms",
	1050	keyvalues={"group_id": group_id, "room_id": room_id},
	1051	updatevalues={"is_public": is_public},
	1052	desc="update_room_in_group_visibility",
	1053	)
	1054
	1055	def remove_room_from_group(self, group_id, room_id):
	1056	def _remove_room_from_group_txn(txn):
	1057	self.db_pool.simple_delete_txn(
	1058	txn,
	1059	table="group_rooms",
	1060	keyvalues={"group_id": group_id, "room_id": room_id},
	1061	)
	1062
	1063	self.db_pool.simple_delete_txn(
	1064	txn,
	1065	table="group_summary_rooms",
	1066	keyvalues={"group_id": group_id, "room_id": room_id},
	1067	)
	1068
	1069	return self.db_pool.runInteraction(
	1070	"remove_room_from_group", _remove_room_from_group_txn
	1071	)
	1072
	1073	def update_group_publicity(self, group_id, user_id, publicise):
	1074	"""Update whether the user is publicising their membership of the group
	1075	"""
	1076	return self.db_pool.simple_update_one(
	1077	table="local_group_membership",
	1078	keyvalues={"group_id": group_id, "user_id": user_id},
	1079	updatevalues={"is_publicised": publicise},
	1080	desc="update_group_publicity",
	1081	)
	1082
	1083	async def register_user_group_membership(
	1084	self,
	1085	group_id: str,
	1086	user_id: str,
	1087	membership: str,
	1088	is_admin: bool = False,
	1089	content: JsonDict = {},
	1090	local_attestation: Optional[dict] = None,
	1091	remote_attestation: Optional[dict] = None,
	1092	is_publicised: bool = False,
	1093	) -> int:
	1094	"""Registers that a local user is a member of a (local or remote) group.
	1095
	1096	Args:
	1097	group_id: The group the member is being added to.
	1098	user_id: THe user ID to add to the group.
	1099	membership: The type of group membership.
	1100	is_admin: Whether the user should be added as a group admin.
	1101	content: Content of the membership, e.g. includes the inviter
	1102	if the user has been invited.
	1103	local_attestation: If remote group then store the fact that we
	1104	have given out an attestation, else None.
	1105	remote_attestation: If remote group then store the remote
	1106	attestation from the group, else None.
	1107	is_publicised: Whether this should be publicised.
	1108	"""
	1109
	1110	def _register_user_group_membership_txn(txn, next_id):
	1111	# TODO: Upsert?
	1112	self.db_pool.simple_delete_txn(
	1113	txn,
	1114	table="local_group_membership",
	1115	keyvalues={"group_id": group_id, "user_id": user_id},
	1116	)
	1117	self.db_pool.simple_insert_txn(
	1118	txn,
	1119	table="local_group_membership",
	1120	values={
	1121	"group_id": group_id,
	1122	"user_id": user_id,
	1123	"is_admin": is_admin,
	1124	"membership": membership,
	1125	"is_publicised": is_publicised,
	1126	"content": json_encoder.encode(content),
	1127	},
	1128	)
	1129
	1130	self.db_pool.simple_insert_txn(
	1131	txn,
	1132	table="local_group_updates",
	1133	values={
	1134	"stream_id": next_id,
	1135	"group_id": group_id,
	1136	"user_id": user_id,
	1137	"type": "membership",
	1138	"content": json_encoder.encode(
	1139	{"membership": membership, "content": content}
	1140	),
	1141	},
	1142	)
	1143	self._group_updates_stream_cache.entity_has_changed(user_id, next_id)
	1144
	1145	# TODO: Insert profile to ensure it comes down stream if its a join.
	1146
	1147	if membership == "join":
	1148	if local_attestation:
	1149	self.db_pool.simple_insert_txn(
	1150	txn,
	1151	table="group_attestations_renewals",
	1152	values={
	1153	"group_id": group_id,
	1154	"user_id": user_id,
	1155	"valid_until_ms": local_attestation["valid_until_ms"],
	1156	},
	1157	)
	1158	if remote_attestation:
	1159	self.db_pool.simple_insert_txn(
	1160	txn,
	1161	table="group_attestations_remote",
	1162	values={
	1163	"group_id": group_id,
	1164	"user_id": user_id,
	1165	"valid_until_ms": remote_attestation["valid_until_ms"],
	1166	"attestation_json": json_encoder.encode(remote_attestation),
	1167	},
	1168	)
	1169	else:
	1170	self.db_pool.simple_delete_txn(
	1171	txn,
	1172	table="group_attestations_renewals",
	1173	keyvalues={"group_id": group_id, "user_id": user_id},
	1174	)
	1175	self.db_pool.simple_delete_txn(
	1176	txn,
	1177	table="group_attestations_remote",
	1178	keyvalues={"group_id": group_id, "user_id": user_id},
	1179	)
	1180
	1181	return next_id
	1182
	1183	with self._group_updates_id_gen.get_next() as next_id:
	1184	res = await self.db_pool.runInteraction(
	1185	"register_user_group_membership",
	1186	_register_user_group_membership_txn,
	1187	next_id,
	1188	)
	1189	return res
	1190
	1191	async def create_group(
	1192	self, group_id, user_id, name, avatar_url, short_description, long_description
	1193	) -> None:
	1194	await self.db_pool.simple_insert(
	1195	table="groups",
	1196	values={
	1197	"group_id": group_id,
	1198	"name": name,
	1199	"avatar_url": avatar_url,
	1200	"short_description": short_description,
	1201	"long_description": long_description,
	1202	"is_public": True,
	1203	},
	1204	desc="create_group",
	1205	)
	1206
	1207	async def update_group_profile(self, group_id, profile):
	1208	await self.db_pool.simple_update_one(
	1209	table="groups",
	1210	keyvalues={"group_id": group_id},
	1211	updatevalues=profile,
	1212	desc="update_group_profile",
	1213	)
	1214
	1215	def update_attestation_renewal(self, group_id, user_id, attestation):
	1216	"""Update an attestation that we have renewed
	1217	"""
	1218	return self.db_pool.simple_update_one(
	1219	table="group_attestations_renewals",
	1220	keyvalues={"group_id": group_id, "user_id": user_id},
	1221	updatevalues={"valid_until_ms": attestation["valid_until_ms"]},
	1222	desc="update_attestation_renewal",
	1223	)
	1224
	1225	def update_remote_attestion(self, group_id, user_id, attestation):
	1226	"""Update an attestation that a remote has renewed
	1227	"""
	1228	return self.db_pool.simple_update_one(
	1229	table="group_attestations_remote",
	1230	keyvalues={"group_id": group_id, "user_id": user_id},
	1231	updatevalues={
	1232	"valid_until_ms": attestation["valid_until_ms"],
	1233	"attestation_json": json_encoder.encode(attestation),
	1234	},
	1235	desc="update_remote_attestion",
	1236	)
	1237
	1238	def remove_attestation_renewal(self, group_id, user_id):
	1239	"""Remove an attestation that we thought we should renew, but actually
	1240	shouldn't. Ideally this would never get called as we would never
	1241	incorrectly try and do attestations for local users on local groups.
	1242
	1243	Args:
	1244	group_id (str)
	1245	user_id (str)
	1246	"""
	1247	return self.db_pool.simple_delete(
	1248	table="group_attestations_renewals",
	1249	keyvalues={"group_id": group_id, "user_id": user_id},
	1250	desc="remove_attestation_renewal",
	1251	)
	1252
	1253	def get_group_stream_token(self):
	1254	return self._group_updates_id_gen.get_current_token()
	1255
	1256	def delete_group(self, group_id):
	1257	"""Deletes a group fully from the database.
	1258
	1259	Args:
	1260	group_id (str)
	1261
	1262	Returns:
	1263	Deferred
	1264	"""
	1265
	1266	def _delete_group_txn(txn):
	1267	tables = [
	1268	"groups",
	1269	"group_users",
	1270	"group_invites",
	1271	"group_rooms",
	1272	"group_summary_rooms",
	1273	"group_summary_room_categories",
	1274	"group_room_categories",
	1275	"group_summary_users",
	1276	"group_summary_roles",
	1277	"group_roles",
	1278	"group_attestations_renewals",
	1279	"group_attestations_remote",
	1280	]
	1281
	1282	for table in tables:
	1283	self.db_pool.simple_delete_txn(
	1284	txn, table=table, keyvalues={"group_id": group_id}
	1285	)
	1286
	1287	return self.db_pool.runInteraction("delete_group", _delete_group_txn)

+210

-0

synapse/storage/databases/main/keys.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2019 New Vector Ltd.
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import itertools
	17	import logging
	18
	19	from signedjson.key import decode_verify_key_bytes
	20
	21	from synapse.storage._base import SQLBaseStore
	22	from synapse.storage.keys import FetchKeyResult
	23	from synapse.util.caches.descriptors import cached, cachedList
	24	from synapse.util.iterutils import batch_iter
	25
	26	logger = logging.getLogger(__name__)
	27
	28
	29	db_binary_type = memoryview
	30
	31
	32	class KeyStore(SQLBaseStore):
	33	"""Persistence for signature verification keys
	34	"""
	35
	36	@cached()
	37	def _get_server_verify_key(self, server_name_and_key_id):
	38	raise NotImplementedError()
	39
	40	@cachedList(
	41	cached_method_name="_get_server_verify_key", list_name="server_name_and_key_ids"
	42	)
	43	def get_server_verify_keys(self, server_name_and_key_ids):
	44	"""
	45	Args:
	46	server_name_and_key_ids (iterable[Tuple[str, str]]):
	47	iterable of (server_name, key-id) tuples to fetch keys for
	48
	49	Returns:
	50	Deferred: resolves to dict[Tuple[str, str], FetchKeyResult\|None]:
	51	map from (server_name, key_id) -> FetchKeyResult, or None if the key is
	52	unknown
	53	"""
	54	keys = {}
	55
	56	def _get_keys(txn, batch):
	57	"""Processes a batch of keys to fetch, and adds the result to `keys`."""
	58
	59	# batch_iter always returns tuples so it's safe to do len(batch)
	60	sql = (
	61	"SELECT server_name, key_id, verify_key, ts_valid_until_ms "
	62	"FROM server_signature_keys WHERE 1=0"
	63	) + " OR (server_name=? AND key_id=?)" * len(batch)
	64
	65	txn.execute(sql, tuple(itertools.chain.from_iterable(batch)))
	66
	67	for row in txn:
	68	server_name, key_id, key_bytes, ts_valid_until_ms = row
	69
	70	if ts_valid_until_ms is None:
	71	# Old keys may be stored with a ts_valid_until_ms of null,
	72	# in which case we treat this as if it was set to `0`, i.e.
	73	# it won't match key requests that define a minimum
	74	# `ts_valid_until_ms`.
	75	ts_valid_until_ms = 0
	76
	77	res = FetchKeyResult(
	78	verify_key=decode_verify_key_bytes(key_id, bytes(key_bytes)),
	79	valid_until_ts=ts_valid_until_ms,
	80	)
	81	keys[(server_name, key_id)] = res
	82
	83	def _txn(txn):
	84	for batch in batch_iter(server_name_and_key_ids, 50):
	85	_get_keys(txn, batch)
	86	return keys
	87
	88	return self.db_pool.runInteraction("get_server_verify_keys", _txn)
	89
	90	def store_server_verify_keys(self, from_server, ts_added_ms, verify_keys):
	91	"""Stores NACL verification keys for remote servers.
	92	Args:
	93	from_server (str): Where the verification keys were looked up
	94	ts_added_ms (int): The time to record that the key was added
	95	verify_keys (iterable[tuple[str, str, FetchKeyResult]]):
	96	keys to be stored. Each entry is a triplet of
	97	(server_name, key_id, key).
	98	"""
	99	key_values = []
	100	value_values = []
	101	invalidations = []
	102	for server_name, key_id, fetch_result in verify_keys:
	103	key_values.append((server_name, key_id))
	104	value_values.append(
	105	(
	106	from_server,
	107	ts_added_ms,
	108	fetch_result.valid_until_ts,
	109	db_binary_type(fetch_result.verify_key.encode()),
	110	)
	111	)
	112	# invalidate takes a tuple corresponding to the params of
	113	# _get_server_verify_key. _get_server_verify_key only takes one
	114	# param, which is itself the 2-tuple (server_name, key_id).
	115	invalidations.append((server_name, key_id))
	116
	117	def _invalidate(res):
	118	f = self._get_server_verify_key.invalidate
	119	for i in invalidations:
	120	f((i,))
	121	return res
	122
	123	return self.db_pool.runInteraction(
	124	"store_server_verify_keys",
	125	self.db_pool.simple_upsert_many_txn,
	126	table="server_signature_keys",
	127	key_names=("server_name", "key_id"),
	128	key_values=key_values,
	129	value_names=(
	130	"from_server",
	131	"ts_added_ms",
	132	"ts_valid_until_ms",
	133	"verify_key",
	134	),
	135	value_values=value_values,
	136	).addCallback(_invalidate)
	137
	138	def store_server_keys_json(
	139	self, server_name, key_id, from_server, ts_now_ms, ts_expires_ms, key_json_bytes
	140	):
	141	"""Stores the JSON bytes for a set of keys from a server
	142	The JSON should be signed by the originating server, the intermediate
	143	server, and by this server. Updates the value for the
	144	(server_name, key_id, from_server) triplet if one already existed.
	145	Args:
	146	server_name (str): The name of the server.
	147	key_id (str): The identifer of the key this JSON is for.
	148	from_server (str): The server this JSON was fetched from.
	149	ts_now_ms (int): The time now in milliseconds.
	150	ts_valid_until_ms (int): The time when this json stops being valid.
	151	key_json (bytes): The encoded JSON.
	152	"""
	153	return self.db_pool.simple_upsert(
	154	table="server_keys_json",
	155	keyvalues={
	156	"server_name": server_name,
	157	"key_id": key_id,
	158	"from_server": from_server,
	159	},
	160	values={
	161	"server_name": server_name,
	162	"key_id": key_id,
	163	"from_server": from_server,
	164	"ts_added_ms": ts_now_ms,
	165	"ts_valid_until_ms": ts_expires_ms,
	166	"key_json": db_binary_type(key_json_bytes),
	167	},
	168	desc="store_server_keys_json",
	169	)
	170
	171	def get_server_keys_json(self, server_keys):
	172	"""Retrive the key json for a list of server_keys and key ids.
	173	If no keys are found for a given server, key_id and source then
	174	that server, key_id, and source triplet entry will be an empty list.
	175	The JSON is returned as a byte array so that it can be efficiently
	176	used in an HTTP response.
	177	Args:
	178	server_keys (list): List of (server_name, key_id, source) triplets.
	179	Returns:
	180	Deferred[dict[Tuple[str, str, str\|None], list[dict]]]:
	181	Dict mapping (server_name, key_id, source) triplets to lists of dicts
	182	"""
	183
	184	def _get_server_keys_json_txn(txn):
	185	results = {}
	186	for server_name, key_id, from_server in server_keys:
	187	keyvalues = {"server_name": server_name}
	188	if key_id is not None:
	189	keyvalues["key_id"] = key_id
	190	if from_server is not None:
	191	keyvalues["from_server"] = from_server
	192	rows = self.db_pool.simple_select_list_txn(
	193	txn,
	194	"server_keys_json",
	195	keyvalues=keyvalues,
	196	retcols=(
	197	"key_id",
	198	"from_server",
	199	"ts_added_ms",
	200	"ts_valid_until_ms",
	201	"key_json",
	202	),
	203	)
	204	results[(server_name, key_id, from_server)] = rows
	205	return results
	206
	207	return self.db_pool.runInteraction(
	208	"get_server_keys_json", _get_server_keys_json_txn
	209	)

+398

-0

synapse/storage/databases/main/media_repository.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	from synapse.storage._base import SQLBaseStore
	15	from synapse.storage.database import DatabasePool
	16
	17
	18	class MediaRepositoryBackgroundUpdateStore(SQLBaseStore):
	19	def __init__(self, database: DatabasePool, db_conn, hs):
	20	super(MediaRepositoryBackgroundUpdateStore, self).__init__(
	21	database, db_conn, hs
	22	)
	23
	24	self.db_pool.updates.register_background_index_update(
	25	update_name="local_media_repository_url_idx",
	26	index_name="local_media_repository_url_idx",
	27	table="local_media_repository",
	28	columns=["created_ts"],
	29	where_clause="url_cache IS NOT NULL",
	30	)
	31
	32
	33	class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
	34	"""Persistence for attachments and avatars"""
	35
	36	def __init__(self, database: DatabasePool, db_conn, hs):
	37	super(MediaRepositoryStore, self).__init__(database, db_conn, hs)
	38
	39	def get_local_media(self, media_id):
	40	"""Get the metadata for a local piece of media
	41	Returns:
	42	None if the media_id doesn't exist.
	43	"""
	44	return self.db_pool.simple_select_one(
	45	"local_media_repository",
	46	{"media_id": media_id},
	47	(
	48	"media_type",
	49	"media_length",
	50	"upload_name",
	51	"created_ts",
	52	"quarantined_by",
	53	"url_cache",
	54	),
	55	allow_none=True,
	56	desc="get_local_media",
	57	)
	58
	59	def store_local_media(
	60	self,
	61	media_id,
	62	media_type,
	63	time_now_ms,
	64	upload_name,
	65	media_length,
	66	user_id,
	67	url_cache=None,
	68	):
	69	return self.db_pool.simple_insert(
	70	"local_media_repository",
	71	{
	72	"media_id": media_id,
	73	"media_type": media_type,
	74	"created_ts": time_now_ms,
	75	"upload_name": upload_name,
	76	"media_length": media_length,
	77	"user_id": user_id.to_string(),
	78	"url_cache": url_cache,
	79	},
	80	desc="store_local_media",
	81	)
	82
	83	def mark_local_media_as_safe(self, media_id: str):
	84	"""Mark a local media as safe from quarantining."""
	85	return self.db_pool.simple_update_one(
	86	table="local_media_repository",
	87	keyvalues={"media_id": media_id},
	88	updatevalues={"safe_from_quarantine": True},
	89	desc="mark_local_media_as_safe",
	90	)
	91
	92	def get_url_cache(self, url, ts):
	93	"""Get the media_id and ts for a cached URL as of the given timestamp
	94	Returns:
	95	None if the URL isn't cached.
	96	"""
	97
	98	def get_url_cache_txn(txn):
	99	# get the most recently cached result (relative to the given ts)
	100	sql = (
	101	"SELECT response_code, etag, expires_ts, og, media_id, download_ts"
	102	" FROM local_media_repository_url_cache"
	103	" WHERE url = ? AND download_ts <= ?"
	104	" ORDER BY download_ts DESC LIMIT 1"
	105	)
	106	txn.execute(sql, (url, ts))
	107	row = txn.fetchone()
	108
	109	if not row:
	110	# ...or if we've requested a timestamp older than the oldest
	111	# copy in the cache, return the oldest copy (if any)
	112	sql = (
	113	"SELECT response_code, etag, expires_ts, og, media_id, download_ts"
	114	" FROM local_media_repository_url_cache"
	115	" WHERE url = ? AND download_ts > ?"
	116	" ORDER BY download_ts ASC LIMIT 1"
	117	)
	118	txn.execute(sql, (url, ts))
	119	row = txn.fetchone()
	120
	121	if not row:
	122	return None
	123
	124	return dict(
	125	zip(
	126	(
	127	"response_code",
	128	"etag",
	129	"expires_ts",
	130	"og",
	131	"media_id",
	132	"download_ts",
	133	),
	134	row,
	135	)
	136	)
	137
	138	return self.db_pool.runInteraction("get_url_cache", get_url_cache_txn)
	139
	140	def store_url_cache(
	141	self, url, response_code, etag, expires_ts, og, media_id, download_ts
	142	):
	143	return self.db_pool.simple_insert(
	144	"local_media_repository_url_cache",
	145	{
	146	"url": url,
	147	"response_code": response_code,
	148	"etag": etag,
	149	"expires_ts": expires_ts,
	150	"og": og,
	151	"media_id": media_id,
	152	"download_ts": download_ts,
	153	},
	154	desc="store_url_cache",
	155	)
	156
	157	def get_local_media_thumbnails(self, media_id):
	158	return self.db_pool.simple_select_list(
	159	"local_media_repository_thumbnails",
	160	{"media_id": media_id},
	161	(
	162	"thumbnail_width",
	163	"thumbnail_height",
	164	"thumbnail_method",
	165	"thumbnail_type",
	166	"thumbnail_length",
	167	),
	168	desc="get_local_media_thumbnails",
	169	)
	170
	171	def store_local_thumbnail(
	172	self,
	173	media_id,
	174	thumbnail_width,
	175	thumbnail_height,
	176	thumbnail_type,
	177	thumbnail_method,
	178	thumbnail_length,
	179	):
	180	return self.db_pool.simple_insert(
	181	"local_media_repository_thumbnails",
	182	{
	183	"media_id": media_id,
	184	"thumbnail_width": thumbnail_width,
	185	"thumbnail_height": thumbnail_height,
	186	"thumbnail_method": thumbnail_method,
	187	"thumbnail_type": thumbnail_type,
	188	"thumbnail_length": thumbnail_length,
	189	},
	190	desc="store_local_thumbnail",
	191	)
	192
	193	def get_cached_remote_media(self, origin, media_id):
	194	return self.db_pool.simple_select_one(
	195	"remote_media_cache",
	196	{"media_origin": origin, "media_id": media_id},
	197	(
	198	"media_type",
	199	"media_length",
	200	"upload_name",
	201	"created_ts",
	202	"filesystem_id",
	203	"quarantined_by",
	204	),
	205	allow_none=True,
	206	desc="get_cached_remote_media",
	207	)
	208
	209	def store_cached_remote_media(
	210	self,
	211	origin,
	212	media_id,
	213	media_type,
	214	media_length,
	215	time_now_ms,
	216	upload_name,
	217	filesystem_id,
	218	):
	219	return self.db_pool.simple_insert(
	220	"remote_media_cache",
	221	{
	222	"media_origin": origin,
	223	"media_id": media_id,
	224	"media_type": media_type,
	225	"media_length": media_length,
	226	"created_ts": time_now_ms,
	227	"upload_name": upload_name,
	228	"filesystem_id": filesystem_id,
	229	"last_access_ts": time_now_ms,
	230	},
	231	desc="store_cached_remote_media",
	232	)
	233
	234	def update_cached_last_access_time(self, local_media, remote_media, time_ms):
	235	"""Updates the last access time of the given media
	236
	237	Args:
	238	local_media (iterable[str]): Set of media_ids
	239	remote_media (iterable[(str, str)]): Set of (server_name, media_id)
	240	time_ms: Current time in milliseconds
	241	"""
	242
	243	def update_cache_txn(txn):
	244	sql = (
	245	"UPDATE remote_media_cache SET last_access_ts = ?"
	246	" WHERE media_origin = ? AND media_id = ?"
	247	)
	248
	249	txn.executemany(
	250	sql,
	251	(
	252	(time_ms, media_origin, media_id)
	253	for media_origin, media_id in remote_media
	254	),
	255	)
	256
	257	sql = (
	258	"UPDATE local_media_repository SET last_access_ts = ?"
	259	" WHERE media_id = ?"
	260	)
	261
	262	txn.executemany(sql, ((time_ms, media_id) for media_id in local_media))
	263
	264	return self.db_pool.runInteraction(
	265	"update_cached_last_access_time", update_cache_txn
	266	)
	267
	268	def get_remote_media_thumbnails(self, origin, media_id):
	269	return self.db_pool.simple_select_list(
	270	"remote_media_cache_thumbnails",
	271	{"media_origin": origin, "media_id": media_id},
	272	(
	273	"thumbnail_width",
	274	"thumbnail_height",
	275	"thumbnail_method",
	276	"thumbnail_type",
	277	"thumbnail_length",
	278	"filesystem_id",
	279	),
	280	desc="get_remote_media_thumbnails",
	281	)
	282
	283	def store_remote_media_thumbnail(
	284	self,
	285	origin,
	286	media_id,
	287	filesystem_id,
	288	thumbnail_width,
	289	thumbnail_height,
	290	thumbnail_type,
	291	thumbnail_method,
	292	thumbnail_length,
	293	):
	294	return self.db_pool.simple_insert(
	295	"remote_media_cache_thumbnails",
	296	{
	297	"media_origin": origin,
	298	"media_id": media_id,
	299	"thumbnail_width": thumbnail_width,
	300	"thumbnail_height": thumbnail_height,
	301	"thumbnail_method": thumbnail_method,
	302	"thumbnail_type": thumbnail_type,
	303	"thumbnail_length": thumbnail_length,
	304	"filesystem_id": filesystem_id,
	305	},
	306	desc="store_remote_media_thumbnail",
	307	)
	308
	309	def get_remote_media_before(self, before_ts):
	310	sql = (
	311	"SELECT media_origin, media_id, filesystem_id"
	312	" FROM remote_media_cache"
	313	" WHERE last_access_ts < ?"
	314	)
	315
	316	return self.db_pool.execute(
	317	"get_remote_media_before", self.db_pool.cursor_to_dict, sql, before_ts
	318	)
	319
	320	def delete_remote_media(self, media_origin, media_id):
	321	def delete_remote_media_txn(txn):
	322	self.db_pool.simple_delete_txn(
	323	txn,
	324	"remote_media_cache",
	325	keyvalues={"media_origin": media_origin, "media_id": media_id},
	326	)
	327	self.db_pool.simple_delete_txn(
	328	txn,
	329	"remote_media_cache_thumbnails",
	330	keyvalues={"media_origin": media_origin, "media_id": media_id},
	331	)
	332
	333	return self.db_pool.runInteraction(
	334	"delete_remote_media", delete_remote_media_txn
	335	)
	336
	337	def get_expired_url_cache(self, now_ts):
	338	sql = (
	339	"SELECT media_id FROM local_media_repository_url_cache"
	340	" WHERE expires_ts < ?"
	341	" ORDER BY expires_ts ASC"
	342	" LIMIT 500"
	343	)
	344
	345	def _get_expired_url_cache_txn(txn):
	346	txn.execute(sql, (now_ts,))
	347	return [row[0] for row in txn]
	348
	349	return self.db_pool.runInteraction(
	350	"get_expired_url_cache", _get_expired_url_cache_txn
	351	)
	352
	353	async def delete_url_cache(self, media_ids):
	354	if len(media_ids) == 0:
	355	return
	356
	357	sql = "DELETE FROM local_media_repository_url_cache WHERE media_id = ?"
	358
	359	def _delete_url_cache_txn(txn):
	360	txn.executemany(sql, [(media_id,) for media_id in media_ids])
	361
	362	return await self.db_pool.runInteraction(
	363	"delete_url_cache", _delete_url_cache_txn
	364	)
	365
	366	def get_url_cache_media_before(self, before_ts):
	367	sql = (
	368	"SELECT media_id FROM local_media_repository"
	369	" WHERE created_ts < ? AND url_cache IS NOT NULL"
	370	" ORDER BY created_ts ASC"
	371	" LIMIT 500"
	372	)
	373
	374	def _get_url_cache_media_before_txn(txn):
	375	txn.execute(sql, (before_ts,))
	376	return [row[0] for row in txn]
	377
	378	return self.db_pool.runInteraction(
	379	"get_url_cache_media_before", _get_url_cache_media_before_txn
	380	)
	381
	382	async def delete_url_cache_media(self, media_ids):
	383	if len(media_ids) == 0:
	384	return
	385
	386	def _delete_url_cache_media_txn(txn):
	387	sql = "DELETE FROM local_media_repository WHERE media_id = ?"
	388
	389	txn.executemany(sql, [(media_id,) for media_id in media_ids])
	390
	391	sql = "DELETE FROM local_media_repository_thumbnails WHERE media_id = ?"
	392
	393	txn.executemany(sql, [(media_id,) for media_id in media_ids])
	394
	395	return await self.db_pool.runInteraction(
	396	"delete_url_cache_media", _delete_url_cache_media_txn
	397	)

+122

-0

synapse/storage/databases/main/metrics.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	import typing
	15	from collections import Counter
	16
	17	from synapse.metrics import BucketCollector
	18	from synapse.metrics.background_process_metrics import run_as_background_process
	19	from synapse.storage._base import SQLBaseStore
	20	from synapse.storage.database import DatabasePool
	21	from synapse.storage.databases.main.event_push_actions import (
	22	EventPushActionsWorkerStore,
	23	)
	24
	25
	26	class ServerMetricsStore(EventPushActionsWorkerStore, SQLBaseStore):
	27	"""Functions to pull various metrics from the DB, for e.g. phone home
	28	stats and prometheus metrics.
	29	"""
	30
	31	def __init__(self, database: DatabasePool, db_conn, hs):
	32	super().__init__(database, db_conn, hs)
	33
	34	# Collect metrics on the number of forward extremities that exist.
	35	# Counter of number of extremities to count
	36	self._current_forward_extremities_amount = (
	37	Counter()
	38	) # type: typing.Counter[int]
	39
	40	BucketCollector(
	41	"synapse_forward_extremities",
	42	lambda: self._current_forward_extremities_amount,
	43	buckets=[1, 2, 3, 5, 7, 10, 15, 20, 50, 100, 200, 500, "+Inf"],
	44	)
	45
	46	# Read the extrems every 60 minutes
	47	def read_forward_extremities():
	48	# run as a background process to make sure that the database transactions
	49	# have a logcontext to report to
	50	return run_as_background_process(
	51	"read_forward_extremities", self._read_forward_extremities
	52	)
	53
	54	hs.get_clock().looping_call(read_forward_extremities, 60 * 60 * 1000)
	55
	56	async def _read_forward_extremities(self):
	57	def fetch(txn):
	58	txn.execute(
	59	"""
	60	select count(*) c from event_forward_extremities
	61	group by room_id
	62	"""
	63	)
	64	return txn.fetchall()
	65
	66	res = await self.db_pool.runInteraction("read_forward_extremities", fetch)
	67	self._current_forward_extremities_amount = Counter([x[0] for x in res])
	68
	69	async def count_daily_messages(self):
	70	"""
	71	Returns an estimate of the number of messages sent in the last day.
	72
	73	If it has been significantly less or more than one day since the last
	74	call to this function, it will return None.
	75	"""
	76
	77	def _count_messages(txn):
	78	sql = """
	79	SELECT COALESCE(COUNT(*), 0) FROM events
	80	WHERE type = 'm.room.message'
	81	AND stream_ordering > ?
	82	"""
	83	txn.execute(sql, (self.stream_ordering_day_ago,))
	84	(count,) = txn.fetchone()
	85	return count
	86
	87	return await self.db_pool.runInteraction("count_messages", _count_messages)
	88
	89	async def count_daily_sent_messages(self):
	90	def _count_messages(txn):
	91	# This is good enough as if you have silly characters in your own
	92	# hostname then thats your own fault.
	93	like_clause = "%:" + self.hs.hostname
	94
	95	sql = """
	96	SELECT COALESCE(COUNT(*), 0) FROM events
	97	WHERE type = 'm.room.message'
	98	AND sender LIKE ?
	99	AND stream_ordering > ?
	100	"""
	101
	102	txn.execute(sql, (like_clause, self.stream_ordering_day_ago))
	103	(count,) = txn.fetchone()
	104	return count
	105
	106	return await self.db_pool.runInteraction(
	107	"count_daily_sent_messages", _count_messages
	108	)
	109
	110	async def count_daily_active_rooms(self):
	111	def _count(txn):
	112	sql = """
	113	SELECT COALESCE(COUNT(DISTINCT room_id), 0) FROM events
	114	WHERE type = 'm.room.message'
	115	AND stream_ordering > ?
	116	"""
	117	txn.execute(sql, (self.stream_ordering_day_ago,))
	118	(count,) = txn.fetchone()
	119	return count
	120
	121	return await self.db_pool.runInteraction("count_daily_active_rooms", _count)

+354

-0

synapse/storage/databases/main/monthly_active_users.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2018 New Vector
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	import logging
	15	from typing import List
	16
	17	from synapse.storage._base import SQLBaseStore
	18	from synapse.storage.database import DatabasePool, make_in_list_sql_clause
	19	from synapse.util.caches.descriptors import cached
	20
	21	logger = logging.getLogger(__name__)
	22
	23	# Number of msec of granularity to store the monthly_active_user timestamp
	24	# This means it is not necessary to update the table on every request
	25	LAST_SEEN_GRANULARITY = 60 * 60 * 1000
	26
	27
	28	class MonthlyActiveUsersWorkerStore(SQLBaseStore):
	29	def __init__(self, database: DatabasePool, db_conn, hs):
	30	super(MonthlyActiveUsersWorkerStore, self).__init__(database, db_conn, hs)
	31	self._clock = hs.get_clock()
	32	self.hs = hs
	33
	34	@cached(num_args=0)
	35	def get_monthly_active_count(self):
	36	"""Generates current count of monthly active users
	37
	38	Returns:
	39	Defered[int]: Number of current monthly active users
	40	"""
	41
	42	def _count_users(txn):
	43	sql = "SELECT COALESCE(count(*), 0) FROM monthly_active_users"
	44	txn.execute(sql)
	45	(count,) = txn.fetchone()
	46	return count
	47
	48	return self.db_pool.runInteraction("count_users", _count_users)
	49
	50	@cached(num_args=0)
	51	def get_monthly_active_count_by_service(self):
	52	"""Generates current count of monthly active users broken down by service.
	53	A service is typically an appservice but also includes native matrix users.
	54	Since the `monthly_active_users` table is populated from the `user_ips` table
	55	`config.track_appservice_user_ips` must be set to `true` for this
	56	method to return anything other than native matrix users.
	57
	58	Returns:
	59	Deferred[dict]: dict that includes a mapping between app_service_id
	60	and the number of occurrences.
	61
	62	"""
	63
	64	def _count_users_by_service(txn):
	65	sql = """
	66	SELECT COALESCE(appservice_id, 'native'), COALESCE(count(*), 0)
	67	FROM monthly_active_users
	68	LEFT JOIN users ON monthly_active_users.user_id=users.name
	69	GROUP BY appservice_id;
	70	"""
	71
	72	txn.execute(sql)
	73	result = txn.fetchall()
	74	return dict(result)
	75
	76	return self.db_pool.runInteraction(
	77	"count_users_by_service", _count_users_by_service
	78	)
	79
	80	async def get_registered_reserved_users(self) -> List[str]:
	81	"""Of the reserved threepids defined in config, retrieve those that are associated
	82	with registered users
	83
	84	Returns:
	85	User IDs of actual users that are reserved
	86	"""
	87	users = []
	88
	89	for tp in self.hs.config.mau_limits_reserved_threepids[
	90	: self.hs.config.max_mau_value
	91	]:
	92	user_id = await self.hs.get_datastore().get_user_id_by_threepid(
	93	tp["medium"], tp["address"]
	94	)
	95	if user_id:
	96	users.append(user_id)
	97
	98	return users
	99
	100	@cached(num_args=1)
	101	def user_last_seen_monthly_active(self, user_id):
	102	"""
	103	Checks if a given user is part of the monthly active user group
	104	Arguments:
	105	user_id (str): user to add/update
	106	Return:
	107	Deferred[int] : timestamp since last seen, None if never seen
	108
	109	"""
	110
	111	return self.db_pool.simple_select_one_onecol(
	112	table="monthly_active_users",
	113	keyvalues={"user_id": user_id},
	114	retcol="timestamp",
	115	allow_none=True,
	116	desc="user_last_seen_monthly_active",
	117	)
	118
	119
	120	class MonthlyActiveUsersStore(MonthlyActiveUsersWorkerStore):
	121	def __init__(self, database: DatabasePool, db_conn, hs):
	122	super(MonthlyActiveUsersStore, self).__init__(database, db_conn, hs)
	123
	124	self._limit_usage_by_mau = hs.config.limit_usage_by_mau
	125	self._mau_stats_only = hs.config.mau_stats_only
	126	self._max_mau_value = hs.config.max_mau_value
	127
	128	# Do not add more reserved users than the total allowable number
	129	# cur = LoggingTransaction(
	130	self.db_pool.new_transaction(
	131	db_conn,
	132	"initialise_mau_threepids",
	133	[],
	134	[],
	135	self._initialise_reserved_users,
	136	hs.config.mau_limits_reserved_threepids[: self._max_mau_value],
	137	)
	138
	139	def _initialise_reserved_users(self, txn, threepids):
	140	"""Ensures that reserved threepids are accounted for in the MAU table, should
	141	be called on start up.
	142
	143	Args:
	144	txn (cursor):
	145	threepids (list[dict]): List of threepid dicts to reserve
	146	"""
	147
	148	# XXX what is this function trying to achieve? It upserts into
	149	# monthly_active_users for each registered reserved mau user, but why?
	150	#
	151	# - shouldn't there already be an entry for each reserved user (at least
	152	# if they have been active recently)?
	153	#
	154	# - if it's important that the timestamp is kept up to date, why do we only
	155	# run this at startup?
	156
	157	for tp in threepids:
	158	user_id = self.get_user_id_by_threepid_txn(txn, tp["medium"], tp["address"])
	159
	160	if user_id:
	161	is_support = self.is_support_user_txn(txn, user_id)
	162	if not is_support:
	163	# We do this manually here to avoid hitting #6791
	164	self.db_pool.simple_upsert_txn(
	165	txn,
	166	table="monthly_active_users",
	167	keyvalues={"user_id": user_id},
	168	values={"timestamp": int(self._clock.time_msec())},
	169	)
	170	else:
	171	logger.warning("mau limit reserved threepid %s not found in db" % tp)
	172
	173	async def reap_monthly_active_users(self):
	174	"""Cleans out monthly active user table to ensure that no stale
	175	entries exist.
	176	"""
	177
	178	def _reap_users(txn, reserved_users):
	179	"""
	180	Args:
	181	reserved_users (tuple): reserved users to preserve
	182	"""
	183
	184	thirty_days_ago = int(self._clock.time_msec()) - (1000 * 60 * 60 * 24 * 30)
	185
	186	in_clause, in_clause_args = make_in_list_sql_clause(
	187	self.database_engine, "user_id", reserved_users
	188	)
	189
	190	txn.execute(
	191	"DELETE FROM monthly_active_users WHERE timestamp < ? AND NOT %s"
	192	% (in_clause,),
	193	[thirty_days_ago] + in_clause_args,
	194	)
	195
	196	if self._limit_usage_by_mau:
	197	# If MAU user count still exceeds the MAU threshold, then delete on
	198	# a least recently active basis.
	199	# Note it is not possible to write this query using OFFSET due to
	200	# incompatibilities in how sqlite and postgres support the feature.
	201	# Sqlite requires 'LIMIT -1 OFFSET ?', the LIMIT must be present,
	202	# while Postgres does not require 'LIMIT', but also does not support
	203	# negative LIMIT values. So there is no way to write it that both can
	204	# support
	205
	206	# Limit must be >= 0 for postgres
	207	num_of_non_reserved_users_to_remove = max(
	208	self._max_mau_value - len(reserved_users), 0
	209	)
	210
	211	# It is important to filter reserved users twice to guard
	212	# against the case where the reserved user is present in the
	213	# SELECT, meaning that a legitimate mau is deleted.
	214	sql = """
	215	DELETE FROM monthly_active_users
	216	WHERE user_id NOT IN (
	217	SELECT user_id FROM monthly_active_users
	218	WHERE NOT %s
	219	ORDER BY timestamp DESC
	220	LIMIT ?
	221	)
	222	AND NOT %s
	223	""" % (
	224	in_clause,
	225	in_clause,
	226	)
	227
	228	query_args = (
	229	in_clause_args
	230	+ [num_of_non_reserved_users_to_remove]
	231	+ in_clause_args
	232	)
	233	txn.execute(sql, query_args)
	234
	235	# It seems poor to invalidate the whole cache. Postgres supports
	236	# 'Returning' which would allow me to invalidate only the
	237	# specific users, but sqlite has no way to do this and instead
	238	# I would need to SELECT and the DELETE which without locking
	239	# is racy.
	240	# Have resolved to invalidate the whole cache for now and do
	241	# something about it if and when the perf becomes significant
	242	self._invalidate_all_cache_and_stream(
	243	txn, self.user_last_seen_monthly_active
	244	)
	245	self._invalidate_cache_and_stream(txn, self.get_monthly_active_count, ())
	246
	247	reserved_users = await self.get_registered_reserved_users()
	248	await self.db_pool.runInteraction(
	249	"reap_monthly_active_users", _reap_users, reserved_users
	250	)
	251
	252	async def upsert_monthly_active_user(self, user_id: str) -> None:
	253	"""Updates or inserts the user into the monthly active user table, which
	254	is used to track the current MAU usage of the server
	255
	256	Args:
	257	user_id: user to add/update
	258	"""
	259	# Support user never to be included in MAU stats. Note I can't easily call this
	260	# from upsert_monthly_active_user_txn because then I need a _txn form of
	261	# is_support_user which is complicated because I want to cache the result.
	262	# Therefore I call it here and ignore the case where
	263	# upsert_monthly_active_user_txn is called directly from
	264	# _initialise_reserved_users reasoning that it would be very strange to
	265	# include a support user in this context.
	266
	267	is_support = await self.is_support_user(user_id)
	268	if is_support:
	269	return
	270
	271	await self.db_pool.runInteraction(
	272	"upsert_monthly_active_user", self.upsert_monthly_active_user_txn, user_id
	273	)
	274
	275	def upsert_monthly_active_user_txn(self, txn, user_id):
	276	"""Updates or inserts monthly active user member
	277
	278	We consciously do not call is_support_txn from this method because it
	279	is not possible to cache the response. is_support_txn will be false in
	280	almost all cases, so it seems reasonable to call it only for
	281	upsert_monthly_active_user and to call is_support_txn manually
	282	for cases where upsert_monthly_active_user_txn is called directly,
	283	like _initialise_reserved_users
	284
	285	In short, don't call this method with support users. (Support users
	286	should not appear in the MAU stats).
	287
	288	Args:
	289	txn (cursor):
	290	user_id (str): user to add/update
	291
	292	Returns:
	293	bool: True if a new entry was created, False if an
	294	existing one was updated.
	295	"""
	296
	297	# Am consciously deciding to lock the table on the basis that is ought
	298	# never be a big table and alternative approaches (batching multiple
	299	# upserts into a single txn) introduced a lot of extra complexity.
	300	# See https://github.com/matrix-org/synapse/issues/3854 for more
	301	is_insert = self.db_pool.simple_upsert_txn(
	302	txn,
	303	table="monthly_active_users",
	304	keyvalues={"user_id": user_id},
	305	values={"timestamp": int(self._clock.time_msec())},
	306	)
	307
	308	self._invalidate_cache_and_stream(txn, self.get_monthly_active_count, ())
	309	self._invalidate_cache_and_stream(
	310	txn, self.get_monthly_active_count_by_service, ()
	311	)
	312	self._invalidate_cache_and_stream(
	313	txn, self.user_last_seen_monthly_active, (user_id,)
	314	)
	315
	316	return is_insert
	317
	318	async def populate_monthly_active_users(self, user_id):
	319	"""Checks on the state of monthly active user limits and optionally
	320	add the user to the monthly active tables
	321
	322	Args:
	323	user_id(str): the user_id to query
	324	"""
	325	if self._limit_usage_by_mau or self._mau_stats_only:
	326	# Trial users and guests should not be included as part of MAU group
	327	is_guest = await self.is_guest(user_id)
	328	if is_guest:
	329	return
	330	is_trial = await self.is_trial_user(user_id)
	331	if is_trial:
	332	return
	333
	334	last_seen_timestamp = await self.user_last_seen_monthly_active(user_id)
	335	now = self.hs.get_clock().time_msec()
	336
	337	# We want to reduce to the total number of db writes, and are happy
	338	# to trade accuracy of timestamp in order to lighten load. This means
	339	# We always insert new users (where MAU threshold has not been reached),
	340	# but only update if we have not previously seen the user for
	341	# LAST_SEEN_GRANULARITY ms
	342	if last_seen_timestamp is None:
	343	# In the case where mau_stats_only is True and limit_usage_by_mau is
	344	# False, there is no point in checking get_monthly_active_count - it
	345	# adds no value and will break the logic if max_mau_value is exceeded.
	346	if not self._limit_usage_by_mau:
	347	await self.upsert_monthly_active_user(user_id)
	348	else:
	349	count = await self.get_monthly_active_count()
	350	if count < self._max_mau_value:
	351	await self.upsert_monthly_active_user(user_id)
	352	elif now - last_seen_timestamp > LAST_SEEN_GRANULARITY:
	353	await self.upsert_monthly_active_user(user_id)

+33

-0

synapse/storage/databases/main/openid.py less more

	0	from synapse.storage._base import SQLBaseStore
	1
	2
	3	class OpenIdStore(SQLBaseStore):
	4	def insert_open_id_token(self, token, ts_valid_until_ms, user_id):
	5	return self.db_pool.simple_insert(
	6	table="open_id_tokens",
	7	values={
	8	"token": token,
	9	"ts_valid_until_ms": ts_valid_until_ms,
	10	"user_id": user_id,
	11	},
	12	desc="insert_open_id_token",
	13	)
	14
	15	def get_user_id_for_open_id_token(self, token, ts_now_ms):
	16	def get_user_id_for_token_txn(txn):
	17	sql = (
	18	"SELECT user_id FROM open_id_tokens"
	19	" WHERE token = ? AND ? <= ts_valid_until_ms"
	20	)
	21
	22	txn.execute(sql, (token, ts_now_ms))
	23
	24	rows = txn.fetchall()
	25	if not rows:
	26	return None
	27	else:
	28	return rows[0][0]
	29
	30	return self.db_pool.runInteraction(
	31	"get_user_id_for_token", get_user_id_for_token_txn
	32	)

+183

-0

synapse/storage/databases/main/presence.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from typing import List, Tuple
	16
	17	from synapse.storage._base import SQLBaseStore, make_in_list_sql_clause
	18	from synapse.storage.presence import UserPresenceState
	19	from synapse.util.caches.descriptors import cached, cachedList
	20	from synapse.util.iterutils import batch_iter
	21
	22
	23	class PresenceStore(SQLBaseStore):
	24	async def update_presence(self, presence_states):
	25	stream_ordering_manager = self._presence_id_gen.get_next_mult(
	26	len(presence_states)
	27	)
	28
	29	with stream_ordering_manager as stream_orderings:
	30	await self.db_pool.runInteraction(
	31	"update_presence",
	32	self._update_presence_txn,
	33	stream_orderings,
	34	presence_states,
	35	)
	36
	37	return stream_orderings[-1], self._presence_id_gen.get_current_token()
	38
	39	def _update_presence_txn(self, txn, stream_orderings, presence_states):
	40	for stream_id, state in zip(stream_orderings, presence_states):
	41	txn.call_after(
	42	self.presence_stream_cache.entity_has_changed, state.user_id, stream_id
	43	)
	44	txn.call_after(self._get_presence_for_user.invalidate, (state.user_id,))
	45
	46	# Actually insert new rows
	47	self.db_pool.simple_insert_many_txn(
	48	txn,
	49	table="presence_stream",
	50	values=[
	51	{
	52	"stream_id": stream_id,
	53	"user_id": state.user_id,
	54	"state": state.state,
	55	"last_active_ts": state.last_active_ts,
	56	"last_federation_update_ts": state.last_federation_update_ts,
	57	"last_user_sync_ts": state.last_user_sync_ts,
	58	"status_msg": state.status_msg,
	59	"currently_active": state.currently_active,
	60	}
	61	for stream_id, state in zip(stream_orderings, presence_states)
	62	],
	63	)
	64
	65	# Delete old rows to stop database from getting really big
	66	sql = "DELETE FROM presence_stream WHERE stream_id < ? AND "
	67
	68	for states in batch_iter(presence_states, 50):
	69	clause, args = make_in_list_sql_clause(
	70	self.database_engine, "user_id", [s.user_id for s in states]
	71	)
	72	txn.execute(sql + clause, [stream_id] + list(args))
	73
	74	async def get_all_presence_updates(
	75	self, instance_name: str, last_id: int, current_id: int, limit: int
	76	) -> Tuple[List[Tuple[int, list]], int, bool]:
	77	"""Get updates for presence replication stream.
	78
	79	Args:
	80	instance_name: The writer we want to fetch updates from. Unused
	81	here since there is only ever one writer.
	82	last_id: The token to fetch updates from. Exclusive.
	83	current_id: The token to fetch updates up to. Inclusive.
	84	limit: The requested limit for the number of rows to return. The
	85	function may return more or fewer rows.
	86
	87	Returns:
	88	A tuple consisting of: the updates, a token to use to fetch
	89	subsequent updates, and whether we returned fewer rows than exists
	90	between the requested tokens due to the limit.
	91
	92	The token returned can be used in a subsequent call to this
	93	function to get further updatees.
	94
	95	The updates are a list of 2-tuples of stream ID and the row data
	96	"""
	97
	98	if last_id == current_id:
	99	return [], current_id, False
	100
	101	def get_all_presence_updates_txn(txn):
	102	sql = """
	103	SELECT stream_id, user_id, state, last_active_ts,
	104	last_federation_update_ts, last_user_sync_ts,
	105	status_msg,
	106	currently_active
	107	FROM presence_stream
	108	WHERE ? < stream_id AND stream_id <= ?
	109	ORDER BY stream_id ASC
	110	LIMIT ?
	111	"""
	112	txn.execute(sql, (last_id, current_id, limit))
	113	updates = [(row[0], row[1:]) for row in txn]
	114
	115	upper_bound = current_id
	116	limited = False
	117	if len(updates) >= limit:
	118	upper_bound = updates[-1][0]
	119	limited = True
	120
	121	return updates, upper_bound, limited
	122
	123	return await self.db_pool.runInteraction(
	124	"get_all_presence_updates", get_all_presence_updates_txn
	125	)
	126
	127	@cached()
	128	def _get_presence_for_user(self, user_id):
	129	raise NotImplementedError()
	130
	131	@cachedList(
	132	cached_method_name="_get_presence_for_user",
	133	list_name="user_ids",
	134	num_args=1,
	135	inlineCallbacks=True,
	136	)
	137	def get_presence_for_users(self, user_ids):
	138	rows = yield self.db_pool.simple_select_many_batch(
	139	table="presence_stream",
	140	column="user_id",
	141	iterable=user_ids,
	142	keyvalues={},
	143	retcols=(
	144	"user_id",
	145	"state",
	146	"last_active_ts",
	147	"last_federation_update_ts",
	148	"last_user_sync_ts",
	149	"status_msg",
	150	"currently_active",
	151	),
	152	desc="get_presence_for_users",
	153	)
	154
	155	for row in rows:
	156	row["currently_active"] = bool(row["currently_active"])
	157
	158	return {row["user_id"]: UserPresenceState(**row) for row in rows}
	159
	160	def get_current_presence_token(self):
	161	return self._presence_id_gen.get_current_token()
	162
	163	def allow_presence_visible(self, observed_localpart, observer_userid):
	164	return self.db_pool.simple_insert(
	165	table="presence_allow_inbound",
	166	values={
	167	"observed_user_id": observed_localpart,
	168	"observer_user_id": observer_userid,
	169	},
	170	desc="allow_presence_visible",
	171	or_ignore=True,
	172	)
	173
	174	def disallow_presence_visible(self, observed_localpart, observer_userid):
	175	return self.db_pool.simple_delete_one(
	176	table="presence_allow_inbound",
	177	keyvalues={
	178	"observed_user_id": observed_localpart,
	179	"observer_user_id": observer_userid,
	180	},
	181	desc="disallow_presence_visible",
	182	)

+173

-0

synapse/storage/databases/main/profile.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from synapse.api.errors import StoreError
	16	from synapse.storage._base import SQLBaseStore
	17	from synapse.storage.databases.main.roommember import ProfileInfo
	18
	19
	20	class ProfileWorkerStore(SQLBaseStore):
	21	async def get_profileinfo(self, user_localpart):
	22	try:
	23	profile = await self.db_pool.simple_select_one(
	24	table="profiles",
	25	keyvalues={"user_id": user_localpart},
	26	retcols=("displayname", "avatar_url"),
	27	desc="get_profileinfo",
	28	)
	29	except StoreError as e:
	30	if e.code == 404:
	31	# no match
	32	return ProfileInfo(None, None)
	33	else:
	34	raise
	35
	36	return ProfileInfo(
	37	avatar_url=profile["avatar_url"], display_name=profile["displayname"]
	38	)
	39
	40	def get_profile_displayname(self, user_localpart):
	41	return self.db_pool.simple_select_one_onecol(
	42	table="profiles",
	43	keyvalues={"user_id": user_localpart},
	44	retcol="displayname",
	45	desc="get_profile_displayname",
	46	)
	47
	48	def get_profile_avatar_url(self, user_localpart):
	49	return self.db_pool.simple_select_one_onecol(
	50	table="profiles",
	51	keyvalues={"user_id": user_localpart},
	52	retcol="avatar_url",
	53	desc="get_profile_avatar_url",
	54	)
	55
	56	def get_from_remote_profile_cache(self, user_id):
	57	return self.db_pool.simple_select_one(
	58	table="remote_profile_cache",
	59	keyvalues={"user_id": user_id},
	60	retcols=("displayname", "avatar_url"),
	61	allow_none=True,
	62	desc="get_from_remote_profile_cache",
	63	)
	64
	65	def create_profile(self, user_localpart):
	66	return self.db_pool.simple_insert(
	67	table="profiles", values={"user_id": user_localpart}, desc="create_profile"
	68	)
	69
	70	def set_profile_displayname(self, user_localpart, new_displayname):
	71	return self.db_pool.simple_update_one(
	72	table="profiles",
	73	keyvalues={"user_id": user_localpart},
	74	updatevalues={"displayname": new_displayname},
	75	desc="set_profile_displayname",
	76	)
	77
	78	def set_profile_avatar_url(self, user_localpart, new_avatar_url):
	79	return self.db_pool.simple_update_one(
	80	table="profiles",
	81	keyvalues={"user_id": user_localpart},
	82	updatevalues={"avatar_url": new_avatar_url},
	83	desc="set_profile_avatar_url",
	84	)
	85
	86
	87	class ProfileStore(ProfileWorkerStore):
	88	def add_remote_profile_cache(self, user_id, displayname, avatar_url):
	89	"""Ensure we are caching the remote user's profiles.
	90
	91	This should only be called when `is_subscribed_remote_profile_for_user`
	92	would return true for the user.
	93	"""
	94	return self.db_pool.simple_upsert(
	95	table="remote_profile_cache",
	96	keyvalues={"user_id": user_id},
	97	values={
	98	"displayname": displayname,
	99	"avatar_url": avatar_url,
	100	"last_check": self._clock.time_msec(),
	101	},
	102	desc="add_remote_profile_cache",
	103	)
	104
	105	def update_remote_profile_cache(self, user_id, displayname, avatar_url):
	106	return self.db_pool.simple_update(
	107	table="remote_profile_cache",
	108	keyvalues={"user_id": user_id},
	109	updatevalues={
	110	"displayname": displayname,
	111	"avatar_url": avatar_url,
	112	"last_check": self._clock.time_msec(),
	113	},
	114	desc="update_remote_profile_cache",
	115	)
	116
	117	async def maybe_delete_remote_profile_cache(self, user_id):
	118	"""Check if we still care about the remote user's profile, and if we
	119	don't then remove their profile from the cache
	120	"""
	121	subscribed = await self.is_subscribed_remote_profile_for_user(user_id)
	122	if not subscribed:
	123	await self.db_pool.simple_delete(
	124	table="remote_profile_cache",
	125	keyvalues={"user_id": user_id},
	126	desc="delete_remote_profile_cache",
	127	)
	128
	129	def get_remote_profile_cache_entries_that_expire(self, last_checked):
	130	"""Get all users who haven't been checked since `last_checked`
	131	"""
	132
	133	def _get_remote_profile_cache_entries_that_expire_txn(txn):
	134	sql = """
	135	SELECT user_id, displayname, avatar_url
	136	FROM remote_profile_cache
	137	WHERE last_check < ?
	138	"""
	139
	140	txn.execute(sql, (last_checked,))
	141
	142	return self.db_pool.cursor_to_dict(txn)
	143
	144	return self.db_pool.runInteraction(
	145	"get_remote_profile_cache_entries_that_expire",
	146	_get_remote_profile_cache_entries_that_expire_txn,
	147	)
	148
	149	async def is_subscribed_remote_profile_for_user(self, user_id):
	150	"""Check whether we are interested in a remote user's profile.
	151	"""
	152	res = await self.db_pool.simple_select_one_onecol(
	153	table="group_users",
	154	keyvalues={"user_id": user_id},
	155	retcol="user_id",
	156	allow_none=True,
	157	desc="should_update_remote_profile_cache_for_user",
	158	)
	159
	160	if res:
	161	return True
	162
	163	res = await self.db_pool.simple_select_one_onecol(
	164	table="group_invites",
	165	keyvalues={"user_id": user_id},
	166	retcol="user_id",
	167	allow_none=True,
	168	desc="should_update_remote_profile_cache_for_user",
	169	)
	170
	171	if res:
	172	return True

+400

-0

synapse/storage/databases/main/purge_events.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16	from typing import Any, Tuple
	17
	18	from synapse.api.errors import SynapseError
	19	from synapse.storage._base import SQLBaseStore
	20	from synapse.storage.databases.main.state import StateGroupWorkerStore
	21	from synapse.types import RoomStreamToken
	22
	23	logger = logging.getLogger(__name__)
	24
	25
	26	class PurgeEventsStore(StateGroupWorkerStore, SQLBaseStore):
	27	def purge_history(self, room_id, token, delete_local_events):
	28	"""Deletes room history before a certain point
	29
	30	Args:
	31	room_id (str):
	32
	33	token (str): A topological token to delete events before
	34
	35	delete_local_events (bool):
	36	if True, we will delete local events as well as remote ones
	37	(instead of just marking them as outliers and deleting their
	38	state groups).
	39
	40	Returns:
	41	Deferred[set[int]]: The set of state groups that are referenced by
	42	deleted events.
	43	"""
	44
	45	return self.db_pool.runInteraction(
	46	"purge_history",
	47	self._purge_history_txn,
	48	room_id,
	49	token,
	50	delete_local_events,
	51	)
	52
	53	def _purge_history_txn(self, txn, room_id, token_str, delete_local_events):
	54	token = RoomStreamToken.parse(token_str)
	55
	56	# Tables that should be pruned:
	57	# event_auth
	58	# event_backward_extremities
	59	# event_edges
	60	# event_forward_extremities
	61	# event_json
	62	# event_push_actions
	63	# event_reference_hashes
	64	# event_relations
	65	# event_search
	66	# event_to_state_groups
	67	# events
	68	# rejections
	69	# room_depth
	70	# state_groups
	71	# state_groups_state
	72
	73	# we will build a temporary table listing the events so that we don't
	74	# have to keep shovelling the list back and forth across the
	75	# connection. Annoyingly the python sqlite driver commits the
	76	# transaction on CREATE, so let's do this first.
	77	#
	78	# furthermore, we might already have the table from a previous (failed)
	79	# purge attempt, so let's drop the table first.
	80
	81	txn.execute("DROP TABLE IF EXISTS events_to_purge")
	82
	83	txn.execute(
	84	"CREATE TEMPORARY TABLE events_to_purge ("
	85	" event_id TEXT NOT NULL,"
	86	" should_delete BOOLEAN NOT NULL"
	87	")"
	88	)
	89
	90	# First ensure that we're not about to delete all the forward extremeties
	91	txn.execute(
	92	"SELECT e.event_id, e.depth FROM events as e "
	93	"INNER JOIN event_forward_extremities as f "
	94	"ON e.event_id = f.event_id "
	95	"AND e.room_id = f.room_id "
	96	"WHERE f.room_id = ?",
	97	(room_id,),
	98	)
	99	rows = txn.fetchall()
	100	max_depth = max(row[1] for row in rows)
	101
	102	if max_depth < token.topological:
	103	# We need to ensure we don't delete all the events from the database
	104	# otherwise we wouldn't be able to send any events (due to not
	105	# having any backwards extremeties)
	106	raise SynapseError(
	107	400, "topological_ordering is greater than forward extremeties"
	108	)
	109
	110	logger.info("[purge] looking for events to delete")
	111
	112	should_delete_expr = "state_key IS NULL"
	113	should_delete_params = () # type: Tuple[Any, ...]
	114	if not delete_local_events:
	115	should_delete_expr += " AND event_id NOT LIKE ?"
	116
	117	# We include the parameter twice since we use the expression twice
	118	should_delete_params += ("%:" + self.hs.hostname, "%:" + self.hs.hostname)
	119
	120	should_delete_params += (room_id, token.topological)
	121
	122	# Note that we insert events that are outliers and aren't going to be
	123	# deleted, as nothing will happen to them.
	124	txn.execute(
	125	"INSERT INTO events_to_purge"
	126	" SELECT event_id, %s"
	127	" FROM events AS e LEFT JOIN state_events USING (event_id)"
	128	" WHERE (NOT outlier OR (%s)) AND e.room_id = ? AND topological_ordering < ?"
	129	% (should_delete_expr, should_delete_expr),
	130	should_delete_params,
	131	)
	132
	133	# We create the indices after insertion as that's a lot faster.
	134
	135	# create an index on should_delete because later we'll be looking for
	136	# the should_delete / shouldn't_delete subsets
	137	txn.execute(
	138	"CREATE INDEX events_to_purge_should_delete"
	139	" ON events_to_purge(should_delete)"
	140	)
	141
	142	# We do joins against events_to_purge for e.g. calculating state
	143	# groups to purge, etc., so lets make an index.
	144	txn.execute("CREATE INDEX events_to_purge_id ON events_to_purge(event_id)")
	145
	146	txn.execute("SELECT event_id, should_delete FROM events_to_purge")
	147	event_rows = txn.fetchall()
	148	logger.info(
	149	"[purge] found %i events before cutoff, of which %i can be deleted",
	150	len(event_rows),
	151	sum(1 for e in event_rows if e[1]),
	152	)
	153
	154	logger.info("[purge] Finding new backward extremities")
	155
	156	# We calculate the new entries for the backward extremeties by finding
	157	# events to be purged that are pointed to by events we're not going to
	158	# purge.
	159	txn.execute(
	160	"SELECT DISTINCT e.event_id FROM events_to_purge AS e"
	161	" INNER JOIN event_edges AS ed ON e.event_id = ed.prev_event_id"
	162	" LEFT JOIN events_to_purge AS ep2 ON ed.event_id = ep2.event_id"
	163	" WHERE ep2.event_id IS NULL"
	164	)
	165	new_backwards_extrems = txn.fetchall()
	166
	167	logger.info("[purge] replacing backward extremities: %r", new_backwards_extrems)
	168
	169	txn.execute(
	170	"DELETE FROM event_backward_extremities WHERE room_id = ?", (room_id,)
	171	)
	172
	173	# Update backward extremeties
	174	txn.executemany(
	175	"INSERT INTO event_backward_extremities (room_id, event_id)"
	176	" VALUES (?, ?)",
	177	[(room_id, event_id) for event_id, in new_backwards_extrems],
	178	)
	179
	180	logger.info("[purge] finding state groups referenced by deleted events")
	181
	182	# Get all state groups that are referenced by events that are to be
	183	# deleted.
	184	txn.execute(
	185	"""
	186	SELECT DISTINCT state_group FROM events_to_purge
	187	INNER JOIN event_to_state_groups USING (event_id)
	188	"""
	189	)
	190
	191	referenced_state_groups = {sg for sg, in txn}
	192	logger.info(
	193	"[purge] found %i referenced state groups", len(referenced_state_groups)
	194	)
	195
	196	logger.info("[purge] removing events from event_to_state_groups")
	197	txn.execute(
	198	"DELETE FROM event_to_state_groups "
	199	"WHERE event_id IN (SELECT event_id from events_to_purge)"
	200	)
	201	for event_id, _ in event_rows:
	202	txn.call_after(self._get_state_group_for_event.invalidate, (event_id,))
	203
	204	# Delete all remote non-state events
	205	for table in (
	206	"events",
	207	"event_json",
	208	"event_auth",
	209	"event_edges",
	210	"event_forward_extremities",
	211	"event_reference_hashes",
	212	"event_relations",
	213	"event_search",
	214	"rejections",
	215	):
	216	logger.info("[purge] removing events from %s", table)
	217
	218	txn.execute(
	219	"DELETE FROM %s WHERE event_id IN ("
	220	" SELECT event_id FROM events_to_purge WHERE should_delete"
	221	")" % (table,)
	222	)
	223
	224	# event_push_actions lacks an index on event_id, and has one on
	225	# (room_id, event_id) instead.
	226	for table in ("event_push_actions",):
	227	logger.info("[purge] removing events from %s", table)
	228
	229	txn.execute(
	230	"DELETE FROM %s WHERE room_id = ? AND event_id IN ("
	231	" SELECT event_id FROM events_to_purge WHERE should_delete"
	232	")" % (table,),
	233	(room_id,),
	234	)
	235
	236	# Mark all state and own events as outliers
	237	logger.info("[purge] marking remaining events as outliers")
	238	txn.execute(
	239	"UPDATE events SET outlier = ?"
	240	" WHERE event_id IN ("
	241	" SELECT event_id FROM events_to_purge "
	242	" WHERE NOT should_delete"
	243	")",
	244	(True,),
	245	)
	246
	247	# synapse tries to take out an exclusive lock on room_depth whenever it
	248	# persists events (because upsert), and once we run this update, we
	249	# will block that for the rest of our transaction.
	250	#
	251	# So, let's stick it at the end so that we don't block event
	252	# persistence.
	253	#
	254	# We do this by calculating the minimum depth of the backwards
	255	# extremities. However, the events in event_backward_extremities
	256	# are ones we don't have yet so we need to look at the events that
	257	# point to it via event_edges table.
	258	txn.execute(
	259	"""
	260	SELECT COALESCE(MIN(depth), 0)
	261	FROM event_backward_extremities AS eb
	262	INNER JOIN event_edges AS eg ON eg.prev_event_id = eb.event_id
	263	INNER JOIN events AS e ON e.event_id = eg.event_id
	264	WHERE eb.room_id = ?
	265	""",
	266	(room_id,),
	267	)
	268	(min_depth,) = txn.fetchone()
	269
	270	logger.info("[purge] updating room_depth to %d", min_depth)
	271
	272	txn.execute(
	273	"UPDATE room_depth SET min_depth = ? WHERE room_id = ?",
	274	(min_depth, room_id),
	275	)
	276
	277	# finally, drop the temp table. this will commit the txn in sqlite,
	278	# so make sure to keep this actually last.
	279	txn.execute("DROP TABLE events_to_purge")
	280
	281	logger.info("[purge] done")
	282
	283	return referenced_state_groups
	284
	285	def purge_room(self, room_id):
	286	"""Deletes all record of a room
	287
	288	Args:
	289	room_id (str)
	290
	291	Returns:
	292	Deferred[List[int]]: The list of state groups to delete.
	293	"""
	294
	295	return self.db_pool.runInteraction("purge_room", self._purge_room_txn, room_id)
	296
	297	def _purge_room_txn(self, txn, room_id):
	298	# First we fetch all the state groups that should be deleted, before
	299	# we delete that information.
	300	txn.execute(
	301	"""
	302	SELECT DISTINCT state_group FROM events
	303	INNER JOIN event_to_state_groups USING(event_id)
	304	WHERE events.room_id = ?
	305	""",
	306	(room_id,),
	307	)
	308
	309	state_groups = [row[0] for row in txn]
	310
	311	# Now we delete tables which lack an index on room_id but have one on event_id
	312	for table in (
	313	"event_auth",
	314	"event_edges",
	315	"event_push_actions_staging",
	316	"event_reference_hashes",
	317	"event_relations",
	318	"event_to_state_groups",
	319	"redactions",
	320	"rejections",
	321	"state_events",
	322	):
	323	logger.info("[purge] removing %s from %s", room_id, table)
	324
	325	txn.execute(
	326	"""
	327	DELETE FROM %s WHERE event_id IN (
	328	SELECT event_id FROM events WHERE room_id=?
	329	)
	330	"""
	331	% (table,),
	332	(room_id,),
	333	)
	334
	335	# and finally, the tables with an index on room_id (or no useful index)
	336	for table in (
	337	"current_state_events",
	338	"event_backward_extremities",
	339	"event_forward_extremities",
	340	"event_json",
	341	"event_push_actions",
	342	"event_search",
	343	"events",
	344	"group_rooms",
	345	"public_room_list_stream",
	346	"receipts_graph",
	347	"receipts_linearized",
	348	"room_aliases",
	349	"room_depth",
	350	"room_memberships",
	351	"room_stats_state",
	352	"room_stats_current",
	353	"room_stats_historical",
	354	"room_stats_earliest_token",
	355	"rooms",
	356	"stream_ordering_to_exterm",
	357	"users_in_public_rooms",
	358	"users_who_share_private_rooms",
	359	# no useful index, but let's clear them anyway
	360	"appservice_room_list",
	361	"e2e_room_keys",
	362	"event_push_summary",
	363	"pusher_throttle",
	364	"group_summary_rooms",
	365	"room_account_data",
	366	"room_tags",
	367	"local_current_membership",
	368	):
	369	logger.info("[purge] removing %s from %s", room_id, table)
	370	txn.execute("DELETE FROM %s WHERE room_id=?" % (table,), (room_id,))
	371
	372	# Other tables we do NOT need to clear out:
	373	#
	374	# - blocked_rooms
	375	# This is important, to make sure that we don't accidentally rejoin a blocked
	376	# room after it was purged
	377	#
	378	# - user_directory
	379	# This has a room_id column, but it is unused
	380	#
	381
	382	# Other tables that we might want to consider clearing out include:
	383	#
	384	# - event_reports
	385	# Given that these are intended for abuse management my initial
	386	# inclination is to leave them in place.
	387	#
	388	# - current_state_delta_stream
	389	# - ex_outlier_stream
	390	# - room_tags_revisions
	391	# The problem with these is that they are largeish and there is no room_id
	392	# index on them. In any case we should be clearing out 'stream' tables
	393	# periodically anyway (#5888)
	394
	395	# TODO: we could probably usefully do a bunch of cache invalidation here
	396
	397	logger.info("[purge] done")
	398
	399	return state_groups

+691

-0

synapse/storage/databases/main/push_rule.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2018 New Vector Ltd
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import abc
	17	import logging
	18	from typing import List, Tuple, Union
	19
	20	from twisted.internet import defer
	21
	22	from synapse.push.baserules import list_with_base_rules
	23	from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
	24	from synapse.storage._base import SQLBaseStore, db_to_json
	25	from synapse.storage.database import DatabasePool
	26	from synapse.storage.databases.main.appservice import ApplicationServiceWorkerStore
	27	from synapse.storage.databases.main.events_worker import EventsWorkerStore
	28	from synapse.storage.databases.main.pusher import PusherWorkerStore
	29	from synapse.storage.databases.main.receipts import ReceiptsWorkerStore
	30	from synapse.storage.databases.main.roommember import RoomMemberWorkerStore
	31	from synapse.storage.push_rule import InconsistentRuleException, RuleNotFoundException
	32	from synapse.storage.util.id_generators import ChainedIdGenerator
	33	from synapse.util import json_encoder
	34	from synapse.util.caches.descriptors import cachedInlineCallbacks, cachedList
	35	from synapse.util.caches.stream_change_cache import StreamChangeCache
	36
	37	logger = logging.getLogger(__name__)
	38
	39
	40	def _load_rules(rawrules, enabled_map, use_new_defaults=False):
	41	ruleslist = []
	42	for rawrule in rawrules:
	43	rule = dict(rawrule)
	44	rule["conditions"] = db_to_json(rawrule["conditions"])
	45	rule["actions"] = db_to_json(rawrule["actions"])
	46	rule["default"] = False
	47	ruleslist.append(rule)
	48
	49	# We're going to be mutating this a lot, so do a deep copy
	50	rules = list(list_with_base_rules(ruleslist, use_new_defaults))
	51
	52	for i, rule in enumerate(rules):
	53	rule_id = rule["rule_id"]
	54	if rule_id in enabled_map:
	55	if rule.get("enabled", True) != bool(enabled_map[rule_id]):
	56	# Rules are cached across users.
	57	rule = dict(rule)
	58	rule["enabled"] = bool(enabled_map[rule_id])
	59	rules[i] = rule
	60
	61	return rules
	62
	63
	64	class PushRulesWorkerStore(
	65	ApplicationServiceWorkerStore,
	66	ReceiptsWorkerStore,
	67	PusherWorkerStore,
	68	RoomMemberWorkerStore,
	69	EventsWorkerStore,
	70	SQLBaseStore,
	71	):
	72	"""This is an abstract base class where subclasses must implement
	73	`get_max_push_rules_stream_id` which can be called in the initializer.
	74	"""
	75
	76	# This ABCMeta metaclass ensures that we cannot be instantiated without
	77	# the abstract methods being implemented.
	78	__metaclass__ = abc.ABCMeta
	79
	80	def __init__(self, database: DatabasePool, db_conn, hs):
	81	super(PushRulesWorkerStore, self).__init__(database, db_conn, hs)
	82
	83	if hs.config.worker.worker_app is None:
	84	self._push_rules_stream_id_gen = ChainedIdGenerator(
	85	self._stream_id_gen, db_conn, "push_rules_stream", "stream_id"
	86	) # type: Union[ChainedIdGenerator, SlavedIdTracker]
	87	else:
	88	self._push_rules_stream_id_gen = SlavedIdTracker(
	89	db_conn, "push_rules_stream", "stream_id"
	90	)
	91
	92	push_rules_prefill, push_rules_id = self.db_pool.get_cache_dict(
	93	db_conn,
	94	"push_rules_stream",
	95	entity_column="user_id",
	96	stream_column="stream_id",
	97	max_value=self.get_max_push_rules_stream_id(),
	98	)
	99
	100	self.push_rules_stream_cache = StreamChangeCache(
	101	"PushRulesStreamChangeCache",
	102	push_rules_id,
	103	prefilled_cache=push_rules_prefill,
	104	)
	105
	106	self._users_new_default_push_rules = hs.config.users_new_default_push_rules
	107
	108	@abc.abstractmethod
	109	def get_max_push_rules_stream_id(self):
	110	"""Get the position of the push rules stream.
	111
	112	Returns:
	113	int
	114	"""
	115	raise NotImplementedError()
	116
	117	@cachedInlineCallbacks(max_entries=5000)
	118	def get_push_rules_for_user(self, user_id):
	119	rows = yield self.db_pool.simple_select_list(
	120	table="push_rules",
	121	keyvalues={"user_name": user_id},
	122	retcols=(
	123	"user_name",
	124	"rule_id",
	125	"priority_class",
	126	"priority",
	127	"conditions",
	128	"actions",
	129	),
	130	desc="get_push_rules_enabled_for_user",
	131	)
	132
	133	rows.sort(key=lambda row: (-int(row["priority_class"]), -int(row["priority"])))
	134
	135	enabled_map = yield self.get_push_rules_enabled_for_user(user_id)
	136
	137	use_new_defaults = user_id in self._users_new_default_push_rules
	138
	139	rules = _load_rules(rows, enabled_map, use_new_defaults)
	140
	141	return rules
	142
	143	@cachedInlineCallbacks(max_entries=5000)
	144	def get_push_rules_enabled_for_user(self, user_id):
	145	results = yield self.db_pool.simple_select_list(
	146	table="push_rules_enable",
	147	keyvalues={"user_name": user_id},
	148	retcols=("user_name", "rule_id", "enabled"),
	149	desc="get_push_rules_enabled_for_user",
	150	)
	151	return {r["rule_id"]: False if r["enabled"] == 0 else True for r in results}
	152
	153	def have_push_rules_changed_for_user(self, user_id, last_id):
	154	if not self.push_rules_stream_cache.has_entity_changed(user_id, last_id):
	155	return defer.succeed(False)
	156	else:
	157
	158	def have_push_rules_changed_txn(txn):
	159	sql = (
	160	"SELECT COUNT(stream_id) FROM push_rules_stream"
	161	" WHERE user_id = ? AND ? < stream_id"
	162	)
	163	txn.execute(sql, (user_id, last_id))
	164	(count,) = txn.fetchone()
	165	return bool(count)
	166
	167	return self.db_pool.runInteraction(
	168	"have_push_rules_changed", have_push_rules_changed_txn
	169	)
	170
	171	@cachedList(
	172	cached_method_name="get_push_rules_for_user",
	173	list_name="user_ids",
	174	num_args=1,
	175	inlineCallbacks=True,
	176	)
	177	def bulk_get_push_rules(self, user_ids):
	178	if not user_ids:
	179	return {}
	180
	181	results = {user_id: [] for user_id in user_ids}
	182
	183	rows = yield self.db_pool.simple_select_many_batch(
	184	table="push_rules",
	185	column="user_name",
	186	iterable=user_ids,
	187	retcols=("*",),
	188	desc="bulk_get_push_rules",
	189	)
	190
	191	rows.sort(key=lambda row: (-int(row["priority_class"]), -int(row["priority"])))
	192
	193	for row in rows:
	194	results.setdefault(row["user_name"], []).append(row)
	195
	196	enabled_map_by_user = yield self.bulk_get_push_rules_enabled(user_ids)
	197
	198	for user_id, rules in results.items():
	199	use_new_defaults = user_id in self._users_new_default_push_rules
	200
	201	results[user_id] = _load_rules(
	202	rules, enabled_map_by_user.get(user_id, {}), use_new_defaults,
	203	)
	204
	205	return results
	206
	207	@defer.inlineCallbacks
	208	def copy_push_rule_from_room_to_room(self, new_room_id, user_id, rule):
	209	"""Copy a single push rule from one room to another for a specific user.
	210
	211	Args:
	212	new_room_id (str): ID of the new room.
	213	user_id (str): ID of user the push rule belongs to.
	214	rule (Dict): A push rule.
	215	"""
	216	# Create new rule id
	217	rule_id_scope = "/".join(rule["rule_id"].split("/")[:-1])
	218	new_rule_id = rule_id_scope + "/" + new_room_id
	219
	220	# Change room id in each condition
	221	for condition in rule.get("conditions", []):
	222	if condition.get("key") == "room_id":
	223	condition["pattern"] = new_room_id
	224
	225	# Add the rule for the new room
	226	yield self.add_push_rule(
	227	user_id=user_id,
	228	rule_id=new_rule_id,
	229	priority_class=rule["priority_class"],
	230	conditions=rule["conditions"],
	231	actions=rule["actions"],
	232	)
	233
	234	@defer.inlineCallbacks
	235	def copy_push_rules_from_room_to_room_for_user(
	236	self, old_room_id, new_room_id, user_id
	237	):
	238	"""Copy all of the push rules from one room to another for a specific
	239	user.
	240
	241	Args:
	242	old_room_id (str): ID of the old room.
	243	new_room_id (str): ID of the new room.
	244	user_id (str): ID of user to copy push rules for.
	245	"""
	246	# Retrieve push rules for this user
	247	user_push_rules = yield self.get_push_rules_for_user(user_id)
	248
	249	# Get rules relating to the old room and copy them to the new room
	250	for rule in user_push_rules:
	251	conditions = rule.get("conditions", [])
	252	if any(
	253	(c.get("key") == "room_id" and c.get("pattern") == old_room_id)
	254	for c in conditions
	255	):
	256	yield self.copy_push_rule_from_room_to_room(new_room_id, user_id, rule)
	257
	258	@cachedList(
	259	cached_method_name="get_push_rules_enabled_for_user",
	260	list_name="user_ids",
	261	num_args=1,
	262	inlineCallbacks=True,
	263	)
	264	def bulk_get_push_rules_enabled(self, user_ids):
	265	if not user_ids:
	266	return {}
	267
	268	results = {user_id: {} for user_id in user_ids}
	269
	270	rows = yield self.db_pool.simple_select_many_batch(
	271	table="push_rules_enable",
	272	column="user_name",
	273	iterable=user_ids,
	274	retcols=("user_name", "rule_id", "enabled"),
	275	desc="bulk_get_push_rules_enabled",
	276	)
	277	for row in rows:
	278	enabled = bool(row["enabled"])
	279	results.setdefault(row["user_name"], {})[row["rule_id"]] = enabled
	280	return results
	281
	282	async def get_all_push_rule_updates(
	283	self, instance_name: str, last_id: int, current_id: int, limit: int
	284	) -> Tuple[List[Tuple[int, tuple]], int, bool]:
	285	"""Get updates for push_rules replication stream.
	286
	287	Args:
	288	instance_name: The writer we want to fetch updates from. Unused
	289	here since there is only ever one writer.
	290	last_id: The token to fetch updates from. Exclusive.
	291	current_id: The token to fetch updates up to. Inclusive.
	292	limit: The requested limit for the number of rows to return. The
	293	function may return more or fewer rows.
	294
	295	Returns:
	296	A tuple consisting of: the updates, a token to use to fetch
	297	subsequent updates, and whether we returned fewer rows than exists
	298	between the requested tokens due to the limit.
	299
	300	The token returned can be used in a subsequent call to this
	301	function to get further updatees.
	302
	303	The updates are a list of 2-tuples of stream ID and the row data
	304	"""
	305
	306	if last_id == current_id:
	307	return [], current_id, False
	308
	309	def get_all_push_rule_updates_txn(txn):
	310	sql = """
	311	SELECT stream_id, user_id
	312	FROM push_rules_stream
	313	WHERE ? < stream_id AND stream_id <= ?
	314	ORDER BY stream_id ASC
	315	LIMIT ?
	316	"""
	317	txn.execute(sql, (last_id, current_id, limit))
	318	updates = [(stream_id, (user_id,)) for stream_id, user_id in txn]
	319
	320	limited = False
	321	upper_bound = current_id
	322	if len(updates) == limit:
	323	limited = True
	324	upper_bound = updates[-1][0]
	325
	326	return updates, upper_bound, limited
	327
	328	return await self.db_pool.runInteraction(
	329	"get_all_push_rule_updates", get_all_push_rule_updates_txn
	330	)
	331
	332
	333	class PushRuleStore(PushRulesWorkerStore):
	334	@defer.inlineCallbacks
	335	def add_push_rule(
	336	self,
	337	user_id,
	338	rule_id,
	339	priority_class,
	340	conditions,
	341	actions,
	342	before=None,
	343	after=None,
	344	):
	345	conditions_json = json_encoder.encode(conditions)
	346	actions_json = json_encoder.encode(actions)
	347	with self._push_rules_stream_id_gen.get_next() as ids:
	348	stream_id, event_stream_ordering = ids
	349	if before or after:
	350	yield self.db_pool.runInteraction(
	351	"_add_push_rule_relative_txn",
	352	self._add_push_rule_relative_txn,
	353	stream_id,
	354	event_stream_ordering,
	355	user_id,
	356	rule_id,
	357	priority_class,
	358	conditions_json,
	359	actions_json,
	360	before,
	361	after,
	362	)
	363	else:
	364	yield self.db_pool.runInteraction(
	365	"_add_push_rule_highest_priority_txn",
	366	self._add_push_rule_highest_priority_txn,
	367	stream_id,
	368	event_stream_ordering,
	369	user_id,
	370	rule_id,
	371	priority_class,
	372	conditions_json,
	373	actions_json,
	374	)
	375
	376	def _add_push_rule_relative_txn(
	377	self,
	378	txn,
	379	stream_id,
	380	event_stream_ordering,
	381	user_id,
	382	rule_id,
	383	priority_class,
	384	conditions_json,
	385	actions_json,
	386	before,
	387	after,
	388	):
	389	# Lock the table since otherwise we'll have annoying races between the
	390	# SELECT here and the UPSERT below.
	391	self.database_engine.lock_table(txn, "push_rules")
	392
	393	relative_to_rule = before or after
	394
	395	res = self.db_pool.simple_select_one_txn(
	396	txn,
	397	table="push_rules",
	398	keyvalues={"user_name": user_id, "rule_id": relative_to_rule},
	399	retcols=["priority_class", "priority"],
	400	allow_none=True,
	401	)
	402
	403	if not res:
	404	raise RuleNotFoundException(
	405	"before/after rule not found: %s" % (relative_to_rule,)
	406	)
	407
	408	base_priority_class = res["priority_class"]
	409	base_rule_priority = res["priority"]
	410
	411	if base_priority_class != priority_class:
	412	raise InconsistentRuleException(
	413	"Given priority class does not match class of relative rule"
	414	)
	415
	416	if before:
	417	# Higher priority rules are executed first, So adding a rule before
	418	# a rule means giving it a higher priority than that rule.
	419	new_rule_priority = base_rule_priority + 1
	420	else:
	421	# We increment the priority of the existing rules to make space for
	422	# the new rule. Therefore if we want this rule to appear after
	423	# an existing rule we give it the priority of the existing rule,
	424	# and then increment the priority of the existing rule.
	425	new_rule_priority = base_rule_priority
	426
	427	sql = (
	428	"UPDATE push_rules SET priority = priority + 1"
	429	" WHERE user_name = ? AND priority_class = ? AND priority >= ?"
	430	)
	431
	432	txn.execute(sql, (user_id, priority_class, new_rule_priority))
	433
	434	self._upsert_push_rule_txn(
	435	txn,
	436	stream_id,
	437	event_stream_ordering,
	438	user_id,
	439	rule_id,
	440	priority_class,
	441	new_rule_priority,
	442	conditions_json,
	443	actions_json,
	444	)
	445
	446	def _add_push_rule_highest_priority_txn(
	447	self,
	448	txn,
	449	stream_id,
	450	event_stream_ordering,
	451	user_id,
	452	rule_id,
	453	priority_class,
	454	conditions_json,
	455	actions_json,
	456	):
	457	# Lock the table since otherwise we'll have annoying races between the
	458	# SELECT here and the UPSERT below.
	459	self.database_engine.lock_table(txn, "push_rules")
	460
	461	# find the highest priority rule in that class
	462	sql = (
	463	"SELECT COUNT(*), MAX(priority) FROM push_rules"
	464	" WHERE user_name = ? and priority_class = ?"
	465	)
	466	txn.execute(sql, (user_id, priority_class))
	467	res = txn.fetchall()
	468	(how_many, highest_prio) = res[0]
	469
	470	new_prio = 0
	471	if how_many > 0:
	472	new_prio = highest_prio + 1
	473
	474	self._upsert_push_rule_txn(
	475	txn,
	476	stream_id,
	477	event_stream_ordering,
	478	user_id,
	479	rule_id,
	480	priority_class,
	481	new_prio,
	482	conditions_json,
	483	actions_json,
	484	)
	485
	486	def _upsert_push_rule_txn(
	487	self,
	488	txn,
	489	stream_id,
	490	event_stream_ordering,
	491	user_id,
	492	rule_id,
	493	priority_class,
	494	priority,
	495	conditions_json,
	496	actions_json,
	497	update_stream=True,
	498	):
	499	"""Specialised version of simple_upsert_txn that picks a push_rule_id
	500	using the _push_rule_id_gen if it needs to insert the rule. It assumes
	501	that the "push_rules" table is locked"""
	502
	503	sql = (
	504	"UPDATE push_rules"
	505	" SET priority_class = ?, priority = ?, conditions = ?, actions = ?"
	506	" WHERE user_name = ? AND rule_id = ?"
	507	)
	508
	509	txn.execute(
	510	sql,
	511	(priority_class, priority, conditions_json, actions_json, user_id, rule_id),
	512	)
	513
	514	if txn.rowcount == 0:
	515	# We didn't update a row with the given rule_id so insert one
	516	push_rule_id = self._push_rule_id_gen.get_next()
	517
	518	self.db_pool.simple_insert_txn(
	519	txn,
	520	table="push_rules",
	521	values={
	522	"id": push_rule_id,
	523	"user_name": user_id,
	524	"rule_id": rule_id,
	525	"priority_class": priority_class,
	526	"priority": priority,
	527	"conditions": conditions_json,
	528	"actions": actions_json,
	529	},
	530	)
	531
	532	if update_stream:
	533	self._insert_push_rules_update_txn(
	534	txn,
	535	stream_id,
	536	event_stream_ordering,
	537	user_id,
	538	rule_id,
	539	op="ADD",
	540	data={
	541	"priority_class": priority_class,
	542	"priority": priority,
	543	"conditions": conditions_json,
	544	"actions": actions_json,
	545	},
	546	)
	547
	548	@defer.inlineCallbacks
	549	def delete_push_rule(self, user_id, rule_id):
	550	"""
	551	Delete a push rule. Args specify the row to be deleted and can be
	552	any of the columns in the push_rule table, but below are the
	553	standard ones
	554
	555	Args:
	556	user_id (str): The matrix ID of the push rule owner
	557	rule_id (str): The rule_id of the rule to be deleted
	558	"""
	559
	560	def delete_push_rule_txn(txn, stream_id, event_stream_ordering):
	561	self.db_pool.simple_delete_one_txn(
	562	txn, "push_rules", {"user_name": user_id, "rule_id": rule_id}
	563	)
	564
	565	self._insert_push_rules_update_txn(
	566	txn, stream_id, event_stream_ordering, user_id, rule_id, op="DELETE"
	567	)
	568
	569	with self._push_rules_stream_id_gen.get_next() as ids:
	570	stream_id, event_stream_ordering = ids
	571	yield self.db_pool.runInteraction(
	572	"delete_push_rule",
	573	delete_push_rule_txn,
	574	stream_id,
	575	event_stream_ordering,
	576	)
	577
	578	@defer.inlineCallbacks
	579	def set_push_rule_enabled(self, user_id, rule_id, enabled):
	580	with self._push_rules_stream_id_gen.get_next() as ids:
	581	stream_id, event_stream_ordering = ids
	582	yield self.db_pool.runInteraction(
	583	"_set_push_rule_enabled_txn",
	584	self._set_push_rule_enabled_txn,
	585	stream_id,
	586	event_stream_ordering,
	587	user_id,
	588	rule_id,
	589	enabled,
	590	)
	591
	592	def _set_push_rule_enabled_txn(
	593	self, txn, stream_id, event_stream_ordering, user_id, rule_id, enabled
	594	):
	595	new_id = self._push_rules_enable_id_gen.get_next()
	596	self.db_pool.simple_upsert_txn(
	597	txn,
	598	"push_rules_enable",
	599	{"user_name": user_id, "rule_id": rule_id},
	600	{"enabled": 1 if enabled else 0},
	601	{"id": new_id},
	602	)
	603
	604	self._insert_push_rules_update_txn(
	605	txn,
	606	stream_id,
	607	event_stream_ordering,
	608	user_id,
	609	rule_id,
	610	op="ENABLE" if enabled else "DISABLE",
	611	)
	612
	613	@defer.inlineCallbacks
	614	def set_push_rule_actions(self, user_id, rule_id, actions, is_default_rule):
	615	actions_json = json_encoder.encode(actions)
	616
	617	def set_push_rule_actions_txn(txn, stream_id, event_stream_ordering):
	618	if is_default_rule:
	619	# Add a dummy rule to the rules table with the user specified
	620	# actions.
	621	priority_class = -1
	622	priority = 1
	623	self._upsert_push_rule_txn(
	624	txn,
	625	stream_id,
	626	event_stream_ordering,
	627	user_id,
	628	rule_id,
	629	priority_class,
	630	priority,
	631	"[]",
	632	actions_json,
	633	update_stream=False,
	634	)
	635	else:
	636	self.db_pool.simple_update_one_txn(
	637	txn,
	638	"push_rules",
	639	{"user_name": user_id, "rule_id": rule_id},
	640	{"actions": actions_json},
	641	)
	642
	643	self._insert_push_rules_update_txn(
	644	txn,
	645	stream_id,
	646	event_stream_ordering,
	647	user_id,
	648	rule_id,
	649	op="ACTIONS",
	650	data={"actions": actions_json},
	651	)
	652
	653	with self._push_rules_stream_id_gen.get_next() as ids:
	654	stream_id, event_stream_ordering = ids
	655	yield self.db_pool.runInteraction(
	656	"set_push_rule_actions",
	657	set_push_rule_actions_txn,
	658	stream_id,
	659	event_stream_ordering,
	660	)
	661
	662	def _insert_push_rules_update_txn(
	663	self, txn, stream_id, event_stream_ordering, user_id, rule_id, op, data=None
	664	):
	665	values = {
	666	"stream_id": stream_id,
	667	"event_stream_ordering": event_stream_ordering,
	668	"user_id": user_id,
	669	"rule_id": rule_id,
	670	"op": op,
	671	}
	672	if data is not None:
	673	values.update(data)
	674
	675	self.db_pool.simple_insert_txn(txn, "push_rules_stream", values=values)
	676
	677	txn.call_after(self.get_push_rules_for_user.invalidate, (user_id,))
	678	txn.call_after(self.get_push_rules_enabled_for_user.invalidate, (user_id,))
	679	txn.call_after(
	680	self.push_rules_stream_cache.entity_has_changed, user_id, stream_id
	681	)
	682
	683	def get_push_rules_stream_token(self):
	684	"""Get the position of the push rules stream.
	685	Returns a pair of a stream id for the push_rules stream and the
	686	room stream ordering it corresponds to."""
	687	return self._push_rules_stream_id_gen.get_current_token()
	688
	689	def get_max_push_rules_stream_id(self):
	690	return self.get_push_rules_stream_token()[0]

+356

-0

synapse/storage/databases/main/pusher.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2018 New Vector Ltd
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import logging
	17	from typing import Iterable, Iterator, List, Tuple
	18
	19	from canonicaljson import encode_canonical_json
	20
	21	from twisted.internet import defer
	22
	23	from synapse.storage._base import SQLBaseStore, db_to_json
	24	from synapse.util.caches.descriptors import cachedInlineCallbacks, cachedList
	25
	26	logger = logging.getLogger(__name__)
	27
	28
	29	class PusherWorkerStore(SQLBaseStore):
	30	def _decode_pushers_rows(self, rows: Iterable[dict]) -> Iterator[dict]:
	31	"""JSON-decode the data in the rows returned from the `pushers` table
	32
	33	Drops any rows whose data cannot be decoded
	34	"""
	35	for r in rows:
	36	dataJson = r["data"]
	37	try:
	38	r["data"] = db_to_json(dataJson)
	39	except Exception as e:
	40	logger.warning(
	41	"Invalid JSON in data for pusher %d: %s, %s",
	42	r["id"],
	43	dataJson,
	44	e.args[0],
	45	)
	46	continue
	47
	48	yield r
	49
	50	@defer.inlineCallbacks
	51	def user_has_pusher(self, user_id):
	52	ret = yield self.db_pool.simple_select_one_onecol(
	53	"pushers", {"user_name": user_id}, "id", allow_none=True
	54	)
	55	return ret is not None
	56
	57	def get_pushers_by_app_id_and_pushkey(self, app_id, pushkey):
	58	return self.get_pushers_by({"app_id": app_id, "pushkey": pushkey})
	59
	60	def get_pushers_by_user_id(self, user_id):
	61	return self.get_pushers_by({"user_name": user_id})
	62
	63	@defer.inlineCallbacks
	64	def get_pushers_by(self, keyvalues):
	65	ret = yield self.db_pool.simple_select_list(
	66	"pushers",
	67	keyvalues,
	68	[
	69	"id",
	70	"user_name",
	71	"access_token",
	72	"profile_tag",
	73	"kind",
	74	"app_id",
	75	"app_display_name",
	76	"device_display_name",
	77	"pushkey",
	78	"ts",
	79	"lang",
	80	"data",
	81	"last_stream_ordering",
	82	"last_success",
	83	"failing_since",
	84	],
	85	desc="get_pushers_by",
	86	)
	87	return self._decode_pushers_rows(ret)
	88
	89	@defer.inlineCallbacks
	90	def get_all_pushers(self):
	91	def get_pushers(txn):
	92	txn.execute("SELECT * FROM pushers")
	93	rows = self.db_pool.cursor_to_dict(txn)
	94
	95	return self._decode_pushers_rows(rows)
	96
	97	rows = yield self.db_pool.runInteraction("get_all_pushers", get_pushers)
	98	return rows
	99
	100	async def get_all_updated_pushers_rows(
	101	self, instance_name: str, last_id: int, current_id: int, limit: int
	102	) -> Tuple[List[Tuple[int, tuple]], int, bool]:
	103	"""Get updates for pushers replication stream.
	104
	105	Args:
	106	instance_name: The writer we want to fetch updates from. Unused
	107	here since there is only ever one writer.
	108	last_id: The token to fetch updates from. Exclusive.
	109	current_id: The token to fetch updates up to. Inclusive.
	110	limit: The requested limit for the number of rows to return. The
	111	function may return more or fewer rows.
	112
	113	Returns:
	114	A tuple consisting of: the updates, a token to use to fetch
	115	subsequent updates, and whether we returned fewer rows than exists
	116	between the requested tokens due to the limit.
	117
	118	The token returned can be used in a subsequent call to this
	119	function to get further updatees.
	120
	121	The updates are a list of 2-tuples of stream ID and the row data
	122	"""
	123
	124	if last_id == current_id:
	125	return [], current_id, False
	126
	127	def get_all_updated_pushers_rows_txn(txn):
	128	sql = """
	129	SELECT id, user_name, app_id, pushkey
	130	FROM pushers
	131	WHERE ? < id AND id <= ?
	132	ORDER BY id ASC LIMIT ?
	133	"""
	134	txn.execute(sql, (last_id, current_id, limit))
	135	updates = [
	136	(stream_id, (user_name, app_id, pushkey, False))
	137	for stream_id, user_name, app_id, pushkey in txn
	138	]
	139
	140	sql = """
	141	SELECT stream_id, user_id, app_id, pushkey
	142	FROM deleted_pushers
	143	WHERE ? < stream_id AND stream_id <= ?
	144	ORDER BY stream_id ASC LIMIT ?
	145	"""
	146	txn.execute(sql, (last_id, current_id, limit))
	147	updates.extend(
	148	(stream_id, (user_name, app_id, pushkey, True))
	149	for stream_id, user_name, app_id, pushkey in txn
	150	)
	151
	152	updates.sort() # Sort so that they're ordered by stream id
	153
	154	limited = False
	155	upper_bound = current_id
	156	if len(updates) >= limit:
	157	limited = True
	158	upper_bound = updates[-1][0]
	159
	160	return updates, upper_bound, limited
	161
	162	return await self.db_pool.runInteraction(
	163	"get_all_updated_pushers_rows", get_all_updated_pushers_rows_txn
	164	)
	165
	166	@cachedInlineCallbacks(num_args=1, max_entries=15000)
	167	def get_if_user_has_pusher(self, user_id):
	168	# This only exists for the cachedList decorator
	169	raise NotImplementedError()
	170
	171	@cachedList(
	172	cached_method_name="get_if_user_has_pusher",
	173	list_name="user_ids",
	174	num_args=1,
	175	inlineCallbacks=True,
	176	)
	177	def get_if_users_have_pushers(self, user_ids):
	178	rows = yield self.db_pool.simple_select_many_batch(
	179	table="pushers",
	180	column="user_name",
	181	iterable=user_ids,
	182	retcols=["user_name"],
	183	desc="get_if_users_have_pushers",
	184	)
	185
	186	result = {user_id: False for user_id in user_ids}
	187	result.update({r["user_name"]: True for r in rows})
	188
	189	return result
	190
	191	@defer.inlineCallbacks
	192	def update_pusher_last_stream_ordering(
	193	self, app_id, pushkey, user_id, last_stream_ordering
	194	):
	195	yield self.db_pool.simple_update_one(
	196	"pushers",
	197	{"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
	198	{"last_stream_ordering": last_stream_ordering},
	199	desc="update_pusher_last_stream_ordering",
	200	)
	201
	202	@defer.inlineCallbacks
	203	def update_pusher_last_stream_ordering_and_success(
	204	self, app_id, pushkey, user_id, last_stream_ordering, last_success
	205	):
	206	"""Update the last stream ordering position we've processed up to for
	207	the given pusher.
	208
	209	Args:
	210	app_id (str)
	211	pushkey (str)
	212	last_stream_ordering (int)
	213	last_success (int)
	214
	215	Returns:
	216	Deferred[bool]: True if the pusher still exists; False if it has been deleted.
	217	"""
	218	updated = yield self.db_pool.simple_update(
	219	table="pushers",
	220	keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
	221	updatevalues={
	222	"last_stream_ordering": last_stream_ordering,
	223	"last_success": last_success,
	224	},
	225	desc="update_pusher_last_stream_ordering_and_success",
	226	)
	227
	228	return bool(updated)
	229
	230	@defer.inlineCallbacks
	231	def update_pusher_failing_since(self, app_id, pushkey, user_id, failing_since):
	232	yield self.db_pool.simple_update(
	233	table="pushers",
	234	keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
	235	updatevalues={"failing_since": failing_since},
	236	desc="update_pusher_failing_since",
	237	)
	238
	239	@defer.inlineCallbacks
	240	def get_throttle_params_by_room(self, pusher_id):
	241	res = yield self.db_pool.simple_select_list(
	242	"pusher_throttle",
	243	{"pusher": pusher_id},
	244	["room_id", "last_sent_ts", "throttle_ms"],
	245	desc="get_throttle_params_by_room",
	246	)
	247
	248	params_by_room = {}
	249	for row in res:
	250	params_by_room[row["room_id"]] = {
	251	"last_sent_ts": row["last_sent_ts"],
	252	"throttle_ms": row["throttle_ms"],
	253	}
	254
	255	return params_by_room
	256
	257	@defer.inlineCallbacks
	258	def set_throttle_params(self, pusher_id, room_id, params):
	259	# no need to lock because `pusher_throttle` has a primary key on
	260	# (pusher, room_id) so simple_upsert will retry
	261	yield self.db_pool.simple_upsert(
	262	"pusher_throttle",
	263	{"pusher": pusher_id, "room_id": room_id},
	264	params,
	265	desc="set_throttle_params",
	266	lock=False,
	267	)
	268
	269
	270	class PusherStore(PusherWorkerStore):
	271	def get_pushers_stream_token(self):
	272	return self._pushers_id_gen.get_current_token()
	273
	274	@defer.inlineCallbacks
	275	def add_pusher(
	276	self,
	277	user_id,
	278	access_token,
	279	kind,
	280	app_id,
	281	app_display_name,
	282	device_display_name,
	283	pushkey,
	284	pushkey_ts,
	285	lang,
	286	data,
	287	last_stream_ordering,
	288	profile_tag="",
	289	):
	290	with self._pushers_id_gen.get_next() as stream_id:
	291	# no need to lock because `pushers` has a unique key on
	292	# (app_id, pushkey, user_name) so simple_upsert will retry
	293	yield self.db_pool.simple_upsert(
	294	table="pushers",
	295	keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
	296	values={
	297	"access_token": access_token,
	298	"kind": kind,
	299	"app_display_name": app_display_name,
	300	"device_display_name": device_display_name,
	301	"ts": pushkey_ts,
	302	"lang": lang,
	303	"data": bytearray(encode_canonical_json(data)),
	304	"last_stream_ordering": last_stream_ordering,
	305	"profile_tag": profile_tag,
	306	"id": stream_id,
	307	},
	308	desc="add_pusher",
	309	lock=False,
	310	)
	311
	312	user_has_pusher = self.get_if_user_has_pusher.cache.get(
	313	(user_id,), None, update_metrics=False
	314	)
	315
	316	if user_has_pusher is not True:
	317	# invalidate, since we the user might not have had a pusher before
	318	yield self.db_pool.runInteraction(
	319	"add_pusher",
	320	self._invalidate_cache_and_stream,
	321	self.get_if_user_has_pusher,
	322	(user_id,),
	323	)
	324
	325	@defer.inlineCallbacks
	326	def delete_pusher_by_app_id_pushkey_user_id(self, app_id, pushkey, user_id):
	327	def delete_pusher_txn(txn, stream_id):
	328	self._invalidate_cache_and_stream(
	329	txn, self.get_if_user_has_pusher, (user_id,)
	330	)
	331
	332	self.db_pool.simple_delete_one_txn(
	333	txn,
	334	"pushers",
	335	{"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
	336	)
	337
	338	# it's possible for us to end up with duplicate rows for
	339	# (app_id, pushkey, user_id) at different stream_ids, but that
	340	# doesn't really matter.
	341	self.db_pool.simple_insert_txn(
	342	txn,
	343	table="deleted_pushers",
	344	values={
	345	"stream_id": stream_id,
	346	"app_id": app_id,
	347	"pushkey": pushkey,
	348	"user_id": user_id,
	349	},
	350	)
	351
	352	with self._pushers_id_gen.get_next() as stream_id:
	353	yield self.db_pool.runInteraction(
	354	"delete_pusher", delete_pusher_txn, stream_id
	355	)

+590

-0

synapse/storage/databases/main/receipts.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2018 New Vector Ltd
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import abc
	17	import logging
	18	from typing import List, Tuple
	19
	20	from twisted.internet import defer
	21
	22	from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
	23	from synapse.storage.database import DatabasePool
	24	from synapse.storage.util.id_generators import StreamIdGenerator
	25	from synapse.util import json_encoder
	26	from synapse.util.async_helpers import ObservableDeferred
	27	from synapse.util.caches.descriptors import cached, cachedInlineCallbacks, cachedList
	28	from synapse.util.caches.stream_change_cache import StreamChangeCache
	29
	30	logger = logging.getLogger(__name__)
	31
	32
	33	class ReceiptsWorkerStore(SQLBaseStore):
	34	"""This is an abstract base class where subclasses must implement
	35	`get_max_receipt_stream_id` which can be called in the initializer.
	36	"""
	37
	38	# This ABCMeta metaclass ensures that we cannot be instantiated without
	39	# the abstract methods being implemented.
	40	__metaclass__ = abc.ABCMeta
	41
	42	def __init__(self, database: DatabasePool, db_conn, hs):
	43	super(ReceiptsWorkerStore, self).__init__(database, db_conn, hs)
	44
	45	self._receipts_stream_cache = StreamChangeCache(
	46	"ReceiptsRoomChangeCache", self.get_max_receipt_stream_id()
	47	)
	48
	49	@abc.abstractmethod
	50	def get_max_receipt_stream_id(self):
	51	"""Get the current max stream ID for receipts stream
	52
	53	Returns:
	54	int
	55	"""
	56	raise NotImplementedError()
	57
	58	@cachedInlineCallbacks()
	59	def get_users_with_read_receipts_in_room(self, room_id):
	60	receipts = yield self.get_receipts_for_room(room_id, "m.read")
	61	return {r["user_id"] for r in receipts}
	62
	63	@cached(num_args=2)
	64	def get_receipts_for_room(self, room_id, receipt_type):
	65	return self.db_pool.simple_select_list(
	66	table="receipts_linearized",
	67	keyvalues={"room_id": room_id, "receipt_type": receipt_type},
	68	retcols=("user_id", "event_id"),
	69	desc="get_receipts_for_room",
	70	)
	71
	72	@cached(num_args=3)
	73	def get_last_receipt_event_id_for_user(self, user_id, room_id, receipt_type):
	74	return self.db_pool.simple_select_one_onecol(
	75	table="receipts_linearized",
	76	keyvalues={
	77	"room_id": room_id,
	78	"receipt_type": receipt_type,
	79	"user_id": user_id,
	80	},
	81	retcol="event_id",
	82	desc="get_own_receipt_for_user",
	83	allow_none=True,
	84	)
	85
	86	@cachedInlineCallbacks(num_args=2)
	87	def get_receipts_for_user(self, user_id, receipt_type):
	88	rows = yield self.db_pool.simple_select_list(
	89	table="receipts_linearized",
	90	keyvalues={"user_id": user_id, "receipt_type": receipt_type},
	91	retcols=("room_id", "event_id"),
	92	desc="get_receipts_for_user",
	93	)
	94
	95	return {row["room_id"]: row["event_id"] for row in rows}
	96
	97	@defer.inlineCallbacks
	98	def get_receipts_for_user_with_orderings(self, user_id, receipt_type):
	99	def f(txn):
	100	sql = (
	101	"SELECT rl.room_id, rl.event_id,"
	102	" e.topological_ordering, e.stream_ordering"
	103	" FROM receipts_linearized AS rl"
	104	" INNER JOIN events AS e USING (room_id, event_id)"
	105	" WHERE rl.room_id = e.room_id"
	106	" AND rl.event_id = e.event_id"
	107	" AND user_id = ?"
	108	)
	109	txn.execute(sql, (user_id,))
	110	return txn.fetchall()
	111
	112	rows = yield self.db_pool.runInteraction(
	113	"get_receipts_for_user_with_orderings", f
	114	)
	115	return {
	116	row[0]: {
	117	"event_id": row[1],
	118	"topological_ordering": row[2],
	119	"stream_ordering": row[3],
	120	}
	121	for row in rows
	122	}
	123
	124	@defer.inlineCallbacks
	125	def get_linearized_receipts_for_rooms(self, room_ids, to_key, from_key=None):
	126	"""Get receipts for multiple rooms for sending to clients.
	127
	128	Args:
	129	room_ids (list): List of room_ids.
	130	to_key (int): Max stream id to fetch receipts upto.
	131	from_key (int): Min stream id to fetch receipts from. None fetches
	132	from the start.
	133
	134	Returns:
	135	list: A list of receipts.
	136	"""
	137	room_ids = set(room_ids)
	138
	139	if from_key is not None:
	140	# Only ask the database about rooms where there have been new
	141	# receipts added since `from_key`
	142	room_ids = yield self._receipts_stream_cache.get_entities_changed(
	143	room_ids, from_key
	144	)
	145
	146	results = yield self._get_linearized_receipts_for_rooms(
	147	room_ids, to_key, from_key=from_key
	148	)
	149
	150	return [ev for res in results.values() for ev in res]
	151
	152	def get_linearized_receipts_for_room(self, room_id, to_key, from_key=None):
	153	"""Get receipts for a single room for sending to clients.
	154
	155	Args:
	156	room_ids (str): The room id.
	157	to_key (int): Max stream id to fetch receipts upto.
	158	from_key (int): Min stream id to fetch receipts from. None fetches
	159	from the start.
	160
	161	Returns:
	162	Deferred[list]: A list of receipts.
	163	"""
	164	if from_key is not None:
	165	# Check the cache first to see if any new receipts have been added
	166	# since`from_key`. If not we can no-op.
	167	if not self._receipts_stream_cache.has_entity_changed(room_id, from_key):
	168	defer.succeed([])
	169
	170	return self._get_linearized_receipts_for_room(room_id, to_key, from_key)
	171
	172	@cachedInlineCallbacks(num_args=3, tree=True)
	173	def _get_linearized_receipts_for_room(self, room_id, to_key, from_key=None):
	174	"""See get_linearized_receipts_for_room
	175	"""
	176
	177	def f(txn):
	178	if from_key:
	179	sql = (
	180	"SELECT * FROM receipts_linearized WHERE"
	181	" room_id = ? AND stream_id > ? AND stream_id <= ?"
	182	)
	183
	184	txn.execute(sql, (room_id, from_key, to_key))
	185	else:
	186	sql = (
	187	"SELECT * FROM receipts_linearized WHERE"
	188	" room_id = ? AND stream_id <= ?"
	189	)
	190
	191	txn.execute(sql, (room_id, to_key))
	192
	193	rows = self.db_pool.cursor_to_dict(txn)
	194
	195	return rows
	196
	197	rows = yield self.db_pool.runInteraction("get_linearized_receipts_for_room", f)
	198
	199	if not rows:
	200	return []
	201
	202	content = {}
	203	for row in rows:
	204	content.setdefault(row["event_id"], {}).setdefault(row["receipt_type"], {})[
	205	row["user_id"]
	206	] = db_to_json(row["data"])
	207
	208	return [{"type": "m.receipt", "room_id": room_id, "content": content}]
	209
	210	@cachedList(
	211	cached_method_name="_get_linearized_receipts_for_room",
	212	list_name="room_ids",
	213	num_args=3,
	214	inlineCallbacks=True,
	215	)
	216	def _get_linearized_receipts_for_rooms(self, room_ids, to_key, from_key=None):
	217	if not room_ids:
	218	return {}
	219
	220	def f(txn):
	221	if from_key:
	222	sql = """
	223	SELECT * FROM receipts_linearized WHERE
	224	stream_id > ? AND stream_id <= ? AND
	225	"""
	226	clause, args = make_in_list_sql_clause(
	227	self.database_engine, "room_id", room_ids
	228	)
	229
	230	txn.execute(sql + clause, [from_key, to_key] + list(args))
	231	else:
	232	sql = """
	233	SELECT * FROM receipts_linearized WHERE
	234	stream_id <= ? AND
	235	"""
	236
	237	clause, args = make_in_list_sql_clause(
	238	self.database_engine, "room_id", room_ids
	239	)
	240
	241	txn.execute(sql + clause, [to_key] + list(args))
	242
	243	return self.db_pool.cursor_to_dict(txn)
	244
	245	txn_results = yield self.db_pool.runInteraction(
	246	"_get_linearized_receipts_for_rooms", f
	247	)
	248
	249	results = {}
	250	for row in txn_results:
	251	# We want a single event per room, since we want to batch the
	252	# receipts by room, event and type.
	253	room_event = results.setdefault(
	254	row["room_id"],
	255	{"type": "m.receipt", "room_id": row["room_id"], "content": {}},
	256	)
	257
	258	# The content is of the form:
	259	# {"$foo:bar": { "read": { "@user:host": <receipt> }, .. }, .. }
	260	event_entry = room_event["content"].setdefault(row["event_id"], {})
	261	receipt_type = event_entry.setdefault(row["receipt_type"], {})
	262
	263	receipt_type[row["user_id"]] = db_to_json(row["data"])
	264
	265	results = {
	266	room_id: [results[room_id]] if room_id in results else []
	267	for room_id in room_ids
	268	}
	269	return results
	270
	271	def get_users_sent_receipts_between(self, last_id: int, current_id: int):
	272	"""Get all users who sent receipts between `last_id` exclusive and
	273	`current_id` inclusive.
	274
	275	Returns:
	276	Deferred[List[str]]
	277	"""
	278
	279	if last_id == current_id:
	280	return defer.succeed([])
	281
	282	def _get_users_sent_receipts_between_txn(txn):
	283	sql = """
	284	SELECT DISTINCT user_id FROM receipts_linearized
	285	WHERE ? < stream_id AND stream_id <= ?
	286	"""
	287	txn.execute(sql, (last_id, current_id))
	288
	289	return [r[0] for r in txn]
	290
	291	return self.db_pool.runInteraction(
	292	"get_users_sent_receipts_between", _get_users_sent_receipts_between_txn
	293	)
	294
	295	async def get_all_updated_receipts(
	296	self, instance_name: str, last_id: int, current_id: int, limit: int
	297	) -> Tuple[List[Tuple[int, list]], int, bool]:
	298	"""Get updates for receipts replication stream.
	299
	300	Args:
	301	instance_name: The writer we want to fetch updates from. Unused
	302	here since there is only ever one writer.
	303	last_id: The token to fetch updates from. Exclusive.
	304	current_id: The token to fetch updates up to. Inclusive.
	305	limit: The requested limit for the number of rows to return. The
	306	function may return more or fewer rows.
	307
	308	Returns:
	309	A tuple consisting of: the updates, a token to use to fetch
	310	subsequent updates, and whether we returned fewer rows than exists
	311	between the requested tokens due to the limit.
	312
	313	The token returned can be used in a subsequent call to this
	314	function to get further updatees.
	315
	316	The updates are a list of 2-tuples of stream ID and the row data
	317	"""
	318
	319	if last_id == current_id:
	320	return [], current_id, False
	321
	322	def get_all_updated_receipts_txn(txn):
	323	sql = """
	324	SELECT stream_id, room_id, receipt_type, user_id, event_id, data
	325	FROM receipts_linearized
	326	WHERE ? < stream_id AND stream_id <= ?
	327	ORDER BY stream_id ASC
	328	LIMIT ?
	329	"""
	330	txn.execute(sql, (last_id, current_id, limit))
	331
	332	updates = [(r[0], r[1:5] + (db_to_json(r[5]),)) for r in txn]
	333
	334	limited = False
	335	upper_bound = current_id
	336
	337	if len(updates) == limit:
	338	limited = True
	339	upper_bound = updates[-1][0]
	340
	341	return updates, upper_bound, limited
	342
	343	return await self.db_pool.runInteraction(
	344	"get_all_updated_receipts", get_all_updated_receipts_txn
	345	)
	346
	347	def _invalidate_get_users_with_receipts_in_room(
	348	self, room_id, receipt_type, user_id
	349	):
	350	if receipt_type != "m.read":
	351	return
	352
	353	# Returns either an ObservableDeferred or the raw result
	354	res = self.get_users_with_read_receipts_in_room.cache.get(
	355	room_id, None, update_metrics=False
	356	)
	357
	358	# first handle the ObservableDeferred case
	359	if isinstance(res, ObservableDeferred):
	360	if res.has_called():
	361	res = res.get_result()
	362	else:
	363	res = None
	364
	365	if res and user_id in res:
	366	# We'd only be adding to the set, so no point invalidating if the
	367	# user is already there
	368	return
	369
	370	self.get_users_with_read_receipts_in_room.invalidate((room_id,))
	371
	372
	373	class ReceiptsStore(ReceiptsWorkerStore):
	374	def __init__(self, database: DatabasePool, db_conn, hs):
	375	# We instantiate this first as the ReceiptsWorkerStore constructor
	376	# needs to be able to call get_max_receipt_stream_id
	377	self._receipts_id_gen = StreamIdGenerator(
	378	db_conn, "receipts_linearized", "stream_id"
	379	)
	380
	381	super(ReceiptsStore, self).__init__(database, db_conn, hs)
	382
	383	def get_max_receipt_stream_id(self):
	384	return self._receipts_id_gen.get_current_token()
	385
	386	def insert_linearized_receipt_txn(
	387	self, txn, room_id, receipt_type, user_id, event_id, data, stream_id
	388	):
	389	"""Inserts a read-receipt into the database if it's newer than the current RR
	390
	391	Returns: int\|None
	392	None if the RR is older than the current RR
	393	otherwise, the rx timestamp of the event that the RR corresponds to
	394	(or 0 if the event is unknown)
	395	"""
	396	res = self.db_pool.simple_select_one_txn(
	397	txn,
	398	table="events",
	399	retcols=["stream_ordering", "received_ts"],
	400	keyvalues={"event_id": event_id},
	401	allow_none=True,
	402	)
	403
	404	stream_ordering = int(res["stream_ordering"]) if res else None
	405	rx_ts = res["received_ts"] if res else 0
	406
	407	# We don't want to clobber receipts for more recent events, so we
	408	# have to compare orderings of existing receipts
	409	if stream_ordering is not None:
	410	sql = (
	411	"SELECT stream_ordering, event_id FROM events"
	412	" INNER JOIN receipts_linearized as r USING (event_id, room_id)"
	413	" WHERE r.room_id = ? AND r.receipt_type = ? AND r.user_id = ?"
	414	)
	415	txn.execute(sql, (room_id, receipt_type, user_id))
	416
	417	for so, eid in txn:
	418	if int(so) >= stream_ordering:
	419	logger.debug(
	420	"Ignoring new receipt for %s in favour of existing "
	421	"one for later event %s",
	422	event_id,
	423	eid,
	424	)
	425	return None
	426
	427	txn.call_after(self.get_receipts_for_room.invalidate, (room_id, receipt_type))
	428	txn.call_after(
	429	self._invalidate_get_users_with_receipts_in_room,
	430	room_id,
	431	receipt_type,
	432	user_id,
	433	)
	434	txn.call_after(self.get_receipts_for_user.invalidate, (user_id, receipt_type))
	435	# FIXME: This shouldn't invalidate the whole cache
	436	txn.call_after(
	437	self._get_linearized_receipts_for_room.invalidate_many, (room_id,)
	438	)
	439
	440	txn.call_after(
	441	self._receipts_stream_cache.entity_has_changed, room_id, stream_id
	442	)
	443
	444	txn.call_after(
	445	self.get_last_receipt_event_id_for_user.invalidate,
	446	(user_id, room_id, receipt_type),
	447	)
	448
	449	self.db_pool.simple_upsert_txn(
	450	txn,
	451	table="receipts_linearized",
	452	keyvalues={
	453	"room_id": room_id,
	454	"receipt_type": receipt_type,
	455	"user_id": user_id,
	456	},
	457	values={
	458	"stream_id": stream_id,
	459	"event_id": event_id,
	460	"data": json_encoder.encode(data),
	461	},
	462	# receipts_linearized has a unique constraint on
	463	# (user_id, room_id, receipt_type), so no need to lock
	464	lock=False,
	465	)
	466
	467	if receipt_type == "m.read" and stream_ordering is not None:
	468	self._remove_old_push_actions_before_txn(
	469	txn, room_id=room_id, user_id=user_id, stream_ordering=stream_ordering
	470	)
	471
	472	return rx_ts
	473
	474	@defer.inlineCallbacks
	475	def insert_receipt(self, room_id, receipt_type, user_id, event_ids, data):
	476	"""Insert a receipt, either from local client or remote server.
	477
	478	Automatically does conversion between linearized and graph
	479	representations.
	480	"""
	481	if not event_ids:
	482	return
	483
	484	if len(event_ids) == 1:
	485	linearized_event_id = event_ids[0]
	486	else:
	487	# we need to points in graph -> linearized form.
	488	# TODO: Make this better.
	489	def graph_to_linear(txn):
	490	clause, args = make_in_list_sql_clause(
	491	self.database_engine, "event_id", event_ids
	492	)
	493
	494	sql = """
	495	SELECT event_id WHERE room_id = ? AND stream_ordering IN (
	496	SELECT max(stream_ordering) WHERE %s
	497	)
	498	""" % (
	499	clause,
	500	)
	501
	502	txn.execute(sql, [room_id] + list(args))
	503	rows = txn.fetchall()
	504	if rows:
	505	return rows[0][0]
	506	else:
	507	raise RuntimeError("Unrecognized event_ids: %r" % (event_ids,))
	508
	509	linearized_event_id = yield self.db_pool.runInteraction(
	510	"insert_receipt_conv", graph_to_linear
	511	)
	512
	513	stream_id_manager = self._receipts_id_gen.get_next()
	514	with stream_id_manager as stream_id:
	515	event_ts = yield self.db_pool.runInteraction(
	516	"insert_linearized_receipt",
	517	self.insert_linearized_receipt_txn,
	518	room_id,
	519	receipt_type,
	520	user_id,
	521	linearized_event_id,
	522	data,
	523	stream_id=stream_id,
	524	)
	525
	526	if event_ts is None:
	527	return None
	528
	529	now = self._clock.time_msec()
	530	logger.debug(
	531	"RR for event %s in %s (%i ms old)",
	532	linearized_event_id,
	533	room_id,
	534	now - event_ts,
	535	)
	536
	537	yield self.insert_graph_receipt(room_id, receipt_type, user_id, event_ids, data)
	538
	539	max_persisted_id = self._receipts_id_gen.get_current_token()
	540
	541	return stream_id, max_persisted_id
	542
	543	def insert_graph_receipt(self, room_id, receipt_type, user_id, event_ids, data):
	544	return self.db_pool.runInteraction(
	545	"insert_graph_receipt",
	546	self.insert_graph_receipt_txn,
	547	room_id,
	548	receipt_type,
	549	user_id,
	550	event_ids,
	551	data,
	552	)
	553
	554	def insert_graph_receipt_txn(
	555	self, txn, room_id, receipt_type, user_id, event_ids, data
	556	):
	557	txn.call_after(self.get_receipts_for_room.invalidate, (room_id, receipt_type))
	558	txn.call_after(
	559	self._invalidate_get_users_with_receipts_in_room,
	560	room_id,
	561	receipt_type,
	562	user_id,
	563	)
	564	txn.call_after(self.get_receipts_for_user.invalidate, (user_id, receipt_type))
	565	# FIXME: This shouldn't invalidate the whole cache
	566	txn.call_after(
	567	self._get_linearized_receipts_for_room.invalidate_many, (room_id,)
	568	)
	569
	570	self.db_pool.simple_delete_txn(
	571	txn,
	572	table="receipts_graph",
	573	keyvalues={
	574	"room_id": room_id,
	575	"receipt_type": receipt_type,
	576	"user_id": user_id,
	577	},
	578	)
	579	self.db_pool.simple_insert_txn(
	580	txn,
	581	table="receipts_graph",
	582	values={
	583	"room_id": room_id,
	584	"receipt_type": receipt_type,
	585	"user_id": user_id,
	586	"event_ids": json_encoder.encode(event_ids),
	587	"data": json_encoder.encode(data),
	588	},
	589	)

+1565

-0

synapse/storage/databases/main/registration.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2017-2018 New Vector Ltd
	3	# Copyright 2019 The Matrix.org Foundation C.I.C.
	4	#
	5	# Licensed under the Apache License, Version 2.0 (the "License");
	6	# you may not use this file except in compliance with the License.
	7	# You may obtain a copy of the License at
	8	#
	9	# http://www.apache.org/licenses/LICENSE-2.0
	10	#
	11	# Unless required by applicable law or agreed to in writing, software
	12	# distributed under the License is distributed on an "AS IS" BASIS,
	13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	14	# See the License for the specific language governing permissions and
	15	# limitations under the License.
	16
	17	import logging
	18	import re
	19	from typing import Dict, List, Optional
	20
	21	from twisted.internet.defer import Deferred
	22
	23	from synapse.api.constants import UserTypes
	24	from synapse.api.errors import Codes, StoreError, SynapseError, ThreepidValidationError
	25	from synapse.metrics.background_process_metrics import run_as_background_process
	26	from synapse.storage._base import SQLBaseStore
	27	from synapse.storage.database import DatabasePool
	28	from synapse.storage.types import Cursor
	29	from synapse.storage.util.sequence import build_sequence_generator
	30	from synapse.types import UserID
	31	from synapse.util.caches.descriptors import cached
	32
	33	THIRTY_MINUTES_IN_MS = 30 * 60 * 1000
	34
	35	logger = logging.getLogger(__name__)
	36
	37
	38	class RegistrationWorkerStore(SQLBaseStore):
	39	def __init__(self, database: DatabasePool, db_conn, hs):
	40	super(RegistrationWorkerStore, self).__init__(database, db_conn, hs)
	41
	42	self.config = hs.config
	43	self.clock = hs.get_clock()
	44
	45	self._user_id_seq = build_sequence_generator(
	46	database.engine, find_max_generated_user_id_localpart, "user_id_seq",
	47	)
	48
	49	@cached()
	50	def get_user_by_id(self, user_id):
	51	return self.db_pool.simple_select_one(
	52	table="users",
	53	keyvalues={"name": user_id},
	54	retcols=[
	55	"name",
	56	"password_hash",
	57	"is_guest",
	58	"admin",
	59	"consent_version",
	60	"consent_server_notice_sent",
	61	"appservice_id",
	62	"creation_ts",
	63	"user_type",
	64	"deactivated",
	65	],
	66	allow_none=True,
	67	desc="get_user_by_id",
	68	)
	69
	70	async def is_trial_user(self, user_id: str) -> bool:
	71	"""Checks if user is in the "trial" period, i.e. within the first
	72	N days of registration defined by `mau_trial_days` config
	73
	74	Args:
	75	user_id: The user to check for trial status.
	76	"""
	77
	78	info = await self.get_user_by_id(user_id)
	79	if not info:
	80	return False
	81
	82	now = self.clock.time_msec()
	83	trial_duration_ms = self.config.mau_trial_days * 24 * 60 * 60 * 1000
	84	is_trial = (now - info["creation_ts"] * 1000) < trial_duration_ms
	85	return is_trial
	86
	87	@cached()
	88	def get_user_by_access_token(self, token):
	89	"""Get a user from the given access token.
	90
	91	Args:
	92	token (str): The access token of a user.
	93	Returns:
	94	defer.Deferred: None, if the token did not match, otherwise dict
	95	including the keys `name`, `is_guest`, `device_id`, `token_id`,
	96	`valid_until_ms`.
	97	"""
	98	return self.db_pool.runInteraction(
	99	"get_user_by_access_token", self._query_for_auth, token
	100	)
	101
	102	@cached()
	103	async def get_expiration_ts_for_user(self, user_id: str) -> Optional[None]:
	104	"""Get the expiration timestamp for the account bearing a given user ID.
	105
	106	Args:
	107	user_id: The ID of the user.
	108	Returns:
	109	None, if the account has no expiration timestamp, otherwise int
	110	representation of the timestamp (as a number of milliseconds since epoch).
	111	"""
	112	return await self.db_pool.simple_select_one_onecol(
	113	table="account_validity",
	114	keyvalues={"user_id": user_id},
	115	retcol="expiration_ts_ms",
	116	allow_none=True,
	117	desc="get_expiration_ts_for_user",
	118	)
	119
	120	async def set_account_validity_for_user(
	121	self,
	122	user_id: str,
	123	expiration_ts: int,
	124	email_sent: bool,
	125	renewal_token: Optional[str] = None,
	126	) -> None:
	127	"""Updates the account validity properties of the given account, with the
	128	given values.
	129
	130	Args:
	131	user_id: ID of the account to update properties for.
	132	expiration_ts: New expiration date, as a timestamp in milliseconds
	133	since epoch.
	134	email_sent: True means a renewal email has been sent for this account
	135	and there's no need to send another one for the current validity
	136	period.
	137	renewal_token: Renewal token the user can use to extend the validity
	138	of their account. Defaults to no token.
	139	"""
	140
	141	def set_account_validity_for_user_txn(txn):
	142	self.db_pool.simple_update_txn(
	143	txn=txn,
	144	table="account_validity",
	145	keyvalues={"user_id": user_id},
	146	updatevalues={
	147	"expiration_ts_ms": expiration_ts,
	148	"email_sent": email_sent,
	149	"renewal_token": renewal_token,
	150	},
	151	)
	152	self._invalidate_cache_and_stream(
	153	txn, self.get_expiration_ts_for_user, (user_id,)
	154	)
	155
	156	await self.db_pool.runInteraction(
	157	"set_account_validity_for_user", set_account_validity_for_user_txn
	158	)
	159
	160	async def set_renewal_token_for_user(
	161	self, user_id: str, renewal_token: str
	162	) -> None:
	163	"""Defines a renewal token for a given user.
	164
	165	Args:
	166	user_id: ID of the user to set the renewal token for.
	167	renewal_token: Random unique string that will be used to renew the
	168	user's account.
	169
	170	Raises:
	171	StoreError: The provided token is already set for another user.
	172	"""
	173	await self.db_pool.simple_update_one(
	174	table="account_validity",
	175	keyvalues={"user_id": user_id},
	176	updatevalues={"renewal_token": renewal_token},
	177	desc="set_renewal_token_for_user",
	178	)
	179
	180	async def get_user_from_renewal_token(self, renewal_token: str) -> str:
	181	"""Get a user ID from a renewal token.
	182
	183	Args:
	184	renewal_token: The renewal token to perform the lookup with.
	185
	186	Returns:
	187	The ID of the user to which the token belongs.
	188	"""
	189	return await self.db_pool.simple_select_one_onecol(
	190	table="account_validity",
	191	keyvalues={"renewal_token": renewal_token},
	192	retcol="user_id",
	193	desc="get_user_from_renewal_token",
	194	)
	195
	196	async def get_renewal_token_for_user(self, user_id: str) -> str:
	197	"""Get the renewal token associated with a given user ID.
	198
	199	Args:
	200	user_id: The user ID to lookup a token for.
	201
	202	Returns:
	203	The renewal token associated with this user ID.
	204	"""
	205	return await self.db_pool.simple_select_one_onecol(
	206	table="account_validity",
	207	keyvalues={"user_id": user_id},
	208	retcol="renewal_token",
	209	desc="get_renewal_token_for_user",
	210	)
	211
	212	async def get_users_expiring_soon(self) -> List[Dict[str, int]]:
	213	"""Selects users whose account will expire in the [now, now + renew_at] time
	214	window (see configuration for account_validity for information on what renew_at
	215	refers to).
	216
	217	Returns:
	218	A list of dictionaries mapping user ID to expiration time (in milliseconds).
	219	"""
	220
	221	def select_users_txn(txn, now_ms, renew_at):
	222	sql = (
	223	"SELECT user_id, expiration_ts_ms FROM account_validity"
	224	" WHERE email_sent = ? AND (expiration_ts_ms - ?) <= ?"
	225	)
	226	values = [False, now_ms, renew_at]
	227	txn.execute(sql, values)
	228	return self.db_pool.cursor_to_dict(txn)
	229
	230	return await self.db_pool.runInteraction(
	231	"get_users_expiring_soon",
	232	select_users_txn,
	233	self.clock.time_msec(),
	234	self.config.account_validity.renew_at,
	235	)
	236
	237	async def set_renewal_mail_status(self, user_id: str, email_sent: bool) -> None:
	238	"""Sets or unsets the flag that indicates whether a renewal email has been sent
	239	to the user (and the user hasn't renewed their account yet).
	240
	241	Args:
	242	user_id: ID of the user to set/unset the flag for.
	243	email_sent: Flag which indicates whether a renewal email has been sent
	244	to this user.
	245	"""
	246	await self.db_pool.simple_update_one(
	247	table="account_validity",
	248	keyvalues={"user_id": user_id},
	249	updatevalues={"email_sent": email_sent},
	250	desc="set_renewal_mail_status",
	251	)
	252
	253	async def delete_account_validity_for_user(self, user_id: str) -> None:
	254	"""Deletes the entry for the given user in the account validity table, removing
	255	their expiration date and renewal token.
	256
	257	Args:
	258	user_id: ID of the user to remove from the account validity table.
	259	"""
	260	await self.db_pool.simple_delete_one(
	261	table="account_validity",
	262	keyvalues={"user_id": user_id},
	263	desc="delete_account_validity_for_user",
	264	)
	265
	266	async def is_server_admin(self, user: UserID) -> bool:
	267	"""Determines if a user is an admin of this homeserver.
	268
	269	Args:
	270	user: user ID of the user to test
	271
	272	Returns:
	273	true iff the user is a server admin, false otherwise.
	274	"""
	275	res = await self.db_pool.simple_select_one_onecol(
	276	table="users",
	277	keyvalues={"name": user.to_string()},
	278	retcol="admin",
	279	allow_none=True,
	280	desc="is_server_admin",
	281	)
	282
	283	return bool(res) if res else False
	284
	285	def set_server_admin(self, user, admin):
	286	"""Sets whether a user is an admin of this homeserver.
	287
	288	Args:
	289	user (UserID): user ID of the user to test
	290	admin (bool): true iff the user is to be a server admin,
	291	false otherwise.
	292	"""
	293
	294	def set_server_admin_txn(txn):
	295	self.db_pool.simple_update_one_txn(
	296	txn, "users", {"name": user.to_string()}, {"admin": 1 if admin else 0}
	297	)
	298	self._invalidate_cache_and_stream(
	299	txn, self.get_user_by_id, (user.to_string(),)
	300	)
	301
	302	return self.db_pool.runInteraction("set_server_admin", set_server_admin_txn)
	303
	304	def _query_for_auth(self, txn, token):
	305	sql = (
	306	"SELECT users.name, users.is_guest, access_tokens.id as token_id,"
	307	" access_tokens.device_id, access_tokens.valid_until_ms"
	308	" FROM users"
	309	" INNER JOIN access_tokens on users.name = access_tokens.user_id"
	310	" WHERE token = ?"
	311	)
	312
	313	txn.execute(sql, (token,))
	314	rows = self.db_pool.cursor_to_dict(txn)
	315	if rows:
	316	return rows[0]
	317
	318	return None
	319
	320	@cached()
	321	async def is_real_user(self, user_id: str) -> bool:
	322	"""Determines if the user is a real user, ie does not have a 'user_type'.
	323
	324	Args:
	325	user_id: user id to test
	326
	327	Returns:
	328	True if user 'user_type' is null or empty string
	329	"""
	330	return await self.db_pool.runInteraction(
	331	"is_real_user", self.is_real_user_txn, user_id
	332	)
	333
	334	@cached()
	335	async def is_support_user(self, user_id: str) -> bool:
	336	"""Determines if the user is of type UserTypes.SUPPORT
	337
	338	Args:
	339	user_id: user id to test
	340
	341	Returns:
	342	True if user is of type UserTypes.SUPPORT
	343	"""
	344	return await self.db_pool.runInteraction(
	345	"is_support_user", self.is_support_user_txn, user_id
	346	)
	347
	348	def is_real_user_txn(self, txn, user_id):
	349	res = self.db_pool.simple_select_one_onecol_txn(
	350	txn=txn,
	351	table="users",
	352	keyvalues={"name": user_id},
	353	retcol="user_type",
	354	allow_none=True,
	355	)
	356	return res is None
	357
	358	def is_support_user_txn(self, txn, user_id):
	359	res = self.db_pool.simple_select_one_onecol_txn(
	360	txn=txn,
	361	table="users",
	362	keyvalues={"name": user_id},
	363	retcol="user_type",
	364	allow_none=True,
	365	)
	366	return True if res == UserTypes.SUPPORT else False
	367
	368	def get_users_by_id_case_insensitive(self, user_id):
	369	"""Gets users that match user_id case insensitively.
	370	Returns a mapping of user_id -> password_hash.
	371	"""
	372
	373	def f(txn):
	374	sql = "SELECT name, password_hash FROM users WHERE lower(name) = lower(?)"
	375	txn.execute(sql, (user_id,))
	376	return dict(txn)
	377
	378	return self.db_pool.runInteraction("get_users_by_id_case_insensitive", f)
	379
	380	async def get_user_by_external_id(
	381	self, auth_provider: str, external_id: str
	382	) -> str:
	383	"""Look up a user by their external auth id
	384
	385	Args:
	386	auth_provider: identifier for the remote auth provider
	387	external_id: id on that system
	388
	389	Returns:
	390	str\|None: the mxid of the user, or None if they are not known
	391	"""
	392	return await self.db_pool.simple_select_one_onecol(
	393	table="user_external_ids",
	394	keyvalues={"auth_provider": auth_provider, "external_id": external_id},
	395	retcol="user_id",
	396	allow_none=True,
	397	desc="get_user_by_external_id",
	398	)
	399
	400	async def count_all_users(self):
	401	"""Counts all users registered on the homeserver."""
	402
	403	def _count_users(txn):
	404	txn.execute("SELECT COUNT(*) AS users FROM users")
	405	rows = self.db_pool.cursor_to_dict(txn)
	406	if rows:
	407	return rows[0]["users"]
	408	return 0
	409
	410	return await self.db_pool.runInteraction("count_users", _count_users)
	411
	412	def count_daily_user_type(self):
	413	"""
	414	Counts 1) native non guest users
	415	2) native guests users
	416	3) bridged users
	417	who registered on the homeserver in the past 24 hours
	418	"""
	419
	420	def _count_daily_user_type(txn):
	421	yesterday = int(self._clock.time()) - (60 * 60 * 24)
	422
	423	sql = """
	424	SELECT user_type, COALESCE(count(*), 0) AS count FROM (
	425	SELECT
	426	CASE
	427	WHEN is_guest=0 AND appservice_id IS NULL THEN 'native'
	428	WHEN is_guest=1 AND appservice_id IS NULL THEN 'guest'
	429	WHEN is_guest=0 AND appservice_id IS NOT NULL THEN 'bridged'
	430	END AS user_type
	431	FROM users
	432	WHERE creation_ts > ?
	433	) AS t GROUP BY user_type
	434	"""
	435	results = {"native": 0, "guest": 0, "bridged": 0}
	436	txn.execute(sql, (yesterday,))
	437	for row in txn:
	438	results[row[0]] = row[1]
	439	return results
	440
	441	return self.db_pool.runInteraction(
	442	"count_daily_user_type", _count_daily_user_type
	443	)
	444
	445	async def count_nonbridged_users(self):
	446	def _count_users(txn):
	447	txn.execute(
	448	"""
	449	SELECT COALESCE(COUNT(*), 0) FROM users
	450	WHERE appservice_id IS NULL
	451	"""
	452	)
	453	(count,) = txn.fetchone()
	454	return count
	455
	456	return await self.db_pool.runInteraction("count_users", _count_users)
	457
	458	async def count_real_users(self):
	459	"""Counts all users without a special user_type registered on the homeserver."""
	460
	461	def _count_users(txn):
	462	txn.execute("SELECT COUNT(*) AS users FROM users where user_type is null")
	463	rows = self.db_pool.cursor_to_dict(txn)
	464	if rows:
	465	return rows[0]["users"]
	466	return 0
	467
	468	return await self.db_pool.runInteraction("count_real_users", _count_users)
	469
	470	async def generate_user_id(self) -> str:
	471	"""Generate a suitable localpart for a guest user
	472
	473	Returns: a (hopefully) free localpart
	474	"""
	475	next_id = await self.db_pool.runInteraction(
	476	"generate_user_id", self._user_id_seq.get_next_id_txn
	477	)
	478
	479	return str(next_id)
	480
	481	async def get_user_id_by_threepid(self, medium: str, address: str) -> Optional[str]:
	482	"""Returns user id from threepid
	483
	484	Args:
	485	medium: threepid medium e.g. email
	486	address: threepid address e.g. me@example.com
	487
	488	Returns:
	489	The user ID or None if no user id/threepid mapping exists
	490	"""
	491	user_id = await self.db_pool.runInteraction(
	492	"get_user_id_by_threepid", self.get_user_id_by_threepid_txn, medium, address
	493	)
	494	return user_id
	495
	496	def get_user_id_by_threepid_txn(self, txn, medium, address):
	497	"""Returns user id from threepid
	498
	499	Args:
	500	txn (cursor):
	501	medium (str): threepid medium e.g. email
	502	address (str): threepid address e.g. me@example.com
	503
	504	Returns:
	505	str\|None: user id or None if no user id/threepid mapping exists
	506	"""
	507	ret = self.db_pool.simple_select_one_txn(
	508	txn,
	509	"user_threepids",
	510	{"medium": medium, "address": address},
	511	["user_id"],
	512	True,
	513	)
	514	if ret:
	515	return ret["user_id"]
	516	return None
	517
	518	async def user_add_threepid(self, user_id, medium, address, validated_at, added_at):
	519	await self.db_pool.simple_upsert(
	520	"user_threepids",
	521	{"medium": medium, "address": address},
	522	{"user_id": user_id, "validated_at": validated_at, "added_at": added_at},
	523	)
	524
	525	async def user_get_threepids(self, user_id):
	526	return await self.db_pool.simple_select_list(
	527	"user_threepids",
	528	{"user_id": user_id},
	529	["medium", "address", "validated_at", "added_at"],
	530	"user_get_threepids",
	531	)
	532
	533	def user_delete_threepid(self, user_id, medium, address):
	534	return self.db_pool.simple_delete(
	535	"user_threepids",
	536	keyvalues={"user_id": user_id, "medium": medium, "address": address},
	537	desc="user_delete_threepid",
	538	)
	539
	540	def user_delete_threepids(self, user_id: str):
	541	"""Delete all threepid this user has bound
	542
	543	Args:
	544	user_id: The user id to delete all threepids of
	545
	546	"""
	547	return self.db_pool.simple_delete(
	548	"user_threepids",
	549	keyvalues={"user_id": user_id},
	550	desc="user_delete_threepids",
	551	)
	552
	553	def add_user_bound_threepid(self, user_id, medium, address, id_server):
	554	"""The server proxied a bind request to the given identity server on
	555	behalf of the given user. We need to remember this in case the user
	556	asks us to unbind the threepid.
	557
	558	Args:
	559	user_id (str)
	560	medium (str)
	561	address (str)
	562	id_server (str)
	563
	564	Returns:
	565	Deferred
	566	"""
	567	# We need to use an upsert, in case they user had already bound the
	568	# threepid
	569	return self.db_pool.simple_upsert(
	570	table="user_threepid_id_server",
	571	keyvalues={
	572	"user_id": user_id,
	573	"medium": medium,
	574	"address": address,
	575	"id_server": id_server,
	576	},
	577	values={},
	578	insertion_values={},
	579	desc="add_user_bound_threepid",
	580	)
	581
	582	def user_get_bound_threepids(self, user_id):
	583	"""Get the threepids that a user has bound to an identity server through the homeserver
	584	The homeserver remembers where binds to an identity server occurred. Using this
	585	method can retrieve those threepids.
	586
	587	Args:
	588	user_id (str): The ID of the user to retrieve threepids for
	589
	590	Returns:
	591	Deferred[list[dict]]: List of dictionaries containing the following:
	592	medium (str): The medium of the threepid (e.g "email")
	593	address (str): The address of the threepid (e.g "bob@example.com")
	594	"""
	595	return self.db_pool.simple_select_list(
	596	table="user_threepid_id_server",
	597	keyvalues={"user_id": user_id},
	598	retcols=["medium", "address"],
	599	desc="user_get_bound_threepids",
	600	)
	601
	602	def remove_user_bound_threepid(self, user_id, medium, address, id_server):
	603	"""The server proxied an unbind request to the given identity server on
	604	behalf of the given user, so we remove the mapping of threepid to
	605	identity server.
	606
	607	Args:
	608	user_id (str)
	609	medium (str)
	610	address (str)
	611	id_server (str)
	612
	613	Returns:
	614	Deferred
	615	"""
	616	return self.db_pool.simple_delete(
	617	table="user_threepid_id_server",
	618	keyvalues={
	619	"user_id": user_id,
	620	"medium": medium,
	621	"address": address,
	622	"id_server": id_server,
	623	},
	624	desc="remove_user_bound_threepid",
	625	)
	626
	627	def get_id_servers_user_bound(self, user_id, medium, address):
	628	"""Get the list of identity servers that the server proxied bind
	629	requests to for given user and threepid
	630
	631	Args:
	632	user_id (str)
	633	medium (str)
	634	address (str)
	635
	636	Returns:
	637	Deferred[list[str]]: Resolves to a list of identity servers
	638	"""
	639	return self.db_pool.simple_select_onecol(
	640	table="user_threepid_id_server",
	641	keyvalues={"user_id": user_id, "medium": medium, "address": address},
	642	retcol="id_server",
	643	desc="get_id_servers_user_bound",
	644	)
	645
	646	@cached()
	647	async def get_user_deactivated_status(self, user_id: str) -> bool:
	648	"""Retrieve the value for the `deactivated` property for the provided user.
	649
	650	Args:
	651	user_id: The ID of the user to retrieve the status for.
	652
	653	Returns:
	654	True if the user was deactivated, false if the user is still active.
	655	"""
	656
	657	res = await self.db_pool.simple_select_one_onecol(
	658	table="users",
	659	keyvalues={"name": user_id},
	660	retcol="deactivated",
	661	desc="get_user_deactivated_status",
	662	)
	663
	664	# Convert the integer into a boolean.
	665	return res == 1
	666
	667	def get_threepid_validation_session(
	668	self, medium, client_secret, address=None, sid=None, validated=True
	669	):
	670	"""Gets a session_id and last_send_attempt (if available) for a
	671	combination of validation metadata
	672
	673	Args:
	674	medium (str\|None): The medium of the 3PID
	675	address (str\|None): The address of the 3PID
	676	sid (str\|None): The ID of the validation session
	677	client_secret (str): A unique string provided by the client to help identify this
	678	validation attempt
	679	validated (bool\|None): Whether sessions should be filtered by
	680	whether they have been validated already or not. None to
	681	perform no filtering
	682
	683	Returns:
	684	Deferred[dict\|None]: A dict containing the following:
	685	* address - address of the 3pid
	686	* medium - medium of the 3pid
	687	* client_secret - a secret provided by the client for this validation session
	688	* session_id - ID of the validation session
	689	* send_attempt - a number serving to dedupe send attempts for this session
	690	* validated_at - timestamp of when this session was validated if so
	691
	692	Otherwise None if a validation session is not found
	693	"""
	694	if not client_secret:
	695	raise SynapseError(
	696	400, "Missing parameter: client_secret", errcode=Codes.MISSING_PARAM
	697	)
	698
	699	keyvalues = {"client_secret": client_secret}
	700	if medium:
	701	keyvalues["medium"] = medium
	702	if address:
	703	keyvalues["address"] = address
	704	if sid:
	705	keyvalues["session_id"] = sid
	706
	707	assert address or sid
	708
	709	def get_threepid_validation_session_txn(txn):
	710	sql = """
	711	SELECT address, session_id, medium, client_secret,
	712	last_send_attempt, validated_at
	713	FROM threepid_validation_session WHERE %s
	714	""" % (
	715	" AND ".join("%s = ?" % k for k in keyvalues.keys()),
	716	)
	717
	718	if validated is not None:
	719	sql += " AND validated_at IS " + ("NOT NULL" if validated else "NULL")
	720
	721	sql += " LIMIT 1"
	722
	723	txn.execute(sql, list(keyvalues.values()))
	724	rows = self.db_pool.cursor_to_dict(txn)
	725	if not rows:
	726	return None
	727
	728	return rows[0]
	729
	730	return self.db_pool.runInteraction(
	731	"get_threepid_validation_session", get_threepid_validation_session_txn
	732	)
	733
	734	def delete_threepid_session(self, session_id):
	735	"""Removes a threepid validation session from the database. This can
	736	be done after validation has been performed and whatever action was
	737	waiting on it has been carried out
	738
	739	Args:
	740	session_id (str): The ID of the session to delete
	741	"""
	742
	743	def delete_threepid_session_txn(txn):
	744	self.db_pool.simple_delete_txn(
	745	txn,
	746	table="threepid_validation_token",
	747	keyvalues={"session_id": session_id},
	748	)
	749	self.db_pool.simple_delete_txn(
	750	txn,
	751	table="threepid_validation_session",
	752	keyvalues={"session_id": session_id},
	753	)
	754
	755	return self.db_pool.runInteraction(
	756	"delete_threepid_session", delete_threepid_session_txn
	757	)
	758
	759
	760	class RegistrationBackgroundUpdateStore(RegistrationWorkerStore):
	761	def __init__(self, database: DatabasePool, db_conn, hs):
	762	super(RegistrationBackgroundUpdateStore, self).__init__(database, db_conn, hs)
	763
	764	self.clock = hs.get_clock()
	765	self.config = hs.config
	766
	767	self.db_pool.updates.register_background_index_update(
	768	"access_tokens_device_index",
	769	index_name="access_tokens_device_id",
	770	table="access_tokens",
	771	columns=["user_id", "device_id"],
	772	)
	773
	774	self.db_pool.updates.register_background_index_update(
	775	"users_creation_ts",
	776	index_name="users_creation_ts",
	777	table="users",
	778	columns=["creation_ts"],
	779	)
	780
	781	# we no longer use refresh tokens, but it's possible that some people
	782	# might have a background update queued to build this index. Just
	783	# clear the background update.
	784	self.db_pool.updates.register_noop_background_update(
	785	"refresh_tokens_device_index"
	786	)
	787
	788	self.db_pool.updates.register_background_update_handler(
	789	"user_threepids_grandfather", self._bg_user_threepids_grandfather
	790	)
	791
	792	self.db_pool.updates.register_background_update_handler(
	793	"users_set_deactivated_flag", self._background_update_set_deactivated_flag
	794	)
	795
	796	async def _background_update_set_deactivated_flag(self, progress, batch_size):
	797	"""Retrieves a list of all deactivated users and sets the 'deactivated' flag to 1
	798	for each of them.
	799	"""
	800
	801	last_user = progress.get("user_id", "")
	802
	803	def _background_update_set_deactivated_flag_txn(txn):
	804	txn.execute(
	805	"""
	806	SELECT
	807	users.name,
	808	COUNT(access_tokens.token) AS count_tokens,
	809	COUNT(user_threepids.address) AS count_threepids
	810	FROM users
	811	LEFT JOIN access_tokens ON (access_tokens.user_id = users.name)
	812	LEFT JOIN user_threepids ON (user_threepids.user_id = users.name)
	813	WHERE (users.password_hash IS NULL OR users.password_hash = '')
	814	AND (users.appservice_id IS NULL OR users.appservice_id = '')
	815	AND users.is_guest = 0
	816	AND users.name > ?
	817	GROUP BY users.name
	818	ORDER BY users.name ASC
	819	LIMIT ?;
	820	""",
	821	(last_user, batch_size),
	822	)
	823
	824	rows = self.db_pool.cursor_to_dict(txn)
	825
	826	if not rows:
	827	return True, 0
	828
	829	rows_processed_nb = 0
	830
	831	for user in rows:
	832	if not user["count_tokens"] and not user["count_threepids"]:
	833	self.set_user_deactivated_status_txn(txn, user["name"], True)
	834	rows_processed_nb += 1
	835
	836	logger.info("Marked %d rows as deactivated", rows_processed_nb)
	837
	838	self.db_pool.updates._background_update_progress_txn(
	839	txn, "users_set_deactivated_flag", {"user_id": rows[-1]["name"]}
	840	)
	841
	842	if batch_size > len(rows):
	843	return True, len(rows)
	844	else:
	845	return False, len(rows)
	846
	847	end, nb_processed = await self.db_pool.runInteraction(
	848	"users_set_deactivated_flag", _background_update_set_deactivated_flag_txn
	849	)
	850
	851	if end:
	852	await self.db_pool.updates._end_background_update(
	853	"users_set_deactivated_flag"
	854	)
	855
	856	return nb_processed
	857
	858	async def _bg_user_threepids_grandfather(self, progress, batch_size):
	859	"""We now track which identity servers a user binds their 3PID to, so
	860	we need to handle the case of existing bindings where we didn't track
	861	this.
	862
	863	We do this by grandfathering in existing user threepids assuming that
	864	they used one of the server configured trusted identity servers.
	865	"""
	866	id_servers = set(self.config.trusted_third_party_id_servers)
	867
	868	def _bg_user_threepids_grandfather_txn(txn):
	869	sql = """
	870	INSERT INTO user_threepid_id_server
	871	(user_id, medium, address, id_server)
	872	SELECT user_id, medium, address, ?
	873	FROM user_threepids
	874	"""
	875
	876	txn.executemany(sql, [(id_server,) for id_server in id_servers])
	877
	878	if id_servers:
	879	await self.db_pool.runInteraction(
	880	"_bg_user_threepids_grandfather", _bg_user_threepids_grandfather_txn
	881	)
	882
	883	await self.db_pool.updates._end_background_update("user_threepids_grandfather")
	884
	885	return 1
	886
	887
	888	class RegistrationStore(RegistrationBackgroundUpdateStore):
	889	def __init__(self, database: DatabasePool, db_conn, hs):
	890	super(RegistrationStore, self).__init__(database, db_conn, hs)
	891
	892	self._account_validity = hs.config.account_validity
	893
	894	if self._account_validity.enabled:
	895	self._clock.call_later(
	896	0.0,
	897	run_as_background_process,
	898	"account_validity_set_expiration_dates",
	899	self._set_expiration_date_when_missing,
	900	)
	901
	902	# Create a background job for culling expired 3PID validity tokens
	903	def start_cull():
	904	# run as a background process to make sure that the database transactions
	905	# have a logcontext to report to
	906	return run_as_background_process(
	907	"cull_expired_threepid_validation_tokens",
	908	self.cull_expired_threepid_validation_tokens,
	909	)
	910
	911	hs.get_clock().looping_call(start_cull, THIRTY_MINUTES_IN_MS)
	912
	913	async def add_access_token_to_user(
	914	self,
	915	user_id: str,
	916	token: str,
	917	device_id: Optional[str],
	918	valid_until_ms: Optional[int],
	919	) -> None:
	920	"""Adds an access token for the given user.
	921
	922	Args:
	923	user_id: The user ID.
	924	token: The new access token to add.
	925	device_id: ID of the device to associate with the access token
	926	valid_until_ms: when the token is valid until. None for no expiry.
	927	Raises:
	928	StoreError if there was a problem adding this.
	929	"""
	930	next_id = self._access_tokens_id_gen.get_next()
	931
	932	await self.db_pool.simple_insert(
	933	"access_tokens",
	934	{
	935	"id": next_id,
	936	"user_id": user_id,
	937	"token": token,
	938	"device_id": device_id,
	939	"valid_until_ms": valid_until_ms,
	940	},
	941	desc="add_access_token_to_user",
	942	)
	943
	944	def register_user(
	945	self,
	946	user_id,
	947	password_hash=None,
	948	was_guest=False,
	949	make_guest=False,
	950	appservice_id=None,
	951	create_profile_with_displayname=None,
	952	admin=False,
	953	user_type=None,
	954	):
	955	"""Attempts to register an account.
	956
	957	Args:
	958	user_id (str): The desired user ID to register.
	959	password_hash (str\|None): Optional. The password hash for this user.
	960	was_guest (bool): Optional. Whether this is a guest account being
	961	upgraded to a non-guest account.
	962	make_guest (boolean): True if the the new user should be guest,
	963	false to add a regular user account.
	964	appservice_id (str): The ID of the appservice registering the user.
	965	create_profile_with_displayname (unicode): Optionally create a profile for
	966	the user, setting their displayname to the given value
	967	admin (boolean): is an admin user?
	968	user_type (str\|None): type of user. One of the values from
	969	api.constants.UserTypes, or None for a normal user.
	970
	971	Raises:
	972	StoreError if the user_id could not be registered.
	973
	974	Returns:
	975	Deferred
	976	"""
	977	return self.db_pool.runInteraction(
	978	"register_user",
	979	self._register_user,
	980	user_id,
	981	password_hash,
	982	was_guest,
	983	make_guest,
	984	appservice_id,
	985	create_profile_with_displayname,
	986	admin,
	987	user_type,
	988	)
	989
	990	def _register_user(
	991	self,
	992	txn,
	993	user_id,
	994	password_hash,
	995	was_guest,
	996	make_guest,
	997	appservice_id,
	998	create_profile_with_displayname,
	999	admin,
	1000	user_type,
	1001	):
	1002	user_id_obj = UserID.from_string(user_id)
	1003
	1004	now = int(self.clock.time())
	1005
	1006	try:
	1007	if was_guest:
	1008	# Ensure that the guest user actually exists
	1009	# ``allow_none=False`` makes this raise an exception
	1010	# if the row isn't in the database.
	1011	self.db_pool.simple_select_one_txn(
	1012	txn,
	1013	"users",
	1014	keyvalues={"name": user_id, "is_guest": 1},
	1015	retcols=("name",),
	1016	allow_none=False,
	1017	)
	1018
	1019	self.db_pool.simple_update_one_txn(
	1020	txn,
	1021	"users",
	1022	keyvalues={"name": user_id, "is_guest": 1},
	1023	updatevalues={
	1024	"password_hash": password_hash,
	1025	"upgrade_ts": now,
	1026	"is_guest": 1 if make_guest else 0,
	1027	"appservice_id": appservice_id,
	1028	"admin": 1 if admin else 0,
	1029	"user_type": user_type,
	1030	},
	1031	)
	1032	else:
	1033	self.db_pool.simple_insert_txn(
	1034	txn,
	1035	"users",
	1036	values={
	1037	"name": user_id,
	1038	"password_hash": password_hash,
	1039	"creation_ts": now,
	1040	"is_guest": 1 if make_guest else 0,
	1041	"appservice_id": appservice_id,
	1042	"admin": 1 if admin else 0,
	1043	"user_type": user_type,
	1044	},
	1045	)
	1046
	1047	except self.database_engine.module.IntegrityError:
	1048	raise StoreError(400, "User ID already taken.", errcode=Codes.USER_IN_USE)
	1049
	1050	if self._account_validity.enabled:
	1051	self.set_expiration_date_for_user_txn(txn, user_id)
	1052
	1053	if create_profile_with_displayname:
	1054	# set a default displayname serverside to avoid ugly race
	1055	# between auto-joins and clients trying to set displaynames
	1056	#
	1057	# obviously the 'profiles' table uses localpart for user_id
	1058	# while everything else uses the full mxid.
	1059	txn.execute(
	1060	"INSERT INTO profiles(user_id, displayname) VALUES (?,?)",
	1061	(user_id_obj.localpart, create_profile_with_displayname),
	1062	)
	1063
	1064	if self.hs.config.stats_enabled:
	1065	# we create a new completed user statistics row
	1066
	1067	# we don't strictly need current_token since this user really can't
	1068	# have any state deltas before now (as it is a new user), but still,
	1069	# we include it for completeness.
	1070	current_token = self._get_max_stream_id_in_current_state_deltas_txn(txn)
	1071	self._update_stats_delta_txn(
	1072	txn, now, "user", user_id, {}, complete_with_stream_id=current_token
	1073	)
	1074
	1075	self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
	1076
	1077	def record_user_external_id(
	1078	self, auth_provider: str, external_id: str, user_id: str
	1079	) -> Deferred:
	1080	"""Record a mapping from an external user id to a mxid
	1081
	1082	Args:
	1083	auth_provider: identifier for the remote auth provider
	1084	external_id: id on that system
	1085	user_id: complete mxid that it is mapped to
	1086	"""
	1087	return self.db_pool.simple_insert(
	1088	table="user_external_ids",
	1089	values={
	1090	"auth_provider": auth_provider,
	1091	"external_id": external_id,
	1092	"user_id": user_id,
	1093	},
	1094	desc="record_user_external_id",
	1095	)
	1096
	1097	def user_set_password_hash(self, user_id, password_hash):
	1098	"""
	1099	NB. This does not evict any cache because the one use for this
	1100	removes most of the entries subsequently anyway so it would be
	1101	pointless. Use flush_user separately.
	1102	"""
	1103
	1104	def user_set_password_hash_txn(txn):
	1105	self.db_pool.simple_update_one_txn(
	1106	txn, "users", {"name": user_id}, {"password_hash": password_hash}
	1107	)
	1108	self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
	1109
	1110	return self.db_pool.runInteraction(
	1111	"user_set_password_hash", user_set_password_hash_txn
	1112	)
	1113
	1114	def user_set_consent_version(self, user_id, consent_version):
	1115	"""Updates the user table to record privacy policy consent
	1116
	1117	Args:
	1118	user_id (str): full mxid of the user to update
	1119	consent_version (str): version of the policy the user has consented
	1120	to
	1121
	1122	Raises:
	1123	StoreError(404) if user not found
	1124	"""
	1125
	1126	def f(txn):
	1127	self.db_pool.simple_update_one_txn(
	1128	txn,
	1129	table="users",
	1130	keyvalues={"name": user_id},
	1131	updatevalues={"consent_version": consent_version},
	1132	)
	1133	self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
	1134
	1135	return self.db_pool.runInteraction("user_set_consent_version", f)
	1136
	1137	def user_set_consent_server_notice_sent(self, user_id, consent_version):
	1138	"""Updates the user table to record that we have sent the user a server
	1139	notice about privacy policy consent
	1140
	1141	Args:
	1142	user_id (str): full mxid of the user to update
	1143	consent_version (str): version of the policy we have notified the
	1144	user about
	1145
	1146	Raises:
	1147	StoreError(404) if user not found
	1148	"""
	1149
	1150	def f(txn):
	1151	self.db_pool.simple_update_one_txn(
	1152	txn,
	1153	table="users",
	1154	keyvalues={"name": user_id},
	1155	updatevalues={"consent_server_notice_sent": consent_version},
	1156	)
	1157	self._invalidate_cache_and_stream(txn, self.get_user_by_id, (user_id,))
	1158
	1159	return self.db_pool.runInteraction("user_set_consent_server_notice_sent", f)
	1160
	1161	def user_delete_access_tokens(self, user_id, except_token_id=None, device_id=None):
	1162	"""
	1163	Invalidate access tokens belonging to a user
	1164
	1165	Args:
	1166	user_id (str): ID of user the tokens belong to
	1167	except_token_id (str): list of access_tokens IDs which should
	1168	not be deleted
	1169	device_id (str\|None): ID of device the tokens are associated with.
	1170	If None, tokens associated with any device (or no device) will
	1171	be deleted
	1172	Returns:
	1173	defer.Deferred[list[str, int, str\|None, int]]: a list of
	1174	(token, token id, device id) for each of the deleted tokens
	1175	"""
	1176
	1177	def f(txn):
	1178	keyvalues = {"user_id": user_id}
	1179	if device_id is not None:
	1180	keyvalues["device_id"] = device_id
	1181
	1182	items = keyvalues.items()
	1183	where_clause = " AND ".join(k + " = ?" for k, _ in items)
	1184	values = [v for _, v in items]
	1185	if except_token_id:
	1186	where_clause += " AND id != ?"
	1187	values.append(except_token_id)
	1188
	1189	txn.execute(
	1190	"SELECT token, id, device_id FROM access_tokens WHERE %s"
	1191	% where_clause,
	1192	values,
	1193	)
	1194	tokens_and_devices = [(r[0], r[1], r[2]) for r in txn]
	1195
	1196	for token, _, _ in tokens_and_devices:
	1197	self._invalidate_cache_and_stream(
	1198	txn, self.get_user_by_access_token, (token,)
	1199	)
	1200
	1201	txn.execute("DELETE FROM access_tokens WHERE %s" % where_clause, values)
	1202
	1203	return tokens_and_devices
	1204
	1205	return self.db_pool.runInteraction("user_delete_access_tokens", f)
	1206
	1207	def delete_access_token(self, access_token):
	1208	def f(txn):
	1209	self.db_pool.simple_delete_one_txn(
	1210	txn, table="access_tokens", keyvalues={"token": access_token}
	1211	)
	1212
	1213	self._invalidate_cache_and_stream(
	1214	txn, self.get_user_by_access_token, (access_token,)
	1215	)
	1216
	1217	return self.db_pool.runInteraction("delete_access_token", f)
	1218
	1219	@cached()
	1220	async def is_guest(self, user_id: str) -> bool:
	1221	res = await self.db_pool.simple_select_one_onecol(
	1222	table="users",
	1223	keyvalues={"name": user_id},
	1224	retcol="is_guest",
	1225	allow_none=True,
	1226	desc="is_guest",
	1227	)
	1228
	1229	return res if res else False
	1230
	1231	def add_user_pending_deactivation(self, user_id):
	1232	"""
	1233	Adds a user to the table of users who need to be parted from all the rooms they're
	1234	in
	1235	"""
	1236	return self.db_pool.simple_insert(
	1237	"users_pending_deactivation",
	1238	values={"user_id": user_id},
	1239	desc="add_user_pending_deactivation",
	1240	)
	1241
	1242	def del_user_pending_deactivation(self, user_id):
	1243	"""
	1244	Removes the given user to the table of users who need to be parted from all the
	1245	rooms they're in, effectively marking that user as fully deactivated.
	1246	"""
	1247	# XXX: This should be simple_delete_one but we failed to put a unique index on
	1248	# the table, so somehow duplicate entries have ended up in it.
	1249	return self.db_pool.simple_delete(
	1250	"users_pending_deactivation",
	1251	keyvalues={"user_id": user_id},
	1252	desc="del_user_pending_deactivation",
	1253	)
	1254
	1255	def get_user_pending_deactivation(self):
	1256	"""
	1257	Gets one user from the table of users waiting to be parted from all the rooms
	1258	they're in.
	1259	"""
	1260	return self.db_pool.simple_select_one_onecol(
	1261	"users_pending_deactivation",
	1262	keyvalues={},
	1263	retcol="user_id",
	1264	allow_none=True,
	1265	desc="get_users_pending_deactivation",
	1266	)
	1267
	1268	def validate_threepid_session(self, session_id, client_secret, token, current_ts):
	1269	"""Attempt to validate a threepid session using a token
	1270
	1271	Args:
	1272	session_id (str): The id of a validation session
	1273	client_secret (str): A unique string provided by the client to
	1274	help identify this validation attempt
	1275	token (str): A validation token
	1276	current_ts (int): The current unix time in milliseconds. Used for
	1277	checking token expiry status
	1278
	1279	Raises:
	1280	ThreepidValidationError: if a matching validation token was not found or has
	1281	expired
	1282
	1283	Returns:
	1284	deferred str\|None: A str representing a link to redirect the user
	1285	to if there is one.
	1286	"""
	1287
	1288	# Insert everything into a transaction in order to run atomically
	1289	def validate_threepid_session_txn(txn):
	1290	row = self.db_pool.simple_select_one_txn(
	1291	txn,
	1292	table="threepid_validation_session",
	1293	keyvalues={"session_id": session_id},
	1294	retcols=["client_secret", "validated_at"],
	1295	allow_none=True,
	1296	)
	1297
	1298	if not row:
	1299	raise ThreepidValidationError(400, "Unknown session_id")
	1300	retrieved_client_secret = row["client_secret"]
	1301	validated_at = row["validated_at"]
	1302
	1303	if retrieved_client_secret != client_secret:
	1304	raise ThreepidValidationError(
	1305	400, "This client_secret does not match the provided session_id"
	1306	)
	1307
	1308	row = self.db_pool.simple_select_one_txn(
	1309	txn,
	1310	table="threepid_validation_token",
	1311	keyvalues={"session_id": session_id, "token": token},
	1312	retcols=["expires", "next_link"],
	1313	allow_none=True,
	1314	)
	1315
	1316	if not row:
	1317	raise ThreepidValidationError(
	1318	400, "Validation token not found or has expired"
	1319	)
	1320	expires = row["expires"]
	1321	next_link = row["next_link"]
	1322
	1323	# If the session is already validated, no need to revalidate
	1324	if validated_at:
	1325	return next_link
	1326
	1327	if expires <= current_ts:
	1328	raise ThreepidValidationError(
	1329	400, "This token has expired. Please request a new one"
	1330	)
	1331
	1332	# Looks good. Validate the session
	1333	self.db_pool.simple_update_txn(
	1334	txn,
	1335	table="threepid_validation_session",
	1336	keyvalues={"session_id": session_id},
	1337	updatevalues={"validated_at": self.clock.time_msec()},
	1338	)
	1339
	1340	return next_link
	1341
	1342	# Return next_link if it exists
	1343	return self.db_pool.runInteraction(
	1344	"validate_threepid_session_txn", validate_threepid_session_txn
	1345	)
	1346
	1347	def upsert_threepid_validation_session(
	1348	self,
	1349	medium,
	1350	address,
	1351	client_secret,
	1352	send_attempt,
	1353	session_id,
	1354	validated_at=None,
	1355	):
	1356	"""Upsert a threepid validation session
	1357	Args:
	1358	medium (str): The medium of the 3PID
	1359	address (str): The address of the 3PID
	1360	client_secret (str): A unique string provided by the client to
	1361	help identify this validation attempt
	1362	send_attempt (int): The latest send_attempt on this session
	1363	session_id (str): The id of this validation session
	1364	validated_at (int\|None): The unix timestamp in milliseconds of
	1365	when the session was marked as valid
	1366	"""
	1367	insertion_values = {
	1368	"medium": medium,
	1369	"address": address,
	1370	"client_secret": client_secret,
	1371	}
	1372
	1373	if validated_at:
	1374	insertion_values["validated_at"] = validated_at
	1375
	1376	return self.db_pool.simple_upsert(
	1377	table="threepid_validation_session",
	1378	keyvalues={"session_id": session_id},
	1379	values={"last_send_attempt": send_attempt},
	1380	insertion_values=insertion_values,
	1381	desc="upsert_threepid_validation_session",
	1382	)
	1383
	1384	def start_or_continue_validation_session(
	1385	self,
	1386	medium,
	1387	address,
	1388	session_id,
	1389	client_secret,
	1390	send_attempt,
	1391	next_link,
	1392	token,
	1393	token_expires,
	1394	):
	1395	"""Creates a new threepid validation session if it does not already
	1396	exist and associates a new validation token with it
	1397
	1398	Args:
	1399	medium (str): The medium of the 3PID
	1400	address (str): The address of the 3PID
	1401	session_id (str): The id of this validation session
	1402	client_secret (str): A unique string provided by the client to
	1403	help identify this validation attempt
	1404	send_attempt (int): The latest send_attempt on this session
	1405	next_link (str\|None): The link to redirect the user to upon
	1406	successful validation
	1407	token (str): The validation token
	1408	token_expires (int): The timestamp for which after the token
	1409	will no longer be valid
	1410	"""
	1411
	1412	def start_or_continue_validation_session_txn(txn):
	1413	# Create or update a validation session
	1414	self.db_pool.simple_upsert_txn(
	1415	txn,
	1416	table="threepid_validation_session",
	1417	keyvalues={"session_id": session_id},
	1418	values={"last_send_attempt": send_attempt},
	1419	insertion_values={
	1420	"medium": medium,
	1421	"address": address,
	1422	"client_secret": client_secret,
	1423	},
	1424	)
	1425
	1426	# Create a new validation token with this session ID
	1427	self.db_pool.simple_insert_txn(
	1428	txn,
	1429	table="threepid_validation_token",
	1430	values={
	1431	"session_id": session_id,
	1432	"token": token,
	1433	"next_link": next_link,
	1434	"expires": token_expires,
	1435	},
	1436	)
	1437
	1438	return self.db_pool.runInteraction(
	1439	"start_or_continue_validation_session",
	1440	start_or_continue_validation_session_txn,
	1441	)
	1442
	1443	def cull_expired_threepid_validation_tokens(self):
	1444	"""Remove threepid validation tokens with expiry dates that have passed"""
	1445
	1446	def cull_expired_threepid_validation_tokens_txn(txn, ts):
	1447	sql = """
	1448	DELETE FROM threepid_validation_token WHERE
	1449	expires < ?
	1450	"""
	1451	return txn.execute(sql, (ts,))
	1452
	1453	return self.db_pool.runInteraction(
	1454	"cull_expired_threepid_validation_tokens",
	1455	cull_expired_threepid_validation_tokens_txn,
	1456	self.clock.time_msec(),
	1457	)
	1458
	1459	async def set_user_deactivated_status(
	1460	self, user_id: str, deactivated: bool
	1461	) -> None:
	1462	"""Set the `deactivated` property for the provided user to the provided value.
	1463
	1464	Args:
	1465	user_id: The ID of the user to set the status for.
	1466	deactivated: The value to set for `deactivated`.
	1467	"""
	1468
	1469	await self.db_pool.runInteraction(
	1470	"set_user_deactivated_status",
	1471	self.set_user_deactivated_status_txn,
	1472	user_id,
	1473	deactivated,
	1474	)
	1475
	1476	def set_user_deactivated_status_txn(self, txn, user_id, deactivated):
	1477	self.db_pool.simple_update_one_txn(
	1478	txn=txn,
	1479	table="users",
	1480	keyvalues={"name": user_id},
	1481	updatevalues={"deactivated": 1 if deactivated else 0},
	1482	)
	1483	self._invalidate_cache_and_stream(
	1484	txn, self.get_user_deactivated_status, (user_id,)
	1485	)
	1486	txn.call_after(self.is_guest.invalidate, (user_id,))
	1487
	1488	async def _set_expiration_date_when_missing(self):
	1489	"""
	1490	Retrieves the list of registered users that don't have an expiration date, and
	1491	adds an expiration date for each of them.
	1492	"""
	1493
	1494	def select_users_with_no_expiration_date_txn(txn):
	1495	"""Retrieves the list of registered users with no expiration date from the
	1496	database, filtering out deactivated users.
	1497	"""
	1498	sql = (
	1499	"SELECT users.name FROM users"
	1500	" LEFT JOIN account_validity ON (users.name = account_validity.user_id)"
	1501	" WHERE account_validity.user_id is NULL AND users.deactivated = 0;"
	1502	)
	1503	txn.execute(sql, [])
	1504
	1505	res = self.db_pool.cursor_to_dict(txn)
	1506	if res:
	1507	for user in res:
	1508	self.set_expiration_date_for_user_txn(
	1509	txn, user["name"], use_delta=True
	1510	)
	1511
	1512	await self.db_pool.runInteraction(
	1513	"get_users_with_no_expiration_date",
	1514	select_users_with_no_expiration_date_txn,
	1515	)
	1516
	1517	def set_expiration_date_for_user_txn(self, txn, user_id, use_delta=False):
	1518	"""Sets an expiration date to the account with the given user ID.
	1519
	1520	Args:
	1521	user_id (str): User ID to set an expiration date for.
	1522	use_delta (bool): If set to False, the expiration date for the user will be
	1523	now + validity period. If set to True, this expiration date will be a
	1524	random value in the [now + period - d ; now + period] range, d being a
	1525	delta equal to 10% of the validity period.
	1526	"""
	1527	now_ms = self._clock.time_msec()
	1528	expiration_ts = now_ms + self._account_validity.period
	1529
	1530	if use_delta:
	1531	expiration_ts = self.rand.randrange(
	1532	expiration_ts - self._account_validity.startup_job_max_delta,
	1533	expiration_ts,
	1534	)
	1535
	1536	self.db_pool.simple_upsert_txn(
	1537	txn,
	1538	"account_validity",
	1539	keyvalues={"user_id": user_id},
	1540	values={"expiration_ts_ms": expiration_ts, "email_sent": False},
	1541	)
	1542
	1543
	1544	def find_max_generated_user_id_localpart(cur: Cursor) -> int:
	1545	"""
	1546	Gets the localpart of the max current generated user ID.
	1547
	1548	Generated user IDs are integers, so we find the largest integer user ID
	1549	already taken and return that.
	1550	"""
	1551
	1552	# We bound between '@0' and '@a' to avoid pulling the entire table
	1553	# out.
	1554	cur.execute("SELECT name FROM users WHERE '@0' <= name AND name < '@a'")
	1555
	1556	regex = re.compile(r"^@(\d+):")
	1557
	1558	max_found = 0
	1559
	1560	for (user_id,) in cur:
	1561	match = regex.search(user_id)
	1562	if match:
	1563	max_found = max(int(match.group(1)), max_found)
	1564	return max_found

+31

-0

synapse/storage/databases/main/rejections.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16
	17	from synapse.storage._base import SQLBaseStore
	18
	19	logger = logging.getLogger(__name__)
	20
	21
	22	class RejectionsStore(SQLBaseStore):
	23	def get_rejection_reason(self, event_id):
	24	return self.db_pool.simple_select_one_onecol(
	25	table="rejections",
	26	retcol="reason",
	27	keyvalues={"event_id": event_id},
	28	allow_none=True,
	29	desc="get_rejection_reason",
	30	)

+328

-0

synapse/storage/databases/main/relations.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2019 New Vector Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16	from typing import Optional
	17
	18	import attr
	19
	20	from synapse.api.constants import RelationTypes
	21	from synapse.events import EventBase
	22	from synapse.storage._base import SQLBaseStore
	23	from synapse.storage.databases.main.stream import generate_pagination_where_clause
	24	from synapse.storage.relations import (
	25	AggregationPaginationToken,
	26	PaginationChunk,
	27	RelationPaginationToken,
	28	)
	29	from synapse.util.caches.descriptors import cached
	30
	31	logger = logging.getLogger(__name__)
	32
	33
	34	class RelationsWorkerStore(SQLBaseStore):
	35	@cached(tree=True)
	36	def get_relations_for_event(
	37	self,
	38	event_id,
	39	relation_type=None,
	40	event_type=None,
	41	aggregation_key=None,
	42	limit=5,
	43	direction="b",
	44	from_token=None,
	45	to_token=None,
	46	):
	47	"""Get a list of relations for an event, ordered by topological ordering.
	48
	49	Args:
	50	event_id (str): Fetch events that relate to this event ID.
	51	relation_type (str\|None): Only fetch events with this relation
	52	type, if given.
	53	event_type (str\|None): Only fetch events with this event type, if
	54	given.
	55	aggregation_key (str\|None): Only fetch events with this aggregation
	56	key, if given.
	57	limit (int): Only fetch the most recent `limit` events.
	58	direction (str): Whether to fetch the most recent first (`"b"`) or
	59	the oldest first (`"f"`).
	60	from_token (RelationPaginationToken\|None): Fetch rows from the given
	61	token, or from the start if None.
	62	to_token (RelationPaginationToken\|None): Fetch rows up to the given
	63	token, or up to the end if None.
	64
	65	Returns:
	66	Deferred[PaginationChunk]: List of event IDs that match relations
	67	requested. The rows are of the form `{"event_id": "..."}`.
	68	"""
	69
	70	where_clause = ["relates_to_id = ?"]
	71	where_args = [event_id]
	72
	73	if relation_type is not None:
	74	where_clause.append("relation_type = ?")
	75	where_args.append(relation_type)
	76
	77	if event_type is not None:
	78	where_clause.append("type = ?")
	79	where_args.append(event_type)
	80
	81	if aggregation_key:
	82	where_clause.append("aggregation_key = ?")
	83	where_args.append(aggregation_key)
	84
	85	pagination_clause = generate_pagination_where_clause(
	86	direction=direction,
	87	column_names=("topological_ordering", "stream_ordering"),
	88	from_token=attr.astuple(from_token) if from_token else None,
	89	to_token=attr.astuple(to_token) if to_token else None,
	90	engine=self.database_engine,
	91	)
	92
	93	if pagination_clause:
	94	where_clause.append(pagination_clause)
	95
	96	if direction == "b":
	97	order = "DESC"
	98	else:
	99	order = "ASC"
	100
	101	sql = """
	102	SELECT event_id, topological_ordering, stream_ordering
	103	FROM event_relations
	104	INNER JOIN events USING (event_id)
	105	WHERE %s
	106	ORDER BY topological_ordering %s, stream_ordering %s
	107	LIMIT ?
	108	""" % (
	109	" AND ".join(where_clause),
	110	order,
	111	order,
	112	)
	113
	114	def _get_recent_references_for_event_txn(txn):
	115	txn.execute(sql, where_args + [limit + 1])
	116
	117	last_topo_id = None
	118	last_stream_id = None
	119	events = []
	120	for row in txn:
	121	events.append({"event_id": row[0]})
	122	last_topo_id = row[1]
	123	last_stream_id = row[2]
	124
	125	next_batch = None
	126	if len(events) > limit and last_topo_id and last_stream_id:
	127	next_batch = RelationPaginationToken(last_topo_id, last_stream_id)
	128
	129	return PaginationChunk(
	130	chunk=list(events[:limit]), next_batch=next_batch, prev_batch=from_token
	131	)
	132
	133	return self.db_pool.runInteraction(
	134	"get_recent_references_for_event", _get_recent_references_for_event_txn
	135	)
	136
	137	@cached(tree=True)
	138	def get_aggregation_groups_for_event(
	139	self,
	140	event_id,
	141	event_type=None,
	142	limit=5,
	143	direction="b",
	144	from_token=None,
	145	to_token=None,
	146	):
	147	"""Get a list of annotations on the event, grouped by event type and
	148	aggregation key, sorted by count.
	149
	150	This is used e.g. to get the what and how many reactions have happend
	151	on an event.
	152
	153	Args:
	154	event_id (str): Fetch events that relate to this event ID.
	155	event_type (str\|None): Only fetch events with this event type, if
	156	given.
	157	limit (int): Only fetch the `limit` groups.
	158	direction (str): Whether to fetch the highest count first (`"b"`) or
	159	the lowest count first (`"f"`).
	160	from_token (AggregationPaginationToken\|None): Fetch rows from the
	161	given token, or from the start if None.
	162	to_token (AggregationPaginationToken\|None): Fetch rows up to the
	163	given token, or up to the end if None.
	164
	165
	166	Returns:
	167	Deferred[PaginationChunk]: List of groups of annotations that
	168	match. Each row is a dict with `type`, `key` and `count` fields.
	169	"""
	170
	171	where_clause = ["relates_to_id = ?", "relation_type = ?"]
	172	where_args = [event_id, RelationTypes.ANNOTATION]
	173
	174	if event_type:
	175	where_clause.append("type = ?")
	176	where_args.append(event_type)
	177
	178	having_clause = generate_pagination_where_clause(
	179	direction=direction,
	180	column_names=("COUNT(*)", "MAX(stream_ordering)"),
	181	from_token=attr.astuple(from_token) if from_token else None,
	182	to_token=attr.astuple(to_token) if to_token else None,
	183	engine=self.database_engine,
	184	)
	185
	186	if direction == "b":
	187	order = "DESC"
	188	else:
	189	order = "ASC"
	190
	191	if having_clause:
	192	having_clause = "HAVING " + having_clause
	193	else:
	194	having_clause = ""
	195
	196	sql = """
	197	SELECT type, aggregation_key, COUNT(DISTINCT sender), MAX(stream_ordering)
	198	FROM event_relations
	199	INNER JOIN events USING (event_id)
	200	WHERE {where_clause}
	201	GROUP BY relation_type, type, aggregation_key
	202	{having_clause}
	203	ORDER BY COUNT(*) {order}, MAX(stream_ordering) {order}
	204	LIMIT ?
	205	""".format(
	206	where_clause=" AND ".join(where_clause),
	207	order=order,
	208	having_clause=having_clause,
	209	)
	210
	211	def _get_aggregation_groups_for_event_txn(txn):
	212	txn.execute(sql, where_args + [limit + 1])
	213
	214	next_batch = None
	215	events = []
	216	for row in txn:
	217	events.append({"type": row[0], "key": row[1], "count": row[2]})
	218	next_batch = AggregationPaginationToken(row[2], row[3])
	219
	220	if len(events) <= limit:
	221	next_batch = None
	222
	223	return PaginationChunk(
	224	chunk=list(events[:limit]), next_batch=next_batch, prev_batch=from_token
	225	)
	226
	227	return self.db_pool.runInteraction(
	228	"get_aggregation_groups_for_event", _get_aggregation_groups_for_event_txn
	229	)
	230
	231	@cached()
	232	async def get_applicable_edit(self, event_id: str) -> Optional[EventBase]:
	233	"""Get the most recent edit (if any) that has happened for the given
	234	event.
	235
	236	Correctly handles checking whether edits were allowed to happen.
	237
	238	Args:
	239	event_id: The original event ID
	240
	241	Returns:
	242	The most recent edit, if any.
	243	"""
	244
	245	# We only allow edits for `m.room.message` events that have the same sender
	246	# and event type. We can't assert these things during regular event auth so
	247	# we have to do the checks post hoc.
	248
	249	# Fetches latest edit that has the same type and sender as the
	250	# original, and is an `m.room.message`.
	251	sql = """
	252	SELECT edit.event_id FROM events AS edit
	253	INNER JOIN event_relations USING (event_id)
	254	INNER JOIN events AS original ON
	255	original.event_id = relates_to_id
	256	AND edit.type = original.type
	257	AND edit.sender = original.sender
	258	WHERE
	259	relates_to_id = ?
	260	AND relation_type = ?
	261	AND edit.type = 'm.room.message'
	262	ORDER by edit.origin_server_ts DESC, edit.event_id DESC
	263	LIMIT 1
	264	"""
	265
	266	def _get_applicable_edit_txn(txn):
	267	txn.execute(sql, (event_id, RelationTypes.REPLACE))
	268	row = txn.fetchone()
	269	if row:
	270	return row[0]
	271
	272	edit_id = await self.db_pool.runInteraction(
	273	"get_applicable_edit", _get_applicable_edit_txn
	274	)
	275
	276	if not edit_id:
	277	return None
	278
	279	return await self.get_event(edit_id, allow_none=True)
	280
	281	def has_user_annotated_event(self, parent_id, event_type, aggregation_key, sender):
	282	"""Check if a user has already annotated an event with the same key
	283	(e.g. already liked an event).
	284
	285	Args:
	286	parent_id (str): The event being annotated
	287	event_type (str): The event type of the annotation
	288	aggregation_key (str): The aggregation key of the annotation
	289	sender (str): The sender of the annotation
	290
	291	Returns:
	292	Deferred[bool]
	293	"""
	294
	295	sql = """
	296	SELECT 1 FROM event_relations
	297	INNER JOIN events USING (event_id)
	298	WHERE
	299	relates_to_id = ?
	300	AND relation_type = ?
	301	AND type = ?
	302	AND sender = ?
	303	AND aggregation_key = ?
	304	LIMIT 1;
	305	"""
	306
	307	def _get_if_user_has_annotated_event(txn):
	308	txn.execute(
	309	sql,
	310	(
	311	parent_id,
	312	RelationTypes.ANNOTATION,
	313	event_type,
	314	sender,
	315	aggregation_key,
	316	),
	317	)
	318
	319	return bool(txn.fetchone())
	320
	321	return self.db_pool.runInteraction(
	322	"get_if_user_has_annotated_event", _get_if_user_has_annotated_event
	323	)
	324
	325
	326	class RelationsStore(RelationsWorkerStore):
	327	pass

+1429

-0

synapse/storage/databases/main/room.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2019 The Matrix.org Foundation C.I.C.
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import collections
	17	import logging
	18	import re
	19	from abc import abstractmethod
	20	from enum import Enum
	21	from typing import Any, Dict, List, Optional, Tuple
	22
	23	from canonicaljson import json
	24
	25	from synapse.api.constants import EventTypes
	26	from synapse.api.errors import StoreError
	27	from synapse.api.room_versions import RoomVersion, RoomVersions
	28	from synapse.storage._base import SQLBaseStore, db_to_json
	29	from synapse.storage.database import DatabasePool, LoggingTransaction
	30	from synapse.storage.databases.main.search import SearchStore
	31	from synapse.types import ThirdPartyInstanceID
	32	from synapse.util.caches.descriptors import cached
	33
	34	logger = logging.getLogger(__name__)
	35
	36
	37	OpsLevel = collections.namedtuple(
	38	"OpsLevel", ("ban_level", "kick_level", "redact_level")
	39	)
	40
	41	RatelimitOverride = collections.namedtuple(
	42	"RatelimitOverride", ("messages_per_second", "burst_count")
	43	)
	44
	45
	46	class RoomSortOrder(Enum):
	47	"""
	48	Enum to define the sorting method used when returning rooms with get_rooms_paginate
	49
	50	NAME = sort rooms alphabetically by name
	51	JOINED_MEMBERS = sort rooms by membership size, highest to lowest
	52	"""
	53
	54	# ALPHABETICAL and SIZE are deprecated.
	55	# ALPHABETICAL is the same as NAME.
	56	ALPHABETICAL = "alphabetical"
	57	# SIZE is the same as JOINED_MEMBERS.
	58	SIZE = "size"
	59	NAME = "name"
	60	CANONICAL_ALIAS = "canonical_alias"
	61	JOINED_MEMBERS = "joined_members"
	62	JOINED_LOCAL_MEMBERS = "joined_local_members"
	63	VERSION = "version"
	64	CREATOR = "creator"
	65	ENCRYPTION = "encryption"
	66	FEDERATABLE = "federatable"
	67	PUBLIC = "public"
	68	JOIN_RULES = "join_rules"
	69	GUEST_ACCESS = "guest_access"
	70	HISTORY_VISIBILITY = "history_visibility"
	71	STATE_EVENTS = "state_events"
	72
	73
	74	class RoomWorkerStore(SQLBaseStore):
	75	def __init__(self, database: DatabasePool, db_conn, hs):
	76	super(RoomWorkerStore, self).__init__(database, db_conn, hs)
	77
	78	self.config = hs.config
	79
	80	def get_room(self, room_id):
	81	"""Retrieve a room.
	82
	83	Args:
	84	room_id (str): The ID of the room to retrieve.
	85	Returns:
	86	A dict containing the room information, or None if the room is unknown.
	87	"""
	88	return self.db_pool.simple_select_one(
	89	table="rooms",
	90	keyvalues={"room_id": room_id},
	91	retcols=("room_id", "is_public", "creator"),
	92	desc="get_room",
	93	allow_none=True,
	94	)
	95
	96	def get_room_with_stats(self, room_id: str):
	97	"""Retrieve room with statistics.
	98
	99	Args:
	100	room_id: The ID of the room to retrieve.
	101	Returns:
	102	A dict containing the room information, or None if the room is unknown.
	103	"""
	104
	105	def get_room_with_stats_txn(txn, room_id):
	106	sql = """
	107	SELECT room_id, state.name, state.canonical_alias, curr.joined_members,
	108	curr.local_users_in_room AS joined_local_members, rooms.room_version AS version,
	109	rooms.creator, state.encryption, state.is_federatable AS federatable,
	110	rooms.is_public AS public, state.join_rules, state.guest_access,
	111	state.history_visibility, curr.current_state_events AS state_events
	112	FROM rooms
	113	LEFT JOIN room_stats_state state USING (room_id)
	114	LEFT JOIN room_stats_current curr USING (room_id)
	115	WHERE room_id = ?
	116	"""
	117	txn.execute(sql, [room_id])
	118	# Catch error if sql returns empty result to return "None" instead of an error
	119	try:
	120	res = self.db_pool.cursor_to_dict(txn)[0]
	121	except IndexError:
	122	return None
	123
	124	res["federatable"] = bool(res["federatable"])
	125	res["public"] = bool(res["public"])
	126	return res
	127
	128	return self.db_pool.runInteraction(
	129	"get_room_with_stats", get_room_with_stats_txn, room_id
	130	)
	131
	132	def get_public_room_ids(self):
	133	return self.db_pool.simple_select_onecol(
	134	table="rooms",
	135	keyvalues={"is_public": True},
	136	retcol="room_id",
	137	desc="get_public_room_ids",
	138	)
	139
	140	def count_public_rooms(self, network_tuple, ignore_non_federatable):
	141	"""Counts the number of public rooms as tracked in the room_stats_current
	142	and room_stats_state table.
	143
	144	Args:
	145	network_tuple (ThirdPartyInstanceID\|None)
	146	ignore_non_federatable (bool): If true filters out non-federatable rooms
	147	"""
	148
	149	def _count_public_rooms_txn(txn):
	150	query_args = []
	151
	152	if network_tuple:
	153	if network_tuple.appservice_id:
	154	published_sql = """
	155	SELECT room_id from appservice_room_list
	156	WHERE appservice_id = ? AND network_id = ?
	157	"""
	158	query_args.append(network_tuple.appservice_id)
	159	query_args.append(network_tuple.network_id)
	160	else:
	161	published_sql = """
	162	SELECT room_id FROM rooms WHERE is_public
	163	"""
	164	else:
	165	published_sql = """
	166	SELECT room_id FROM rooms WHERE is_public
	167	UNION SELECT room_id from appservice_room_list
	168	"""
	169
	170	sql = """
	171	SELECT
	172	COALESCE(COUNT(*), 0)
	173	FROM (
	174	%(published_sql)s
	175	) published
	176	INNER JOIN room_stats_state USING (room_id)
	177	INNER JOIN room_stats_current USING (room_id)
	178	WHERE
	179	(
	180	join_rules = 'public' OR history_visibility = 'world_readable'
	181	)
	182	AND joined_members > 0
	183	""" % {
	184	"published_sql": published_sql
	185	}
	186
	187	txn.execute(sql, query_args)
	188	return txn.fetchone()[0]
	189
	190	return self.db_pool.runInteraction(
	191	"count_public_rooms", _count_public_rooms_txn
	192	)
	193
	194	async def get_largest_public_rooms(
	195	self,
	196	network_tuple: Optional[ThirdPartyInstanceID],
	197	search_filter: Optional[dict],
	198	limit: Optional[int],
	199	bounds: Optional[Tuple[int, str]],
	200	forwards: bool,
	201	ignore_non_federatable: bool = False,
	202	):
	203	"""Gets the largest public rooms (where largest is in terms of joined
	204	members, as tracked in the statistics table).
	205
	206	Args:
	207	network_tuple
	208	search_filter
	209	limit: Maxmimum number of rows to return, unlimited otherwise.
	210	bounds: An uppoer or lower bound to apply to result set if given,
	211	consists of a joined member count and room_id (these are
	212	excluded from result set).
	213	forwards: true iff going forwards, going backwards otherwise
	214	ignore_non_federatable: If true filters out non-federatable rooms.
	215
	216	Returns:
	217	Rooms in order: biggest number of joined users first.
	218	We then arbitrarily use the room_id as a tie breaker.
	219
	220	"""
	221
	222	where_clauses = []
	223	query_args = []
	224
	225	if network_tuple:
	226	if network_tuple.appservice_id:
	227	published_sql = """
	228	SELECT room_id from appservice_room_list
	229	WHERE appservice_id = ? AND network_id = ?
	230	"""
	231	query_args.append(network_tuple.appservice_id)
	232	query_args.append(network_tuple.network_id)
	233	else:
	234	published_sql = """
	235	SELECT room_id FROM rooms WHERE is_public
	236	"""
	237	else:
	238	published_sql = """
	239	SELECT room_id FROM rooms WHERE is_public
	240	UNION SELECT room_id from appservice_room_list
	241	"""
	242
	243	# Work out the bounds if we're given them, these bounds look slightly
	244	# odd, but are designed to help query planner use indices by pulling
	245	# out a common bound.
	246	if bounds:
	247	last_joined_members, last_room_id = bounds
	248	if forwards:
	249	where_clauses.append(
	250	"""
	251	joined_members <= ? AND (
	252	joined_members < ? OR room_id < ?
	253	)
	254	"""
	255	)
	256	else:
	257	where_clauses.append(
	258	"""
	259	joined_members >= ? AND (
	260	joined_members > ? OR room_id > ?
	261	)
	262	"""
	263	)
	264
	265	query_args += [last_joined_members, last_joined_members, last_room_id]
	266
	267	if ignore_non_federatable:
	268	where_clauses.append("is_federatable")
	269
	270	if search_filter and search_filter.get("generic_search_term", None):
	271	search_term = "%" + search_filter["generic_search_term"] + "%"
	272
	273	where_clauses.append(
	274	"""
	275	(
	276	LOWER(name) LIKE ?
	277	OR LOWER(topic) LIKE ?
	278	OR LOWER(canonical_alias) LIKE ?
	279	)
	280	"""
	281	)
	282	query_args += [
	283	search_term.lower(),
	284	search_term.lower(),
	285	search_term.lower(),
	286	]
	287
	288	where_clause = ""
	289	if where_clauses:
	290	where_clause = " AND " + " AND ".join(where_clauses)
	291
	292	sql = """
	293	SELECT
	294	room_id, name, topic, canonical_alias, joined_members,
	295	avatar, history_visibility, joined_members, guest_access
	296	FROM (
	297	%(published_sql)s
	298	) published
	299	INNER JOIN room_stats_state USING (room_id)
	300	INNER JOIN room_stats_current USING (room_id)
	301	WHERE
	302	(
	303	join_rules = 'public' OR history_visibility = 'world_readable'
	304	)
	305	AND joined_members > 0
	306	%(where_clause)s
	307	ORDER BY joined_members %(dir)s, room_id %(dir)s
	308	""" % {
	309	"published_sql": published_sql,
	310	"where_clause": where_clause,
	311	"dir": "DESC" if forwards else "ASC",
	312	}
	313
	314	if limit is not None:
	315	query_args.append(limit)
	316
	317	sql += """
	318	LIMIT ?
	319	"""
	320
	321	def _get_largest_public_rooms_txn(txn):
	322	txn.execute(sql, query_args)
	323
	324	results = self.db_pool.cursor_to_dict(txn)
	325
	326	if not forwards:
	327	results.reverse()
	328
	329	return results
	330
	331	ret_val = await self.db_pool.runInteraction(
	332	"get_largest_public_rooms", _get_largest_public_rooms_txn
	333	)
	334	return ret_val
	335
	336	@cached(max_entries=10000)
	337	def is_room_blocked(self, room_id):
	338	return self.db_pool.simple_select_one_onecol(
	339	table="blocked_rooms",
	340	keyvalues={"room_id": room_id},
	341	retcol="1",
	342	allow_none=True,
	343	desc="is_room_blocked",
	344	)
	345
	346	async def get_rooms_paginate(
	347	self,
	348	start: int,
	349	limit: int,
	350	order_by: RoomSortOrder,
	351	reverse_order: bool,
	352	search_term: Optional[str],
	353	) -> Tuple[List[Dict[str, Any]], int]:
	354	"""Function to retrieve a paginated list of rooms as json.
	355
	356	Args:
	357	start: offset in the list
	358	limit: maximum amount of rooms to retrieve
	359	order_by: the sort order of the returned list
	360	reverse_order: whether to reverse the room list
	361	search_term: a string to filter room names by
	362	Returns:
	363	A list of room dicts and an integer representing the total number of
	364	rooms that exist given this query
	365	"""
	366	# Filter room names by a string
	367	where_statement = ""
	368	if search_term:
	369	where_statement = "WHERE state.name LIKE ?"
	370
	371	# Our postgres db driver converts ? -> %s in SQL strings as that's the
	372	# placeholder for postgres.
	373	# HOWEVER, if you put a % into your SQL then everything goes wibbly.
	374	# To get around this, we're going to surround search_term with %'s
	375	# before giving it to the database in python instead
	376	search_term = "%" + search_term + "%"
	377
	378	# Set ordering
	379	if RoomSortOrder(order_by) == RoomSortOrder.SIZE:
	380	# Deprecated in favour of RoomSortOrder.JOINED_MEMBERS
	381	order_by_column = "curr.joined_members"
	382	order_by_asc = False
	383	elif RoomSortOrder(order_by) == RoomSortOrder.ALPHABETICAL:
	384	# Deprecated in favour of RoomSortOrder.NAME
	385	order_by_column = "state.name"
	386	order_by_asc = True
	387	elif RoomSortOrder(order_by) == RoomSortOrder.NAME:
	388	order_by_column = "state.name"
	389	order_by_asc = True
	390	elif RoomSortOrder(order_by) == RoomSortOrder.CANONICAL_ALIAS:
	391	order_by_column = "state.canonical_alias"
	392	order_by_asc = True
	393	elif RoomSortOrder(order_by) == RoomSortOrder.JOINED_MEMBERS:
	394	order_by_column = "curr.joined_members"
	395	order_by_asc = False
	396	elif RoomSortOrder(order_by) == RoomSortOrder.JOINED_LOCAL_MEMBERS:
	397	order_by_column = "curr.local_users_in_room"
	398	order_by_asc = False
	399	elif RoomSortOrder(order_by) == RoomSortOrder.VERSION:
	400	order_by_column = "rooms.room_version"
	401	order_by_asc = False
	402	elif RoomSortOrder(order_by) == RoomSortOrder.CREATOR:
	403	order_by_column = "rooms.creator"
	404	order_by_asc = True
	405	elif RoomSortOrder(order_by) == RoomSortOrder.ENCRYPTION:
	406	order_by_column = "state.encryption"
	407	order_by_asc = True
	408	elif RoomSortOrder(order_by) == RoomSortOrder.FEDERATABLE:
	409	order_by_column = "state.is_federatable"
	410	order_by_asc = True
	411	elif RoomSortOrder(order_by) == RoomSortOrder.PUBLIC:
	412	order_by_column = "rooms.is_public"
	413	order_by_asc = True
	414	elif RoomSortOrder(order_by) == RoomSortOrder.JOIN_RULES:
	415	order_by_column = "state.join_rules"
	416	order_by_asc = True
	417	elif RoomSortOrder(order_by) == RoomSortOrder.GUEST_ACCESS:
	418	order_by_column = "state.guest_access"
	419	order_by_asc = True
	420	elif RoomSortOrder(order_by) == RoomSortOrder.HISTORY_VISIBILITY:
	421	order_by_column = "state.history_visibility"
	422	order_by_asc = True
	423	elif RoomSortOrder(order_by) == RoomSortOrder.STATE_EVENTS:
	424	order_by_column = "curr.current_state_events"
	425	order_by_asc = False
	426	else:
	427	raise StoreError(
	428	500, "Incorrect value for order_by provided: %s" % order_by
	429	)
	430
	431	# Whether to return the list in reverse order
	432	if reverse_order:
	433	# Flip the boolean
	434	order_by_asc = not order_by_asc
	435
	436	# Create one query for getting the limited number of events that the user asked
	437	# for, and another query for getting the total number of events that could be
	438	# returned. Thus allowing us to see if there are more events to paginate through
	439	info_sql = """
	440	SELECT state.room_id, state.name, state.canonical_alias, curr.joined_members,
	441	curr.local_users_in_room, rooms.room_version, rooms.creator,
	442	state.encryption, state.is_federatable, rooms.is_public, state.join_rules,
	443	state.guest_access, state.history_visibility, curr.current_state_events
	444	FROM room_stats_state state
	445	INNER JOIN room_stats_current curr USING (room_id)
	446	INNER JOIN rooms USING (room_id)
	447	%s
	448	ORDER BY %s %s
	449	LIMIT ?
	450	OFFSET ?
	451	""" % (
	452	where_statement,
	453	order_by_column,
	454	"ASC" if order_by_asc else "DESC",
	455	)
	456
	457	# Use a nested SELECT statement as SQL can't count(*) with an OFFSET
	458	count_sql = """
	459	SELECT count(*) FROM (
	460	SELECT room_id FROM room_stats_state state
	461	%s
	462	) AS get_room_ids
	463	""" % (
	464	where_statement,
	465	)
	466
	467	def _get_rooms_paginate_txn(txn):
	468	# Execute the data query
	469	sql_values = (limit, start)
	470	if search_term:
	471	# Add the search term into the WHERE clause
	472	sql_values = (search_term,) + sql_values
	473	txn.execute(info_sql, sql_values)
	474
	475	# Refactor room query data into a structured dictionary
	476	rooms = []
	477	for room in txn:
	478	rooms.append(
	479	{
	480	"room_id": room[0],
	481	"name": room[1],
	482	"canonical_alias": room[2],
	483	"joined_members": room[3],
	484	"joined_local_members": room[4],
	485	"version": room[5],
	486	"creator": room[6],
	487	"encryption": room[7],
	488	"federatable": room[8],
	489	"public": room[9],
	490	"join_rules": room[10],
	491	"guest_access": room[11],
	492	"history_visibility": room[12],
	493	"state_events": room[13],
	494	}
	495	)
	496
	497	# Execute the count query
	498
	499	# Add the search term into the WHERE clause if present
	500	sql_values = (search_term,) if search_term else ()
	501	txn.execute(count_sql, sql_values)
	502
	503	room_count = txn.fetchone()
	504	return rooms, room_count[0]
	505
	506	return await self.db_pool.runInteraction(
	507	"get_rooms_paginate", _get_rooms_paginate_txn,
	508	)
	509
	510	@cached(max_entries=10000)
	511	async def get_ratelimit_for_user(self, user_id):
	512	"""Check if there are any overrides for ratelimiting for the given
	513	user
	514
	515	Args:
	516	user_id (str)
	517
	518	Returns:
	519	RatelimitOverride if there is an override, else None. If the contents
	520	of RatelimitOverride are None or 0 then ratelimitng has been
	521	disabled for that user entirely.
	522	"""
	523	row = await self.db_pool.simple_select_one(
	524	table="ratelimit_override",
	525	keyvalues={"user_id": user_id},
	526	retcols=("messages_per_second", "burst_count"),
	527	allow_none=True,
	528	desc="get_ratelimit_for_user",
	529	)
	530
	531	if row:
	532	return RatelimitOverride(
	533	messages_per_second=row["messages_per_second"],
	534	burst_count=row["burst_count"],
	535	)
	536	else:
	537	return None
	538
	539	@cached()
	540	async def get_retention_policy_for_room(self, room_id):
	541	"""Get the retention policy for a given room.
	542
	543	If no retention policy has been found for this room, returns a policy defined
	544	by the configured default policy (which has None as both the 'min_lifetime' and
	545	the 'max_lifetime' if no default policy has been defined in the server's
	546	configuration).
	547
	548	Args:
	549	room_id (str): The ID of the room to get the retention policy of.
	550
	551	Returns:
	552	dict[int, int]: "min_lifetime" and "max_lifetime" for this room.
	553	"""
	554
	555	def get_retention_policy_for_room_txn(txn):
	556	txn.execute(
	557	"""
	558	SELECT min_lifetime, max_lifetime FROM room_retention
	559	INNER JOIN current_state_events USING (event_id, room_id)
	560	WHERE room_id = ?;
	561	""",
	562	(room_id,),
	563	)
	564
	565	return self.db_pool.cursor_to_dict(txn)
	566
	567	ret = await self.db_pool.runInteraction(
	568	"get_retention_policy_for_room", get_retention_policy_for_room_txn,
	569	)
	570
	571	# If we don't know this room ID, ret will be None, in this case return the default
	572	# policy.
	573	if not ret:
	574	return {
	575	"min_lifetime": self.config.retention_default_min_lifetime,
	576	"max_lifetime": self.config.retention_default_max_lifetime,
	577	}
	578
	579	row = ret[0]
	580
	581	# If one of the room's policy's attributes isn't defined, use the matching
	582	# attribute from the default policy.
	583	# The default values will be None if no default policy has been defined, or if one
	584	# of the attributes is missing from the default policy.
	585	if row["min_lifetime"] is None:
	586	row["min_lifetime"] = self.config.retention_default_min_lifetime
	587
	588	if row["max_lifetime"] is None:
	589	row["max_lifetime"] = self.config.retention_default_max_lifetime
	590
	591	return row
	592
	593	def get_media_mxcs_in_room(self, room_id):
	594	"""Retrieves all the local and remote media MXC URIs in a given room
	595
	596	Args:
	597	room_id (str)
	598
	599	Returns:
	600	The local and remote media as a lists of tuples where the key is
	601	the hostname and the value is the media ID.
	602	"""
	603
	604	def _get_media_mxcs_in_room_txn(txn):
	605	local_mxcs, remote_mxcs = self._get_media_mxcs_in_room_txn(txn, room_id)
	606	local_media_mxcs = []
	607	remote_media_mxcs = []
	608
	609	# Convert the IDs to MXC URIs
	610	for media_id in local_mxcs:
	611	local_media_mxcs.append("mxc://%s/%s" % (self.hs.hostname, media_id))
	612	for hostname, media_id in remote_mxcs:
	613	remote_media_mxcs.append("mxc://%s/%s" % (hostname, media_id))
	614
	615	return local_media_mxcs, remote_media_mxcs
	616
	617	return self.db_pool.runInteraction(
	618	"get_media_ids_in_room", _get_media_mxcs_in_room_txn
	619	)
	620
	621	def quarantine_media_ids_in_room(self, room_id, quarantined_by):
	622	"""For a room loops through all events with media and quarantines
	623	the associated media
	624	"""
	625
	626	logger.info("Quarantining media in room: %s", room_id)
	627
	628	def _quarantine_media_in_room_txn(txn):
	629	local_mxcs, remote_mxcs = self._get_media_mxcs_in_room_txn(txn, room_id)
	630	return self._quarantine_media_txn(
	631	txn, local_mxcs, remote_mxcs, quarantined_by
	632	)
	633
	634	return self.db_pool.runInteraction(
	635	"quarantine_media_in_room", _quarantine_media_in_room_txn
	636	)
	637
	638	def _get_media_mxcs_in_room_txn(self, txn, room_id):
	639	"""Retrieves all the local and remote media MXC URIs in a given room
	640
	641	Args:
	642	txn (cursor)
	643	room_id (str)
	644
	645	Returns:
	646	The local and remote media as a lists of tuples where the key is
	647	the hostname and the value is the media ID.
	648	"""
	649	mxc_re = re.compile("^mxc://([^/]+)/([^/#?]+)")
	650
	651	sql = """
	652	SELECT stream_ordering, json FROM events
	653	JOIN event_json USING (room_id, event_id)
	654	WHERE room_id = ?
	655	%(where_clause)s
	656	AND contains_url = ? AND outlier = ?
	657	ORDER BY stream_ordering DESC
	658	LIMIT ?
	659	"""
	660	txn.execute(sql % {"where_clause": ""}, (room_id, True, False, 100))
	661
	662	local_media_mxcs = []
	663	remote_media_mxcs = []
	664
	665	while True:
	666	next_token = None
	667	for stream_ordering, content_json in txn:
	668	next_token = stream_ordering
	669	event_json = db_to_json(content_json)
	670	content = event_json["content"]
	671	content_url = content.get("url")
	672	thumbnail_url = content.get("info", {}).get("thumbnail_url")
	673
	674	for url in (content_url, thumbnail_url):
	675	if not url:
	676	continue
	677	matches = mxc_re.match(url)
	678	if matches:
	679	hostname = matches.group(1)
	680	media_id = matches.group(2)
	681	if hostname == self.hs.hostname:
	682	local_media_mxcs.append(media_id)
	683	else:
	684	remote_media_mxcs.append((hostname, media_id))
	685
	686	if next_token is None:
	687	# We've gone through the whole room, so we're finished.
	688	break
	689
	690	txn.execute(
	691	sql % {"where_clause": "AND stream_ordering < ?"},
	692	(room_id, next_token, True, False, 100),
	693	)
	694
	695	return local_media_mxcs, remote_media_mxcs
	696
	697	def quarantine_media_by_id(
	698	self, server_name: str, media_id: str, quarantined_by: str,
	699	):
	700	"""quarantines a single local or remote media id
	701
	702	Args:
	703	server_name: The name of the server that holds this media
	704	media_id: The ID of the media to be quarantined
	705	quarantined_by: The user ID that initiated the quarantine request
	706	"""
	707	logger.info("Quarantining media: %s/%s", server_name, media_id)
	708	is_local = server_name == self.config.server_name
	709
	710	def _quarantine_media_by_id_txn(txn):
	711	local_mxcs = [media_id] if is_local else []
	712	remote_mxcs = [(server_name, media_id)] if not is_local else []
	713
	714	return self._quarantine_media_txn(
	715	txn, local_mxcs, remote_mxcs, quarantined_by
	716	)
	717
	718	return self.db_pool.runInteraction(
	719	"quarantine_media_by_user", _quarantine_media_by_id_txn
	720	)
	721
	722	def quarantine_media_ids_by_user(self, user_id: str, quarantined_by: str):
	723	"""quarantines all local media associated with a single user
	724
	725	Args:
	726	user_id: The ID of the user to quarantine media of
	727	quarantined_by: The ID of the user who made the quarantine request
	728	"""
	729
	730	def _quarantine_media_by_user_txn(txn):
	731	local_media_ids = self._get_media_ids_by_user_txn(txn, user_id)
	732	return self._quarantine_media_txn(txn, local_media_ids, [], quarantined_by)
	733
	734	return self.db_pool.runInteraction(
	735	"quarantine_media_by_user", _quarantine_media_by_user_txn
	736	)
	737
	738	def _get_media_ids_by_user_txn(self, txn, user_id: str, filter_quarantined=True):
	739	"""Retrieves local media IDs by a given user
	740
	741	Args:
	742	txn (cursor)
	743	user_id: The ID of the user to retrieve media IDs of
	744
	745	Returns:
	746	The local and remote media as a lists of tuples where the key is
	747	the hostname and the value is the media ID.
	748	"""
	749	# Local media
	750	sql = """
	751	SELECT media_id
	752	FROM local_media_repository
	753	WHERE user_id = ?
	754	"""
	755	if filter_quarantined:
	756	sql += "AND quarantined_by IS NULL"
	757	txn.execute(sql, (user_id,))
	758
	759	local_media_ids = [row[0] for row in txn]
	760
	761	# TODO: Figure out all remote media a user has referenced in a message
	762
	763	return local_media_ids
	764
	765	def _quarantine_media_txn(
	766	self,
	767	txn,
	768	local_mxcs: List[str],
	769	remote_mxcs: List[Tuple[str, str]],
	770	quarantined_by: str,
	771	) -> int:
	772	"""Quarantine local and remote media items
	773
	774	Args:
	775	txn (cursor)
	776	local_mxcs: A list of local mxc URLs
	777	remote_mxcs: A list of (remote server, media id) tuples representing
	778	remote mxc URLs
	779	quarantined_by: The ID of the user who initiated the quarantine request
	780	Returns:
	781	The total number of media items quarantined
	782	"""
	783	# Update all the tables to set the quarantined_by flag
	784	txn.executemany(
	785	"""
	786	UPDATE local_media_repository
	787	SET quarantined_by = ?
	788	WHERE media_id = ? AND safe_from_quarantine = ?
	789	""",
	790	((quarantined_by, media_id, False) for media_id in local_mxcs),
	791	)
	792	# Note that a rowcount of -1 can be used to indicate no rows were affected.
	793	total_media_quarantined = txn.rowcount if txn.rowcount > 0 else 0
	794
	795	txn.executemany(
	796	"""
	797	UPDATE remote_media_cache
	798	SET quarantined_by = ?
	799	WHERE media_origin = ? AND media_id = ?
	800	""",
	801	((quarantined_by, origin, media_id) for origin, media_id in remote_mxcs),
	802	)
	803	total_media_quarantined += txn.rowcount if txn.rowcount > 0 else 0
	804
	805	return total_media_quarantined
	806
	807	async def get_all_new_public_rooms(
	808	self, instance_name: str, last_id: int, current_id: int, limit: int
	809	) -> Tuple[List[Tuple[int, tuple]], int, bool]:
	810	"""Get updates for public rooms replication stream.
	811
	812	Args:
	813	instance_name: The writer we want to fetch updates from. Unused
	814	here since there is only ever one writer.
	815	last_id: The token to fetch updates from. Exclusive.
	816	current_id: The token to fetch updates up to. Inclusive.
	817	limit: The requested limit for the number of rows to return. The
	818	function may return more or fewer rows.
	819
	820	Returns:
	821	A tuple consisting of: the updates, a token to use to fetch
	822	subsequent updates, and whether we returned fewer rows than exists
	823	between the requested tokens due to the limit.
	824
	825	The token returned can be used in a subsequent call to this
	826	function to get further updatees.
	827
	828	The updates are a list of 2-tuples of stream ID and the row data
	829	"""
	830	if last_id == current_id:
	831	return [], current_id, False
	832
	833	def get_all_new_public_rooms(txn):
	834	sql = """
	835	SELECT stream_id, room_id, visibility, appservice_id, network_id
	836	FROM public_room_list_stream
	837	WHERE stream_id > ? AND stream_id <= ?
	838	ORDER BY stream_id ASC
	839	LIMIT ?
	840	"""
	841
	842	txn.execute(sql, (last_id, current_id, limit))
	843	updates = [(row[0], row[1:]) for row in txn]
	844	limited = False
	845	upto_token = current_id
	846	if len(updates) >= limit:
	847	upto_token = updates[-1][0]
	848	limited = True
	849
	850	return updates, upto_token, limited
	851
	852	return await self.db_pool.runInteraction(
	853	"get_all_new_public_rooms", get_all_new_public_rooms
	854	)
	855
	856
	857	class RoomBackgroundUpdateStore(SQLBaseStore):
	858	REMOVE_TOMESTONED_ROOMS_BG_UPDATE = "remove_tombstoned_rooms_from_directory"
	859	ADD_ROOMS_ROOM_VERSION_COLUMN = "add_rooms_room_version_column"
	860
	861	def __init__(self, database: DatabasePool, db_conn, hs):
	862	super(RoomBackgroundUpdateStore, self).__init__(database, db_conn, hs)
	863
	864	self.config = hs.config
	865
	866	self.db_pool.updates.register_background_update_handler(
	867	"insert_room_retention", self._background_insert_retention,
	868	)
	869
	870	self.db_pool.updates.register_background_update_handler(
	871	self.REMOVE_TOMESTONED_ROOMS_BG_UPDATE,
	872	self._remove_tombstoned_rooms_from_directory,
	873	)
	874
	875	self.db_pool.updates.register_background_update_handler(
	876	self.ADD_ROOMS_ROOM_VERSION_COLUMN,
	877	self._background_add_rooms_room_version_column,
	878	)
	879
	880	async def _background_insert_retention(self, progress, batch_size):
	881	"""Retrieves a list of all rooms within a range and inserts an entry for each of
	882	them into the room_retention table.
	883	NULLs the property's columns if missing from the retention event in the room's
	884	state (or NULLs all of them if there's no retention event in the room's state),
	885	so that we fall back to the server's retention policy.
	886	"""
	887
	888	last_room = progress.get("room_id", "")
	889
	890	def _background_insert_retention_txn(txn):
	891	txn.execute(
	892	"""
	893	SELECT state.room_id, state.event_id, events.json
	894	FROM current_state_events as state
	895	LEFT JOIN event_json AS events ON (state.event_id = events.event_id)
	896	WHERE state.room_id > ? AND state.type = '%s'
	897	ORDER BY state.room_id ASC
	898	LIMIT ?;
	899	"""
	900	% EventTypes.Retention,
	901	(last_room, batch_size),
	902	)
	903
	904	rows = self.db_pool.cursor_to_dict(txn)
	905
	906	if not rows:
	907	return True
	908
	909	for row in rows:
	910	if not row["json"]:
	911	retention_policy = {}
	912	else:
	913	ev = db_to_json(row["json"])
	914	retention_policy = ev["content"]
	915
	916	self.db_pool.simple_insert_txn(
	917	txn=txn,
	918	table="room_retention",
	919	values={
	920	"room_id": row["room_id"],
	921	"event_id": row["event_id"],
	922	"min_lifetime": retention_policy.get("min_lifetime"),
	923	"max_lifetime": retention_policy.get("max_lifetime"),
	924	},
	925	)
	926
	927	logger.info("Inserted %d rows into room_retention", len(rows))
	928
	929	self.db_pool.updates._background_update_progress_txn(
	930	txn, "insert_room_retention", {"room_id": rows[-1]["room_id"]}
	931	)
	932
	933	if batch_size > len(rows):
	934	return True
	935	else:
	936	return False
	937
	938	end = await self.db_pool.runInteraction(
	939	"insert_room_retention", _background_insert_retention_txn,
	940	)
	941
	942	if end:
	943	await self.db_pool.updates._end_background_update("insert_room_retention")
	944
	945	return batch_size
	946
	947	async def _background_add_rooms_room_version_column(
	948	self, progress: dict, batch_size: int
	949	):
	950	"""Background update to go and add room version inforamtion to `rooms`
	951	table from `current_state_events` table.
	952	"""
	953
	954	last_room_id = progress.get("room_id", "")
	955
	956	def _background_add_rooms_room_version_column_txn(txn: LoggingTransaction):
	957	sql = """
	958	SELECT room_id, json FROM current_state_events
	959	INNER JOIN event_json USING (room_id, event_id)
	960	WHERE room_id > ? AND type = 'm.room.create' AND state_key = ''
	961	ORDER BY room_id
	962	LIMIT ?
	963	"""
	964
	965	txn.execute(sql, (last_room_id, batch_size))
	966
	967	updates = []
	968	for room_id, event_json in txn:
	969	event_dict = db_to_json(event_json)
	970	room_version_id = event_dict.get("content", {}).get(
	971	"room_version", RoomVersions.V1.identifier
	972	)
	973
	974	creator = event_dict.get("content").get("creator")
	975
	976	updates.append((room_id, creator, room_version_id))
	977
	978	if not updates:
	979	return True
	980
	981	new_last_room_id = ""
	982	for room_id, creator, room_version_id in updates:
	983	# We upsert here just in case we don't already have a row,
	984	# mainly for paranoia as much badness would happen if we don't
	985	# insert the row and then try and get the room version for the
	986	# room.
	987	self.db_pool.simple_upsert_txn(
	988	txn,
	989	table="rooms",
	990	keyvalues={"room_id": room_id},
	991	values={"room_version": room_version_id},
	992	insertion_values={"is_public": False, "creator": creator},
	993	)
	994	new_last_room_id = room_id
	995
	996	self.db_pool.updates._background_update_progress_txn(
	997	txn, self.ADD_ROOMS_ROOM_VERSION_COLUMN, {"room_id": new_last_room_id}
	998	)
	999
	1000	return False
	1001
	1002	end = await self.db_pool.runInteraction(
	1003	"_background_add_rooms_room_version_column",
	1004	_background_add_rooms_room_version_column_txn,
	1005	)
	1006
	1007	if end:
	1008	await self.db_pool.updates._end_background_update(
	1009	self.ADD_ROOMS_ROOM_VERSION_COLUMN
	1010	)
	1011
	1012	return batch_size
	1013
	1014	async def _remove_tombstoned_rooms_from_directory(
	1015	self, progress, batch_size
	1016	) -> int:
	1017	"""Removes any rooms with tombstone events from the room directory
	1018
	1019	Nowadays this is handled by the room upgrade handler, but we may have some
	1020	that got left behind
	1021	"""
	1022
	1023	last_room = progress.get("room_id", "")
	1024
	1025	def _get_rooms(txn):
	1026	txn.execute(
	1027	"""
	1028	SELECT room_id
	1029	FROM rooms r
	1030	INNER JOIN current_state_events cse USING (room_id)
	1031	WHERE room_id > ? AND r.is_public
	1032	AND cse.type = '%s' AND cse.state_key = ''
	1033	ORDER BY room_id ASC
	1034	LIMIT ?;
	1035	"""
	1036	% EventTypes.Tombstone,
	1037	(last_room, batch_size),
	1038	)
	1039
	1040	return [row[0] for row in txn]
	1041
	1042	rooms = await self.db_pool.runInteraction(
	1043	"get_tombstoned_directory_rooms", _get_rooms
	1044	)
	1045
	1046	if not rooms:
	1047	await self.db_pool.updates._end_background_update(
	1048	self.REMOVE_TOMESTONED_ROOMS_BG_UPDATE
	1049	)
	1050	return 0
	1051
	1052	for room_id in rooms:
	1053	logger.info("Removing tombstoned room %s from the directory", room_id)
	1054	await self.set_room_is_public(room_id, False)
	1055
	1056	await self.db_pool.updates._background_update_progress(
	1057	self.REMOVE_TOMESTONED_ROOMS_BG_UPDATE, {"room_id": rooms[-1]}
	1058	)
	1059
	1060	return len(rooms)
	1061
	1062	@abstractmethod
	1063	def set_room_is_public(self, room_id, is_public):
	1064	# this will need to be implemented if a background update is performed with
	1065	# existing (tombstoned, public) rooms in the database.
	1066	#
	1067	# It's overridden by RoomStore for the synapse master.
	1068	raise NotImplementedError()
	1069
	1070
	1071	class RoomStore(RoomBackgroundUpdateStore, RoomWorkerStore, SearchStore):
	1072	def __init__(self, database: DatabasePool, db_conn, hs):
	1073	super(RoomStore, self).__init__(database, db_conn, hs)
	1074
	1075	self.config = hs.config
	1076
	1077	async def upsert_room_on_join(self, room_id: str, room_version: RoomVersion):
	1078	"""Ensure that the room is stored in the table
	1079
	1080	Called when we join a room over federation, and overwrites any room version
	1081	currently in the table.
	1082	"""
	1083	await self.db_pool.simple_upsert(
	1084	desc="upsert_room_on_join",
	1085	table="rooms",
	1086	keyvalues={"room_id": room_id},
	1087	values={"room_version": room_version.identifier},
	1088	insertion_values={"is_public": False, "creator": ""},
	1089	# rooms has a unique constraint on room_id, so no need to lock when doing an
	1090	# emulated upsert.
	1091	lock=False,
	1092	)
	1093
	1094	async def store_room(
	1095	self,
	1096	room_id: str,
	1097	room_creator_user_id: str,
	1098	is_public: bool,
	1099	room_version: RoomVersion,
	1100	):
	1101	"""Stores a room.
	1102
	1103	Args:
	1104	room_id: The desired room ID, can be None.
	1105	room_creator_user_id: The user ID of the room creator.
	1106	is_public: True to indicate that this room should appear in
	1107	public room lists.
	1108	room_version: The version of the room
	1109	Raises:
	1110	StoreError if the room could not be stored.
	1111	"""
	1112	try:
	1113
	1114	def store_room_txn(txn, next_id):
	1115	self.db_pool.simple_insert_txn(
	1116	txn,
	1117	"rooms",
	1118	{
	1119	"room_id": room_id,
	1120	"creator": room_creator_user_id,
	1121	"is_public": is_public,
	1122	"room_version": room_version.identifier,
	1123	},
	1124	)
	1125	if is_public:
	1126	self.db_pool.simple_insert_txn(
	1127	txn,
	1128	table="public_room_list_stream",
	1129	values={
	1130	"stream_id": next_id,
	1131	"room_id": room_id,
	1132	"visibility": is_public,
	1133	},
	1134	)
	1135
	1136	with self._public_room_id_gen.get_next() as next_id:
	1137	await self.db_pool.runInteraction(
	1138	"store_room_txn", store_room_txn, next_id
	1139	)
	1140	except Exception as e:
	1141	logger.error("store_room with room_id=%s failed: %s", room_id, e)
	1142	raise StoreError(500, "Problem creating room.")
	1143
	1144	async def maybe_store_room_on_invite(self, room_id: str, room_version: RoomVersion):
	1145	"""
	1146	When we receive an invite over federation, store the version of the room if we
	1147	don't already know the room version.
	1148	"""
	1149	await self.db_pool.simple_upsert(
	1150	desc="maybe_store_room_on_invite",
	1151	table="rooms",
	1152	keyvalues={"room_id": room_id},
	1153	values={},
	1154	insertion_values={
	1155	"room_version": room_version.identifier,
	1156	"is_public": False,
	1157	"creator": "",
	1158	},
	1159	# rooms has a unique constraint on room_id, so no need to lock when doing an
	1160	# emulated upsert.
	1161	lock=False,
	1162	)
	1163
	1164	async def set_room_is_public(self, room_id, is_public):
	1165	def set_room_is_public_txn(txn, next_id):
	1166	self.db_pool.simple_update_one_txn(
	1167	txn,
	1168	table="rooms",
	1169	keyvalues={"room_id": room_id},
	1170	updatevalues={"is_public": is_public},
	1171	)
	1172
	1173	entries = self.db_pool.simple_select_list_txn(
	1174	txn,
	1175	table="public_room_list_stream",
	1176	keyvalues={
	1177	"room_id": room_id,
	1178	"appservice_id": None,
	1179	"network_id": None,
	1180	},
	1181	retcols=("stream_id", "visibility"),
	1182	)
	1183
	1184	entries.sort(key=lambda r: r["stream_id"])
	1185
	1186	add_to_stream = True
	1187	if entries:
	1188	add_to_stream = bool(entries[-1]["visibility"]) != is_public
	1189
	1190	if add_to_stream:
	1191	self.db_pool.simple_insert_txn(
	1192	txn,
	1193	table="public_room_list_stream",
	1194	values={
	1195	"stream_id": next_id,
	1196	"room_id": room_id,
	1197	"visibility": is_public,
	1198	"appservice_id": None,
	1199	"network_id": None,
	1200	},
	1201	)
	1202
	1203	with self._public_room_id_gen.get_next() as next_id:
	1204	await self.db_pool.runInteraction(
	1205	"set_room_is_public", set_room_is_public_txn, next_id
	1206	)
	1207	self.hs.get_notifier().on_new_replication_data()
	1208
	1209	async def set_room_is_public_appservice(
	1210	self, room_id, appservice_id, network_id, is_public
	1211	):
	1212	"""Edit the appservice/network specific public room list.
	1213
	1214	Each appservice can have a number of published room lists associated
	1215	with them, keyed off of an appservice defined `network_id`, which
	1216	basically represents a single instance of a bridge to a third party
	1217	network.
	1218
	1219	Args:
	1220	room_id (str)
	1221	appservice_id (str)
	1222	network_id (str)
	1223	is_public (bool): Whether to publish or unpublish the room from the
	1224	list.
	1225	"""
	1226
	1227	def set_room_is_public_appservice_txn(txn, next_id):
	1228	if is_public:
	1229	try:
	1230	self.db_pool.simple_insert_txn(
	1231	txn,
	1232	table="appservice_room_list",
	1233	values={
	1234	"appservice_id": appservice_id,
	1235	"network_id": network_id,
	1236	"room_id": room_id,
	1237	},
	1238	)
	1239	except self.database_engine.module.IntegrityError:
	1240	# We've already inserted, nothing to do.
	1241	return
	1242	else:
	1243	self.db_pool.simple_delete_txn(
	1244	txn,
	1245	table="appservice_room_list",
	1246	keyvalues={
	1247	"appservice_id": appservice_id,
	1248	"network_id": network_id,
	1249	"room_id": room_id,
	1250	},
	1251	)
	1252
	1253	entries = self.db_pool.simple_select_list_txn(
	1254	txn,
	1255	table="public_room_list_stream",
	1256	keyvalues={
	1257	"room_id": room_id,
	1258	"appservice_id": appservice_id,
	1259	"network_id": network_id,
	1260	},
	1261	retcols=("stream_id", "visibility"),
	1262	)
	1263
	1264	entries.sort(key=lambda r: r["stream_id"])
	1265
	1266	add_to_stream = True
	1267	if entries:
	1268	add_to_stream = bool(entries[-1]["visibility"]) != is_public
	1269
	1270	if add_to_stream:
	1271	self.db_pool.simple_insert_txn(
	1272	txn,
	1273	table="public_room_list_stream",
	1274	values={
	1275	"stream_id": next_id,
	1276	"room_id": room_id,
	1277	"visibility": is_public,
	1278	"appservice_id": appservice_id,
	1279	"network_id": network_id,
	1280	},
	1281	)
	1282
	1283	with self._public_room_id_gen.get_next() as next_id:
	1284	await self.db_pool.runInteraction(
	1285	"set_room_is_public_appservice",
	1286	set_room_is_public_appservice_txn,
	1287	next_id,
	1288	)
	1289	self.hs.get_notifier().on_new_replication_data()
	1290
	1291	def get_room_count(self):
	1292	"""Retrieve a list of all rooms
	1293	"""
	1294
	1295	def f(txn):
	1296	sql = "SELECT count(*) FROM rooms"
	1297	txn.execute(sql)
	1298	row = txn.fetchone()
	1299	return row[0] or 0
	1300
	1301	return self.db_pool.runInteraction("get_rooms", f)
	1302
	1303	def add_event_report(
	1304	self, room_id, event_id, user_id, reason, content, received_ts
	1305	):
	1306	next_id = self._event_reports_id_gen.get_next()
	1307	return self.db_pool.simple_insert(
	1308	table="event_reports",
	1309	values={
	1310	"id": next_id,
	1311	"received_ts": received_ts,
	1312	"room_id": room_id,
	1313	"event_id": event_id,
	1314	"user_id": user_id,
	1315	"reason": reason,
	1316	"content": json.dumps(content),
	1317	},
	1318	desc="add_event_report",
	1319	)
	1320
	1321	def get_current_public_room_stream_id(self):
	1322	return self._public_room_id_gen.get_current_token()
	1323
	1324	async def block_room(self, room_id: str, user_id: str) -> None:
	1325	"""Marks the room as blocked. Can be called multiple times.
	1326
	1327	Args:
	1328	room_id: Room to block
	1329	user_id: Who blocked it
	1330	"""
	1331	await self.db_pool.simple_upsert(
	1332	table="blocked_rooms",
	1333	keyvalues={"room_id": room_id},
	1334	values={},
	1335	insertion_values={"user_id": user_id},
	1336	desc="block_room",
	1337	)
	1338	await self.db_pool.runInteraction(
	1339	"block_room_invalidation",
	1340	self._invalidate_cache_and_stream,
	1341	self.is_room_blocked,
	1342	(room_id,),
	1343	)
	1344
	1345	async def get_rooms_for_retention_period_in_range(
	1346	self, min_ms: Optional[int], max_ms: Optional[int], include_null: bool = False
	1347	) -> Dict[str, dict]:
	1348	"""Retrieves all of the rooms within the given retention range.
	1349
	1350	Optionally includes the rooms which don't have a retention policy.
	1351
	1352	Args:
	1353	min_ms: Duration in milliseconds that define the lower limit of
	1354	the range to handle (exclusive). If None, doesn't set a lower limit.
	1355	max_ms: Duration in milliseconds that define the upper limit of
	1356	the range to handle (inclusive). If None, doesn't set an upper limit.
	1357	include_null: Whether to include rooms which retention policy is NULL
	1358	in the returned set.
	1359
	1360	Returns:
	1361	The rooms within this range, along with their retention
	1362	policy. The key is "room_id", and maps to a dict describing the retention
	1363	policy associated with this room ID. The keys for this nested dict are
	1364	"min_lifetime" (int\|None), and "max_lifetime" (int\|None).
	1365	"""
	1366
	1367	def get_rooms_for_retention_period_in_range_txn(txn):
	1368	range_conditions = []
	1369	args = []
	1370
	1371	if min_ms is not None:
	1372	range_conditions.append("max_lifetime > ?")
	1373	args.append(min_ms)
	1374
	1375	if max_ms is not None:
	1376	range_conditions.append("max_lifetime <= ?")
	1377	args.append(max_ms)
	1378
	1379	# Do a first query which will retrieve the rooms that have a retention policy
	1380	# in their current state.
	1381	sql = """
	1382	SELECT room_id, min_lifetime, max_lifetime FROM room_retention
	1383	INNER JOIN current_state_events USING (event_id, room_id)
	1384	"""
	1385
	1386	if len(range_conditions):
	1387	sql += " WHERE (" + " AND ".join(range_conditions) + ")"
	1388
	1389	if include_null:
	1390	sql += " OR max_lifetime IS NULL"
	1391
	1392	txn.execute(sql, args)
	1393
	1394	rows = self.db_pool.cursor_to_dict(txn)
	1395	rooms_dict = {}
	1396
	1397	for row in rows:
	1398	rooms_dict[row["room_id"]] = {
	1399	"min_lifetime": row["min_lifetime"],
	1400	"max_lifetime": row["max_lifetime"],
	1401	}
	1402
	1403	if include_null:
	1404	# If required, do a second query that retrieves all of the rooms we know
	1405	# of so we can handle rooms with no retention policy.
	1406	sql = "SELECT DISTINCT room_id FROM current_state_events"
	1407
	1408	txn.execute(sql)
	1409
	1410	rows = self.db_pool.cursor_to_dict(txn)
	1411
	1412	# If a room isn't already in the dict (i.e. it doesn't have a retention
	1413	# policy in its state), add it with a null policy.
	1414	for row in rows:
	1415	if row["room_id"] not in rooms_dict:
	1416	rooms_dict[row["room_id"]] = {
	1417	"min_lifetime": None,
	1418	"max_lifetime": None,
	1419	}
	1420
	1421	return rooms_dict
	1422
	1423	rooms = await self.db_pool.runInteraction(
	1424	"get_rooms_for_retention_period_in_range",
	1425	get_rooms_for_retention_period_in_range_txn,
	1426	)
	1427
	1428	return rooms

+1072

-0

synapse/storage/databases/main/roommember.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2018 New Vector Ltd
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import logging
	17	from typing import TYPE_CHECKING, Awaitable, Iterable, List, Optional, Set
	18
	19	from twisted.internet import defer
	20
	21	from synapse.api.constants import EventTypes, Membership
	22	from synapse.events import EventBase
	23	from synapse.events.snapshot import EventContext
	24	from synapse.metrics import LaterGauge
	25	from synapse.metrics.background_process_metrics import run_as_background_process
	26	from synapse.storage._base import (
	27	LoggingTransaction,
	28	SQLBaseStore,
	29	db_to_json,
	30	make_in_list_sql_clause,
	31	)
	32	from synapse.storage.database import DatabasePool
	33	from synapse.storage.databases.main.events_worker import EventsWorkerStore
	34	from synapse.storage.engines import Sqlite3Engine
	35	from synapse.storage.roommember import (
	36	GetRoomsForUserWithStreamOrdering,
	37	MemberSummary,
	38	ProfileInfo,
	39	RoomsForUser,
	40	)
	41	from synapse.types import Collection, get_domain_from_id
	42	from synapse.util.async_helpers import Linearizer
	43	from synapse.util.caches import intern_string
	44	from synapse.util.caches.descriptors import _CacheContext, cached, cachedList
	45	from synapse.util.metrics import Measure
	46
	47	if TYPE_CHECKING:
	48	from synapse.state import _StateCacheEntry
	49
	50	logger = logging.getLogger(__name__)
	51
	52
	53	_MEMBERSHIP_PROFILE_UPDATE_NAME = "room_membership_profile_update"
	54	_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME = "current_state_events_membership"
	55
	56
	57	class RoomMemberWorkerStore(EventsWorkerStore):
	58	def __init__(self, database: DatabasePool, db_conn, hs):
	59	super(RoomMemberWorkerStore, self).__init__(database, db_conn, hs)
	60
	61	# Is the current_state_events.membership up to date? Or is the
	62	# background update still running?
	63	self._current_state_events_membership_up_to_date = False
	64
	65	txn = LoggingTransaction(
	66	db_conn.cursor(),
	67	name="_check_safe_current_state_events_membership_updated",
	68	database_engine=self.database_engine,
	69	)
	70	self._check_safe_current_state_events_membership_updated_txn(txn)
	71	txn.close()
	72
	73	if self.hs.config.metrics_flags.known_servers:
	74	self._known_servers_count = 1
	75	self.hs.get_clock().looping_call(
	76	run_as_background_process,
	77	60 * 1000,
	78	"_count_known_servers",
	79	self._count_known_servers,
	80	)
	81	self.hs.get_clock().call_later(
	82	1000,
	83	run_as_background_process,
	84	"_count_known_servers",
	85	self._count_known_servers,
	86	)
	87	LaterGauge(
	88	"synapse_federation_known_servers",
	89	"",
	90	[],
	91	lambda: self._known_servers_count,
	92	)
	93
	94	@defer.inlineCallbacks
	95	def _count_known_servers(self):
	96	"""
	97	Count the servers that this server knows about.
	98
	99	The statistic is stored on the class for the
	100	`synapse_federation_known_servers` LaterGauge to collect.
	101	"""
	102
	103	def _transact(txn):
	104	if isinstance(self.database_engine, Sqlite3Engine):
	105	query = """
	106	SELECT COUNT(DISTINCT substr(out.user_id, pos+1))
	107	FROM (
	108	SELECT rm.user_id as user_id, instr(rm.user_id, ':')
	109	AS pos FROM room_memberships as rm
	110	INNER JOIN current_state_events as c ON rm.event_id = c.event_id
	111	WHERE c.type = 'm.room.member'
	112	) as out
	113	"""
	114	else:
	115	query = """
	116	SELECT COUNT(DISTINCT split_part(state_key, ':', 2))
	117	FROM current_state_events
	118	WHERE type = 'm.room.member' AND membership = 'join';
	119	"""
	120	txn.execute(query)
	121	return list(txn)[0][0]
	122
	123	count = yield self.db_pool.runInteraction("get_known_servers", _transact)
	124
	125	# We always know about ourselves, even if we have nothing in
	126	# room_memberships (for example, the server is new).
	127	self._known_servers_count = max([count, 1])
	128	return self._known_servers_count
	129
	130	def _check_safe_current_state_events_membership_updated_txn(self, txn):
	131	"""Checks if it is safe to assume the new current_state_events
	132	membership column is up to date
	133	"""
	134
	135	pending_update = self.db_pool.simple_select_one_txn(
	136	txn,
	137	table="background_updates",
	138	keyvalues={"update_name": _CURRENT_STATE_MEMBERSHIP_UPDATE_NAME},
	139	retcols=["update_name"],
	140	allow_none=True,
	141	)
	142
	143	self._current_state_events_membership_up_to_date = not pending_update
	144
	145	# If the update is still running, reschedule to run.
	146	if pending_update:
	147	self._clock.call_later(
	148	15.0,
	149	run_as_background_process,
	150	"_check_safe_current_state_events_membership_updated",
	151	self.db_pool.runInteraction,
	152	"_check_safe_current_state_events_membership_updated",
	153	self._check_safe_current_state_events_membership_updated_txn,
	154	)
	155
	156	@cached(max_entries=100000, iterable=True)
	157	def get_users_in_room(self, room_id: str):
	158	return self.db_pool.runInteraction(
	159	"get_users_in_room", self.get_users_in_room_txn, room_id
	160	)
	161
	162	def get_users_in_room_txn(self, txn, room_id: str) -> List[str]:
	163	# If we can assume current_state_events.membership is up to date
	164	# then we can avoid a join, which is a Very Good Thing given how
	165	# frequently this function gets called.
	166	if self._current_state_events_membership_up_to_date:
	167	sql = """
	168	SELECT state_key FROM current_state_events
	169	WHERE type = 'm.room.member' AND room_id = ? AND membership = ?
	170	"""
	171	else:
	172	sql = """
	173	SELECT state_key FROM room_memberships as m
	174	INNER JOIN current_state_events as c
	175	ON m.event_id = c.event_id
	176	AND m.room_id = c.room_id
	177	AND m.user_id = c.state_key
	178	WHERE c.type = 'm.room.member' AND c.room_id = ? AND m.membership = ?
	179	"""
	180
	181	txn.execute(sql, (room_id, Membership.JOIN))
	182	return [r[0] for r in txn]
	183
	184	@cached(max_entries=100000)
	185	def get_room_summary(self, room_id: str):
	186	""" Get the details of a room roughly suitable for use by the room
	187	summary extension to /sync. Useful when lazy loading room members.
	188	Args:
	189	room_id: The room ID to query
	190	Returns:
	191	Deferred[dict[str, MemberSummary]:
	192	dict of membership states, pointing to a MemberSummary named tuple.
	193	"""
	194
	195	def _get_room_summary_txn(txn):
	196	# first get counts.
	197	# We do this all in one transaction to keep the cache small.
	198	# FIXME: get rid of this when we have room_stats
	199
	200	# If we can assume current_state_events.membership is up to date
	201	# then we can avoid a join, which is a Very Good Thing given how
	202	# frequently this function gets called.
	203	if self._current_state_events_membership_up_to_date:
	204	# Note, rejected events will have a null membership field, so
	205	# we we manually filter them out.
	206	sql = """
	207	SELECT count(*), membership FROM current_state_events
	208	WHERE type = 'm.room.member' AND room_id = ?
	209	AND membership IS NOT NULL
	210	GROUP BY membership
	211	"""
	212	else:
	213	sql = """
	214	SELECT count(*), m.membership FROM room_memberships as m
	215	INNER JOIN current_state_events as c
	216	ON m.event_id = c.event_id
	217	AND m.room_id = c.room_id
	218	AND m.user_id = c.state_key
	219	WHERE c.type = 'm.room.member' AND c.room_id = ?
	220	GROUP BY m.membership
	221	"""
	222
	223	txn.execute(sql, (room_id,))
	224	res = {}
	225	for count, membership in txn:
	226	summary = res.setdefault(membership, MemberSummary([], count))
	227
	228	# we order by membership and then fairly arbitrarily by event_id so
	229	# heroes are consistent
	230	if self._current_state_events_membership_up_to_date:
	231	# Note, rejected events will have a null membership field, so
	232	# we we manually filter them out.
	233	sql = """
	234	SELECT state_key, membership, event_id
	235	FROM current_state_events
	236	WHERE type = 'm.room.member' AND room_id = ?
	237	AND membership IS NOT NULL
	238	ORDER BY
	239	CASE membership WHEN ? THEN 1 WHEN ? THEN 2 ELSE 3 END ASC,
	240	event_id ASC
	241	LIMIT ?
	242	"""
	243	else:
	244	sql = """
	245	SELECT c.state_key, m.membership, c.event_id
	246	FROM room_memberships as m
	247	INNER JOIN current_state_events as c USING (room_id, event_id)
	248	WHERE c.type = 'm.room.member' AND c.room_id = ?
	249	ORDER BY
	250	CASE m.membership WHEN ? THEN 1 WHEN ? THEN 2 ELSE 3 END ASC,
	251	c.event_id ASC
	252	LIMIT ?
	253	"""
	254
	255	# 6 is 5 (number of heroes) plus 1, in case one of them is the calling user.
	256	txn.execute(sql, (room_id, Membership.JOIN, Membership.INVITE, 6))
	257	for user_id, membership, event_id in txn:
	258	summary = res[membership]
	259	# we will always have a summary for this membership type at this
	260	# point given the summary currently contains the counts.
	261	members = summary.members
	262	members.append((user_id, event_id))
	263
	264	return res
	265
	266	return self.db_pool.runInteraction("get_room_summary", _get_room_summary_txn)
	267
	268	@cached()
	269	def get_invited_rooms_for_local_user(self, user_id: str) -> Awaitable[RoomsForUser]:
	270	"""Get all the rooms the local user is invited to.
	271
	272	Args:
	273	user_id: The user ID.
	274
	275	Returns:
	276	A awaitable list of RoomsForUser.
	277	"""
	278
	279	return self.get_rooms_for_local_user_where_membership_is(
	280	user_id, [Membership.INVITE]
	281	)
	282
	283	async def get_invite_for_local_user_in_room(
	284	self, user_id: str, room_id: str
	285	) -> Optional[RoomsForUser]:
	286	"""Gets the invite for the given local user and room.
	287
	288	Args:
	289	user_id: The user ID to find the invite of.
	290	room_id: The room to user was invited to.
	291
	292	Returns:
	293	Either a RoomsForUser or None if no invite was found.
	294	"""
	295	invites = await self.get_invited_rooms_for_local_user(user_id)
	296	for invite in invites:
	297	if invite.room_id == room_id:
	298	return invite
	299	return None
	300
	301	async def get_rooms_for_local_user_where_membership_is(
	302	self, user_id: str, membership_list: List[str]
	303	) -> Optional[List[RoomsForUser]]:
	304	"""Get all the rooms for this local user where the membership for this user
	305	matches one in the membership list.
	306
	307	Filters out forgotten rooms.
	308
	309	Args:
	310	user_id: The user ID.
	311	membership_list: A list of synapse.api.constants.Membership
	312	values which the user must be in.
	313
	314	Returns:
	315	The RoomsForUser that the user matches the membership types.
	316	"""
	317	if not membership_list:
	318	return None
	319
	320	rooms = await self.db_pool.runInteraction(
	321	"get_rooms_for_local_user_where_membership_is",
	322	self._get_rooms_for_local_user_where_membership_is_txn,
	323	user_id,
	324	membership_list,
	325	)
	326
	327	# Now we filter out forgotten rooms
	328	forgotten_rooms = await self.get_forgotten_rooms_for_user(user_id)
	329	return [room for room in rooms if room.room_id not in forgotten_rooms]
	330
	331	def _get_rooms_for_local_user_where_membership_is_txn(
	332	self, txn, user_id: str, membership_list: List[str]
	333	) -> List[RoomsForUser]:
	334	# Paranoia check.
	335	if not self.hs.is_mine_id(user_id):
	336	raise Exception(
	337	"Cannot call 'get_rooms_for_local_user_where_membership_is' on non-local user %r"
	338	% (user_id,),
	339	)
	340
	341	clause, args = make_in_list_sql_clause(
	342	self.database_engine, "c.membership", membership_list
	343	)
	344
	345	sql = """
	346	SELECT room_id, e.sender, c.membership, event_id, e.stream_ordering
	347	FROM local_current_membership AS c
	348	INNER JOIN events AS e USING (room_id, event_id)
	349	WHERE
	350	user_id = ?
	351	AND %s
	352	""" % (
	353	clause,
	354	)
	355
	356	txn.execute(sql, (user_id, *args))
	357	results = [RoomsForUser(**r) for r in self.db_pool.cursor_to_dict(txn)]
	358
	359	return results
	360
	361	@cached(max_entries=500000, iterable=True)
	362	def get_rooms_for_user_with_stream_ordering(self, user_id: str):
	363	"""Returns a set of room_ids the user is currently joined to.
	364
	365	If a remote user only returns rooms this server is currently
	366	participating in.
	367
	368	Args:
	369	user_id
	370
	371	Returns:
	372	Deferred[frozenset[GetRoomsForUserWithStreamOrdering]]: Returns
	373	the rooms the user is in currently, along with the stream ordering
	374	of the most recent join for that user and room.
	375	"""
	376	return self.db_pool.runInteraction(
	377	"get_rooms_for_user_with_stream_ordering",
	378	self._get_rooms_for_user_with_stream_ordering_txn,
	379	user_id,
	380	)
	381
	382	def _get_rooms_for_user_with_stream_ordering_txn(self, txn, user_id: str):
	383	# We use `current_state_events` here and not `local_current_membership`
	384	# as a) this gets called with remote users and b) this only gets called
	385	# for rooms the server is participating in.
	386	if self._current_state_events_membership_up_to_date:
	387	sql = """
	388	SELECT room_id, e.stream_ordering
	389	FROM current_state_events AS c
	390	INNER JOIN events AS e USING (room_id, event_id)
	391	WHERE
	392	c.type = 'm.room.member'
	393	AND state_key = ?
	394	AND c.membership = ?
	395	"""
	396	else:
	397	sql = """
	398	SELECT room_id, e.stream_ordering
	399	FROM current_state_events AS c
	400	INNER JOIN room_memberships AS m USING (room_id, event_id)
	401	INNER JOIN events AS e USING (room_id, event_id)
	402	WHERE
	403	c.type = 'm.room.member'
	404	AND state_key = ?
	405	AND m.membership = ?
	406	"""
	407
	408	txn.execute(sql, (user_id, Membership.JOIN))
	409	results = frozenset(GetRoomsForUserWithStreamOrdering(*row) for row in txn)
	410
	411	return results
	412
	413	async def get_users_server_still_shares_room_with(
	414	self, user_ids: Collection[str]
	415	) -> Set[str]:
	416	"""Given a list of users return the set that the server still share a
	417	room with.
	418	"""
	419
	420	if not user_ids:
	421	return set()
	422
	423	def _get_users_server_still_shares_room_with_txn(txn):
	424	sql = """
	425	SELECT state_key FROM current_state_events
	426	WHERE
	427	type = 'm.room.member'
	428	AND membership = 'join'
	429	AND %s
	430	GROUP BY state_key
	431	"""
	432
	433	clause, args = make_in_list_sql_clause(
	434	self.database_engine, "state_key", user_ids
	435	)
	436
	437	txn.execute(sql % (clause,), args)
	438
	439	return {row[0] for row in txn}
	440
	441	return await self.db_pool.runInteraction(
	442	"get_users_server_still_shares_room_with",
	443	_get_users_server_still_shares_room_with_txn,
	444	)
	445
	446	async def get_rooms_for_user(self, user_id: str, on_invalidate=None):
	447	"""Returns a set of room_ids the user is currently joined to.
	448
	449	If a remote user only returns rooms this server is currently
	450	participating in.
	451	"""
	452	rooms = await self.get_rooms_for_user_with_stream_ordering(
	453	user_id, on_invalidate=on_invalidate
	454	)
	455	return frozenset(r.room_id for r in rooms)
	456
	457	@cached(max_entries=500000, cache_context=True, iterable=True)
	458	async def get_users_who_share_room_with_user(
	459	self, user_id: str, cache_context: _CacheContext
	460	) -> Set[str]:
	461	"""Returns the set of users who share a room with `user_id`
	462	"""
	463	room_ids = await self.get_rooms_for_user(
	464	user_id, on_invalidate=cache_context.invalidate
	465	)
	466
	467	user_who_share_room = set()
	468	for room_id in room_ids:
	469	user_ids = await self.get_users_in_room(
	470	room_id, on_invalidate=cache_context.invalidate
	471	)
	472	user_who_share_room.update(user_ids)
	473
	474	return user_who_share_room
	475
	476	async def get_joined_users_from_context(
	477	self, event: EventBase, context: EventContext
	478	):
	479	state_group = context.state_group
	480	if not state_group:
	481	# If state_group is None it means it has yet to be assigned a
	482	# state group, i.e. we need to make sure that calls with a state_group
	483	# of None don't hit previous cached calls with a None state_group.
	484	# To do this we set the state_group to a new object as object() != object()
	485	state_group = object()
	486
	487	current_state_ids = await context.get_current_state_ids()
	488	return await self._get_joined_users_from_context(
	489	event.room_id, state_group, current_state_ids, event=event, context=context
	490	)
	491
	492	async def get_joined_users_from_state(self, room_id, state_entry):
	493	state_group = state_entry.state_group
	494	if not state_group:
	495	# If state_group is None it means it has yet to be assigned a
	496	# state group, i.e. we need to make sure that calls with a state_group
	497	# of None don't hit previous cached calls with a None state_group.
	498	# To do this we set the state_group to a new object as object() != object()
	499	state_group = object()
	500
	501	with Measure(self._clock, "get_joined_users_from_state"):
	502	return await self._get_joined_users_from_context(
	503	room_id, state_group, state_entry.state, context=state_entry
	504	)
	505
	506	@cached(num_args=2, cache_context=True, iterable=True, max_entries=100000)
	507	async def _get_joined_users_from_context(
	508	self,
	509	room_id,
	510	state_group,
	511	current_state_ids,
	512	cache_context,
	513	event=None,
	514	context=None,
	515	):
	516	# We don't use `state_group`, it's there so that we can cache based
	517	# on it. However, it's important that it's never None, since two current_states
	518	# with a state_group of None are likely to be different.
	519	assert state_group is not None
	520
	521	users_in_room = {}
	522	member_event_ids = [
	523	e_id
	524	for key, e_id in current_state_ids.items()
	525	if key[0] == EventTypes.Member
	526	]
	527
	528	if context is not None:
	529	# If we have a context with a delta from a previous state group,
	530	# check if we also have the result from the previous group in cache.
	531	# If we do then we can reuse that result and simply update it with
	532	# any membership changes in `delta_ids`
	533	if context.prev_group and context.delta_ids:
	534	prev_res = self._get_joined_users_from_context.cache.get(
	535	(room_id, context.prev_group), None
	536	)
	537	if prev_res and isinstance(prev_res, dict):
	538	users_in_room = dict(prev_res)
	539	member_event_ids = [
	540	e_id
	541	for key, e_id in context.delta_ids.items()
	542	if key[0] == EventTypes.Member
	543	]
	544	for etype, state_key in context.delta_ids:
	545	if etype == EventTypes.Member:
	546	users_in_room.pop(state_key, None)
	547
	548	# We check if we have any of the member event ids in the event cache
	549	# before we ask the DB
	550
	551	# We don't update the event cache hit ratio as it completely throws off
	552	# the hit ratio counts. After all, we don't populate the cache if we
	553	# miss it here
	554	event_map = self._get_events_from_cache(
	555	member_event_ids, allow_rejected=False, update_metrics=False
	556	)
	557
	558	missing_member_event_ids = []
	559	for event_id in member_event_ids:
	560	ev_entry = event_map.get(event_id)
	561	if ev_entry:
	562	if ev_entry.event.membership == Membership.JOIN:
	563	users_in_room[ev_entry.event.state_key] = ProfileInfo(
	564	display_name=ev_entry.event.content.get("displayname", None),
	565	avatar_url=ev_entry.event.content.get("avatar_url", None),
	566	)
	567	else:
	568	missing_member_event_ids.append(event_id)
	569
	570	if missing_member_event_ids:
	571	event_to_memberships = await self._get_joined_profiles_from_event_ids(
	572	missing_member_event_ids
	573	)
	574	users_in_room.update((row for row in event_to_memberships.values() if row))
	575
	576	if event is not None and event.type == EventTypes.Member:
	577	if event.membership == Membership.JOIN:
	578	if event.event_id in member_event_ids:
	579	users_in_room[event.state_key] = ProfileInfo(
	580	display_name=event.content.get("displayname", None),
	581	avatar_url=event.content.get("avatar_url", None),
	582	)
	583
	584	return users_in_room
	585
	586	@cached(max_entries=10000)
	587	def _get_joined_profile_from_event_id(self, event_id):
	588	raise NotImplementedError()
	589
	590	@cachedList(
	591	cached_method_name="_get_joined_profile_from_event_id",
	592	list_name="event_ids",
	593	inlineCallbacks=True,
	594	)
	595	def _get_joined_profiles_from_event_ids(self, event_ids: Iterable[str]):
	596	"""For given set of member event_ids check if they point to a join
	597	event and if so return the associated user and profile info.
	598
	599	Args:
	600	event_ids: The member event IDs to lookup
	601
	602	Returns:
	603	Deferred[dict[str, Tuple[str, ProfileInfo]\|None]]: Map from event ID
	604	to `user_id` and ProfileInfo (or None if not join event).
	605	"""
	606
	607	rows = yield self.db_pool.simple_select_many_batch(
	608	table="room_memberships",
	609	column="event_id",
	610	iterable=event_ids,
	611	retcols=("user_id", "display_name", "avatar_url", "event_id"),
	612	keyvalues={"membership": Membership.JOIN},
	613	batch_size=500,
	614	desc="_get_membership_from_event_ids",
	615	)
	616
	617	return {
	618	row["event_id"]: (
	619	row["user_id"],
	620	ProfileInfo(
	621	avatar_url=row["avatar_url"], display_name=row["display_name"]
	622	),
	623	)
	624	for row in rows
	625	}
	626
	627	@cached(max_entries=10000)
	628	async def is_host_joined(self, room_id: str, host: str) -> bool:
	629	if "%" in host or "_" in host:
	630	raise Exception("Invalid host name")
	631
	632	sql = """
	633	SELECT state_key FROM current_state_events AS c
	634	INNER JOIN room_memberships AS m USING (event_id)
	635	WHERE m.membership = 'join'
	636	AND type = 'm.room.member'
	637	AND c.room_id = ?
	638	AND state_key LIKE ?
	639	LIMIT 1
	640	"""
	641
	642	# We do need to be careful to ensure that host doesn't have any wild cards
	643	# in it, but we checked above for known ones and we'll check below that
	644	# the returned user actually has the correct domain.
	645	like_clause = "%:" + host
	646
	647	rows = await self.db_pool.execute(
	648	"is_host_joined", None, sql, room_id, like_clause
	649	)
	650
	651	if not rows:
	652	return False
	653
	654	user_id = rows[0][0]
	655	if get_domain_from_id(user_id) != host:
	656	# This can only happen if the host name has something funky in it
	657	raise Exception("Invalid host name")
	658
	659	return True
	660
	661	async def get_joined_hosts(self, room_id: str, state_entry):
	662	state_group = state_entry.state_group
	663	if not state_group:
	664	# If state_group is None it means it has yet to be assigned a
	665	# state group, i.e. we need to make sure that calls with a state_group
	666	# of None don't hit previous cached calls with a None state_group.
	667	# To do this we set the state_group to a new object as object() != object()
	668	state_group = object()
	669
	670	with Measure(self._clock, "get_joined_hosts"):
	671	return await self._get_joined_hosts(
	672	room_id, state_group, state_entry.state, state_entry=state_entry
	673	)
	674
	675	@cached(num_args=2, max_entries=10000, iterable=True)
	676	async def _get_joined_hosts(
	677	self, room_id, state_group, current_state_ids, state_entry
	678	):
	679	# We don't use `state_group`, its there so that we can cache based
	680	# on it. However, its important that its never None, since two current_state's
	681	# with a state_group of None are likely to be different.
	682	assert state_group is not None
	683
	684	cache = await self._get_joined_hosts_cache(room_id)
	685	return await cache.get_destinations(state_entry)
	686
	687	@cached(max_entries=10000)
	688	def _get_joined_hosts_cache(self, room_id: str) -> "_JoinedHostsCache":
	689	return _JoinedHostsCache(self, room_id)
	690
	691	@cached(num_args=2)
	692	async def did_forget(self, user_id: str, room_id: str) -> bool:
	693	"""Returns whether user_id has elected to discard history for room_id.
	694
	695	Returns False if they have since re-joined."""
	696
	697	def f(txn):
	698	sql = (
	699	"SELECT"
	700	" COUNT(*)"
	701	" FROM"
	702	" room_memberships"
	703	" WHERE"
	704	" user_id = ?"
	705	" AND"
	706	" room_id = ?"
	707	" AND"
	708	" forgotten = 0"
	709	)
	710	txn.execute(sql, (user_id, room_id))
	711	rows = txn.fetchall()
	712	return rows[0][0]
	713
	714	count = await self.db_pool.runInteraction("did_forget_membership", f)
	715	return count == 0
	716
	717	@cached()
	718	def get_forgotten_rooms_for_user(self, user_id: str):
	719	"""Gets all rooms the user has forgotten.
	720
	721	Args:
	722	user_id
	723
	724	Returns:
	725	Deferred[set[str]]
	726	"""
	727
	728	def _get_forgotten_rooms_for_user_txn(txn):
	729	# This is a slightly convoluted query that first looks up all rooms
	730	# that the user has forgotten in the past, then rechecks that list
	731	# to see if any have subsequently been updated. This is done so that
	732	# we can use a partial index on `forgotten = 1` on the assumption
	733	# that few users will actually forget many rooms.
	734	#
	735	# Note that a room is considered "forgotten" if all membership
	736	# events for that user and room have the forgotten field set (as
	737	# when a user forgets a room we update all rows for that user and
	738	# room, not just the current one).
	739	sql = """
	740	SELECT room_id, (
	741	SELECT count(*) FROM room_memberships
	742	WHERE room_id = m.room_id AND user_id = m.user_id AND forgotten = 0
	743	) AS count
	744	FROM room_memberships AS m
	745	WHERE user_id = ? AND forgotten = 1
	746	GROUP BY room_id, user_id;
	747	"""
	748	txn.execute(sql, (user_id,))
	749	return {row[0] for row in txn if row[1] == 0}
	750
	751	return self.db_pool.runInteraction(
	752	"get_forgotten_rooms_for_user", _get_forgotten_rooms_for_user_txn
	753	)
	754
	755	async def get_rooms_user_has_been_in(self, user_id: str) -> Set[str]:
	756	"""Get all rooms that the user has ever been in.
	757
	758	Args:
	759	user_id: The user ID to get the rooms of.
	760
	761	Returns:
	762	Set of room IDs.
	763	"""
	764
	765	room_ids = await self.db_pool.simple_select_onecol(
	766	table="room_memberships",
	767	keyvalues={"membership": Membership.JOIN, "user_id": user_id},
	768	retcol="room_id",
	769	desc="get_rooms_user_has_been_in",
	770	)
	771
	772	return set(room_ids)
	773
	774	def get_membership_from_event_ids(
	775	self, member_event_ids: Iterable[str]
	776	) -> List[dict]:
	777	"""Get user_id and membership of a set of event IDs.
	778	"""
	779
	780	return self.db_pool.simple_select_many_batch(
	781	table="room_memberships",
	782	column="event_id",
	783	iterable=member_event_ids,
	784	retcols=("user_id", "membership", "event_id"),
	785	keyvalues={},
	786	batch_size=500,
	787	desc="get_membership_from_event_ids",
	788	)
	789
	790	async def is_local_host_in_room_ignoring_users(
	791	self, room_id: str, ignore_users: Collection[str]
	792	) -> bool:
	793	"""Check if there are any local users, excluding those in the given
	794	list, in the room.
	795	"""
	796
	797	clause, args = make_in_list_sql_clause(
	798	self.database_engine, "user_id", ignore_users
	799	)
	800
	801	sql = """
	802	SELECT 1 FROM local_current_membership
	803	WHERE
	804	room_id = ? AND membership = ?
	805	AND NOT (%s)
	806	LIMIT 1
	807	""" % (
	808	clause,
	809	)
	810
	811	def _is_local_host_in_room_ignoring_users_txn(txn):
	812	txn.execute(sql, (room_id, Membership.JOIN, *args))
	813
	814	return bool(txn.fetchone())
	815
	816	return await self.db_pool.runInteraction(
	817	"is_local_host_in_room_ignoring_users",
	818	_is_local_host_in_room_ignoring_users_txn,
	819	)
	820
	821
	822	class RoomMemberBackgroundUpdateStore(SQLBaseStore):
	823	def __init__(self, database: DatabasePool, db_conn, hs):
	824	super(RoomMemberBackgroundUpdateStore, self).__init__(database, db_conn, hs)
	825	self.db_pool.updates.register_background_update_handler(
	826	_MEMBERSHIP_PROFILE_UPDATE_NAME, self._background_add_membership_profile
	827	)
	828	self.db_pool.updates.register_background_update_handler(
	829	_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME,
	830	self._background_current_state_membership,
	831	)
	832	self.db_pool.updates.register_background_index_update(
	833	"room_membership_forgotten_idx",
	834	index_name="room_memberships_user_room_forgotten",
	835	table="room_memberships",
	836	columns=["user_id", "room_id"],
	837	where_clause="forgotten = 1",
	838	)
	839
	840	async def _background_add_membership_profile(self, progress, batch_size):
	841	target_min_stream_id = progress.get(
	842	"target_min_stream_id_inclusive", self._min_stream_order_on_start
	843	)
	844	max_stream_id = progress.get(
	845	"max_stream_id_exclusive", self._stream_order_on_start + 1
	846	)
	847
	848	INSERT_CLUMP_SIZE = 1000
	849
	850	def add_membership_profile_txn(txn):
	851	sql = """
	852	SELECT stream_ordering, event_id, events.room_id, event_json.json
	853	FROM events
	854	INNER JOIN event_json USING (event_id)
	855	INNER JOIN room_memberships USING (event_id)
	856	WHERE ? <= stream_ordering AND stream_ordering < ?
	857	AND type = 'm.room.member'
	858	ORDER BY stream_ordering DESC
	859	LIMIT ?
	860	"""
	861
	862	txn.execute(sql, (target_min_stream_id, max_stream_id, batch_size))
	863
	864	rows = self.db_pool.cursor_to_dict(txn)
	865	if not rows:
	866	return 0
	867
	868	min_stream_id = rows[-1]["stream_ordering"]
	869
	870	to_update = []
	871	for row in rows:
	872	event_id = row["event_id"]
	873	room_id = row["room_id"]
	874	try:
	875	event_json = db_to_json(row["json"])
	876	content = event_json["content"]
	877	except Exception:
	878	continue
	879
	880	display_name = content.get("displayname", None)
	881	avatar_url = content.get("avatar_url", None)
	882
	883	if display_name or avatar_url:
	884	to_update.append((display_name, avatar_url, event_id, room_id))
	885
	886	to_update_sql = """
	887	UPDATE room_memberships SET display_name = ?, avatar_url = ?
	888	WHERE event_id = ? AND room_id = ?
	889	"""
	890	for index in range(0, len(to_update), INSERT_CLUMP_SIZE):
	891	clump = to_update[index : index + INSERT_CLUMP_SIZE]
	892	txn.executemany(to_update_sql, clump)
	893
	894	progress = {
	895	"target_min_stream_id_inclusive": target_min_stream_id,
	896	"max_stream_id_exclusive": min_stream_id,
	897	}
	898
	899	self.db_pool.updates._background_update_progress_txn(
	900	txn, _MEMBERSHIP_PROFILE_UPDATE_NAME, progress
	901	)
	902
	903	return len(rows)
	904
	905	result = await self.db_pool.runInteraction(
	906	_MEMBERSHIP_PROFILE_UPDATE_NAME, add_membership_profile_txn
	907	)
	908
	909	if not result:
	910	await self.db_pool.updates._end_background_update(
	911	_MEMBERSHIP_PROFILE_UPDATE_NAME
	912	)
	913
	914	return result
	915
	916	async def _background_current_state_membership(self, progress, batch_size):
	917	"""Update the new membership column on current_state_events.
	918
	919	This works by iterating over all rooms in alphebetical order.
	920	"""
	921
	922	def _background_current_state_membership_txn(txn, last_processed_room):
	923	processed = 0
	924	while processed < batch_size:
	925	txn.execute(
	926	"""
	927	SELECT MIN(room_id) FROM current_state_events WHERE room_id > ?
	928	""",
	929	(last_processed_room,),
	930	)
	931	row = txn.fetchone()
	932	if not row or not row[0]:
	933	return processed, True
	934
	935	(next_room,) = row
	936
	937	sql = """
	938	UPDATE current_state_events
	939	SET membership = (
	940	SELECT membership FROM room_memberships
	941	WHERE event_id = current_state_events.event_id
	942	)
	943	WHERE room_id = ?
	944	"""
	945	txn.execute(sql, (next_room,))
	946	processed += txn.rowcount
	947
	948	last_processed_room = next_room
	949
	950	self.db_pool.updates._background_update_progress_txn(
	951	txn,
	952	_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME,
	953	{"last_processed_room": last_processed_room},
	954	)
	955
	956	return processed, False
	957
	958	# If we haven't got a last processed room then just use the empty
	959	# string, which will compare before all room IDs correctly.
	960	last_processed_room = progress.get("last_processed_room", "")
	961
	962	row_count, finished = await self.db_pool.runInteraction(
	963	"_background_current_state_membership_update",
	964	_background_current_state_membership_txn,
	965	last_processed_room,
	966	)
	967
	968	if finished:
	969	await self.db_pool.updates._end_background_update(
	970	_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME
	971	)
	972
	973	return row_count
	974
	975
	976	class RoomMemberStore(RoomMemberWorkerStore, RoomMemberBackgroundUpdateStore):
	977	def __init__(self, database: DatabasePool, db_conn, hs):
	978	super(RoomMemberStore, self).__init__(database, db_conn, hs)
	979
	980	def forget(self, user_id: str, room_id: str):
	981	"""Indicate that user_id wishes to discard history for room_id."""
	982
	983	def f(txn):
	984	sql = (
	985	"UPDATE"
	986	" room_memberships"
	987	" SET"
	988	" forgotten = 1"
	989	" WHERE"
	990	" user_id = ?"
	991	" AND"
	992	" room_id = ?"
	993	)
	994	txn.execute(sql, (user_id, room_id))
	995
	996	self._invalidate_cache_and_stream(txn, self.did_forget, (user_id, room_id))
	997	self._invalidate_cache_and_stream(
	998	txn, self.get_forgotten_rooms_for_user, (user_id,)
	999	)
	1000
	1001	return self.db_pool.runInteraction("forget_membership", f)
	1002
	1003
	1004	class _JoinedHostsCache(object):
	1005	"""Cache for joined hosts in a room that is optimised to handle updates
	1006	via state deltas.
	1007	"""
	1008
	1009	def __init__(self, store, room_id):
	1010	self.store = store
	1011	self.room_id = room_id
	1012
	1013	self.hosts_to_joined_users = {}
	1014
	1015	self.state_group = object()
	1016
	1017	self.linearizer = Linearizer("_JoinedHostsCache")
	1018
	1019	self._len = 0
	1020
	1021	async def get_destinations(self, state_entry: "_StateCacheEntry") -> Set[str]:
	1022	"""Get set of destinations for a state entry
	1023
	1024	Args:
	1025	state_entry
	1026
	1027	Returns:
	1028	The destinations as a set.
	1029	"""
	1030	if state_entry.state_group == self.state_group:
	1031	return frozenset(self.hosts_to_joined_users)
	1032
	1033	with (await self.linearizer.queue(())):
	1034	if state_entry.state_group == self.state_group:
	1035	pass
	1036	elif state_entry.prev_group == self.state_group:
	1037	for (typ, state_key), event_id in state_entry.delta_ids.items():
	1038	if typ != EventTypes.Member:
	1039	continue
	1040
	1041	host = intern_string(get_domain_from_id(state_key))
	1042	user_id = state_key
	1043	known_joins = self.hosts_to_joined_users.setdefault(host, set())
	1044
	1045	event = await self.store.get_event(event_id)
	1046	if event.membership == Membership.JOIN:
	1047	known_joins.add(user_id)
	1048	else:
	1049	known_joins.discard(user_id)
	1050
	1051	if not known_joins:
	1052	self.hosts_to_joined_users.pop(host, None)
	1053	else:
	1054	joined_users = await self.store.get_joined_users_from_state(
	1055	self.room_id, state_entry
	1056	)
	1057
	1058	self.hosts_to_joined_users = {}
	1059	for user_id in joined_users:
	1060	host = intern_string(get_domain_from_id(user_id))
	1061	self.hosts_to_joined_users.setdefault(host, set()).add(user_id)
	1062
	1063	if state_entry.state_group:
	1064	self.state_group = state_entry.state_group
	1065	else:
	1066	self.state_group = object()
	1067	self._len = sum(len(v) for v in self.hosts_to_joined_users.values())
	1068	return frozenset(self.hosts_to_joined_users)
	1069
	1070	def __len__(self):
	1071	return self._len

+63

-0

synapse/storage/databases/main/schema/delta/12/v12.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS rejections(
	16	event_id TEXT NOT NULL,
	17	reason TEXT NOT NULL,
	18	last_check TEXT NOT NULL,
	19	UNIQUE (event_id)
	20	);
	21
	22	-- Push notification endpoints that users have configured
	23	CREATE TABLE IF NOT EXISTS pushers (
	24	id INTEGER PRIMARY KEY AUTOINCREMENT,
	25	user_name TEXT NOT NULL,
	26	profile_tag VARCHAR(32) NOT NULL,
	27	kind VARCHAR(8) NOT NULL,
	28	app_id VARCHAR(64) NOT NULL,
	29	app_display_name VARCHAR(64) NOT NULL,
	30	device_display_name VARCHAR(128) NOT NULL,
	31	pushkey VARBINARY(512) NOT NULL,
	32	ts BIGINT UNSIGNED NOT NULL,
	33	lang VARCHAR(8),
	34	data LONGBLOB,
	35	last_token TEXT,
	36	last_success BIGINT UNSIGNED,
	37	failing_since BIGINT UNSIGNED,
	38	UNIQUE (app_id, pushkey)
	39	);
	40
	41	CREATE TABLE IF NOT EXISTS push_rules (
	42	id INTEGER PRIMARY KEY AUTOINCREMENT,
	43	user_name TEXT NOT NULL,
	44	rule_id TEXT NOT NULL,
	45	priority_class TINYINT NOT NULL,
	46	priority INTEGER NOT NULL DEFAULT 0,
	47	conditions TEXT NOT NULL,
	48	actions TEXT NOT NULL,
	49	UNIQUE(user_name, rule_id)
	50	);
	51
	52	CREATE INDEX IF NOT EXISTS push_rules_user_name on push_rules (user_name);
	53
	54	CREATE TABLE IF NOT EXISTS user_filters(
	55	user_id TEXT,
	56	filter_id BIGINT UNSIGNED,
	57	filter_json LONGBLOB
	58	);
	59
	60	CREATE INDEX IF NOT EXISTS user_filters_by_user_id_filter_id ON user_filters(
	61	user_id, filter_id
	62	);

+19

-0

synapse/storage/databases/main/schema/delta/13/v13.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* We used to create a tables called application_services and
	16	* application_services_regex, but these are no longer used and are removed in
	17	* delta 54.
	18	*/

+23

-0

synapse/storage/databases/main/schema/delta/14/v14.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14	CREATE TABLE IF NOT EXISTS push_rules_enable (
	15	id INTEGER PRIMARY KEY AUTOINCREMENT,
	16	user_name TEXT NOT NULL,
	17	rule_id TEXT NOT NULL,
	18	enabled TINYINT,
	19	UNIQUE(user_name, rule_id)
	20	);
	21
	22	CREATE INDEX IF NOT EXISTS push_rules_enable_user_name on push_rules_enable (user_name);

+31

-0

synapse/storage/databases/main/schema/delta/15/appservice_txns.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS application_services_state(
	16	as_id TEXT PRIMARY KEY,
	17	state VARCHAR(5),
	18	last_txn INTEGER
	19	);
	20
	21	CREATE TABLE IF NOT EXISTS application_services_txns(
	22	as_id TEXT NOT NULL,
	23	txn_id INTEGER NOT NULL,
	24	event_ids TEXT NOT NULL,
	25	UNIQUE(as_id, txn_id)
	26	);
	27
	28	CREATE INDEX IF NOT EXISTS application_services_txns_id ON application_services_txns (
	29	as_id
	30	);

+2

-0

synapse/storage/databases/main/schema/delta/15/presence_indices.sql less more

	0
	1	CREATE INDEX IF NOT EXISTS presence_list_user_id ON presence_list (user_id);

+24

-0

synapse/storage/databases/main/schema/delta/15/v15.sql less more

	0	-- Drop, copy & recreate pushers table to change unique key
	1	-- Also add access_token column at the same time
	2	CREATE TABLE IF NOT EXISTS pushers2 (
	3	id BIGINT PRIMARY KEY,
	4	user_name TEXT NOT NULL,
	5	access_token BIGINT DEFAULT NULL,
	6	profile_tag VARCHAR(32) NOT NULL,
	7	kind VARCHAR(8) NOT NULL,
	8	app_id VARCHAR(64) NOT NULL,
	9	app_display_name VARCHAR(64) NOT NULL,
	10	device_display_name VARCHAR(128) NOT NULL,
	11	pushkey bytea NOT NULL,
	12	ts BIGINT NOT NULL,
	13	lang VARCHAR(8),
	14	data bytea,
	15	last_token TEXT,
	16	last_success BIGINT,
	17	failing_since BIGINT,
	18	UNIQUE (app_id, pushkey)
	19	);
	20	INSERT INTO pushers2 (id, user_name, profile_tag, kind, app_id, app_display_name, device_display_name, pushkey, ts, lang, data, last_token, last_success, failing_since)
	21	SELECT id, user_name, profile_tag, kind, app_id, app_display_name, device_display_name, pushkey, ts, lang, data, last_token, last_success, failing_since FROM pushers;
	22	DROP TABLE pushers;
	23	ALTER TABLE pushers2 RENAME TO pushers;

+4

-0

synapse/storage/databases/main/schema/delta/16/events_order_index.sql less more

	0	CREATE INDEX events_order ON events (topological_ordering, stream_ordering);
	1	CREATE INDEX events_order_room ON events (
	2	room_id, topological_ordering, stream_ordering
	3	);

+2

-0

synapse/storage/databases/main/schema/delta/16/remote_media_cache_index.sql less more

	0	CREATE INDEX IF NOT EXISTS remote_media_cache_thumbnails_media_id
	1	ON remote_media_cache_thumbnails (media_id);⏎

+9

-0

synapse/storage/databases/main/schema/delta/16/remove_duplicates.sql less more

	0
	1
	2	DELETE FROM event_to_state_groups WHERE state_group not in (
	3	SELECT MAX(state_group) FROM event_to_state_groups GROUP BY event_id
	4	);
	5
	6	DELETE FROM event_to_state_groups WHERE rowid not in (
	7	SELECT MIN(rowid) FROM event_to_state_groups GROUP BY event_id
	8	);

+3

-0

synapse/storage/databases/main/schema/delta/16/room_alias_index.sql less more

	0
	1	CREATE INDEX IF NOT EXISTS room_aliases_id ON room_aliases(room_id);
	2	CREATE INDEX IF NOT EXISTS room_alias_servers_alias ON room_alias_servers(room_alias);

+72

-0

synapse/storage/databases/main/schema/delta/16/unique_constraints.sql less more

	0
	1	-- We can use SQLite features here, since other db support was only added in v16
	2
	3	--
	4	DELETE FROM current_state_events WHERE rowid not in (
	5	SELECT MIN(rowid) FROM current_state_events GROUP BY event_id
	6	);
	7
	8	DROP INDEX IF EXISTS current_state_events_event_id;
	9	CREATE UNIQUE INDEX current_state_events_event_id ON current_state_events(event_id);
	10
	11	--
	12	DELETE FROM room_memberships WHERE rowid not in (
	13	SELECT MIN(rowid) FROM room_memberships GROUP BY event_id
	14	);
	15
	16	DROP INDEX IF EXISTS room_memberships_event_id;
	17	CREATE UNIQUE INDEX room_memberships_event_id ON room_memberships(event_id);
	18
	19	--
	20	DELETE FROM topics WHERE rowid not in (
	21	SELECT MIN(rowid) FROM topics GROUP BY event_id
	22	);
	23
	24	DROP INDEX IF EXISTS topics_event_id;
	25	CREATE UNIQUE INDEX topics_event_id ON topics(event_id);
	26
	27	--
	28	DELETE FROM room_names WHERE rowid not in (
	29	SELECT MIN(rowid) FROM room_names GROUP BY event_id
	30	);
	31
	32	DROP INDEX IF EXISTS room_names_id;
	33	CREATE UNIQUE INDEX room_names_id ON room_names(event_id);
	34
	35	--
	36	DELETE FROM presence WHERE rowid not in (
	37	SELECT MIN(rowid) FROM presence GROUP BY user_id
	38	);
	39
	40	DROP INDEX IF EXISTS presence_id;
	41	CREATE UNIQUE INDEX presence_id ON presence(user_id);
	42
	43	--
	44	DELETE FROM presence_allow_inbound WHERE rowid not in (
	45	SELECT MIN(rowid) FROM presence_allow_inbound
	46	GROUP BY observed_user_id, observer_user_id
	47	);
	48
	49	DROP INDEX IF EXISTS presence_allow_inbound_observers;
	50	CREATE UNIQUE INDEX presence_allow_inbound_observers ON presence_allow_inbound(
	51	observed_user_id, observer_user_id
	52	);
	53
	54	--
	55	DELETE FROM presence_list WHERE rowid not in (
	56	SELECT MIN(rowid) FROM presence_list
	57	GROUP BY user_id, observed_user_id
	58	);
	59
	60	DROP INDEX IF EXISTS presence_list_observers;
	61	CREATE UNIQUE INDEX presence_list_observers ON presence_list(
	62	user_id, observed_user_id
	63	);
	64
	65	--
	66	DELETE FROM room_aliases WHERE rowid not in (
	67	SELECT MIN(rowid) FROM room_aliases GROUP BY room_alias
	68	);
	69
	70	DROP INDEX IF EXISTS room_aliases_id;
	71	CREATE INDEX room_aliases_id ON room_aliases(room_id);

+56

-0

synapse/storage/databases/main/schema/delta/16/users.sql less more

	0	-- Convert `access_tokens`.user from rowids to user strings.
	1	-- MUST BE DONE BEFORE REMOVING ID COLUMN FROM USERS TABLE BELOW
	2	CREATE TABLE IF NOT EXISTS new_access_tokens(
	3	id BIGINT UNSIGNED PRIMARY KEY,
	4	user_id TEXT NOT NULL,
	5	device_id TEXT,
	6	token TEXT NOT NULL,
	7	last_used BIGINT UNSIGNED,
	8	UNIQUE(token)
	9	);
	10
	11	INSERT INTO new_access_tokens
	12	SELECT a.id, u.name, a.device_id, a.token, a.last_used
	13	FROM access_tokens as a
	14	INNER JOIN users as u ON u.id = a.user_id;
	15
	16	DROP TABLE access_tokens;
	17
	18	ALTER TABLE new_access_tokens RENAME TO access_tokens;
	19
	20	-- Remove ID column from `users` table
	21	CREATE TABLE IF NOT EXISTS new_users(
	22	name TEXT,
	23	password_hash TEXT,
	24	creation_ts BIGINT UNSIGNED,
	25	admin BOOL DEFAULT 0 NOT NULL,
	26	UNIQUE(name)
	27	);
	28
	29	INSERT INTO new_users SELECT name, password_hash, creation_ts, admin FROM users;
	30
	31	DROP TABLE users;
	32
	33	ALTER TABLE new_users RENAME TO users;
	34
	35
	36	-- Remove UNIQUE constraint from `user_ips` table
	37	CREATE TABLE IF NOT EXISTS new_user_ips (
	38	user_id TEXT NOT NULL,
	39	access_token TEXT NOT NULL,
	40	device_id TEXT,
	41	ip TEXT NOT NULL,
	42	user_agent TEXT NOT NULL,
	43	last_seen BIGINT UNSIGNED NOT NULL
	44	);
	45
	46	INSERT INTO new_user_ips
	47	SELECT user, access_token, device_id, ip, user_agent, last_seen FROM user_ips;
	48
	49	DROP TABLE user_ips;
	50
	51	ALTER TABLE new_user_ips RENAME TO user_ips;
	52
	53	CREATE INDEX IF NOT EXISTS user_ips_user ON user_ips(user_id);
	54	CREATE INDEX IF NOT EXISTS user_ips_user_ip ON user_ips(user_id, access_token, ip);
	55

+18

-0

synapse/storage/databases/main/schema/delta/17/drop_indexes.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	DROP INDEX IF EXISTS sent_transaction_dest;
	16	DROP INDEX IF EXISTS sent_transaction_sent;
	17	DROP INDEX IF EXISTS user_ips_user;

+24

-0

synapse/storage/databases/main/schema/delta/17/server_keys.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS server_keys_json (
	16	server_name TEXT, -- Server name.
	17	key_id TEXT, -- Requested key id.
	18	from_server TEXT, -- Which server the keys were fetched from.
	19	ts_added_ms INTEGER, -- When the keys were fetched
	20	ts_valid_until_ms INTEGER, -- When this version of the keys exipires.
	21	key_json bytea, -- JSON certificate for the remote server.
	22	CONSTRAINT uniqueness UNIQUE (server_name, key_id, from_server)
	23	);

+9

-0

synapse/storage/databases/main/schema/delta/17/user_threepids.sql less more

	0	CREATE TABLE user_threepids (
	1	user_id TEXT NOT NULL,
	2	medium TEXT NOT NULL,
	3	address TEXT NOT NULL,
	4	validated_at BIGINT NOT NULL,
	5	added_at BIGINT NOT NULL,
	6	CONSTRAINT user_medium_address UNIQUE (user_id, medium, address)
	7	);
	8	CREATE INDEX user_threepids_user_id ON user_threepids(user_id);

+32

-0

synapse/storage/databases/main/schema/delta/18/server_keys_bigger_ints.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE IF NOT EXISTS new_server_keys_json (
	17	server_name TEXT NOT NULL, -- Server name.
	18	key_id TEXT NOT NULL, -- Requested key id.
	19	from_server TEXT NOT NULL, -- Which server the keys were fetched from.
	20	ts_added_ms BIGINT NOT NULL, -- When the keys were fetched
	21	ts_valid_until_ms BIGINT NOT NULL, -- When this version of the keys exipires.
	22	key_json bytea NOT NULL, -- JSON certificate for the remote server.
	23	CONSTRAINT server_keys_json_uniqueness UNIQUE (server_name, key_id, from_server)
	24	);
	25
	26	INSERT INTO new_server_keys_json
	27	SELECT server_name, key_id, from_server,ts_added_ms, ts_valid_until_ms, key_json FROM server_keys_json ;
	28
	29	DROP TABLE server_keys_json;
	30
	31	ALTER TABLE new_server_keys_json RENAME TO server_keys_json;

+19

-0

synapse/storage/databases/main/schema/delta/19/event_index.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE INDEX events_order_topo_stream_room ON events(
	17	topological_ordering, stream_ordering, room_id
	18	);

+1

-0

synapse/storage/databases/main/schema/delta/20/dummy.sql less more

0

SELECT 1;

+88

-0

synapse/storage/databases/main/schema/delta/20/pushers.py less more

	0	# Copyright 2015, 2016 OpenMarket Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14
	15	"""
	16	Main purpose of this upgrade is to change the unique key on the
	17	pushers table again (it was missed when the v16 full schema was
	18	made) but this also changes the pushkey and data columns to text.
	19	When selecting a bytea column into a text column, postgres inserts
	20	the hex encoded data, and there's no portable way of getting the
	21	UTF-8 bytes, so we have to do it in Python.
	22	"""
	23
	24	import logging
	25
	26	logger = logging.getLogger(__name__)
	27
	28
	29	def run_create(cur, database_engine, args, *kwargs):
	30	logger.info("Porting pushers table...")
	31	cur.execute(
	32	"""
	33	CREATE TABLE IF NOT EXISTS pushers2 (
	34	id BIGINT PRIMARY KEY,
	35	user_name TEXT NOT NULL,
	36	access_token BIGINT DEFAULT NULL,
	37	profile_tag VARCHAR(32) NOT NULL,
	38	kind VARCHAR(8) NOT NULL,
	39	app_id VARCHAR(64) NOT NULL,
	40	app_display_name VARCHAR(64) NOT NULL,
	41	device_display_name VARCHAR(128) NOT NULL,
	42	pushkey TEXT NOT NULL,
	43	ts BIGINT NOT NULL,
	44	lang VARCHAR(8),
	45	data TEXT,
	46	last_token TEXT,
	47	last_success BIGINT,
	48	failing_since BIGINT,
	49	UNIQUE (app_id, pushkey, user_name)
	50	)
	51	"""
	52	)
	53	cur.execute(
	54	"""SELECT
	55	id, user_name, access_token, profile_tag, kind,
	56	app_id, app_display_name, device_display_name,
	57	pushkey, ts, lang, data, last_token, last_success,
	58	failing_since
	59	FROM pushers
	60	"""
	61	)
	62	count = 0
	63	for row in cur.fetchall():
	64	row = list(row)
	65	row[8] = bytes(row[8]).decode("utf-8")
	66	row[11] = bytes(row[11]).decode("utf-8")
	67	cur.execute(
	68	database_engine.convert_param_style(
	69	"""
	70	INSERT into pushers2 (
	71	id, user_name, access_token, profile_tag, kind,
	72	app_id, app_display_name, device_display_name,
	73	pushkey, ts, lang, data, last_token, last_success,
	74	failing_since
	75	) values (%s)"""
	76	% (",".join(["?" for _ in range(len(row))]))
	77	),
	78	row,
	79	)
	80	count += 1
	81	cur.execute("DROP TABLE pushers")
	82	cur.execute("ALTER TABLE pushers2 RENAME TO pushers")
	83	logger.info("Moved %d pushers to new table", count)
	84
	85
	86	def run_upgrade(args, *kwargs):
	87	pass

+34

-0

synapse/storage/databases/main/schema/delta/21/end_to_end_keys.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE IF NOT EXISTS e2e_device_keys_json (
	17	user_id TEXT NOT NULL, -- The user these keys are for.
	18	device_id TEXT NOT NULL, -- Which of the user's devices these keys are for.
	19	ts_added_ms BIGINT NOT NULL, -- When the keys were uploaded.
	20	key_json TEXT NOT NULL, -- The keys for the device as a JSON blob.
	21	CONSTRAINT e2e_device_keys_json_uniqueness UNIQUE (user_id, device_id)
	22	);
	23
	24
	25	CREATE TABLE IF NOT EXISTS e2e_one_time_keys_json (
	26	user_id TEXT NOT NULL, -- The user this one-time key is for.
	27	device_id TEXT NOT NULL, -- The device this one-time key is for.
	28	algorithm TEXT NOT NULL, -- Which algorithm this one-time key is for.
	29	key_id TEXT NOT NULL, -- An id for suppressing duplicate uploads.
	30	ts_added_ms BIGINT NOT NULL, -- When this key was uploaded.
	31	key_json TEXT NOT NULL, -- The key as a JSON blob.
	32	CONSTRAINT e2e_one_time_keys_json_uniqueness UNIQUE (user_id, device_id, algorithm, key_id)
	33	);

+38

-0

synapse/storage/databases/main/schema/delta/21/receipts.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE IF NOT EXISTS receipts_graph(
	17	room_id TEXT NOT NULL,
	18	receipt_type TEXT NOT NULL,
	19	user_id TEXT NOT NULL,
	20	event_ids TEXT NOT NULL,
	21	data TEXT NOT NULL,
	22	CONSTRAINT receipts_graph_uniqueness UNIQUE (room_id, receipt_type, user_id)
	23	);
	24
	25	CREATE TABLE IF NOT EXISTS receipts_linearized (
	26	stream_id BIGINT NOT NULL,
	27	room_id TEXT NOT NULL,
	28	receipt_type TEXT NOT NULL,
	29	user_id TEXT NOT NULL,
	30	event_id TEXT NOT NULL,
	31	data TEXT NOT NULL,
	32	CONSTRAINT receipts_linearized_uniqueness UNIQUE (room_id, receipt_type, user_id)
	33	);
	34
	35	CREATE INDEX receipts_linearized_id ON receipts_linearized(
	36	stream_id
	37	);

+22

-0

synapse/storage/databases/main/schema/delta/22/receipts_index.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/** Using CREATE INDEX directly is deprecated in favour of using background
	16	* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
	17	* and synapse/storage/registration.py for an example using
	18	* "access_tokens_device_index" **/
	19	CREATE INDEX receipts_linearized_room_stream ON receipts_linearized(
	20	room_id, stream_id
	21	);

+19

-0

synapse/storage/databases/main/schema/delta/22/user_threepids_unique.sql less more

	0	CREATE TABLE IF NOT EXISTS user_threepids2 (
	1	user_id TEXT NOT NULL,
	2	medium TEXT NOT NULL,
	3	address TEXT NOT NULL,
	4	validated_at BIGINT NOT NULL,
	5	added_at BIGINT NOT NULL,
	6	CONSTRAINT medium_address UNIQUE (medium, address)
	7	);
	8
	9	INSERT INTO user_threepids2
	10	SELECT * FROM user_threepids WHERE added_at IN (
	11	SELECT max(added_at) FROM user_threepids GROUP BY medium, address
	12	)
	13	;
	14
	15	DROP TABLE user_threepids;
	16	ALTER TABLE user_threepids2 RENAME TO user_threepids;
	17
	18	CREATE INDEX user_threepids_user_id ON user_threepids(user_id);

+18

-0

synapse/storage/databases/main/schema/delta/24/stats_reporting.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* We used to create a table called stats_reporting, but this is no longer
	16	* used and is removed in delta 54.
	17	*/⏎

+80

-0

synapse/storage/databases/main/schema/delta/25/fts.py less more

	0	# Copyright 2015, 2016 OpenMarket Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13	import json
	14	import logging
	15
	16	from synapse.storage.engines import PostgresEngine, Sqlite3Engine
	17	from synapse.storage.prepare_database import get_statements
	18
	19	logger = logging.getLogger(__name__)
	20
	21
	22	POSTGRES_TABLE = """
	23	CREATE TABLE IF NOT EXISTS event_search (
	24	event_id TEXT,
	25	room_id TEXT,
	26	sender TEXT,
	27	key TEXT,
	28	vector tsvector
	29	);
	30
	31	CREATE INDEX event_search_fts_idx ON event_search USING gin(vector);
	32	CREATE INDEX event_search_ev_idx ON event_search(event_id);
	33	CREATE INDEX event_search_ev_ridx ON event_search(room_id);
	34	"""
	35
	36
	37	SQLITE_TABLE = (
	38	"CREATE VIRTUAL TABLE event_search"
	39	" USING fts4 ( event_id, room_id, sender, key, value )"
	40	)
	41
	42
	43	def run_create(cur, database_engine, args, *kwargs):
	44	if isinstance(database_engine, PostgresEngine):
	45	for statement in get_statements(POSTGRES_TABLE.splitlines()):
	46	cur.execute(statement)
	47	elif isinstance(database_engine, Sqlite3Engine):
	48	cur.execute(SQLITE_TABLE)
	49	else:
	50	raise Exception("Unrecognized database engine")
	51
	52	cur.execute("SELECT MIN(stream_ordering) FROM events")
	53	rows = cur.fetchall()
	54	min_stream_id = rows[0][0]
	55
	56	cur.execute("SELECT MAX(stream_ordering) FROM events")
	57	rows = cur.fetchall()
	58	max_stream_id = rows[0][0]
	59
	60	if min_stream_id is not None and max_stream_id is not None:
	61	progress = {
	62	"target_min_stream_id_inclusive": min_stream_id,
	63	"max_stream_id_exclusive": max_stream_id + 1,
	64	"rows_inserted": 0,
	65	}
	66	progress_json = json.dumps(progress)
	67
	68	sql = (
	69	"INSERT into background_updates (update_name, progress_json)"
	70	" VALUES (?, ?)"
	71	)
	72
	73	sql = database_engine.convert_param_style(sql)
	74
	75	cur.execute(sql, ("event_search", progress_json))
	76
	77
	78	def run_upgrade(args, *kwargs):
	79	pass

+25

-0

synapse/storage/databases/main/schema/delta/25/guest_access.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/*
	16	* This is a manual index of guest_access content of state events,
	17	* so that we can join on them in SELECT statements.
	18	*/
	19	CREATE TABLE IF NOT EXISTS guest_access(
	20	event_id TEXT NOT NULL,
	21	room_id TEXT NOT NULL,
	22	guest_access TEXT NOT NULL,
	23	UNIQUE (event_id)
	24	);

+25

-0

synapse/storage/databases/main/schema/delta/25/history_visibility.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/*
	16	* This is a manual index of history_visibility content of state events,
	17	* so that we can join on them in SELECT statements.
	18	*/
	19	CREATE TABLE IF NOT EXISTS history_visibility(
	20	event_id TEXT NOT NULL,
	21	room_id TEXT NOT NULL,
	22	history_visibility TEXT NOT NULL,
	23	UNIQUE (event_id)
	24	);

+38

-0

synapse/storage/databases/main/schema/delta/25/tags.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE IF NOT EXISTS room_tags(
	17	user_id TEXT NOT NULL,
	18	room_id TEXT NOT NULL,
	19	tag TEXT NOT NULL, -- The name of the tag.
	20	content TEXT NOT NULL, -- The JSON content of the tag.
	21	CONSTRAINT room_tag_uniqueness UNIQUE (user_id, room_id, tag)
	22	);
	23
	24	CREATE TABLE IF NOT EXISTS room_tags_revisions (
	25	user_id TEXT NOT NULL,
	26	room_id TEXT NOT NULL,
	27	stream_id BIGINT NOT NULL, -- The current version of the room tags.
	28	CONSTRAINT room_tag_revisions_uniqueness UNIQUE (user_id, room_id)
	29	);
	30
	31	CREATE TABLE IF NOT EXISTS private_user_data_max_stream_id(
	32	Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
	33	stream_id BIGINT NOT NULL,
	34	CHECK (Lock='X')
	35	);
	36
	37	INSERT INTO private_user_data_max_stream_id (stream_id) VALUES (0);

+17

-0

synapse/storage/databases/main/schema/delta/26/account_data.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	ALTER TABLE private_user_data_max_stream_id RENAME TO account_data_max_stream_id;

+36

-0

synapse/storage/databases/main/schema/delta/27/account_data.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS account_data(
	16	user_id TEXT NOT NULL,
	17	account_data_type TEXT NOT NULL, -- The type of the account_data.
	18	stream_id BIGINT NOT NULL, -- The version of the account_data.
	19	content TEXT NOT NULL, -- The JSON content of the account_data
	20	CONSTRAINT account_data_uniqueness UNIQUE (user_id, account_data_type)
	21	);
	22
	23
	24	CREATE TABLE IF NOT EXISTS room_account_data(
	25	user_id TEXT NOT NULL,
	26	room_id TEXT NOT NULL,
	27	account_data_type TEXT NOT NULL, -- The type of the account_data.
	28	stream_id BIGINT NOT NULL, -- The version of the account_data.
	29	content TEXT NOT NULL, -- The JSON content of the account_data
	30	CONSTRAINT room_account_data_uniqueness UNIQUE (user_id, room_id, account_data_type)
	31	);
	32
	33
	34	CREATE INDEX account_data_stream_id on account_data(user_id, stream_id);
	35	CREATE INDEX room_account_data_stream_id on room_account_data(user_id, stream_id);

+26

-0

synapse/storage/databases/main/schema/delta/27/forgotten_memberships.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/*
	16	* Keeps track of what rooms users have left and don't want to be able to
	17	* access again.
	18	*
	19	* If all users on this server have left a room, we can delete the room
	20	* entirely.
	21	*
	22	* This column should always contain either 0 or 1.
	23	*/
	24
	25	ALTER TABLE room_memberships ADD COLUMN forgotten INTEGER DEFAULT 0;

+59

-0

synapse/storage/databases/main/schema/delta/27/ts.py less more

	0	# Copyright 2015, 2016 OpenMarket Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13	import json
	14	import logging
	15
	16	from synapse.storage.prepare_database import get_statements
	17
	18	logger = logging.getLogger(__name__)
	19
	20
	21	ALTER_TABLE = (
	22	"ALTER TABLE events ADD COLUMN origin_server_ts BIGINT;"
	23	"CREATE INDEX events_ts ON events(origin_server_ts, stream_ordering);"
	24	)
	25
	26
	27	def run_create(cur, database_engine, args, *kwargs):
	28	for statement in get_statements(ALTER_TABLE.splitlines()):
	29	cur.execute(statement)
	30
	31	cur.execute("SELECT MIN(stream_ordering) FROM events")
	32	rows = cur.fetchall()
	33	min_stream_id = rows[0][0]
	34
	35	cur.execute("SELECT MAX(stream_ordering) FROM events")
	36	rows = cur.fetchall()
	37	max_stream_id = rows[0][0]
	38
	39	if min_stream_id is not None and max_stream_id is not None:
	40	progress = {
	41	"target_min_stream_id_inclusive": min_stream_id,
	42	"max_stream_id_exclusive": max_stream_id + 1,
	43	"rows_inserted": 0,
	44	}
	45	progress_json = json.dumps(progress)
	46
	47	sql = (
	48	"INSERT into background_updates (update_name, progress_json)"
	49	" VALUES (?, ?)"
	50	)
	51
	52	sql = database_engine.convert_param_style(sql)
	53
	54	cur.execute(sql, ("event_origin_server_ts", progress_json))
	55
	56
	57	def run_upgrade(args, *kwargs):
	58	pass

+27

-0

synapse/storage/databases/main/schema/delta/28/event_push_actions.sql less more

	0	/* Copyright 2015 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS event_push_actions(
	16	room_id TEXT NOT NULL,
	17	event_id TEXT NOT NULL,
	18	user_id TEXT NOT NULL,
	19	profile_tag VARCHAR(32),
	20	actions TEXT NOT NULL,
	21	CONSTRAINT event_id_user_id_profile_tag_uniqueness UNIQUE (room_id, event_id, user_id, profile_tag)
	22	);
	23
	24
	25	CREATE INDEX event_push_actions_room_id_event_id_user_id_profile_tag on event_push_actions(room_id, event_id, user_id, profile_tag);
	26	CREATE INDEX event_push_actions_room_id_user_id on event_push_actions(room_id, user_id);

+20

-0

synapse/storage/databases/main/schema/delta/28/events_room_stream.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/** Using CREATE INDEX directly is deprecated in favour of using background
	16	* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
	17	* and synapse/storage/registration.py for an example using
	18	* "access_tokens_device_index" **/
	19	CREATE INDEX events_room_stream on events(room_id, stream_ordering);

+20

-0

synapse/storage/databases/main/schema/delta/28/public_roms_index.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/** Using CREATE INDEX directly is deprecated in favour of using background
	16	* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
	17	* and synapse/storage/registration.py for an example using
	18	* "access_tokens_device_index" **/
	19	CREATE INDEX public_room_index on rooms(is_public);

+22

-0

synapse/storage/databases/main/schema/delta/28/receipts_user_id_index.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/** Using CREATE INDEX directly is deprecated in favour of using background
	16	* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
	17	* and synapse/storage/registration.py for an example using
	18	* "access_tokens_device_index" **/
	19	CREATE INDEX receipts_linearized_user ON receipts_linearized(
	20	user_id
	21	);

+21

-0

synapse/storage/databases/main/schema/delta/28/upgrade_times.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/*
	16	* Stores the timestamp when a user upgraded from a guest to a full user, if
	17	* that happened.
	18	*/
	19
	20	ALTER TABLE users ADD COLUMN upgrade_ts BIGINT;

+22

-0

synapse/storage/databases/main/schema/delta/28/users_is_guest.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE users ADD is_guest SMALLINT DEFAULT 0 NOT NULL;
	16	/*
	17	* NB: any guest users created between 27 and 28 will be incorrectly
	18	* marked as not guests: we don't bother to fill these in correctly
	19	* because guest access is not really complete in 27 anyway so it's
	20	* very unlikley there will be any guest users created.
	21	*/

+35

-0

synapse/storage/databases/main/schema/delta/29/push_actions.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE event_push_actions ADD COLUMN topological_ordering BIGINT;
	16	ALTER TABLE event_push_actions ADD COLUMN stream_ordering BIGINT;
	17	ALTER TABLE event_push_actions ADD COLUMN notif SMALLINT;
	18	ALTER TABLE event_push_actions ADD COLUMN highlight SMALLINT;
	19
	20	UPDATE event_push_actions SET stream_ordering = (
	21	SELECT stream_ordering FROM events WHERE event_id = event_push_actions.event_id
	22	), topological_ordering = (
	23	SELECT topological_ordering FROM events WHERE event_id = event_push_actions.event_id
	24	);
	25
	26	UPDATE event_push_actions SET notif = 1, highlight = 0;
	27
	28	/** Using CREATE INDEX directly is deprecated in favour of using background
	29	* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
	30	* and synapse/storage/registration.py for an example using
	31	* "access_tokens_device_index" **/
	32	CREATE INDEX event_push_actions_rm_tokens on event_push_actions(
	33	user_id, room_id, topological_ordering, stream_ordering
	34	);

+16

-0

synapse/storage/databases/main/schema/delta/30/alias_creator.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE room_aliases ADD COLUMN creator TEXT;

+67

-0

synapse/storage/databases/main/schema/delta/30/as_users.py less more

	0	# Copyright 2016 OpenMarket Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13	import logging
	14
	15	from synapse.config.appservice import load_appservices
	16
	17	logger = logging.getLogger(__name__)
	18
	19
	20	def run_create(cur, database_engine, args, *kwargs):
	21	# NULL indicates user was not registered by an appservice.
	22	try:
	23	cur.execute("ALTER TABLE users ADD COLUMN appservice_id TEXT")
	24	except Exception:
	25	# Maybe we already added the column? Hope so...
	26	pass
	27
	28
	29	def run_upgrade(cur, database_engine, config, args, *kwargs):
	30	cur.execute("SELECT name FROM users")
	31	rows = cur.fetchall()
	32
	33	config_files = []
	34	try:
	35	config_files = config.app_service_config_files
	36	except AttributeError:
	37	logger.warning("Could not get app_service_config_files from config")
	38	pass
	39
	40	appservices = load_appservices(config.server_name, config_files)
	41
	42	owned = {}
	43
	44	for row in rows:
	45	user_id = row[0]
	46	for appservice in appservices:
	47	if appservice.is_exclusive_user(user_id):
	48	if user_id in owned.keys():
	49	logger.error(
	50	"user_id %s was owned by more than one application"
	51	" service (IDs %s and %s); assigning arbitrarily to %s"
	52	% (user_id, owned[user_id], appservice.id, owned[user_id])
	53	)
	54	owned.setdefault(appservice.id, []).append(user_id)
	55
	56	for as_id, user_ids in owned.items():
	57	n = 100
	58	user_chunks = (user_ids[i : i + 100] for i in range(0, len(user_ids), n))
	59	for chunk in user_chunks:
	60	cur.execute(
	61	database_engine.convert_param_style(
	62	"UPDATE users SET appservice_id = ? WHERE name IN (%s)"
	63	% (",".join("?" for _ in chunk),)
	64	),
	65	[as_id] + chunk,
	66	)

+25

-0

synapse/storage/databases/main/schema/delta/30/deleted_pushers.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS deleted_pushers(
	16	stream_id BIGINT NOT NULL,
	17	app_id TEXT NOT NULL,
	18	pushkey TEXT NOT NULL,
	19	user_id TEXT NOT NULL,
	20	/* We only track the most recent delete for each app_id, pushkey and user_id. */
	21	UNIQUE (app_id, pushkey, user_id)
	22	);
	23
	24	CREATE INDEX deleted_pushers_stream_id ON deleted_pushers (stream_id);

+30

-0

synapse/storage/databases/main/schema/delta/30/presence_stream.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE presence_stream(
	17	stream_id BIGINT,
	18	user_id TEXT,
	19	state TEXT,
	20	last_active_ts BIGINT,
	21	last_federation_update_ts BIGINT,
	22	last_user_sync_ts BIGINT,
	23	status_msg TEXT,
	24	currently_active BOOLEAN
	25	);
	26
	27	CREATE INDEX presence_stream_id ON presence_stream(stream_id, user_id);
	28	CREATE INDEX presence_stream_user_id ON presence_stream(user_id);
	29	CREATE INDEX presence_stream_state ON presence_stream(state);

+23

-0

synapse/storage/databases/main/schema/delta/30/public_rooms.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	/* This release removes the restriction that published rooms must have an alias,
	17	* so we go back and ensure the only 'public' rooms are ones with an alias.
	18	* We use (1 = 0) and (1 = 1) so that it works in both postgres and sqlite
	19	*/
	20	UPDATE rooms SET is_public = (1 = 0) WHERE is_public = (1 = 1) AND room_id not in (
	21	SELECT room_id FROM room_aliases
	22	);

+38

-0

synapse/storage/databases/main/schema/delta/30/push_rule_stream.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16
	17	CREATE TABLE push_rules_stream(
	18	stream_id BIGINT NOT NULL,
	19	event_stream_ordering BIGINT NOT NULL,
	20	user_id TEXT NOT NULL,
	21	rule_id TEXT NOT NULL,
	22	op TEXT NOT NULL, -- One of "ENABLE", "DISABLE", "ACTIONS", "ADD", "DELETE"
	23	priority_class SMALLINT,
	24	priority INTEGER,
	25	conditions TEXT,
	26	actions TEXT
	27	);
	28
	29	-- The extra data for each operation is:
	30	-- * ENABLE, DISABLE, DELETE: []
	31	-- * ACTIONS: ["actions"]
	32	-- * ADD: ["priority_class", "priority", "actions", "conditions"]
	33
	34	-- Index for replication queries.
	35	CREATE INDEX push_rules_stream_id ON push_rules_stream(stream_id);
	36	-- Index for /sync queries.
	37	CREATE INDEX push_rules_stream_user_stream_id on push_rules_stream(user_id, stream_id);

+24

-0

synapse/storage/databases/main/schema/delta/30/threepid_guest_access_tokens.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Stores guest account access tokens generated for unbound 3pids.
	16	CREATE TABLE threepid_guest_access_tokens(
	17	medium TEXT, -- The medium of the 3pid. Must be "email".
	18	address TEXT, -- The 3pid address.
	19	guest_access_token TEXT, -- The access token for a guest user for this 3pid.
	20	first_inviter TEXT -- User ID of the first user to invite this 3pid to a room.
	21	);
	22
	23	CREATE UNIQUE INDEX threepid_guest_access_tokens_index ON threepid_guest_access_tokens(medium, address);

+42

-0

synapse/storage/databases/main/schema/delta/31/invites.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE local_invites(
	17	stream_id BIGINT NOT NULL,
	18	inviter TEXT NOT NULL,
	19	invitee TEXT NOT NULL,
	20	event_id TEXT NOT NULL,
	21	room_id TEXT NOT NULL,
	22	locally_rejected TEXT,
	23	replaced_by TEXT
	24	);
	25
	26	-- Insert all invites for local users into new `invites` table
	27	INSERT INTO local_invites SELECT
	28	stream_ordering as stream_id,
	29	sender as inviter,
	30	state_key as invitee,
	31	event_id,
	32	room_id,
	33	NULL as locally_rejected,
	34	NULL as replaced_by
	35	FROM events
	36	NATURAL JOIN current_state_events
	37	NATURAL JOIN room_memberships
	38	WHERE membership = 'invite' AND state_key IN (SELECT name FROM users);
	39
	40	CREATE INDEX local_invites_id ON local_invites(stream_id);
	41	CREATE INDEX local_invites_for_user_idx ON local_invites(invitee, locally_rejected, replaced_by, room_id);

+27

-0

synapse/storage/databases/main/schema/delta/31/local_media_repository_url_cache.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE local_media_repository_url_cache(
	16	url TEXT, -- the URL being cached
	17	response_code INTEGER, -- the HTTP response code of this download attempt
	18	etag TEXT, -- the etag header of this response
	19	expires INTEGER, -- the number of ms this response was valid for
	20	og TEXT, -- cache of the OG metadata of this URL as JSON
	21	media_id TEXT, -- the media_id, if any, of the URL's content in the repo
	22	download_ts BIGINT -- the timestamp of this download attempt
	23	);
	24
	25	CREATE INDEX local_media_repository_url_cache_by_url_download_ts
	26	ON local_media_repository_url_cache(url, download_ts);

+87

-0

synapse/storage/databases/main/schema/delta/31/pushers.py less more

	0	# Copyright 2016 OpenMarket Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14
	15	# Change the last_token to last_stream_ordering now that pushers no longer
	16	# listen on an event stream but instead select out of the event_push_actions
	17	# table.
	18
	19
	20	import logging
	21
	22	logger = logging.getLogger(__name__)
	23
	24
	25	def token_to_stream_ordering(token):
	26	return int(token[1:].split("_")[0])
	27
	28
	29	def run_create(cur, database_engine, args, *kwargs):
	30	logger.info("Porting pushers table, delta 31...")
	31	cur.execute(
	32	"""
	33	CREATE TABLE IF NOT EXISTS pushers2 (
	34	id BIGINT PRIMARY KEY,
	35	user_name TEXT NOT NULL,
	36	access_token BIGINT DEFAULT NULL,
	37	profile_tag VARCHAR(32) NOT NULL,
	38	kind VARCHAR(8) NOT NULL,
	39	app_id VARCHAR(64) NOT NULL,
	40	app_display_name VARCHAR(64) NOT NULL,
	41	device_display_name VARCHAR(128) NOT NULL,
	42	pushkey TEXT NOT NULL,
	43	ts BIGINT NOT NULL,
	44	lang VARCHAR(8),
	45	data TEXT,
	46	last_stream_ordering INTEGER,
	47	last_success BIGINT,
	48	failing_since BIGINT,
	49	UNIQUE (app_id, pushkey, user_name)
	50	)
	51	"""
	52	)
	53	cur.execute(
	54	"""SELECT
	55	id, user_name, access_token, profile_tag, kind,
	56	app_id, app_display_name, device_display_name,
	57	pushkey, ts, lang, data, last_token, last_success,
	58	failing_since
	59	FROM pushers
	60	"""
	61	)
	62	count = 0
	63	for row in cur.fetchall():
	64	row = list(row)
	65	row[12] = token_to_stream_ordering(row[12])
	66	cur.execute(
	67	database_engine.convert_param_style(
	68	"""
	69	INSERT into pushers2 (
	70	id, user_name, access_token, profile_tag, kind,
	71	app_id, app_display_name, device_display_name,
	72	pushkey, ts, lang, data, last_stream_ordering, last_success,
	73	failing_since
	74	) values (%s)"""
	75	% (",".join(["?" for _ in range(len(row))]))
	76	),
	77	row,
	78	)
	79	count += 1
	80	cur.execute("DROP TABLE pushers")
	81	cur.execute("ALTER TABLE pushers2 RENAME TO pushers")
	82	logger.info("Moved %d pushers to new table", count)
	83
	84
	85	def run_upgrade(cur, database_engine, args, *kwargs):
	86	pass

+22

-0

synapse/storage/databases/main/schema/delta/31/pushers_index.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/** Using CREATE INDEX directly is deprecated in favour of using background
	16	* update see synapse/storage/schema/delta/33/access_tokens_device_index.sql
	17	* and synapse/storage/registration.py for an example using
	18	* "access_tokens_device_index" **/
	19	CREATE INDEX event_push_actions_stream_ordering on event_push_actions(
	20	stream_ordering, user_id
	21	);

+64

-0

synapse/storage/databases/main/schema/delta/31/search_update.py less more

	0	# Copyright 2016 OpenMarket Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13	import json
	14	import logging
	15
	16	from synapse.storage.engines import PostgresEngine
	17	from synapse.storage.prepare_database import get_statements
	18
	19	logger = logging.getLogger(__name__)
	20
	21
	22	ALTER_TABLE = """
	23	ALTER TABLE event_search ADD COLUMN origin_server_ts BIGINT;
	24	ALTER TABLE event_search ADD COLUMN stream_ordering BIGINT;
	25	"""
	26
	27
	28	def run_create(cur, database_engine, args, *kwargs):
	29	if not isinstance(database_engine, PostgresEngine):
	30	return
	31
	32	for statement in get_statements(ALTER_TABLE.splitlines()):
	33	cur.execute(statement)
	34
	35	cur.execute("SELECT MIN(stream_ordering) FROM events")
	36	rows = cur.fetchall()
	37	min_stream_id = rows[0][0]
	38
	39	cur.execute("SELECT MAX(stream_ordering) FROM events")
	40	rows = cur.fetchall()
	41	max_stream_id = rows[0][0]
	42
	43	if min_stream_id is not None and max_stream_id is not None:
	44	progress = {
	45	"target_min_stream_id_inclusive": min_stream_id,
	46	"max_stream_id_exclusive": max_stream_id + 1,
	47	"rows_inserted": 0,
	48	"have_added_indexes": False,
	49	}
	50	progress_json = json.dumps(progress)
	51
	52	sql = (
	53	"INSERT into background_updates (update_name, progress_json)"
	54	" VALUES (?, ?)"
	55	)
	56
	57	sql = database_engine.convert_param_style(sql)
	58
	59	cur.execute(sql, ("event_search_order", progress_json))
	60
	61
	62	def run_upgrade(cur, database_engine, args, *kwargs):
	63	pass

+16

-0

synapse/storage/databases/main/schema/delta/32/events.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE events ADD COLUMN received_ts BIGINT;

+9

-0

synapse/storage/databases/main/schema/delta/32/openid.sql less more

	0
	1	CREATE TABLE open_id_tokens (
	2	token TEXT NOT NULL PRIMARY KEY,
	3	ts_valid_until_ms bigint NOT NULL,
	4	user_id TEXT NOT NULL,
	5	UNIQUE (token)
	6	);
	7
	8	CREATE index open_id_tokens_ts_valid_until_ms ON open_id_tokens(ts_valid_until_ms);

+23

-0

synapse/storage/databases/main/schema/delta/32/pusher_throttle.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE pusher_throttle(
	17	pusher BIGINT NOT NULL,
	18	room_id TEXT NOT NULL,
	19	last_sent_ts BIGINT,
	20	throttle_ms BIGINT,
	21	PRIMARY KEY (pusher, room_id)
	22	);

+33

-0

synapse/storage/databases/main/schema/delta/32/remove_indices.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	-- The following indices are redundant, other indices are equivalent or
	17	-- supersets
	18	DROP INDEX IF EXISTS events_room_id; -- Prefix of events_room_stream
	19	DROP INDEX IF EXISTS events_order; -- Prefix of events_order_topo_stream_room
	20	DROP INDEX IF EXISTS events_topological_ordering; -- Prefix of events_order_topo_stream_room
	21	DROP INDEX IF EXISTS events_stream_ordering; -- Duplicate of PRIMARY KEY
	22	DROP INDEX IF EXISTS event_to_state_groups_id; -- Duplicate of PRIMARY KEY
	23	DROP INDEX IF EXISTS event_push_actions_room_id_event_id_user_id_profile_tag; -- Duplicate of UNIQUE CONSTRAINT
	24
	25	DROP INDEX IF EXISTS st_extrem_id; -- Prefix of UNIQUE CONSTRAINT
	26	DROP INDEX IF EXISTS event_signatures_id; -- Prefix of UNIQUE CONSTRAINT
	27	DROP INDEX IF EXISTS redactions_event_id; -- Duplicate of UNIQUE CONSTRAINT
	28
	29	-- The following indices were unused
	30	DROP INDEX IF EXISTS remote_media_cache_thumbnails_media_id;
	31	DROP INDEX IF EXISTS evauth_edges_auth_id;
	32	DROP INDEX IF EXISTS presence_stream_state;

+25

-0

synapse/storage/databases/main/schema/delta/32/reports.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE event_reports(
	17	id BIGINT NOT NULL PRIMARY KEY,
	18	received_ts BIGINT NOT NULL,
	19	room_id TEXT NOT NULL,
	20	event_id TEXT NOT NULL,
	21	user_id TEXT NOT NULL,
	22	reason TEXT,
	23	content TEXT
	24	);

+17

-0

synapse/storage/databases/main/schema/delta/33/access_tokens_device_index.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT INTO background_updates (update_name, progress_json) VALUES
	16	('access_tokens_device_index', '{}');

+21

-0

synapse/storage/databases/main/schema/delta/33/devices.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE devices (
	16	user_id TEXT NOT NULL,
	17	device_id TEXT NOT NULL,
	18	display_name TEXT,
	19	CONSTRAINT device_uniqueness UNIQUE (user_id, device_id)
	20	);

+19

-0

synapse/storage/databases/main/schema/delta/33/devices_for_e2e_keys.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- make sure that we have a device record for each set of E2E keys, so that the
	16	-- user can delete them if they like.
	17	INSERT INTO devices
	18	SELECT user_id, device_id, NULL FROM e2e_device_keys_json;

+20

-0

synapse/storage/databases/main/schema/delta/33/devices_for_e2e_keys_clear_unknown_device.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- a previous version of the "devices_for_e2e_keys" delta set all the device
	16	-- names to "unknown device". This wasn't terribly helpful
	17	UPDATE devices
	18	SET display_name = NULL
	19	WHERE display_name = 'unknown device';

+59

-0

synapse/storage/databases/main/schema/delta/33/event_fields.py less more

	0	# Copyright 2016 OpenMarket Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13	import json
	14	import logging
	15
	16	from synapse.storage.prepare_database import get_statements
	17
	18	logger = logging.getLogger(__name__)
	19
	20
	21	ALTER_TABLE = """
	22	ALTER TABLE events ADD COLUMN sender TEXT;
	23	ALTER TABLE events ADD COLUMN contains_url BOOLEAN;
	24	"""
	25
	26
	27	def run_create(cur, database_engine, args, *kwargs):
	28	for statement in get_statements(ALTER_TABLE.splitlines()):
	29	cur.execute(statement)
	30
	31	cur.execute("SELECT MIN(stream_ordering) FROM events")
	32	rows = cur.fetchall()
	33	min_stream_id = rows[0][0]
	34
	35	cur.execute("SELECT MAX(stream_ordering) FROM events")
	36	rows = cur.fetchall()
	37	max_stream_id = rows[0][0]
	38
	39	if min_stream_id is not None and max_stream_id is not None:
	40	progress = {
	41	"target_min_stream_id_inclusive": min_stream_id,
	42	"max_stream_id_exclusive": max_stream_id + 1,
	43	"rows_inserted": 0,
	44	}
	45	progress_json = json.dumps(progress)
	46
	47	sql = (
	48	"INSERT into background_updates (update_name, progress_json)"
	49	" VALUES (?, ?)"
	50	)
	51
	52	sql = database_engine.convert_param_style(sql)
	53
	54	cur.execute(sql, ("event_fields_sender_url", progress_json))
	55
	56
	57	def run_upgrade(cur, database_engine, args, *kwargs):
	58	pass

+30

-0

synapse/storage/databases/main/schema/delta/33/remote_media_ts.py less more

	0	# Copyright 2016 OpenMarket Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14	import time
	15
	16	ALTER_TABLE = "ALTER TABLE remote_media_cache ADD COLUMN last_access_ts BIGINT"
	17
	18
	19	def run_create(cur, database_engine, args, *kwargs):
	20	cur.execute(ALTER_TABLE)
	21
	22
	23	def run_upgrade(cur, database_engine, args, *kwargs):
	24	cur.execute(
	25	database_engine.convert_param_style(
	26	"UPDATE remote_media_cache SET last_access_ts = ?"
	27	),
	28	(int(time.time() * 1000),),
	29	)

+17

-0

synapse/storage/databases/main/schema/delta/33/user_ips_index.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT INTO background_updates (update_name, progress_json) VALUES
	16	('user_ips_device_index', '{}');

+23

-0

synapse/storage/databases/main/schema/delta/34/appservice_stream.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS appservice_stream_position(
	16	Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
	17	stream_ordering BIGINT,
	18	CHECK (Lock='X')
	19	);
	20
	21	INSERT INTO appservice_stream_position (stream_ordering)
	22	SELECT COALESCE(MAX(stream_ordering), 0) FROM events;

+46

-0

synapse/storage/databases/main/schema/delta/34/cache_stream.py less more

	0	# Copyright 2016 OpenMarket Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14	import logging
	15
	16	from synapse.storage.engines import PostgresEngine
	17	from synapse.storage.prepare_database import get_statements
	18
	19	logger = logging.getLogger(__name__)
	20
	21
	22	# This stream is used to notify replication slaves that some caches have
	23	# been invalidated that they cannot infer from the other streams.
	24	CREATE_TABLE = """
	25	CREATE TABLE cache_invalidation_stream (
	26	stream_id BIGINT,
	27	cache_func TEXT,
	28	keys TEXT[],
	29	invalidation_ts BIGINT
	30	);
	31
	32	CREATE INDEX cache_invalidation_stream_id ON cache_invalidation_stream(stream_id);
	33	"""
	34
	35
	36	def run_create(cur, database_engine, args, *kwargs):
	37	if not isinstance(database_engine, PostgresEngine):
	38	return
	39
	40	for statement in get_statements(CREATE_TABLE.splitlines()):
	41	cur.execute(statement)
	42
	43
	44	def run_upgrade(cur, database_engine, args, *kwargs):
	45	pass

+24

-0

synapse/storage/databases/main/schema/delta/34/device_inbox.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE device_inbox (
	16	user_id TEXT NOT NULL,
	17	device_id TEXT NOT NULL,
	18	stream_id BIGINT NOT NULL,
	19	message_json TEXT NOT NULL -- {"type":, "sender":, "content",}
	20	);
	21
	22	CREATE INDEX device_inbox_user_stream_id ON device_inbox(user_id, device_id, stream_id);
	23	CREATE INDEX device_inbox_stream_id ON device_inbox(stream_id);

+20

-0

synapse/storage/databases/main/schema/delta/34/push_display_name_rename.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	DELETE FROM push_rules WHERE rule_id = 'global/override/.m.rule.contains_display_name';
	16	UPDATE push_rules SET rule_id = 'global/override/.m.rule.contains_display_name' WHERE rule_id = 'global/underride/.m.rule.contains_display_name';
	17
	18	DELETE FROM push_rules_enable WHERE rule_id = 'global/override/.m.rule.contains_display_name';
	19	UPDATE push_rules_enable SET rule_id = 'global/override/.m.rule.contains_display_name' WHERE rule_id = 'global/underride/.m.rule.contains_display_name';

+32

-0

synapse/storage/databases/main/schema/delta/34/received_txn_purge.py less more

	0	# Copyright 2016 OpenMarket Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14	import logging
	15
	16	from synapse.storage.engines import PostgresEngine
	17
	18	logger = logging.getLogger(__name__)
	19
	20
	21	def run_create(cur, database_engine, args, *kwargs):
	22	if isinstance(database_engine, PostgresEngine):
	23	cur.execute("TRUNCATE received_transactions")
	24	else:
	25	cur.execute("DELETE FROM received_transactions")
	26
	27	cur.execute("CREATE INDEX received_transactions_ts ON received_transactions(ts)")
	28
	29
	30	def run_upgrade(cur, database_engine, args, *kwargs):
	31	pass

+17

-0

synapse/storage/databases/main/schema/delta/35/contains_url.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT into background_updates (update_name, progress_json)
	16	VALUES ('event_contains_url_index', '{}');

+39

-0

synapse/storage/databases/main/schema/delta/35/device_outbox.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	DROP TABLE IF EXISTS device_federation_outbox;
	16	CREATE TABLE device_federation_outbox (
	17	destination TEXT NOT NULL,
	18	stream_id BIGINT NOT NULL,
	19	queued_ts BIGINT NOT NULL,
	20	messages_json TEXT NOT NULL
	21	);
	22
	23
	24	DROP INDEX IF EXISTS device_federation_outbox_destination_id;
	25	CREATE INDEX device_federation_outbox_destination_id
	26	ON device_federation_outbox(destination, stream_id);
	27
	28
	29	DROP TABLE IF EXISTS device_federation_inbox;
	30	CREATE TABLE device_federation_inbox (
	31	origin TEXT NOT NULL,
	32	message_id TEXT NOT NULL,
	33	received_ts BIGINT NOT NULL
	34	);
	35
	36	DROP INDEX IF EXISTS device_federation_inbox_sender_id;
	37	CREATE INDEX device_federation_inbox_sender_id
	38	ON device_federation_inbox(origin, message_id);

+21

-0

synapse/storage/databases/main/schema/delta/35/device_stream_id.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE device_max_stream_id (
	16	stream_id BIGINT NOT NULL
	17	);
	18
	19	INSERT INTO device_max_stream_id (stream_id)
	20	SELECT COALESCE(MAX(stream_id), 0) FROM device_inbox;

+17

-0

synapse/storage/databases/main/schema/delta/35/event_push_actions_index.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT into background_updates (update_name, progress_json)
	16	VALUES ('epa_highlight_index', '{}');

+33

-0

synapse/storage/databases/main/schema/delta/35/public_room_list_change_stream.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE public_room_list_stream (
	17	stream_id BIGINT NOT NULL,
	18	room_id TEXT NOT NULL,
	19	visibility BOOLEAN NOT NULL
	20	);
	21
	22	INSERT INTO public_room_list_stream (stream_id, room_id, visibility)
	23	SELECT 1, room_id, is_public FROM rooms
	24	WHERE is_public = CAST(1 AS BOOLEAN);
	25
	26	CREATE INDEX public_room_list_stream_idx on public_room_list_stream(
	27	stream_id
	28	);
	29
	30	CREATE INDEX public_room_list_stream_rm_idx on public_room_list_stream(
	31	room_id, stream_id
	32	);

+37

-0

synapse/storage/databases/main/schema/delta/35/stream_order_to_extrem.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE stream_ordering_to_exterm (
	17	stream_ordering BIGINT NOT NULL,
	18	room_id TEXT NOT NULL,
	19	event_id TEXT NOT NULL
	20	);
	21
	22	INSERT INTO stream_ordering_to_exterm (stream_ordering, room_id, event_id)
	23	SELECT stream_ordering, room_id, event_id FROM event_forward_extremities
	24	INNER JOIN (
	25	SELECT room_id, max(stream_ordering) as stream_ordering FROM events
	26	INNER JOIN event_forward_extremities USING (room_id, event_id)
	27	GROUP BY room_id
	28	) AS rms USING (room_id);
	29
	30	CREATE INDEX stream_ordering_to_exterm_idx on stream_ordering_to_exterm(
	31	stream_ordering
	32	);
	33
	34	CREATE INDEX stream_ordering_to_exterm_rm_idx on stream_ordering_to_exterm(
	35	room_id, stream_ordering
	36	);

+26

-0

synapse/storage/databases/main/schema/delta/36/readd_public_rooms.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Re-add some entries to stream_ordering_to_exterm that were incorrectly deleted
	16	INSERT INTO stream_ordering_to_exterm (stream_ordering, room_id, event_id)
	17	SELECT
	18	(SELECT stream_ordering FROM events where event_id = e.event_id) AS stream_ordering,
	19	room_id,
	20	event_id
	21	FROM event_forward_extremities AS e
	22	WHERE NOT EXISTS (
	23	SELECT room_id FROM stream_ordering_to_exterm AS s
	24	WHERE s.room_id = e.room_id
	25	);

+85

-0

synapse/storage/databases/main/schema/delta/37/remove_auth_idx.py less more

	0	# Copyright 2016 OpenMarket Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14	import logging
	15
	16	from synapse.storage.engines import PostgresEngine
	17	from synapse.storage.prepare_database import get_statements
	18
	19	logger = logging.getLogger(__name__)
	20
	21	DROP_INDICES = """
	22	-- We only ever query based on event_id
	23	DROP INDEX IF EXISTS state_events_room_id;
	24	DROP INDEX IF EXISTS state_events_type;
	25	DROP INDEX IF EXISTS state_events_state_key;
	26
	27	-- room_id is indexed elsewhere
	28	DROP INDEX IF EXISTS current_state_events_room_id;
	29	DROP INDEX IF EXISTS current_state_events_state_key;
	30	DROP INDEX IF EXISTS current_state_events_type;
	31
	32	DROP INDEX IF EXISTS transactions_have_ref;
	33
	34	-- (topological_ordering, stream_ordering, room_id) seems like a strange index,
	35	-- and is used incredibly rarely.
	36	DROP INDEX IF EXISTS events_order_topo_stream_room;
	37
	38	-- an equivalent index to this actually gets re-created in delta 41, because it
	39	-- turned out that deleting it wasn't a great plan :/. In any case, let's
	40	-- delete it here, and delta 41 will create a new one with an added UNIQUE
	41	-- constraint
	42	DROP INDEX IF EXISTS event_search_ev_idx;
	43	"""
	44
	45	POSTGRES_DROP_CONSTRAINT = """
	46	ALTER TABLE event_auth DROP CONSTRAINT IF EXISTS event_auth_event_id_auth_id_room_id_key;
	47	"""
	48
	49	SQLITE_DROP_CONSTRAINT = """
	50	DROP INDEX IF EXISTS evauth_edges_id;
	51
	52	CREATE TABLE IF NOT EXISTS event_auth_new(
	53	event_id TEXT NOT NULL,
	54	auth_id TEXT NOT NULL,
	55	room_id TEXT NOT NULL
	56	);
	57
	58	INSERT INTO event_auth_new
	59	SELECT event_id, auth_id, room_id
	60	FROM event_auth;
	61
	62	DROP TABLE event_auth;
	63
	64	ALTER TABLE event_auth_new RENAME TO event_auth;
	65
	66	CREATE INDEX evauth_edges_id ON event_auth(event_id);
	67	"""
	68
	69
	70	def run_create(cur, database_engine, args, *kwargs):
	71	for statement in get_statements(DROP_INDICES.splitlines()):
	72	cur.execute(statement)
	73
	74	if isinstance(database_engine, PostgresEngine):
	75	drop_constraint = POSTGRES_DROP_CONSTRAINT
	76	else:
	77	drop_constraint = SQLITE_DROP_CONSTRAINT
	78
	79	for statement in get_statements(drop_constraint.splitlines()):
	80	cur.execute(statement)
	81
	82
	83	def run_upgrade(cur, database_engine, args, *kwargs):
	84	pass

+52

-0

synapse/storage/databases/main/schema/delta/37/user_threepids.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/*
	16	* Update any email addresses that were stored with mixed case into all
	17	* lowercase
	18	*/
	19
	20	-- There may be "duplicate" emails (with different case) already in the table,
	21	-- so we find them and move all but the most recently used account.
	22	UPDATE user_threepids
	23	SET medium = 'email_old'
	24	WHERE medium = 'email'
	25	AND address IN (
	26	-- We select all the addresses that are linked to the user_id that is NOT
	27	-- the most recently created.
	28	SELECT u.address
	29	FROM
	30	user_threepids AS u,
	31	-- `duplicate_addresses` is a table of all the email addresses that
	32	-- appear multiple times and when the binding was created
	33	(
	34	SELECT lower(u1.address) AS address, max(u1.added_at) AS max_ts
	35	FROM user_threepids AS u1
	36	INNER JOIN user_threepids AS u2 ON u1.medium = u2.medium AND lower(u1.address) = lower(u2.address) AND u1.address != u2.address
	37	WHERE u1.medium = 'email' AND u2.medium = 'email'
	38	GROUP BY lower(u1.address)
	39	) AS duplicate_addresses
	40	WHERE
	41	lower(u.address) = duplicate_addresses.address
	42	AND u.added_at != max_ts -- NOT the most recently created
	43	);
	44
	45
	46	-- This update is now safe since we've removed the duplicate addresses.
	47	UPDATE user_threepids SET address = LOWER(address) WHERE medium = 'email';
	48
	49
	50	/* Add an index for the select we do on passwored reset */
	51	CREATE INDEX user_threepids_medium_address on user_threepids (medium, address);

+19

-0

synapse/storage/databases/main/schema/delta/38/postgres_fts_gist.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- We no longer do this given we back it out again in schema 47
	16
	17	-- INSERT into background_updates (update_name, progress_json)
	18	-- VALUES ('event_search_postgres_gist', '{}');

+29

-0

synapse/storage/databases/main/schema/delta/39/appservice_room_list.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE appservice_room_list(
	16	appservice_id TEXT NOT NULL,
	17	network_id TEXT NOT NULL,
	18	room_id TEXT NOT NULL
	19	);
	20
	21	-- Each appservice can have multiple published room lists associated with them,
	22	-- keyed of a particular network_id
	23	CREATE UNIQUE INDEX appservice_room_list_idx ON appservice_room_list(
	24	appservice_id, network_id, room_id
	25	);
	26
	27	ALTER TABLE public_room_list_stream ADD COLUMN appservice_id TEXT;
	28	ALTER TABLE public_room_list_stream ADD COLUMN network_id TEXT;

+16

-0

synapse/storage/databases/main/schema/delta/39/device_federation_stream_idx.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE INDEX device_federation_outbox_id ON device_federation_outbox(stream_id);

+17

-0

synapse/storage/databases/main/schema/delta/39/event_push_index.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT INTO background_updates (update_name, progress_json) VALUES
	16	('event_push_actions_highlights_index', '{}');

+22

-0

synapse/storage/databases/main/schema/delta/39/federation_out_position.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE federation_stream_position(
	16	type TEXT NOT NULL,
	17	stream_id INTEGER NOT NULL
	18	);
	19
	20	INSERT INTO federation_stream_position (type, stream_id) VALUES ('federation', -1);
	21	INSERT INTO federation_stream_position (type, stream_id) SELECT 'events', coalesce(max(stream_ordering), -1) FROM events;

+20

-0

synapse/storage/databases/main/schema/delta/39/membership_profile.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE room_memberships ADD COLUMN display_name TEXT;
	16	ALTER TABLE room_memberships ADD COLUMN avatar_url TEXT;
	17
	18	INSERT into background_updates (update_name, progress_json)
	19	VALUES ('room_membership_profile_update', '{}');

+17

-0

synapse/storage/databases/main/schema/delta/40/current_state_idx.sql less more

	0	/* Copyright 2017 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT INTO background_updates (update_name, progress_json) VALUES
	16	('current_state_members_idx', '{}');

+21

-0

synapse/storage/databases/main/schema/delta/40/device_inbox.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- turn the pre-fill startup query into a index-only scan on postgresql.
	16	INSERT into background_updates (update_name, progress_json)
	17	VALUES ('device_inbox_stream_index', '{}');
	18
	19	INSERT into background_updates (update_name, progress_json, depends_on)
	20	VALUES ('device_inbox_stream_drop', '{}', 'device_inbox_stream_index');

+60

-0

synapse/storage/databases/main/schema/delta/40/device_list_streams.sql less more

	0	/* Copyright 2017 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Cache of remote devices.
	16	CREATE TABLE device_lists_remote_cache (
	17	user_id TEXT NOT NULL,
	18	device_id TEXT NOT NULL,
	19	content TEXT NOT NULL
	20	);
	21
	22	-- The last update we got for a user. Empty if we're not receiving updates for
	23	-- that user.
	24	CREATE TABLE device_lists_remote_extremeties (
	25	user_id TEXT NOT NULL,
	26	stream_id TEXT NOT NULL
	27	);
	28
	29	-- we used to create non-unique indexes on these tables, but as of update 52 we create
	30	-- unique indexes concurrently:
	31	--
	32	-- CREATE INDEX device_lists_remote_cache_id ON device_lists_remote_cache(user_id, device_id);
	33	-- CREATE INDEX device_lists_remote_extremeties_id ON device_lists_remote_extremeties(user_id, stream_id);
	34
	35
	36	-- Stream of device lists updates. Includes both local and remotes
	37	CREATE TABLE device_lists_stream (
	38	stream_id BIGINT NOT NULL,
	39	user_id TEXT NOT NULL,
	40	device_id TEXT NOT NULL
	41	);
	42
	43	CREATE INDEX device_lists_stream_id ON device_lists_stream(stream_id, user_id);
	44
	45
	46	-- The stream of updates to send to other servers. We keep at least one row
	47	-- per user that was sent so that the prev_id for any new updates can be
	48	-- calculated
	49	CREATE TABLE device_lists_outbound_pokes (
	50	destination TEXT NOT NULL,
	51	stream_id BIGINT NOT NULL,
	52	user_id TEXT NOT NULL,
	53	device_id TEXT NOT NULL,
	54	sent BOOLEAN NOT NULL,
	55	ts BIGINT NOT NULL -- So that in future we can clear out pokes to dead servers
	56	);
	57
	58	CREATE INDEX device_lists_outbound_pokes_id ON device_lists_outbound_pokes(destination, stream_id);
	59	CREATE INDEX device_lists_outbound_pokes_user ON device_lists_outbound_pokes(destination, user_id);

+37

-0

synapse/storage/databases/main/schema/delta/40/event_push_summary.sql less more

	0	/* Copyright 2017 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Aggregate of old notification counts that have been deleted out of the
	16	-- main event_push_actions table. This count does not include those that were
	17	-- highlights, as they remain in the event_push_actions table.
	18	CREATE TABLE event_push_summary (
	19	user_id TEXT NOT NULL,
	20	room_id TEXT NOT NULL,
	21	notif_count BIGINT NOT NULL,
	22	stream_ordering BIGINT NOT NULL
	23	);
	24
	25	CREATE INDEX event_push_summary_user_rm ON event_push_summary(user_id, room_id);
	26
	27
	28	-- The stream ordering up to which we have aggregated the event_push_actions
	29	-- table into event_push_summary
	30	CREATE TABLE event_push_summary_stream_ordering (
	31	Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
	32	stream_ordering BIGINT NOT NULL,
	33	CHECK (Lock='X')
	34	);
	35
	36	INSERT INTO event_push_summary_stream_ordering (stream_ordering) VALUES (0);

+39

-0

synapse/storage/databases/main/schema/delta/40/pushers.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS pushers2 (
	16	id BIGINT PRIMARY KEY,
	17	user_name TEXT NOT NULL,
	18	access_token BIGINT DEFAULT NULL,
	19	profile_tag TEXT NOT NULL,
	20	kind TEXT NOT NULL,
	21	app_id TEXT NOT NULL,
	22	app_display_name TEXT NOT NULL,
	23	device_display_name TEXT NOT NULL,
	24	pushkey TEXT NOT NULL,
	25	ts BIGINT NOT NULL,
	26	lang TEXT,
	27	data TEXT,
	28	last_stream_ordering INTEGER,
	29	last_success BIGINT,
	30	failing_since BIGINT,
	31	UNIQUE (app_id, pushkey, user_name)
	32	);
	33
	34	INSERT INTO pushers2 SELECT * FROM PUSHERS;
	35
	36	DROP TABLE PUSHERS;
	37
	38	ALTER TABLE pushers2 RENAME TO pushers;

+17

-0

synapse/storage/databases/main/schema/delta/41/device_list_stream_idx.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT into background_updates (update_name, progress_json)
	16	VALUES ('device_lists_stream_idx', '{}');

+16

-0

synapse/storage/databases/main/schema/delta/41/device_outbound_index.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE INDEX device_lists_outbound_pokes_stream ON device_lists_outbound_pokes(stream_id);

+17

-0

synapse/storage/databases/main/schema/delta/41/event_search_event_id_idx.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT into background_updates (update_name, progress_json)
	16	VALUES ('event_search_event_id_idx', '{}');

+22

-0

synapse/storage/databases/main/schema/delta/41/ratelimit.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE ratelimit_override (
	16	user_id TEXT NOT NULL,
	17	messages_per_second BIGINT,
	18	burst_count BIGINT
	19	);
	20
	21	CREATE UNIQUE INDEX ratelimit_override_idx ON ratelimit_override(user_id);

+26

-0

synapse/storage/databases/main/schema/delta/42/current_state_delta.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE current_state_delta_stream (
	17	stream_id BIGINT NOT NULL,
	18	room_id TEXT NOT NULL,
	19	type TEXT NOT NULL,
	20	state_key TEXT NOT NULL,
	21	event_id TEXT, -- Is null if the key was removed
	22	prev_event_id TEXT -- Is null if the key was added
	23	);
	24
	25	CREATE INDEX current_state_delta_stream_idx ON current_state_delta_stream(stream_id);

+33

-0

synapse/storage/databases/main/schema/delta/42/device_list_last_id.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	-- Table of last stream_id that we sent to destination for user_id. This is
	17	-- used to fill out the `prev_id` fields of outbound device list updates.
	18	CREATE TABLE device_lists_outbound_last_success (
	19	destination TEXT NOT NULL,
	20	user_id TEXT NOT NULL,
	21	stream_id BIGINT NOT NULL
	22	);
	23
	24	INSERT INTO device_lists_outbound_last_success
	25	SELECT destination, user_id, coalesce(max(stream_id), 0) as stream_id
	26	FROM device_lists_outbound_pokes
	27	WHERE sent = (1 = 1) -- sqlite doesn't have inbuilt boolean values
	28	GROUP BY destination, user_id;
	29
	30	CREATE INDEX device_lists_outbound_last_success_idx ON device_lists_outbound_last_success(
	31	destination, user_id, stream_id
	32	);

+17

-0

synapse/storage/databases/main/schema/delta/42/event_auth_state_only.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT INTO background_updates (update_name, progress_json) VALUES
	16	('event_auth_state_only', '{}');

+84

-0

synapse/storage/databases/main/schema/delta/42/user_dir.py less more

	0	# Copyright 2017 Vector Creations Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14	import logging
	15
	16	from synapse.storage.engines import PostgresEngine, Sqlite3Engine
	17	from synapse.storage.prepare_database import get_statements
	18
	19	logger = logging.getLogger(__name__)
	20
	21
	22	BOTH_TABLES = """
	23	CREATE TABLE user_directory_stream_pos (
	24	Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
	25	stream_id BIGINT,
	26	CHECK (Lock='X')
	27	);
	28
	29	INSERT INTO user_directory_stream_pos (stream_id) VALUES (null);
	30
	31	CREATE TABLE user_directory (
	32	user_id TEXT NOT NULL,
	33	room_id TEXT NOT NULL, -- A room_id that we know the user is joined to
	34	display_name TEXT,
	35	avatar_url TEXT
	36	);
	37
	38	CREATE INDEX user_directory_room_idx ON user_directory(room_id);
	39	CREATE UNIQUE INDEX user_directory_user_idx ON user_directory(user_id);
	40
	41	CREATE TABLE users_in_pubic_room (
	42	user_id TEXT NOT NULL,
	43	room_id TEXT NOT NULL -- A room_id that we know is public
	44	);
	45
	46	CREATE INDEX users_in_pubic_room_room_idx ON users_in_pubic_room(room_id);
	47	CREATE UNIQUE INDEX users_in_pubic_room_user_idx ON users_in_pubic_room(user_id);
	48	"""
	49
	50
	51	POSTGRES_TABLE = """
	52	CREATE TABLE user_directory_search (
	53	user_id TEXT NOT NULL,
	54	vector tsvector
	55	);
	56
	57	CREATE INDEX user_directory_search_fts_idx ON user_directory_search USING gin(vector);
	58	CREATE UNIQUE INDEX user_directory_search_user_idx ON user_directory_search(user_id);
	59	"""
	60
	61
	62	SQLITE_TABLE = """
	63	CREATE VIRTUAL TABLE user_directory_search
	64	USING fts4 ( user_id, value );
	65	"""
	66
	67
	68	def run_create(cur, database_engine, args, *kwargs):
	69	for statement in get_statements(BOTH_TABLES.splitlines()):
	70	cur.execute(statement)
	71
	72	if isinstance(database_engine, PostgresEngine):
	73	for statement in get_statements(POSTGRES_TABLE.splitlines()):
	74	cur.execute(statement)
	75	elif isinstance(database_engine, Sqlite3Engine):
	76	for statement in get_statements(SQLITE_TABLE.splitlines()):
	77	cur.execute(statement)
	78	else:
	79	raise Exception("Unrecognized database engine")
	80
	81
	82	def run_upgrade(args, *kwargs):
	83	pass

+21

-0

synapse/storage/databases/main/schema/delta/43/blocked_rooms.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE blocked_rooms (
	16	room_id TEXT NOT NULL,
	17	user_id TEXT NOT NULL -- Admin who blocked the room
	18	);
	19
	20	CREATE UNIQUE INDEX blocked_rooms_idx ON blocked_rooms(room_id);

+17

-0

synapse/storage/databases/main/schema/delta/43/quarantine_media.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE local_media_repository ADD COLUMN quarantined_by TEXT;
	16	ALTER TABLE remote_media_cache ADD COLUMN quarantined_by TEXT;

+16

-0

synapse/storage/databases/main/schema/delta/43/url_cache.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE local_media_repository ADD COLUMN url_cache TEXT;

+33

-0

synapse/storage/databases/main/schema/delta/43/user_share.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Table keeping track of who shares a room with who. We only keep track
	16	-- of this for local users, so `user_id` is local users only (but we do keep track
	17	-- of which remote users share a room)
	18	CREATE TABLE users_who_share_rooms (
	19	user_id TEXT NOT NULL,
	20	other_user_id TEXT NOT NULL,
	21	room_id TEXT NOT NULL,
	22	share_private BOOLEAN NOT NULL -- is the shared room private? i.e. they share a private room
	23	);
	24
	25
	26	CREATE UNIQUE INDEX users_who_share_rooms_u_idx ON users_who_share_rooms(user_id, other_user_id);
	27	CREATE INDEX users_who_share_rooms_r_idx ON users_who_share_rooms(room_id);
	28	CREATE INDEX users_who_share_rooms_o_idx ON users_who_share_rooms(other_user_id);
	29
	30
	31	-- Make sure that we populate the table initially
	32	UPDATE user_directory_stream_pos SET stream_id = NULL;

+41

-0

synapse/storage/databases/main/schema/delta/44/expire_url_cache.sql less more

	0	/* Copyright 2017 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- this didn't work on SQLite 3.7 (because of lack of partial indexes), so was
	16	-- removed and replaced with 46/local_media_repository_url_idx.sql.
	17	--
	18	-- CREATE INDEX local_media_repository_url_idx ON local_media_repository(created_ts) WHERE url_cache IS NOT NULL;
	19
	20	-- we need to change `expires` to `expires_ts` so that we can index on it. SQLite doesn't support
	21	-- indices on expressions until 3.9.
	22	CREATE TABLE local_media_repository_url_cache_new(
	23	url TEXT,
	24	response_code INTEGER,
	25	etag TEXT,
	26	expires_ts BIGINT,
	27	og TEXT,
	28	media_id TEXT,
	29	download_ts BIGINT
	30	);
	31
	32	INSERT INTO local_media_repository_url_cache_new
	33	SELECT url, response_code, etag, expires + download_ts, og, media_id, download_ts FROM local_media_repository_url_cache;
	34
	35	DROP TABLE local_media_repository_url_cache;
	36	ALTER TABLE local_media_repository_url_cache_new RENAME TO local_media_repository_url_cache;
	37
	38	CREATE INDEX local_media_repository_url_cache_expires_idx ON local_media_repository_url_cache(expires_ts);
	39	CREATE INDEX local_media_repository_url_cache_by_url_download_ts ON local_media_repository_url_cache(url, download_ts);
	40	CREATE INDEX local_media_repository_url_cache_media_idx ON local_media_repository_url_cache(media_id);

+167

-0

synapse/storage/databases/main/schema/delta/45/group_server.sql less more

	0	/* Copyright 2017 Vector Creations Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE groups (
	16	group_id TEXT NOT NULL,
	17	name TEXT, -- the display name of the room
	18	avatar_url TEXT,
	19	short_description TEXT,
	20	long_description TEXT
	21	);
	22
	23	CREATE UNIQUE INDEX groups_idx ON groups(group_id);
	24
	25
	26	-- list of users the group server thinks are joined
	27	CREATE TABLE group_users (
	28	group_id TEXT NOT NULL,
	29	user_id TEXT NOT NULL,
	30	is_admin BOOLEAN NOT NULL,
	31	is_public BOOLEAN NOT NULL -- whether the users membership can be seen by everyone
	32	);
	33
	34
	35	CREATE INDEX groups_users_g_idx ON group_users(group_id, user_id);
	36	CREATE INDEX groups_users_u_idx ON group_users(user_id);
	37
	38	-- list of users the group server thinks are invited
	39	CREATE TABLE group_invites (
	40	group_id TEXT NOT NULL,
	41	user_id TEXT NOT NULL
	42	);
	43
	44	CREATE INDEX groups_invites_g_idx ON group_invites(group_id, user_id);
	45	CREATE INDEX groups_invites_u_idx ON group_invites(user_id);
	46
	47
	48	CREATE TABLE group_rooms (
	49	group_id TEXT NOT NULL,
	50	room_id TEXT NOT NULL,
	51	is_public BOOLEAN NOT NULL -- whether the room can be seen by everyone
	52	);
	53
	54	CREATE UNIQUE INDEX groups_rooms_g_idx ON group_rooms(group_id, room_id);
	55	CREATE INDEX groups_rooms_r_idx ON group_rooms(room_id);
	56
	57
	58	-- Rooms to include in the summary
	59	CREATE TABLE group_summary_rooms (
	60	group_id TEXT NOT NULL,
	61	room_id TEXT NOT NULL,
	62	category_id TEXT NOT NULL,
	63	room_order BIGINT NOT NULL,
	64	is_public BOOLEAN NOT NULL, -- whether the room should be show to everyone
	65	UNIQUE (group_id, category_id, room_id, room_order),
	66	CHECK (room_order > 0)
	67	);
	68
	69	CREATE UNIQUE INDEX group_summary_rooms_g_idx ON group_summary_rooms(group_id, room_id, category_id);
	70
	71
	72	-- Categories to include in the summary
	73	CREATE TABLE group_summary_room_categories (
	74	group_id TEXT NOT NULL,
	75	category_id TEXT NOT NULL,
	76	cat_order BIGINT NOT NULL,
	77	UNIQUE (group_id, category_id, cat_order),
	78	CHECK (cat_order > 0)
	79	);
	80
	81	-- The categories in the group
	82	CREATE TABLE group_room_categories (
	83	group_id TEXT NOT NULL,
	84	category_id TEXT NOT NULL,
	85	profile TEXT NOT NULL,
	86	is_public BOOLEAN NOT NULL, -- whether the category should be show to everyone
	87	UNIQUE (group_id, category_id)
	88	);
	89
	90	-- The users to include in the group summary
	91	CREATE TABLE group_summary_users (
	92	group_id TEXT NOT NULL,
	93	user_id TEXT NOT NULL,
	94	role_id TEXT NOT NULL,
	95	user_order BIGINT NOT NULL,
	96	is_public BOOLEAN NOT NULL -- whether the user should be show to everyone
	97	);
	98
	99	CREATE INDEX group_summary_users_g_idx ON group_summary_users(group_id);
	100
	101	-- The roles to include in the group summary
	102	CREATE TABLE group_summary_roles (
	103	group_id TEXT NOT NULL,
	104	role_id TEXT NOT NULL,
	105	role_order BIGINT NOT NULL,
	106	UNIQUE (group_id, role_id, role_order),
	107	CHECK (role_order > 0)
	108	);
	109
	110
	111	-- The roles in a groups
	112	CREATE TABLE group_roles (
	113	group_id TEXT NOT NULL,
	114	role_id TEXT NOT NULL,
	115	profile TEXT NOT NULL,
	116	is_public BOOLEAN NOT NULL, -- whether the role should be show to everyone
	117	UNIQUE (group_id, role_id)
	118	);
	119
	120
	121	-- List of attestations we've given out and need to renew
	122	CREATE TABLE group_attestations_renewals (
	123	group_id TEXT NOT NULL,
	124	user_id TEXT NOT NULL,
	125	valid_until_ms BIGINT NOT NULL
	126	);
	127
	128	CREATE INDEX group_attestations_renewals_g_idx ON group_attestations_renewals(group_id, user_id);
	129	CREATE INDEX group_attestations_renewals_u_idx ON group_attestations_renewals(user_id);
	130	CREATE INDEX group_attestations_renewals_v_idx ON group_attestations_renewals(valid_until_ms);
	131
	132
	133	-- List of attestations we've received from remotes and are interested in.
	134	CREATE TABLE group_attestations_remote (
	135	group_id TEXT NOT NULL,
	136	user_id TEXT NOT NULL,
	137	valid_until_ms BIGINT NOT NULL,
	138	attestation_json TEXT NOT NULL
	139	);
	140
	141	CREATE INDEX group_attestations_remote_g_idx ON group_attestations_remote(group_id, user_id);
	142	CREATE INDEX group_attestations_remote_u_idx ON group_attestations_remote(user_id);
	143	CREATE INDEX group_attestations_remote_v_idx ON group_attestations_remote(valid_until_ms);
	144
	145
	146	-- The group membership for the HS's users
	147	CREATE TABLE local_group_membership (
	148	group_id TEXT NOT NULL,
	149	user_id TEXT NOT NULL,
	150	is_admin BOOLEAN NOT NULL,
	151	membership TEXT NOT NULL,
	152	is_publicised BOOLEAN NOT NULL, -- if the user is publicising their membership
	153	content TEXT NOT NULL
	154	);
	155
	156	CREATE INDEX local_group_membership_u_idx ON local_group_membership(user_id, group_id);
	157	CREATE INDEX local_group_membership_g_idx ON local_group_membership(group_id);
	158
	159
	160	CREATE TABLE local_group_updates (
	161	stream_id BIGINT NOT NULL,
	162	group_id TEXT NOT NULL,
	163	user_id TEXT NOT NULL,
	164	type TEXT NOT NULL,
	165	content TEXT NOT NULL
	166	);

+28

-0

synapse/storage/databases/main/schema/delta/45/profile_cache.sql less more

	0	/* Copyright 2017 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	-- A subset of remote users whose profiles we have cached.
	17	-- Whether a user is in this table or not is defined by the storage function
	18	-- `is_subscribed_remote_profile_for_user`
	19	CREATE TABLE remote_profile_cache (
	20	user_id TEXT NOT NULL,
	21	displayname TEXT,
	22	avatar_url TEXT,
	23	last_check BIGINT NOT NULL
	24	);
	25
	26	CREATE UNIQUE INDEX remote_profile_cache_user_id ON remote_profile_cache(user_id);
	27	CREATE INDEX remote_profile_cache_time ON remote_profile_cache(last_check);

+17

-0

synapse/storage/databases/main/schema/delta/46/drop_refresh_tokens.sql less more

	0	/* Copyright 2017 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* we no longer use (or create) the refresh_tokens table */
	16	DROP TABLE IF EXISTS refresh_tokens;

+35

-0

synapse/storage/databases/main/schema/delta/46/drop_unique_deleted_pushers.sql less more

	0	/* Copyright 2017 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- drop the unique constraint on deleted_pushers so that we can just insert
	16	-- into it rather than upserting.
	17
	18	CREATE TABLE deleted_pushers2 (
	19	stream_id BIGINT NOT NULL,
	20	app_id TEXT NOT NULL,
	21	pushkey TEXT NOT NULL,
	22	user_id TEXT NOT NULL
	23	);
	24
	25	INSERT INTO deleted_pushers2 (stream_id, app_id, pushkey, user_id)
	26	SELECT stream_id, app_id, pushkey, user_id from deleted_pushers;
	27
	28	DROP TABLE deleted_pushers;
	29	ALTER TABLE deleted_pushers2 RENAME TO deleted_pushers;
	30
	31	-- create the index after doing the inserts because that's more efficient.
	32	-- it also means we can give it the same name as the old one without renaming.
	33	CREATE INDEX deleted_pushers_stream_id ON deleted_pushers (stream_id);
	34

+32

-0

synapse/storage/databases/main/schema/delta/46/group_server.sql less more

	0	/* Copyright 2017 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE groups_new (
	16	group_id TEXT NOT NULL,
	17	name TEXT, -- the display name of the room
	18	avatar_url TEXT,
	19	short_description TEXT,
	20	long_description TEXT,
	21	is_public BOOL NOT NULL -- whether non-members can access group APIs
	22	);
	23
	24	-- NB: awful hack to get the default to be true on postgres and 1 on sqlite
	25	INSERT INTO groups_new
	26	SELECT group_id, name, avatar_url, short_description, long_description, (1=1) FROM groups;
	27
	28	DROP TABLE groups;
	29	ALTER TABLE groups_new RENAME TO groups;
	30
	31	CREATE UNIQUE INDEX groups_idx ON groups(group_id);

+24

-0

synapse/storage/databases/main/schema/delta/46/local_media_repository_url_idx.sql less more

	0	/* Copyright 2017 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- register a background update which will recreate the
	16	-- local_media_repository_url_idx index.
	17	--
	18	-- We do this as a bg update not because it is a particularly onerous
	19	-- operation, but because we'd like it to be a partial index if possible, and
	20	-- the background_index_update code will understand whether we are on
	21	-- postgres or sqlite and behave accordingly.
	22	INSERT INTO background_updates (update_name, progress_json) VALUES
	23	('local_media_repository_url_idx', '{}');

+35

-0

synapse/storage/databases/main/schema/delta/46/user_dir_null_room_ids.sql less more

	0	/* Copyright 2017 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- change the user_directory table to also cover global local user profiles
	16	-- rather than just profiles within specific rooms.
	17
	18	CREATE TABLE user_directory2 (
	19	user_id TEXT NOT NULL,
	20	room_id TEXT,
	21	display_name TEXT,
	22	avatar_url TEXT
	23	);
	24
	25	INSERT INTO user_directory2(user_id, room_id, display_name, avatar_url)
	26	SELECT user_id, room_id, display_name, avatar_url from user_directory;
	27
	28	DROP TABLE user_directory;
	29	ALTER TABLE user_directory2 RENAME TO user_directory;
	30
	31	-- create indexes after doing the inserts because that's more efficient.
	32	-- it also means we can give it the same name as the old one without renaming.
	33	CREATE INDEX user_directory_room_idx ON user_directory(room_id);
	34	CREATE UNIQUE INDEX user_directory_user_idx ON user_directory(user_id);

+24

-0

synapse/storage/databases/main/schema/delta/46/user_dir_typos.sql less more

	0	/* Copyright 2017 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- this is just embarassing :\|
	16	ALTER TABLE users_in_pubic_room RENAME TO users_in_public_rooms;
	17
	18	-- this is only 300K rows on matrix.org and takes ~3s to generate the index,
	19	-- so is hopefully not going to block anyone else for that long...
	20	CREATE INDEX users_in_public_rooms_room_idx ON users_in_public_rooms(room_id);
	21	CREATE UNIQUE INDEX users_in_public_rooms_user_idx ON users_in_public_rooms(user_id);
	22	DROP INDEX users_in_pubic_room_room_idx;
	23	DROP INDEX users_in_pubic_room_user_idx;

+16

-0

synapse/storage/databases/main/schema/delta/47/last_access_media.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE local_media_repository ADD COLUMN last_access_ts BIGINT;

+17

-0

synapse/storage/databases/main/schema/delta/47/postgres_fts_gin.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT into background_updates (update_name, progress_json)
	16	VALUES ('event_search_postgres_gin', '{}');

+28

-0

synapse/storage/databases/main/schema/delta/47/push_actions_staging.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Temporary staging area for push actions that have been calculated for an
	16	-- event, but the event hasn't yet been persisted.
	17	-- When the event is persisted the rows are moved over to the
	18	-- event_push_actions table.
	19	CREATE TABLE event_push_actions_staging (
	20	event_id TEXT NOT NULL,
	21	user_id TEXT NOT NULL,
	22	actions TEXT NOT NULL,
	23	notif SMALLINT NOT NULL,
	24	highlight SMALLINT NOT NULL
	25	);
	26
	27	CREATE INDEX event_push_actions_staging_id ON event_push_actions_staging(event_id);

+18

-0

synapse/storage/databases/main/schema/delta/48/add_user_consent.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* record the version of the privacy policy the user has consented to
	16	*/
	17	ALTER TABLE users ADD COLUMN consent_version TEXT;

+17

-0

synapse/storage/databases/main/schema/delta/48/add_user_ips_last_seen_index.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT into background_updates (update_name, progress_json)
	16	VALUES ('user_ips_last_seen_index', '{}');

+25

-0

synapse/storage/databases/main/schema/delta/48/deactivated_users.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/*
	16	* Store any accounts that have been requested to be deactivated.
	17	* We part the account from all the rooms its in when its
	18	* deactivated. This can take some time and synapse may be restarted
	19	* before it completes, so store the user IDs here until the process
	20	* is complete.
	21	*/
	22	CREATE TABLE users_pending_deactivation (
	23	user_id TEXT NOT NULL
	24	);

+63

-0

synapse/storage/databases/main/schema/delta/48/group_unique_indexes.py less more

	0	# Copyright 2018 New Vector Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14	from synapse.storage.engines import PostgresEngine
	15	from synapse.storage.prepare_database import get_statements
	16
	17	FIX_INDEXES = """
	18	-- rebuild indexes as uniques
	19	DROP INDEX groups_invites_g_idx;
	20	CREATE UNIQUE INDEX group_invites_g_idx ON group_invites(group_id, user_id);
	21	DROP INDEX groups_users_g_idx;
	22	CREATE UNIQUE INDEX group_users_g_idx ON group_users(group_id, user_id);
	23
	24	-- rename other indexes to actually match their table names..
	25	DROP INDEX groups_users_u_idx;
	26	CREATE INDEX group_users_u_idx ON group_users(user_id);
	27	DROP INDEX groups_invites_u_idx;
	28	CREATE INDEX group_invites_u_idx ON group_invites(user_id);
	29	DROP INDEX groups_rooms_g_idx;
	30	CREATE UNIQUE INDEX group_rooms_g_idx ON group_rooms(group_id, room_id);
	31	DROP INDEX groups_rooms_r_idx;
	32	CREATE INDEX group_rooms_r_idx ON group_rooms(room_id);
	33	"""
	34
	35
	36	def run_create(cur, database_engine, args, *kwargs):
	37	rowid = "ctid" if isinstance(database_engine, PostgresEngine) else "rowid"
	38
	39	# remove duplicates from group_users & group_invites tables
	40	cur.execute(
	41	"""
	42	DELETE FROM group_users WHERE %s NOT IN (
	43	SELECT min(%s) FROM group_users GROUP BY group_id, user_id
	44	);
	45	"""
	46	% (rowid, rowid)
	47	)
	48	cur.execute(
	49	"""
	50	DELETE FROM group_invites WHERE %s NOT IN (
	51	SELECT min(%s) FROM group_invites GROUP BY group_id, user_id
	52	);
	53	"""
	54	% (rowid, rowid)
	55	)
	56
	57	for statement in get_statements(FIX_INDEXES.splitlines()):
	58	cur.execute(statement)
	59
	60
	61	def run_upgrade(args, *kwargs):
	62	pass

+22

-0

synapse/storage/databases/main/schema/delta/48/groups_joinable.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/*
	16	* This isn't a real ENUM because sqlite doesn't support it
	17	* and we use a default of NULL for inserted rows and interpret
	18	* NULL at the python store level as necessary so that existing
	19	* rows are given the correct default policy.
	20	*/
	21	ALTER TABLE groups ADD COLUMN join_policy TEXT NOT NULL DEFAULT 'invite';

+20

-0

synapse/storage/databases/main/schema/delta/49/add_user_consent_server_notice_sent.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* record whether we have sent a server notice about consenting to the
	16	* privacy policy. Specifically records the version of the policy we sent
	17	* a message about.
	18	*/
	19	ALTER TABLE users ADD COLUMN consent_server_notice_sent TEXT;

+21

-0

synapse/storage/databases/main/schema/delta/49/add_user_daily_visits.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	CREATE TABLE user_daily_visits ( user_id TEXT NOT NULL,
	17	device_id TEXT,
	18	timestamp BIGINT NOT NULL );
	19	CREATE INDEX user_daily_visits_uts_idx ON user_daily_visits(user_id, timestamp);
	20	CREATE INDEX user_daily_visits_ts_idx ON user_daily_visits(timestamp);

+17

-0

synapse/storage/databases/main/schema/delta/49/add_user_ips_last_seen_only_index.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT into background_updates (update_name, progress_json)
	16	VALUES ('user_ips_last_seen_only_index', '{}');

+19

-0

synapse/storage/databases/main/schema/delta/50/add_creation_ts_users_index.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16
	17	INSERT into background_updates (update_name, progress_json)
	18	VALUES ('users_creation_ts', '{}');

+21

-0

synapse/storage/databases/main/schema/delta/50/erasure_store.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- a table of users who have requested that their details be erased
	16	CREATE TABLE erased_users (
	17	user_id TEXT NOT NULL
	18	);
	19
	20	CREATE UNIQUE INDEX erased_users_user ON erased_users(user_id);

+96

-0

synapse/storage/databases/main/schema/delta/50/make_event_content_nullable.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2018 New Vector Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	"""
	16	We want to stop populating 'event.content', so we need to make it nullable.
	17
	18	If this has to be rolled back, then the following should populate the missing data:
	19
	20	Postgres:
	21
	22	UPDATE events SET content=(ej.json::json)->'content' FROM event_json ej
	23	WHERE ej.event_id = events.event_id AND
	24	stream_ordering < (
	25	SELECT stream_ordering FROM events WHERE content IS NOT NULL
	26	ORDER BY stream_ordering LIMIT 1
	27	);
	28
	29	UPDATE events SET content=(ej.json::json)->'content' FROM event_json ej
	30	WHERE ej.event_id = events.event_id AND
	31	stream_ordering > (
	32	SELECT stream_ordering FROM events WHERE content IS NOT NULL
	33	ORDER BY stream_ordering DESC LIMIT 1
	34	);
	35
	36	SQLite:
	37
	38	UPDATE events SET content=(
	39	SELECT json_extract(json,'$.content') FROM event_json ej
	40	WHERE ej.event_id = events.event_id
	41	)
	42	WHERE
	43	stream_ordering < (
	44	SELECT stream_ordering FROM events WHERE content IS NOT NULL
	45	ORDER BY stream_ordering LIMIT 1
	46	)
	47	OR stream_ordering > (
	48	SELECT stream_ordering FROM events WHERE content IS NOT NULL
	49	ORDER BY stream_ordering DESC LIMIT 1
	50	);
	51
	52	"""
	53
	54	import logging
	55
	56	from synapse.storage.engines import PostgresEngine
	57
	58	logger = logging.getLogger(__name__)
	59
	60
	61	def run_create(cur, database_engine, args, *kwargs):
	62	pass
	63
	64
	65	def run_upgrade(cur, database_engine, args, *kwargs):
	66	if isinstance(database_engine, PostgresEngine):
	67	cur.execute(
	68	"""
	69	ALTER TABLE events ALTER COLUMN content DROP NOT NULL;
	70	"""
	71	)
	72	return
	73
	74	# sqlite is an arse about this. ref: https://www.sqlite.org/lang_altertable.html
	75
	76	cur.execute(
	77	"SELECT sql FROM sqlite_master WHERE tbl_name='events' AND type='table'"
	78	)
	79	(oldsql,) = cur.fetchone()
	80
	81	sql = oldsql.replace("content TEXT NOT NULL", "content TEXT")
	82	if sql == oldsql:
	83	raise Exception("Couldn't find null constraint to drop in %s" % oldsql)
	84
	85	logger.info("Replacing definition of 'events' with: %s", sql)
	86
	87	cur.execute("PRAGMA schema_version")
	88	(oldver,) = cur.fetchone()
	89	cur.execute("PRAGMA writable_schema=ON")
	90	cur.execute(
	91	"UPDATE sqlite_master SET sql=? WHERE tbl_name='events' AND type='table'",
	92	(sql,),
	93	)
	94	cur.execute("PRAGMA schema_version=%i" % (oldver + 1,))
	95	cur.execute("PRAGMA writable_schema=OFF")

+39

-0

synapse/storage/databases/main/schema/delta/51/e2e_room_keys.sql less more

	0	/* Copyright 2017 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- users' optionally backed up encrypted e2e sessions
	16	CREATE TABLE e2e_room_keys (
	17	user_id TEXT NOT NULL,
	18	room_id TEXT NOT NULL,
	19	session_id TEXT NOT NULL,
	20	version TEXT NOT NULL,
	21	first_message_index INT,
	22	forwarded_count INT,
	23	is_verified BOOLEAN,
	24	session_data TEXT NOT NULL
	25	);
	26
	27	CREATE UNIQUE INDEX e2e_room_keys_idx ON e2e_room_keys(user_id, room_id, session_id);
	28
	29	-- the metadata for each generation of encrypted e2e session backups
	30	CREATE TABLE e2e_room_keys_versions (
	31	user_id TEXT NOT NULL,
	32	version TEXT NOT NULL,
	33	algorithm TEXT NOT NULL,
	34	auth_data TEXT NOT NULL,
	35	deleted SMALLINT DEFAULT 0 NOT NULL
	36	);
	37
	38	CREATE UNIQUE INDEX e2e_room_keys_versions_idx ON e2e_room_keys_versions(user_id, version);

+27

-0

synapse/storage/databases/main/schema/delta/51/monthly_active_users.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- a table of monthly active users, for use where blocking based on mau limits
	16	CREATE TABLE monthly_active_users (
	17	user_id TEXT NOT NULL,
	18	-- Last time we saw the user. Not guaranteed to be accurate due to rate limiting
	19	-- on updates, Granularity of updates governed by
	20	-- synapse.storage.monthly_active_users.LAST_SEEN_GRANULARITY
	21	-- Measured in ms since epoch.
	22	timestamp BIGINT NOT NULL
	23	);
	24
	25	CREATE UNIQUE INDEX monthly_active_users_users ON monthly_active_users(user_id);
	26	CREATE INDEX monthly_active_users_time_stamp ON monthly_active_users(timestamp);

+19

-0

synapse/storage/databases/main/schema/delta/52/add_event_to_state_group_index.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- This is needed to efficiently check for unreferenced state groups during
	16	-- purge. Added events_to_state_group(state_group) index
	17	INSERT into background_updates (update_name, progress_json)
	18	VALUES ('event_to_state_groups_sg_index', '{}');

+36

-0

synapse/storage/databases/main/schema/delta/52/device_list_streams_unique_idx.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- register a background update which will create a unique index on
	16	-- device_lists_remote_cache
	17	INSERT into background_updates (update_name, progress_json)
	18	VALUES ('device_lists_remote_cache_unique_idx', '{}');
	19
	20	-- and one on device_lists_remote_extremeties
	21	INSERT into background_updates (update_name, progress_json, depends_on)
	22	VALUES (
	23	'device_lists_remote_extremeties_unique_idx', '{}',
	24
	25	-- doesn't really depend on this, but we need to make sure both happen
	26	-- before we drop the old indexes.
	27	'device_lists_remote_cache_unique_idx'
	28	);
	29
	30	-- once they complete, we can drop the old indexes.
	31	INSERT into background_updates (update_name, progress_json, depends_on)
	32	VALUES (
	33	'drop_device_list_streams_non_unique_indexes', '{}',
	34	'device_lists_remote_extremeties_unique_idx'
	35	);

+53

-0

synapse/storage/databases/main/schema/delta/52/e2e_room_keys.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* Change version column to an integer so we can do MAX() sensibly
	16	*/
	17	CREATE TABLE e2e_room_keys_versions_new (
	18	user_id TEXT NOT NULL,
	19	version BIGINT NOT NULL,
	20	algorithm TEXT NOT NULL,
	21	auth_data TEXT NOT NULL,
	22	deleted SMALLINT DEFAULT 0 NOT NULL
	23	);
	24
	25	INSERT INTO e2e_room_keys_versions_new
	26	SELECT user_id, CAST(version as BIGINT), algorithm, auth_data, deleted FROM e2e_room_keys_versions;
	27
	28	DROP TABLE e2e_room_keys_versions;
	29	ALTER TABLE e2e_room_keys_versions_new RENAME TO e2e_room_keys_versions;
	30
	31	CREATE UNIQUE INDEX e2e_room_keys_versions_idx ON e2e_room_keys_versions(user_id, version);
	32
	33	/* Change e2e_rooms_keys to match
	34	*/
	35	CREATE TABLE e2e_room_keys_new (
	36	user_id TEXT NOT NULL,
	37	room_id TEXT NOT NULL,
	38	session_id TEXT NOT NULL,
	39	version BIGINT NOT NULL,
	40	first_message_index INT,
	41	forwarded_count INT,
	42	is_verified BOOLEAN,
	43	session_data TEXT NOT NULL
	44	);
	45
	46	INSERT INTO e2e_room_keys_new
	47	SELECT user_id, room_id, session_id, CAST(version as BIGINT), first_message_index, forwarded_count, is_verified, session_data FROM e2e_room_keys;
	48
	49	DROP TABLE e2e_room_keys;
	50	ALTER TABLE e2e_room_keys_new RENAME TO e2e_room_keys;
	51
	52	CREATE UNIQUE INDEX e2e_room_keys_idx ON e2e_room_keys(user_id, room_id, session_id);

+19

-0

synapse/storage/databases/main/schema/delta/53/add_user_type_to_users.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* The type of the user: NULL for a regular user, or one of the constants in
	16	* synapse.api.constants.UserTypes
	17	*/
	18	ALTER TABLE users ADD COLUMN user_type TEXT DEFAULT NULL;

+16

-0

synapse/storage/databases/main/schema/delta/53/drop_sent_transactions.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	DROP TABLE IF EXISTS sent_transactions;

+16

-0

synapse/storage/databases/main/schema/delta/53/event_format_version.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE event_json ADD COLUMN format_version INTEGER;

+30

-0

synapse/storage/databases/main/schema/delta/53/user_dir_populate.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Set up staging tables
	16	INSERT INTO background_updates (update_name, progress_json) VALUES
	17	('populate_user_directory_createtables', '{}');
	18
	19	-- Run through each room and update the user directory according to who is in it
	20	INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
	21	('populate_user_directory_process_rooms', '{}', 'populate_user_directory_createtables');
	22
	23	-- Insert all users, if search_all_users is on
	24	INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
	25	('populate_user_directory_process_users', '{}', 'populate_user_directory_process_rooms');
	26
	27	-- Clean up staging tables
	28	INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
	29	('populate_user_directory_cleanup', '{}', 'populate_user_directory_process_users');

+30

-0

synapse/storage/databases/main/schema/delta/53/user_ips_index.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- analyze user_ips, to help ensure the correct indices are used
	16	INSERT INTO background_updates (update_name, progress_json) VALUES
	17	('user_ips_analyze', '{}');
	18
	19	-- delete duplicates
	20	INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
	21	('user_ips_remove_dupes', '{}', 'user_ips_analyze');
	22
	23	-- add a new unique index to user_ips table
	24	INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
	25	('user_ips_device_unique_index', '{}', 'user_ips_remove_dupes');
	26
	27	-- drop the old original index
	28	INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
	29	('user_ips_drop_nonunique_index', '{}', 'user_ips_device_unique_index');

+44

-0

synapse/storage/databases/main/schema/delta/53/user_share.sql less more

	0	/* Copyright 2017 Vector Creations Ltd, 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Old disused version of the tables below.
	16	DROP TABLE IF EXISTS users_who_share_rooms;
	17
	18	-- Tables keeping track of what users share rooms. This is a map of local users
	19	-- to local or remote users, per room. Remote users cannot be in the user_id
	20	-- column, only the other_user_id column. There are two tables, one for public
	21	-- rooms and those for private rooms.
	22	CREATE TABLE IF NOT EXISTS users_who_share_public_rooms (
	23	user_id TEXT NOT NULL,
	24	other_user_id TEXT NOT NULL,
	25	room_id TEXT NOT NULL
	26	);
	27
	28	CREATE TABLE IF NOT EXISTS users_who_share_private_rooms (
	29	user_id TEXT NOT NULL,
	30	other_user_id TEXT NOT NULL,
	31	room_id TEXT NOT NULL
	32	);
	33
	34	CREATE UNIQUE INDEX users_who_share_public_rooms_u_idx ON users_who_share_public_rooms(user_id, other_user_id, room_id);
	35	CREATE INDEX users_who_share_public_rooms_r_idx ON users_who_share_public_rooms(room_id);
	36	CREATE INDEX users_who_share_public_rooms_o_idx ON users_who_share_public_rooms(other_user_id);
	37
	38	CREATE UNIQUE INDEX users_who_share_private_rooms_u_idx ON users_who_share_private_rooms(user_id, other_user_id, room_id);
	39	CREATE INDEX users_who_share_private_rooms_r_idx ON users_who_share_private_rooms(room_id);
	40	CREATE INDEX users_who_share_private_rooms_o_idx ON users_who_share_private_rooms(other_user_id);
	41
	42	-- Make sure that we populate the tables initially by resetting the stream ID
	43	UPDATE user_directory_stream_pos SET stream_id = NULL;

+29

-0

synapse/storage/databases/main/schema/delta/53/user_threepid_id.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Tracks which identity server a user bound their threepid via.
	16	CREATE TABLE user_threepid_id_server (
	17	user_id TEXT NOT NULL,
	18	medium TEXT NOT NULL,
	19	address TEXT NOT NULL,
	20	id_server TEXT NOT NULL
	21	);
	22
	23	CREATE UNIQUE INDEX user_threepid_id_server_idx ON user_threepid_id_server(
	24	user_id, medium, address, id_server
	25	);
	26
	27	INSERT INTO background_updates (update_name, progress_json) VALUES
	28	('user_threepids_grandfather', '{}');

+28

-0

synapse/storage/databases/main/schema/delta/53/users_in_public_rooms.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- We don't need the old version of this table.
	16	DROP TABLE IF EXISTS users_in_public_rooms;
	17
	18	-- Old version of users_in_public_rooms
	19	DROP TABLE IF EXISTS users_who_share_public_rooms;
	20
	21	-- Track what users are in public rooms.
	22	CREATE TABLE IF NOT EXISTS users_in_public_rooms (
	23	user_id TEXT NOT NULL,
	24	room_id TEXT NOT NULL
	25	);
	26
	27	CREATE UNIQUE INDEX users_in_public_rooms_u_idx ON users_in_public_rooms(user_id, room_id);

+30

-0

synapse/storage/databases/main/schema/delta/54/account_validity_with_renewal.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- We previously changed the schema for this table without renaming the file, which means
	16	-- that some databases might still be using the old schema. This ensures Synapse uses the
	17	-- right schema for the table.
	18	DROP TABLE IF EXISTS account_validity;
	19
	20	-- Track what users are in public rooms.
	21	CREATE TABLE IF NOT EXISTS account_validity (
	22	user_id TEXT PRIMARY KEY,
	23	expiration_ts_ms BIGINT NOT NULL,
	24	email_sent BOOLEAN NOT NULL,
	25	renewal_token TEXT
	26	);
	27
	28	CREATE INDEX account_validity_email_sent_idx ON account_validity(email_sent, expiration_ts_ms)
	29	CREATE UNIQUE INDEX account_validity_renewal_string_idx ON account_validity(renewal_token)

+23

-0

synapse/storage/databases/main/schema/delta/54/add_validity_to_server_keys.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* When we can use this key until, before we have to refresh it. */
	16	ALTER TABLE server_signature_keys ADD COLUMN ts_valid_until_ms BIGINT;
	17
	18	UPDATE server_signature_keys SET ts_valid_until_ms = (
	19	SELECT MAX(ts_valid_until_ms) FROM server_keys_json skj WHERE
	20	skj.server_name = server_signature_keys.server_name AND
	21	skj.key_id = server_signature_keys.key_id
	22	);

+23

-0

synapse/storage/databases/main/schema/delta/54/delete_forward_extremities.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Start a background job to cleanup extremities that were incorrectly added
	16	-- by bug #5269.
	17	INSERT INTO background_updates (update_name, progress_json) VALUES
	18	('delete_soft_failed_extremities', '{}');
	19
	20	DROP TABLE IF EXISTS _extremities_to_check; -- To make this delta schema file idempotent.
	21	CREATE TABLE _extremities_to_check AS SELECT event_id FROM event_forward_extremities;
	22	CREATE INDEX _extremities_to_check_id ON _extremities_to_check(event_id);

+30

-0

synapse/storage/databases/main/schema/delta/54/drop_legacy_tables.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- we need to do this first due to foreign constraints
	16	DROP TABLE IF EXISTS application_services_regex;
	17
	18	DROP TABLE IF EXISTS application_services;
	19	DROP TABLE IF EXISTS transaction_id_to_pdu;
	20	DROP TABLE IF EXISTS stats_reporting;
	21	DROP TABLE IF EXISTS current_state_resets;
	22	DROP TABLE IF EXISTS event_content_hashes;
	23	DROP TABLE IF EXISTS event_destinations;
	24	DROP TABLE IF EXISTS event_edge_hashes;
	25	DROP TABLE IF EXISTS event_signatures;
	26	DROP TABLE IF EXISTS feedback;
	27	DROP TABLE IF EXISTS room_hosts;
	28	DROP TABLE IF EXISTS server_tls_certificates;
	29	DROP TABLE IF EXISTS state_forward_extremities;

+16

-0

synapse/storage/databases/main/schema/delta/54/drop_presence_list.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	DROP TABLE IF EXISTS presence_list;

+27

-0

synapse/storage/databases/main/schema/delta/54/relations.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Tracks related events, like reactions, replies, edits, etc. Note that things
	16	-- in this table are not necessarily "valid", e.g. it may contain edits from
	17	-- people who don't have power to edit other peoples events.
	18	CREATE TABLE IF NOT EXISTS event_relations (
	19	event_id TEXT NOT NULL,
	20	relates_to_id TEXT NOT NULL,
	21	relation_type TEXT NOT NULL,
	22	aggregation_key TEXT
	23	);
	24
	25	CREATE UNIQUE INDEX event_relations_id ON event_relations(event_id);
	26	CREATE INDEX event_relations_relates ON event_relations(relates_to_id, relation_type, aggregation_key);

+80

-0

synapse/storage/databases/main/schema/delta/54/stats.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE stats_stream_pos (
	16	Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
	17	stream_id BIGINT,
	18	CHECK (Lock='X')
	19	);
	20
	21	INSERT INTO stats_stream_pos (stream_id) VALUES (null);
	22
	23	CREATE TABLE user_stats (
	24	user_id TEXT NOT NULL,
	25	ts BIGINT NOT NULL,
	26	bucket_size INT NOT NULL,
	27	public_rooms INT NOT NULL,
	28	private_rooms INT NOT NULL
	29	);
	30
	31	CREATE UNIQUE INDEX user_stats_user_ts ON user_stats(user_id, ts);
	32
	33	CREATE TABLE room_stats (
	34	room_id TEXT NOT NULL,
	35	ts BIGINT NOT NULL,
	36	bucket_size INT NOT NULL,
	37	current_state_events INT NOT NULL,
	38	joined_members INT NOT NULL,
	39	invited_members INT NOT NULL,
	40	left_members INT NOT NULL,
	41	banned_members INT NOT NULL,
	42	state_events INT NOT NULL
	43	);
	44
	45	CREATE UNIQUE INDEX room_stats_room_ts ON room_stats(room_id, ts);
	46
	47	-- cache of current room state; useful for the publicRooms list
	48	CREATE TABLE room_state (
	49	room_id TEXT NOT NULL,
	50	join_rules TEXT,
	51	history_visibility TEXT,
	52	encryption TEXT,
	53	name TEXT,
	54	topic TEXT,
	55	avatar TEXT,
	56	canonical_alias TEXT
	57	-- get aliases straight from the right table
	58	);
	59
	60	CREATE UNIQUE INDEX room_state_room ON room_state(room_id);
	61
	62	CREATE TABLE room_stats_earliest_token (
	63	room_id TEXT NOT NULL,
	64	token BIGINT NOT NULL
	65	);
	66
	67	CREATE UNIQUE INDEX room_stats_earliest_token_idx ON room_stats_earliest_token(room_id);
	68
	69	-- Set up staging tables
	70	INSERT INTO background_updates (update_name, progress_json) VALUES
	71	('populate_stats_createtables', '{}');
	72
	73	-- Run through each room and update stats
	74	INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
	75	('populate_stats_process_rooms', '{}', 'populate_stats_createtables');
	76
	77	-- Clean up staging tables
	78	INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
	79	('populate_stats_cleanup', '{}', 'populate_stats_process_rooms');

+28

-0

synapse/storage/databases/main/schema/delta/54/stats2.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- This delta file gets run after `54/stats.sql` delta.
	16
	17	-- We want to add some indices to the temporary stats table, so we re-insert
	18	-- 'populate_stats_createtables' if we are still processing the rooms update.
	19	INSERT INTO background_updates (update_name, progress_json)
	20	SELECT 'populate_stats_createtables', '{}'
	21	WHERE
	22	'populate_stats_process_rooms' IN (
	23	SELECT update_name FROM background_updates
	24	)
	25	AND 'populate_stats_createtables' NOT IN ( -- don't insert if already exists
	26	SELECT update_name FROM background_updates
	27	);

+18

-0

synapse/storage/databases/main/schema/delta/55/access_token_expiry.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- when this access token can be used until, in ms since the epoch. NULL means the token
	16	-- never expires.
	17	ALTER TABLE access_tokens ADD COLUMN valid_until_ms BIGINT;

+31

-0

synapse/storage/databases/main/schema/delta/55/track_threepid_validations.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14	CREATE TABLE IF NOT EXISTS threepid_validation_session (
	15	session_id TEXT PRIMARY KEY,
	16	medium TEXT NOT NULL,
	17	address TEXT NOT NULL,
	18	client_secret TEXT NOT NULL,
	19	last_send_attempt BIGINT NOT NULL,
	20	validated_at BIGINT
	21	);
	22
	23	CREATE TABLE IF NOT EXISTS threepid_validation_token (
	24	token TEXT PRIMARY KEY,
	25	session_id TEXT NOT NULL,
	26	next_link TEXT,
	27	expires BIGINT NOT NULL
	28	);
	29
	30	CREATE INDEX threepid_validation_token_session_id ON threepid_validation_token(session_id);

+19

-0

synapse/storage/databases/main/schema/delta/55/users_alter_deactivated.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE users ADD deactivated SMALLINT DEFAULT 0 NOT NULL;
	16
	17	INSERT INTO background_updates (update_name, progress_json) VALUES
	18	('users_set_deactivated_flag', '{}');

+20

-0

synapse/storage/databases/main/schema/delta/56/add_spans_to_device_lists.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/*
	16	* Opentracing context data for inclusion in the device_list_update EDUs, as a
	17	* json-encoded dictionary. NULL if opentracing is disabled (or not enabled for this destination).
	18	*/
	19	ALTER TABLE device_lists_outbound_pokes ADD opentracing_context TEXT;

+22

-0

synapse/storage/databases/main/schema/delta/56/current_state_events_membership.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- We add membership to current state so that we don't need to join against
	16	-- room_memberships, which can be surprisingly costly (we do such queries
	17	-- very frequently).
	18	-- This will be null for non-membership events and the content.membership key
	19	-- for membership events. (Will also be null for membership events until the
	20	-- background update job has finished).
	21	ALTER TABLE current_state_events ADD membership TEXT;

+24

-0

synapse/storage/databases/main/schema/delta/56/current_state_events_membership_mk2.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- We add membership to current state so that we don't need to join against
	16	-- room_memberships, which can be surprisingly costly (we do such queries
	17	-- very frequently).
	18	-- This will be null for non-membership events and the content.membership key
	19	-- for membership events. (Will also be null for membership events until the
	20	-- background update job has finished).
	21
	22	INSERT INTO background_updates (update_name, progress_json) VALUES
	23	('current_state_events_membership', '{}');

+25

-0

synapse/storage/databases/main/schema/delta/56/delete_keys_from_deleted_backups.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* delete room keys that belong to deleted room key version, or to room key
	16	* versions that don't exist (anymore)
	17	*/
	18	DELETE FROM e2e_room_keys
	19	WHERE version NOT IN (
	20	SELECT version
	21	FROM e2e_room_keys_versions
	22	WHERE e2e_room_keys.user_id = e2e_room_keys_versions.user_id
	23	AND e2e_room_keys_versions.deleted = 0
	24	);

+25

-0

synapse/storage/databases/main/schema/delta/56/destinations_failure_ts.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/*
	16	* Record the timestamp when a given server started failing
	17	*/
	18	ALTER TABLE destinations ADD failure_ts BIGINT;
	19
	20	/* as a rough approximation, we assume that the server started failing at
	21	* retry_interval before the last retry
	22	*/
	23	UPDATE destinations SET failure_ts = retry_last_ts - retry_interval
	24	WHERE retry_last_ts > 0;

+18

-0

synapse/storage/databases/main/schema/delta/56/destinations_retry_interval_type.sql.postgres less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- We want to store large retry intervals so we upgrade the column from INT
	16	-- to BIGINT. We don't need to do this on SQLite.
	17	ALTER TABLE destinations ALTER retry_interval SET DATA TYPE BIGINT;

+20

-0

synapse/storage/databases/main/schema/delta/56/device_stream_id_insert.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- This line already existed in deltas/35/device_stream_id but was not included in the
	16	-- 54 full schema SQL. Add some SQL here to insert the missing row if it does not exist
	17	INSERT INTO device_max_stream_id (stream_id) SELECT 0 WHERE NOT EXISTS (
	18	SELECT * from device_max_stream_id
	19	);⏎

+24

-0

synapse/storage/databases/main/schema/delta/56/devices_last_seen.sql less more

	0	/* Copyright 2019 Matrix.org Foundation CIC
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Track last seen information for a device in the devices table, rather
	16	-- than relying on it being in the user_ips table (which we want to be able
	17	-- to purge old entries from)
	18	ALTER TABLE devices ADD COLUMN last_seen BIGINT;
	19	ALTER TABLE devices ADD COLUMN ip TEXT;
	20	ALTER TABLE devices ADD COLUMN user_agent TEXT;
	21
	22	INSERT INTO background_updates (update_name, progress_json) VALUES
	23	('devices_last_seen', '{}');

+20

-0

synapse/storage/databases/main/schema/delta/56/drop_unused_event_tables.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- these tables are never used.
	16	DROP TABLE IF EXISTS room_names;
	17	DROP TABLE IF EXISTS topics;
	18	DROP TABLE IF EXISTS history_visibility;
	19	DROP TABLE IF EXISTS guest_access;

+21

-0

synapse/storage/databases/main/schema/delta/56/event_expiry.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS event_expiry (
	16	event_id TEXT PRIMARY KEY,
	17	expiry_ts BIGINT NOT NULL
	18	);
	19
	20	CREATE INDEX event_expiry_expiry_ts_idx ON event_expiry(expiry_ts);

+30

-0

synapse/storage/databases/main/schema/delta/56/event_labels.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- room_id and topoligical_ordering are denormalised from the events table in order to
	16	-- make the index work.
	17	CREATE TABLE IF NOT EXISTS event_labels (
	18	event_id TEXT,
	19	label TEXT,
	20	room_id TEXT NOT NULL,
	21	topological_ordering BIGINT NOT NULL,
	22	PRIMARY KEY(event_id, label)
	23	);
	24
	25
	26	-- This index enables an event pagination looking for a particular label to index the
	27	-- event_labels table first, which is much quicker than scanning the events table and then
	28	-- filtering by label, if the label is rarely used relative to the size of the room.
	29	CREATE INDEX event_labels_room_id_label_idx ON event_labels(room_id, label, topological_ordering);

+17

-0

synapse/storage/databases/main/schema/delta/56/event_labels_background_update.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT INTO background_updates (update_name, progress_json) VALUES
	16	('event_store_labels', '{}');

+18

-0

synapse/storage/databases/main/schema/delta/56/fix_room_keys_index.sql less more

	0	/* Copyright 2019 Matrix.org Foundation CIC
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- version is supposed to be part of the room keys index
	16	CREATE UNIQUE INDEX e2e_room_keys_with_version_idx ON e2e_room_keys(user_id, version, room_id, session_id);
	17	DROP INDEX IF EXISTS e2e_room_keys_idx;

+18

-0

synapse/storage/databases/main/schema/delta/56/hidden_devices.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- device list needs to know which ones are "real" devices, and which ones are
	16	-- just used to avoid collisions
	17	ALTER TABLE devices ADD COLUMN hidden BOOLEAN DEFAULT FALSE;

+42

-0

synapse/storage/databases/main/schema/delta/56/hidden_devices_fix.sql.sqlite less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* Change the hidden column from a default value of FALSE to a default value of
	16	* 0, because sqlite3 prior to 3.23.0 caused the hidden column to contain the
	17	* string 'FALSE', which is truthy.
	18	*
	19	* Since sqlite doesn't allow us to just change the default value, we have to
	20	* recreate the table, copy the data, fix the rows that have incorrect data, and
	21	* replace the old table with the new table.
	22	*/
	23
	24	CREATE TABLE IF NOT EXISTS devices2 (
	25	user_id TEXT NOT NULL,
	26	device_id TEXT NOT NULL,
	27	display_name TEXT,
	28	last_seen BIGINT,
	29	ip TEXT,
	30	user_agent TEXT,
	31	hidden BOOLEAN DEFAULT 0,
	32	CONSTRAINT device_uniqueness UNIQUE (user_id, device_id)
	33	);
	34
	35	INSERT INTO devices2 SELECT * FROM devices;
	36
	37	UPDATE devices2 SET hidden = 0 WHERE hidden = 'FALSE';
	38
	39	DROP TABLE devices;
	40
	41	ALTER TABLE devices2 RENAME TO devices;

+29

-0

synapse/storage/databases/main/schema/delta/56/nuke_empty_communities_from_db.sql less more

	0	/* Copyright 2019 Werner Sembach
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Groups/communities now get deleted when the last member leaves. This is a one time cleanup to remove old groups/communities that were already empty before that change was made.
	16	DELETE FROM group_attestations_remote WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	17	DELETE FROM group_attestations_renewals WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	18	DELETE FROM group_invites WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	19	DELETE FROM group_roles WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	20	DELETE FROM group_room_categories WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	21	DELETE FROM group_rooms WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	22	DELETE FROM group_summary_roles WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	23	DELETE FROM group_summary_room_categories WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	24	DELETE FROM group_summary_rooms WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	25	DELETE FROM group_summary_users WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	26	DELETE FROM local_group_membership WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	27	DELETE FROM local_group_updates WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));
	28	DELETE FROM groups WHERE group_id IN (SELECT group_id FROM groups WHERE NOT EXISTS (SELECT group_id FROM group_users WHERE group_id = groups.group_id));

+16

-0

synapse/storage/databases/main/schema/delta/56/public_room_list_idx.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE INDEX public_room_list_stream_network ON public_room_list_stream (appservice_id, network_id, room_id);

+16

-0

synapse/storage/databases/main/schema/delta/56/redaction_censor.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE redactions ADD COLUMN have_censored BOOL NOT NULL DEFAULT false;

+22

-0

synapse/storage/databases/main/schema/delta/56/redaction_censor2.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	ALTER TABLE redactions ADD COLUMN received_ts BIGINT;
	16
	17	INSERT INTO background_updates (update_name, progress_json) VALUES
	18	('redactions_received_ts', '{}');
	19
	20	INSERT INTO background_updates (update_name, progress_json) VALUES
	21	('redactions_have_censored_ts_idx', '{}');

+25

-0

synapse/storage/databases/main/schema/delta/56/redaction_censor3_fix_update.sql.postgres less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	-- There was a bug where we may have updated censored redactions as bytes,
	17	-- which can (somehow) cause json to be inserted hex encoded. These updates go
	18	-- and undoes any such hex encoded JSON.
	19
	20	INSERT into background_updates (update_name, progress_json)
	21	VALUES ('event_fix_redactions_bytes_create_index', '{}');
	22
	23	INSERT into background_updates (update_name, progress_json, depends_on)
	24	VALUES ('event_fix_redactions_bytes', '{}', 'event_fix_redactions_bytes_create_index');

+16

-0

synapse/storage/databases/main/schema/delta/56/redaction_censor4.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	DROP INDEX IF EXISTS redactions_have_censored;

+18

-0

synapse/storage/databases/main/schema/delta/56/remove_tombstoned_rooms_from_directory.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Now that #6232 is a thing, we can remove old rooms from the directory.
	16	INSERT INTO background_updates (update_name, progress_json) VALUES
	17	('remove_tombstoned_rooms_from_directory', '{}');

+17

-0

synapse/storage/databases/main/schema/delta/56/room_key_etag.sql less more

	0	/* Copyright 2019 Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- store the current etag of backup version
	16	ALTER TABLE e2e_room_keys_versions ADD COLUMN etag BIGINT;

+18

-0

synapse/storage/databases/main/schema/delta/56/room_membership_idx.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Adds an index on room_memberships for fetching all forgotten rooms for a user
	16	INSERT INTO background_updates (update_name, progress_json) VALUES
	17	('room_membership_forgotten_idx', '{}');

+33

-0

synapse/storage/databases/main/schema/delta/56/room_retention.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Tracks the retention policy of a room.
	16	-- A NULL max_lifetime or min_lifetime means that the matching property is not defined in
	17	-- the room's retention policy state event.
	18	-- If a room doesn't have a retention policy state event in its state, both max_lifetime
	19	-- and min_lifetime are NULL.
	20	CREATE TABLE IF NOT EXISTS room_retention(
	21	room_id TEXT,
	22	event_id TEXT,
	23	min_lifetime BIGINT,
	24	max_lifetime BIGINT,
	25
	26	PRIMARY KEY(room_id, event_id)
	27	);
	28
	29	CREATE INDEX room_retention_max_lifetime_idx on room_retention(max_lifetime);
	30
	31	INSERT INTO background_updates (update_name, progress_json) VALUES
	32	('insert_room_retention', '{}');

+56

-0

synapse/storage/databases/main/schema/delta/56/signing_keys.sql less more

	0	/* Copyright 2019 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- cross-signing keys
	16	CREATE TABLE IF NOT EXISTS e2e_cross_signing_keys (
	17	user_id TEXT NOT NULL,
	18	-- the type of cross-signing key (master, user_signing, or self_signing)
	19	keytype TEXT NOT NULL,
	20	-- the full key information, as a json-encoded dict
	21	keydata TEXT NOT NULL,
	22	-- for keeping the keys in order, so that we can fetch the latest one
	23	stream_id BIGINT NOT NULL
	24	);
	25
	26	CREATE UNIQUE INDEX e2e_cross_signing_keys_idx ON e2e_cross_signing_keys(user_id, keytype, stream_id);
	27
	28	-- cross-signing signatures
	29	CREATE TABLE IF NOT EXISTS e2e_cross_signing_signatures (
	30	-- user who did the signing
	31	user_id TEXT NOT NULL,
	32	-- key used to sign
	33	key_id TEXT NOT NULL,
	34	-- user who was signed
	35	target_user_id TEXT NOT NULL,
	36	-- device/key that was signed
	37	target_device_id TEXT NOT NULL,
	38	-- the actual signature
	39	signature TEXT NOT NULL
	40	);
	41
	42	-- replaced by the index created in signing_keys_nonunique_signatures.sql
	43	-- CREATE UNIQUE INDEX e2e_cross_signing_signatures_idx ON e2e_cross_signing_signatures(user_id, target_user_id, target_device_id);
	44
	45	-- stream of user signature updates
	46	CREATE TABLE IF NOT EXISTS user_signature_stream (
	47	-- uses the same stream ID as device list stream
	48	stream_id BIGINT NOT NULL,
	49	-- user who did the signing
	50	from_user_id TEXT NOT NULL,
	51	-- list of users who were signed, as a JSON array
	52	user_ids TEXT NOT NULL
	53	);
	54
	55	CREATE UNIQUE INDEX user_signature_stream_idx ON user_signature_stream(stream_id);

+22

-0

synapse/storage/databases/main/schema/delta/56/signing_keys_nonunique_signatures.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* The cross-signing signatures index should not be a unique index, because a
	16	* user may upload multiple signatures for the same target user. The previous
	17	* index was unique, so delete it if it's there and create a new non-unique
	18	* index. */
	19
	20	DROP INDEX IF EXISTS e2e_cross_signing_signatures_idx; CREATE INDEX IF NOT
	21	EXISTS e2e_cross_signing_signatures2_idx ON e2e_cross_signing_signatures(user_id, target_user_id, target_device_id);

+156

-0

synapse/storage/databases/main/schema/delta/56/stats_separated.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	* Copyright 2019 The Matrix.org Foundation C.I.C.
	2	*
	3	* Licensed under the Apache License, Version 2.0 (the "License");
	4	* you may not use this file except in compliance with the License.
	5	* You may obtain a copy of the License at
	6	*
	7	* http://www.apache.org/licenses/LICENSE-2.0
	8	*
	9	* Unless required by applicable law or agreed to in writing, software
	10	* distributed under the License is distributed on an "AS IS" BASIS,
	11	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	* See the License for the specific language governing permissions and
	13	* limitations under the License.
	14	*/
	15
	16
	17	----- First clean up from previous versions of room stats.
	18
	19	-- First remove old stats stuff
	20	DROP TABLE IF EXISTS room_stats;
	21	DROP TABLE IF EXISTS room_state;
	22	DROP TABLE IF EXISTS room_stats_state;
	23	DROP TABLE IF EXISTS user_stats;
	24	DROP TABLE IF EXISTS room_stats_earliest_tokens;
	25	DROP TABLE IF EXISTS _temp_populate_stats_position;
	26	DROP TABLE IF EXISTS _temp_populate_stats_rooms;
	27	DROP TABLE IF EXISTS stats_stream_pos;
	28
	29	-- Unschedule old background updates if they're still scheduled
	30	DELETE FROM background_updates WHERE update_name IN (
	31	'populate_stats_createtables',
	32	'populate_stats_process_rooms',
	33	'populate_stats_process_users',
	34	'populate_stats_cleanup'
	35	);
	36
	37	-- this relies on current_state_events.membership having been populated, so add
	38	-- a dependency on current_state_events_membership.
	39	INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
	40	('populate_stats_process_rooms', '{}', 'current_state_events_membership');
	41
	42	-- this also relies on current_state_events.membership having been populated, but
	43	-- we get that as a side-effect of depending on populate_stats_process_rooms.
	44	INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
	45	('populate_stats_process_users', '{}', 'populate_stats_process_rooms');
	46
	47	----- Create tables for our version of room stats.
	48
	49	-- single-row table to track position of incremental updates
	50	DROP TABLE IF EXISTS stats_incremental_position;
	51	CREATE TABLE stats_incremental_position (
	52	Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
	53	stream_id BIGINT NOT NULL,
	54	CHECK (Lock='X')
	55	);
	56
	57	-- insert a null row and make sure it is the only one.
	58	INSERT INTO stats_incremental_position (
	59	stream_id
	60	) SELECT COALESCE(MAX(stream_ordering), 0) from events;
	61
	62	-- represents PRESENT room statistics for a room
	63	-- only holds absolute fields
	64	DROP TABLE IF EXISTS room_stats_current;
	65	CREATE TABLE room_stats_current (
	66	room_id TEXT NOT NULL PRIMARY KEY,
	67
	68	-- These are absolute counts
	69	current_state_events INT NOT NULL,
	70	joined_members INT NOT NULL,
	71	invited_members INT NOT NULL,
	72	left_members INT NOT NULL,
	73	banned_members INT NOT NULL,
	74
	75	local_users_in_room INT NOT NULL,
	76
	77	-- The maximum delta stream position that this row takes into account.
	78	completed_delta_stream_id BIGINT NOT NULL
	79	);
	80
	81
	82	-- represents HISTORICAL room statistics for a room
	83	DROP TABLE IF EXISTS room_stats_historical;
	84	CREATE TABLE room_stats_historical (
	85	room_id TEXT NOT NULL,
	86	-- These stats cover the time from (end_ts - bucket_size)...end_ts (in ms).
	87	-- Note that end_ts is quantised.
	88	end_ts BIGINT NOT NULL,
	89	bucket_size BIGINT NOT NULL,
	90
	91	-- These stats are absolute counts
	92	current_state_events BIGINT NOT NULL,
	93	joined_members BIGINT NOT NULL,
	94	invited_members BIGINT NOT NULL,
	95	left_members BIGINT NOT NULL,
	96	banned_members BIGINT NOT NULL,
	97	local_users_in_room BIGINT NOT NULL,
	98
	99	-- These stats are per time slice
	100	total_events BIGINT NOT NULL,
	101	total_event_bytes BIGINT NOT NULL,
	102
	103	PRIMARY KEY (room_id, end_ts)
	104	);
	105
	106	-- We use this index to speed up deletion of ancient room stats.
	107	CREATE INDEX room_stats_historical_end_ts ON room_stats_historical (end_ts);
	108
	109	-- represents PRESENT statistics for a user
	110	-- only holds absolute fields
	111	DROP TABLE IF EXISTS user_stats_current;
	112	CREATE TABLE user_stats_current (
	113	user_id TEXT NOT NULL PRIMARY KEY,
	114
	115	joined_rooms BIGINT NOT NULL,
	116
	117	-- The maximum delta stream position that this row takes into account.
	118	completed_delta_stream_id BIGINT NOT NULL
	119	);
	120
	121	-- represents HISTORICAL statistics for a user
	122	DROP TABLE IF EXISTS user_stats_historical;
	123	CREATE TABLE user_stats_historical (
	124	user_id TEXT NOT NULL,
	125	end_ts BIGINT NOT NULL,
	126	bucket_size BIGINT NOT NULL,
	127
	128	joined_rooms BIGINT NOT NULL,
	129
	130	invites_sent BIGINT NOT NULL,
	131	rooms_created BIGINT NOT NULL,
	132	total_events BIGINT NOT NULL,
	133	total_event_bytes BIGINT NOT NULL,
	134
	135	PRIMARY KEY (user_id, end_ts)
	136	);
	137
	138	-- We use this index to speed up deletion of ancient user stats.
	139	CREATE INDEX user_stats_historical_end_ts ON user_stats_historical (end_ts);
	140
	141
	142	CREATE TABLE room_stats_state (
	143	room_id TEXT NOT NULL,
	144	name TEXT,
	145	canonical_alias TEXT,
	146	join_rules TEXT,
	147	history_visibility TEXT,
	148	encryption TEXT,
	149	avatar TEXT,
	150	guest_access TEXT,
	151	is_federatable BOOLEAN,
	152	topic TEXT
	153	);
	154
	155	CREATE UNIQUE INDEX room_stats_state_room ON room_stats_state(room_id);

+52

-0

synapse/storage/databases/main/schema/delta/56/unique_user_filter_index.py less more

	0	import logging
	1
	2	from synapse.storage.engines import PostgresEngine
	3
	4	logger = logging.getLogger(__name__)
	5
	6
	7	"""
	8	This migration updates the user_filters table as follows:
	9
	10	- drops any (user_id, filter_id) duplicates
	11	- makes the columns NON-NULLable
	12	- turns the index into a UNIQUE index
	13	"""
	14
	15
	16	def run_upgrade(cur, database_engine, args, *kwargs):
	17	pass
	18
	19
	20	def run_create(cur, database_engine, args, *kwargs):
	21	if isinstance(database_engine, PostgresEngine):
	22	select_clause = """
	23	SELECT DISTINCT ON (user_id, filter_id) user_id, filter_id, filter_json
	24	FROM user_filters
	25	"""
	26	else:
	27	select_clause = """
	28	SELECT * FROM user_filters GROUP BY user_id, filter_id
	29	"""
	30	sql = """
	31	DROP TABLE IF EXISTS user_filters_migration;
	32	DROP INDEX IF EXISTS user_filters_unique;
	33	CREATE TABLE user_filters_migration (
	34	user_id TEXT NOT NULL,
	35	filter_id BIGINT NOT NULL,
	36	filter_json BYTEA NOT NULL
	37	);
	38	INSERT INTO user_filters_migration (user_id, filter_id, filter_json)
	39	%s;
	40	CREATE UNIQUE INDEX user_filters_unique ON user_filters_migration
	41	(user_id, filter_id);
	42	DROP TABLE user_filters;
	43	ALTER TABLE user_filters_migration RENAME TO user_filters;
	44	""" % (
	45	select_clause,
	46	)
	47
	48	if isinstance(database_engine, PostgresEngine):
	49	cur.execute(sql)
	50	else:
	51	cur.executescript(sql)

+24

-0

synapse/storage/databases/main/schema/delta/56/user_external_ids.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/*
	16	* a table which records mappings from external auth providers to mxids
	17	*/
	18	CREATE TABLE IF NOT EXISTS user_external_ids (
	19	auth_provider TEXT NOT NULL,
	20	external_id TEXT NOT NULL,
	21	user_id TEXT NOT NULL,
	22	UNIQUE (auth_provider, external_id)
	23	);

+17

-0

synapse/storage/databases/main/schema/delta/56/users_in_public_rooms_idx.sql less more

	0	/* Copyright 2019 Matrix.org Foundation CIC
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- this was apparently forgotten when the table was created back in delta 53.
	16	CREATE INDEX users_in_public_rooms_r_idx ON users_in_public_rooms(room_id);

+22

-0

synapse/storage/databases/main/schema/delta/57/delete_old_current_state_events.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Add background update to go and delete current state events for rooms the
	16	-- server is no longer in.
	17	--
	18	-- this relies on the 'membership' column of current_state_events, so make sure
	19	-- that's populated first!
	20	INSERT into background_updates (update_name, progress_json, depends_on)
	21	VALUES ('delete_old_current_state_events', '{}', 'current_state_events_membership');

+25

-0

synapse/storage/databases/main/schema/delta/57/device_list_remote_cache_stale.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Records whether the server thinks that the remote users cached device lists
	16	-- may be out of date (e.g. if we have received a to device message from a
	17	-- device we don't know about).
	18	CREATE TABLE IF NOT EXISTS device_lists_remote_resync (
	19	user_id TEXT NOT NULL,
	20	added_ts BIGINT NOT NULL
	21	);
	22
	23	CREATE UNIQUE INDEX device_lists_remote_resync_idx ON device_lists_remote_resync (user_id);
	24	CREATE INDEX device_lists_remote_resync_ts_idx ON device_lists_remote_resync (added_ts);

+98

-0

synapse/storage/databases/main/schema/delta/57/local_current_membership.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 New Vector Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15
	16	# We create a new table called `local_current_membership` that stores the latest
	17	# membership state of local users in rooms, which helps track leaves/bans/etc
	18	# even if the server has left the room (and so has deleted the room from
	19	# `current_state_events`). This will also include outstanding invites for local
	20	# users for rooms the server isn't in.
	21	#
	22	# If the server isn't and hasn't been in the room then it will only include
	23	# outsstanding invites, and not e.g. pre-emptive bans of local users.
	24	#
	25	# If the server later rejoins a room `local_current_membership` can simply be
	26	# replaced with the new current state of the room (which results in the
	27	# equivalent behaviour as if the server had remained in the room).
	28
	29
	30	def run_upgrade(cur, database_engine, config, args, *kwargs):
	31	# We need to do the insert in `run_upgrade` section as we don't have access
	32	# to `config` in `run_create`.
	33
	34	# This upgrade may take a bit of time for large servers (e.g. one minute for
	35	# matrix.org) but means we avoid a lots of book keeping required to do it as
	36	# a background update.
	37
	38	# We check if the `current_state_events.membership` is up to date by
	39	# checking if the relevant background update has finished. If it has
	40	# finished we can avoid doing a join against `room_memberships`, which
	41	# speesd things up.
	42	cur.execute(
	43	"""SELECT 1 FROM background_updates
	44	WHERE update_name = 'current_state_events_membership'
	45	"""
	46	)
	47	current_state_membership_up_to_date = not bool(cur.fetchone())
	48
	49	# Cheekily drop and recreate indices, as that is faster.
	50	cur.execute("DROP INDEX local_current_membership_idx")
	51	cur.execute("DROP INDEX local_current_membership_room_idx")
	52
	53	if current_state_membership_up_to_date:
	54	sql = """
	55	INSERT INTO local_current_membership (room_id, user_id, event_id, membership)
	56	SELECT c.room_id, state_key AS user_id, event_id, c.membership
	57	FROM current_state_events AS c
	58	WHERE type = 'm.room.member' AND c.membership IS NOT NULL AND state_key LIKE ?
	59	"""
	60	else:
	61	# We can't rely on the membership column, so we need to join against
	62	# `room_memberships`.
	63	sql = """
	64	INSERT INTO local_current_membership (room_id, user_id, event_id, membership)
	65	SELECT c.room_id, state_key AS user_id, event_id, r.membership
	66	FROM current_state_events AS c
	67	INNER JOIN room_memberships AS r USING (event_id)
	68	WHERE type = 'm.room.member' AND state_key LIKE ?
	69	"""
	70	sql = database_engine.convert_param_style(sql)
	71	cur.execute(sql, ("%:" + config.server_name,))
	72
	73	cur.execute(
	74	"CREATE UNIQUE INDEX local_current_membership_idx ON local_current_membership(user_id, room_id)"
	75	)
	76	cur.execute(
	77	"CREATE INDEX local_current_membership_room_idx ON local_current_membership(room_id)"
	78	)
	79
	80
	81	def run_create(cur, database_engine, args, *kwargs):
	82	cur.execute(
	83	"""
	84	CREATE TABLE local_current_membership (
	85	room_id TEXT NOT NULL,
	86	user_id TEXT NOT NULL,
	87	event_id TEXT NOT NULL,
	88	membership TEXT NOT NULL
	89	)"""
	90	)
	91
	92	cur.execute(
	93	"CREATE UNIQUE INDEX local_current_membership_idx ON local_current_membership(user_id, room_id)"
	94	)
	95	cur.execute(
	96	"CREATE INDEX local_current_membership_room_idx ON local_current_membership(room_id)"
	97	)

+21

-0

synapse/storage/databases/main/schema/delta/57/remove_sent_outbound_pokes.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- we no longer keep sent outbound device pokes in the db; clear them out
	16	-- so that we don't have to worry about them.
	17	--
	18	-- This is a sequence scan, but it doesn't take too long.
	19
	20	DELETE FROM device_lists_outbound_pokes WHERE sent;

+24

-0

synapse/storage/databases/main/schema/delta/57/rooms_version_column.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	-- We want to start storing the room version independently of
	17	-- `current_state_events` so that we can delete stale entries from it without
	18	-- losing the information.
	19	ALTER TABLE rooms ADD COLUMN room_version TEXT;
	20
	21
	22	INSERT into background_updates (update_name, progress_json)
	23	VALUES ('add_rooms_room_version_column', '{}');

+35

-0

synapse/storage/databases/main/schema/delta/57/rooms_version_column_2.sql.postgres less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- when we first added the room_version column, it was populated via a background
	16	-- update. We now need it to be populated before synapse starts, so we populate
	17	-- any remaining rows with a NULL room version now. For servers which have completed
	18	-- the background update, this will be pretty quick.
	19
	20	-- the following query will set room_version to NULL if no create event is found for
	21	-- the room in current_state_events, and will set it to '1' if a create event with no
	22	-- room_version is found.
	23
	24	UPDATE rooms SET room_version=(
	25	SELECT COALESCE(json::json->'content'->>'room_version','1')
	26	FROM current_state_events cse INNER JOIN event_json ej USING (event_id)
	27	WHERE cse.room_id=rooms.room_id AND cse.type='m.room.create' AND cse.state_key=''
	28	) WHERE rooms.room_version IS NULL;
	29
	30	-- we still allow the background update to complete: it has the useful side-effect of
	31	-- populating `rooms` with any missing rooms (based on the current_state_events table).
	32
	33	-- see also rooms_version_column_2.sql.sqlite which has a copy of the above query, using
	34	-- sqlite syntax for the json extraction.

+22

-0

synapse/storage/databases/main/schema/delta/57/rooms_version_column_2.sql.sqlite less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- see rooms_version_column_2.sql.postgres for details of what's going on here.
	16
	17	UPDATE rooms SET room_version=(
	18	SELECT COALESCE(json_extract(ej.json, '$.content.room_version'), '1')
	19	FROM current_state_events cse INNER JOIN event_json ej USING (event_id)
	20	WHERE cse.room_id=rooms.room_id AND cse.type='m.room.create' AND cse.state_key=''
	21	) WHERE rooms.room_version IS NULL;

+39

-0

synapse/storage/databases/main/schema/delta/57/rooms_version_column_3.sql.postgres less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- When we first added the room_version column to the rooms table, it was populated from
	16	-- the current_state_events table. However, there was an issue causing a background
	17	-- update to clean up the current_state_events table for rooms where the server is no
	18	-- longer participating, before that column could be populated. Therefore, some rooms had
	19	-- a NULL room_version.
	20
	21	-- The rooms_version_column_2.sql.* delta files were introduced to make the populating
	22	-- synchronous instead of running it in a background update, which fixed this issue.
	23	-- However, all of the instances of Synapse installed or updated in the meantime got
	24	-- their rooms table corrupted with NULL room_versions.
	25
	26	-- This query fishes out the room versions from the create event using the state_events
	27	-- table instead of the current_state_events one, as the former still have all of the
	28	-- create events.
	29
	30	UPDATE rooms SET room_version=(
	31	SELECT COALESCE(json::json->'content'->>'room_version','1')
	32	FROM state_events se INNER JOIN event_json ej USING (event_id)
	33	WHERE se.room_id=rooms.room_id AND se.type='m.room.create' AND se.state_key=''
	34	LIMIT 1
	35	) WHERE rooms.room_version IS NULL;
	36
	37	-- see also rooms_version_column_3.sql.sqlite which has a copy of the above query, using
	38	-- sqlite syntax for the json extraction.

+23

-0

synapse/storage/databases/main/schema/delta/57/rooms_version_column_3.sql.sqlite less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- see rooms_version_column_3.sql.postgres for details of what's going on here.
	16
	17	UPDATE rooms SET room_version=(
	18	SELECT COALESCE(json_extract(ej.json, '$.content.room_version'), '1')
	19	FROM state_events se INNER JOIN event_json ej USING (event_id)
	20	WHERE se.room_id=rooms.room_id AND se.type='m.room.create' AND se.state_key=''
	21	LIMIT 1
	22	) WHERE rooms.room_version IS NULL;

+22

-0

synapse/storage/databases/main/schema/delta/58/02remove_dup_outbound_pokes.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* for some reason, we have accumulated duplicate entries in
	16	* device_lists_outbound_pokes, which makes prune_outbound_device_list_pokes less
	17	* efficient.
	18	*/
	19
	20	INSERT INTO background_updates (ordering, update_name, progress_json)
	21	VALUES (5800, 'remove_dup_outbound_pokes', '{}');

+36

-0

synapse/storage/databases/main/schema/delta/58/03persist_ui_auth.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS ui_auth_sessions(
	16	session_id TEXT NOT NULL, -- The session ID passed to the client.
	17	creation_time BIGINT NOT NULL, -- The time this session was created (epoch time in milliseconds).
	18	serverdict TEXT NOT NULL, -- A JSON dictionary of arbitrary data added by Synapse.
	19	clientdict TEXT NOT NULL, -- A JSON dictionary of arbitrary data from the client.
	20	uri TEXT NOT NULL, -- The URI the UI authentication session is using.
	21	method TEXT NOT NULL, -- The HTTP method the UI authentication session is using.
	22	-- The clientdict, uri, and method make up an tuple that must be immutable
	23	-- throughout the lifetime of the UI Auth session.
	24	description TEXT NOT NULL, -- A human readable description of the operation which caused the UI Auth flow to occur.
	25	UNIQUE (session_id)
	26	);
	27
	28	CREATE TABLE IF NOT EXISTS ui_auth_sessions_credentials(
	29	session_id TEXT NOT NULL, -- The corresponding UI Auth session.
	30	stage_type TEXT NOT NULL, -- The stage type.
	31	result TEXT NOT NULL, -- The result of the stage verification, stored as JSON.
	32	UNIQUE (session_id, stage_type),
	33	FOREIGN KEY (session_id)
	34	REFERENCES ui_auth_sessions (session_id)
	35	);

+30

-0

synapse/storage/databases/main/schema/delta/58/05cache_instance.sql.postgres less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- We keep the old table here to enable us to roll back. It doesn't matter
	16	-- that we have dropped all the data here.
	17	TRUNCATE cache_invalidation_stream;
	18
	19	CREATE TABLE cache_invalidation_stream_by_instance (
	20	stream_id BIGINT NOT NULL,
	21	instance_name TEXT NOT NULL,
	22	cache_func TEXT NOT NULL,
	23	keys TEXT[],
	24	invalidation_ts BIGINT
	25	);
	26
	27	CREATE UNIQUE INDEX cache_invalidation_stream_by_instance_id ON cache_invalidation_stream_by_instance(stream_id);
	28
	29	CREATE SEQUENCE cache_invalidation_stream_seq;

+80

-0

synapse/storage/databases/main/schema/delta/58/06dlols_unique_idx.py less more

	0	# Copyright 2020 The Matrix.org Foundation C.I.C.
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14	"""
	15	This migration rebuilds the device_lists_outbound_last_success table without duplicate
	16	entries, and with a UNIQUE index.
	17	"""
	18
	19	import logging
	20	from io import StringIO
	21
	22	from synapse.storage.engines import BaseDatabaseEngine, PostgresEngine
	23	from synapse.storage.prepare_database import execute_statements_from_stream
	24	from synapse.storage.types import Cursor
	25
	26	logger = logging.getLogger(__name__)
	27
	28
	29	def run_upgrade(args, *kwargs):
	30	pass
	31
	32
	33	def run_create(cur: Cursor, database_engine: BaseDatabaseEngine, args, *kwargs):
	34	# some instances might already have this index, in which case we can skip this
	35	if isinstance(database_engine, PostgresEngine):
	36	cur.execute(
	37	"""
	38	SELECT 1 FROM pg_class WHERE relkind = 'i'
	39	AND relname = 'device_lists_outbound_last_success_unique_idx'
	40	"""
	41	)
	42
	43	if cur.rowcount:
	44	logger.info(
	45	"Unique index exists on device_lists_outbound_last_success: "
	46	"skipping rebuild"
	47	)
	48	return
	49
	50	logger.info("Rebuilding device_lists_outbound_last_success with unique index")
	51	execute_statements_from_stream(cur, StringIO(_rebuild_commands))
	52
	53
	54	# there might be duplicates, so the easiest way to achieve this is to create a new
	55	# table with the right data, and renaming it into place
	56
	57	_rebuild_commands = """
	58	DROP TABLE IF EXISTS device_lists_outbound_last_success_new;
	59
	60	CREATE TABLE device_lists_outbound_last_success_new (
	61	destination TEXT NOT NULL,
	62	user_id TEXT NOT NULL,
	63	stream_id BIGINT NOT NULL
	64	);
	65
	66	-- this took about 30 seconds on matrix.org's 16 million rows.
	67	INSERT INTO device_lists_outbound_last_success_new
	68	SELECT destination, user_id, MAX(stream_id) FROM device_lists_outbound_last_success
	69	GROUP BY destination, user_id;
	70
	71	-- and this another 30 seconds.
	72	CREATE UNIQUE INDEX device_lists_outbound_last_success_unique_idx
	73	ON device_lists_outbound_last_success_new (destination, user_id);
	74
	75	DROP TABLE device_lists_outbound_last_success;
	76
	77	ALTER TABLE device_lists_outbound_last_success_new
	78	RENAME TO device_lists_outbound_last_success;
	79	"""

+18

-0

synapse/storage/databases/main/schema/delta/58/08_media_safe_from_quarantine.sql.postgres less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- The local_media_repository should have files which do not get quarantined,
	16	-- e.g. files from sticker packs.
	17	ALTER TABLE local_media_repository ADD COLUMN safe_from_quarantine BOOLEAN NOT NULL DEFAULT FALSE;

+18

-0

synapse/storage/databases/main/schema/delta/58/08_media_safe_from_quarantine.sql.sqlite less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- The local_media_repository should have files which do not get quarantined,
	16	-- e.g. files from sticker packs.
	17	ALTER TABLE local_media_repository ADD COLUMN safe_from_quarantine BOOLEAN NOT NULL DEFAULT 0;

+22

-0

synapse/storage/databases/main/schema/delta/58/10drop_local_rejections_stream.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/*
	16	The version of synapse 1.16.0 on pypi incorrectly contained a migration which
	17	added a table called 'local_rejections_stream'. This table is not used, and
	18	we drop it here for anyone who was affected.
	19	*/
	20
	21	DROP TABLE IF EXISTS local_rejections_stream;

+22

-0

synapse/storage/databases/main/schema/delta/58/10federation_pos_instance_name.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- We need to store the stream positions by instance in a sharded config world.
	16	--
	17	-- We default to master as we want the column to be NOT NULL and we correctly
	18	-- reset the instance name to match the config each time we start up.
	19	ALTER TABLE federation_stream_position ADD COLUMN instance_name TEXT NOT NULL DEFAULT 'master';
	20
	21	CREATE UNIQUE INDEX federation_stream_position_instance ON federation_stream_position(type, instance_name);

+34

-0

synapse/storage/databases/main/schema/delta/58/11user_id_seq.py less more

	0	# Copyright 2020 The Matrix.org Foundation C.I.C.
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14	"""
	15	Adds a postgres SEQUENCE for generating guest user IDs.
	16	"""
	17
	18	from synapse.storage.databases.main.registration import (
	19	find_max_generated_user_id_localpart,
	20	)
	21	from synapse.storage.engines import PostgresEngine
	22
	23
	24	def run_create(cur, database_engine, args, *kwargs):
	25	if not isinstance(database_engine, PostgresEngine):
	26	return
	27
	28	next_id = find_max_generated_user_id_localpart(cur) + 1
	29	cur.execute("CREATE SEQUENCE user_id_seq START WITH %s", (next_id,))
	30
	31
	32	def run_upgrade(args, *kwargs):
	33	pass

+32

-0

synapse/storage/databases/main/schema/delta/58/12room_stats.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Recalculate the stats for all rooms after the fix to joined_members erroneously
	16	-- incrementing on per-room profile changes.
	17
	18	-- Note that the populate_stats_process_rooms background update is already set to
	19	-- run if you're upgrading from Synapse <1.0.0.
	20
	21	-- Additionally, if you've upgraded to v1.18.0 (which doesn't include this fix),
	22	-- this bg job runs, and then update to v1.19.0, you'd end up with only half of
	23	-- your rooms having room stats recalculated after this fix was in place.
	24
	25	-- So we've switched the old `populate_stats_process_rooms` background job to a
	26	-- no-op, and then kick off a bg job with a new name, but with the same
	27	-- functionality as the old one. This effectively restarts the background job
	28	-- from the beginning, without running it twice in a row, supporting both
	29	-- upgrade usecases.
	30	INSERT INTO background_updates (update_name, progress_json) VALUES
	31	('populate_stats_process_rooms_2', '{}');

+37

-0

synapse/storage/databases/main/schema/full_schemas/16/application_services.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* We used to create tables called application_services and
	16	* application_services_regex, but these are no longer used and are removed in
	17	* delta 54.
	18	*/
	19
	20
	21	CREATE TABLE IF NOT EXISTS application_services_state(
	22	as_id TEXT PRIMARY KEY,
	23	state VARCHAR(5),
	24	last_txn INTEGER
	25	);
	26
	27	CREATE TABLE IF NOT EXISTS application_services_txns(
	28	as_id TEXT NOT NULL,
	29	txn_id INTEGER NOT NULL,
	30	event_ids TEXT NOT NULL,
	31	UNIQUE(as_id, txn_id)
	32	);
	33
	34	CREATE INDEX application_services_txns_id ON application_services_txns (
	35	as_id
	36	);

+70

-0

synapse/storage/databases/main/schema/full_schemas/16/event_edges.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* We used to create tables called event_destinations and
	16	* state_forward_extremities, but these are no longer used and are removed in
	17	* delta 54.
	18	*/
	19
	20	CREATE TABLE IF NOT EXISTS event_forward_extremities(
	21	event_id TEXT NOT NULL,
	22	room_id TEXT NOT NULL,
	23	UNIQUE (event_id, room_id)
	24	);
	25
	26	CREATE INDEX ev_extrem_room ON event_forward_extremities(room_id);
	27	CREATE INDEX ev_extrem_id ON event_forward_extremities(event_id);
	28
	29
	30	CREATE TABLE IF NOT EXISTS event_backward_extremities(
	31	event_id TEXT NOT NULL,
	32	room_id TEXT NOT NULL,
	33	UNIQUE (event_id, room_id)
	34	);
	35
	36	CREATE INDEX ev_b_extrem_room ON event_backward_extremities(room_id);
	37	CREATE INDEX ev_b_extrem_id ON event_backward_extremities(event_id);
	38
	39
	40	CREATE TABLE IF NOT EXISTS event_edges(
	41	event_id TEXT NOT NULL,
	42	prev_event_id TEXT NOT NULL,
	43	room_id TEXT NOT NULL,
	44	is_state BOOL NOT NULL, -- true if this is a prev_state edge rather than a regular
	45	-- event dag edge.
	46	UNIQUE (event_id, prev_event_id, room_id, is_state)
	47	);
	48
	49	CREATE INDEX ev_edges_id ON event_edges(event_id);
	50	CREATE INDEX ev_edges_prev_id ON event_edges(prev_event_id);
	51
	52
	53	CREATE TABLE IF NOT EXISTS room_depth(
	54	room_id TEXT NOT NULL,
	55	min_depth INTEGER NOT NULL,
	56	UNIQUE (room_id)
	57	);
	58
	59	CREATE INDEX room_depth_room ON room_depth(room_id);
	60
	61	CREATE TABLE IF NOT EXISTS event_auth(
	62	event_id TEXT NOT NULL,
	63	auth_id TEXT NOT NULL,
	64	room_id TEXT NOT NULL,
	65	UNIQUE (event_id, auth_id, room_id)
	66	);
	67
	68	CREATE INDEX evauth_edges_id ON event_auth(event_id);
	69	CREATE INDEX evauth_edges_auth_id ON event_auth(auth_id);

+38

-0

synapse/storage/databases/main/schema/full_schemas/16/event_signatures.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* We used to create tables called event_content_hashes and event_edge_hashes,
	16	* but these are no longer used and are removed in delta 54.
	17	*/
	18
	19	CREATE TABLE IF NOT EXISTS event_reference_hashes (
	20	event_id TEXT,
	21	algorithm TEXT,
	22	hash bytea,
	23	UNIQUE (event_id, algorithm)
	24	);
	25
	26	CREATE INDEX event_reference_hashes_id ON event_reference_hashes(event_id);
	27
	28
	29	CREATE TABLE IF NOT EXISTS event_signatures (
	30	event_id TEXT,
	31	signature_name TEXT,
	32	key_id TEXT,
	33	signature bytea,
	34	UNIQUE (event_id, signature_name, key_id)
	35	);
	36
	37	CREATE INDEX event_signatures_id ON event_signatures(event_id);

+120

-0

synapse/storage/databases/main/schema/full_schemas/16/im.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* We used to create tables called room_hosts and feedback,
	16	* but these are no longer used and are removed in delta 54.
	17	*/
	18
	19	CREATE TABLE IF NOT EXISTS events(
	20	stream_ordering INTEGER PRIMARY KEY,
	21	topological_ordering BIGINT NOT NULL,
	22	event_id TEXT NOT NULL,
	23	type TEXT NOT NULL,
	24	room_id TEXT NOT NULL,
	25
	26	-- 'content' used to be created NULLable, but as of delta 50 we drop that constraint.
	27	-- the hack we use to drop the constraint doesn't work for an in-memory sqlite
	28	-- database, which breaks the sytests. Hence, we no longer make it nullable.
	29	content TEXT,
	30
	31	unrecognized_keys TEXT,
	32	processed BOOL NOT NULL,
	33	outlier BOOL NOT NULL,
	34	depth BIGINT DEFAULT 0 NOT NULL,
	35	UNIQUE (event_id)
	36	);
	37
	38	CREATE INDEX events_stream_ordering ON events (stream_ordering);
	39	CREATE INDEX events_topological_ordering ON events (topological_ordering);
	40	CREATE INDEX events_order ON events (topological_ordering, stream_ordering);
	41	CREATE INDEX events_room_id ON events (room_id);
	42	CREATE INDEX events_order_room ON events (
	43	room_id, topological_ordering, stream_ordering
	44	);
	45
	46
	47	CREATE TABLE IF NOT EXISTS event_json(
	48	event_id TEXT NOT NULL,
	49	room_id TEXT NOT NULL,
	50	internal_metadata TEXT NOT NULL,
	51	json TEXT NOT NULL,
	52	UNIQUE (event_id)
	53	);
	54
	55	CREATE INDEX event_json_room_id ON event_json(room_id);
	56
	57
	58	CREATE TABLE IF NOT EXISTS state_events(
	59	event_id TEXT NOT NULL,
	60	room_id TEXT NOT NULL,
	61	type TEXT NOT NULL,
	62	state_key TEXT NOT NULL,
	63	prev_state TEXT,
	64	UNIQUE (event_id)
	65	);
	66
	67	CREATE INDEX state_events_room_id ON state_events (room_id);
	68	CREATE INDEX state_events_type ON state_events (type);
	69	CREATE INDEX state_events_state_key ON state_events (state_key);
	70
	71
	72	CREATE TABLE IF NOT EXISTS current_state_events(
	73	event_id TEXT NOT NULL,
	74	room_id TEXT NOT NULL,
	75	type TEXT NOT NULL,
	76	state_key TEXT NOT NULL,
	77	UNIQUE (event_id),
	78	UNIQUE (room_id, type, state_key)
	79	);
	80
	81	CREATE INDEX current_state_events_room_id ON current_state_events (room_id);
	82	CREATE INDEX current_state_events_type ON current_state_events (type);
	83	CREATE INDEX current_state_events_state_key ON current_state_events (state_key);
	84
	85	CREATE TABLE IF NOT EXISTS room_memberships(
	86	event_id TEXT NOT NULL,
	87	user_id TEXT NOT NULL,
	88	sender TEXT NOT NULL,
	89	room_id TEXT NOT NULL,
	90	membership TEXT NOT NULL,
	91	UNIQUE (event_id)
	92	);
	93
	94	CREATE INDEX room_memberships_room_id ON room_memberships (room_id);
	95	CREATE INDEX room_memberships_user_id ON room_memberships (user_id);
	96
	97	CREATE TABLE IF NOT EXISTS topics(
	98	event_id TEXT NOT NULL,
	99	room_id TEXT NOT NULL,
	100	topic TEXT NOT NULL,
	101	UNIQUE (event_id)
	102	);
	103
	104	CREATE INDEX topics_room_id ON topics(room_id);
	105
	106	CREATE TABLE IF NOT EXISTS room_names(
	107	event_id TEXT NOT NULL,
	108	room_id TEXT NOT NULL,
	109	name TEXT NOT NULL,
	110	UNIQUE (event_id)
	111	);
	112
	113	CREATE INDEX room_names_room_id ON room_names(room_id);
	114
	115	CREATE TABLE IF NOT EXISTS rooms(
	116	room_id TEXT PRIMARY KEY NOT NULL,
	117	is_public BOOL,
	118	creator TEXT
	119	);

+26

-0

synapse/storage/databases/main/schema/full_schemas/16/keys.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- we used to create a table called server_tls_certificates, but this is no
	16	-- longer used, and is removed in delta 54.
	17
	18	CREATE TABLE IF NOT EXISTS server_signature_keys(
	19	server_name TEXT, -- Server name.
	20	key_id TEXT, -- Key version.
	21	from_server TEXT, -- Which key server the key was fetched form.
	22	ts_added_ms BIGINT, -- When the key was added.
	23	verify_key bytea, -- NACL verification key.
	24	UNIQUE (server_name, key_id)
	25	);

+68

-0

synapse/storage/databases/main/schema/full_schemas/16/media_repository.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS local_media_repository (
	16	media_id TEXT, -- The id used to refer to the media.
	17	media_type TEXT, -- The MIME-type of the media.
	18	media_length INTEGER, -- Length of the media in bytes.
	19	created_ts BIGINT, -- When the content was uploaded in ms.
	20	upload_name TEXT, -- The name the media was uploaded with.
	21	user_id TEXT, -- The user who uploaded the file.
	22	UNIQUE (media_id)
	23	);
	24
	25	CREATE TABLE IF NOT EXISTS local_media_repository_thumbnails (
	26	media_id TEXT, -- The id used to refer to the media.
	27	thumbnail_width INTEGER, -- The width of the thumbnail in pixels.
	28	thumbnail_height INTEGER, -- The height of the thumbnail in pixels.
	29	thumbnail_type TEXT, -- The MIME-type of the thumbnail.
	30	thumbnail_method TEXT, -- The method used to make the thumbnail.
	31	thumbnail_length INTEGER, -- The length of the thumbnail in bytes.
	32	UNIQUE (
	33	media_id, thumbnail_width, thumbnail_height, thumbnail_type
	34	)
	35	);
	36
	37	CREATE INDEX local_media_repository_thumbnails_media_id
	38	ON local_media_repository_thumbnails (media_id);
	39
	40	CREATE TABLE IF NOT EXISTS remote_media_cache (
	41	media_origin TEXT, -- The remote HS the media came from.
	42	media_id TEXT, -- The id used to refer to the media on that server.
	43	media_type TEXT, -- The MIME-type of the media.
	44	created_ts BIGINT, -- When the content was uploaded in ms.
	45	upload_name TEXT, -- The name the media was uploaded with.
	46	media_length INTEGER, -- Length of the media in bytes.
	47	filesystem_id TEXT, -- The name used to store the media on disk.
	48	UNIQUE (media_origin, media_id)
	49	);
	50
	51	CREATE TABLE IF NOT EXISTS remote_media_cache_thumbnails (
	52	media_origin TEXT, -- The remote HS the media came from.
	53	media_id TEXT, -- The id used to refer to the media.
	54	thumbnail_width INTEGER, -- The width of the thumbnail in pixels.
	55	thumbnail_height INTEGER, -- The height of the thumbnail in pixels.
	56	thumbnail_method TEXT, -- The method used to make the thumbnail
	57	thumbnail_type TEXT, -- The MIME-type of the thumbnail.
	58	thumbnail_length INTEGER, -- The length of the thumbnail in bytes.
	59	filesystem_id TEXT, -- The name used to store the media on disk.
	60	UNIQUE (
	61	media_origin, media_id, thumbnail_width, thumbnail_height,
	62	thumbnail_type
	63	)
	64	);
	65
	66	CREATE INDEX remote_media_cache_thumbnails_media_id
	67	ON remote_media_cache_thumbnails (media_id);

+32

-0

synapse/storage/databases/main/schema/full_schemas/16/presence.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14	CREATE TABLE IF NOT EXISTS presence(
	15	user_id TEXT NOT NULL,
	16	state VARCHAR(20),
	17	status_msg TEXT,
	18	mtime BIGINT, -- miliseconds since last state change
	19	UNIQUE (user_id)
	20	);
	21
	22	-- For each of /my/ users which possibly-remote users are allowed to see their
	23	-- presence state
	24	CREATE TABLE IF NOT EXISTS presence_allow_inbound(
	25	observed_user_id TEXT NOT NULL,
	26	observer_user_id TEXT NOT NULL, -- a UserID,
	27	UNIQUE (observed_user_id, observer_user_id)
	28	);
	29
	30	-- We used to create a table called presence_list, but this is no longer used
	31	-- and is removed in delta 54.⏎

+20

-0

synapse/storage/databases/main/schema/full_schemas/16/profiles.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14	CREATE TABLE IF NOT EXISTS profiles(
	15	user_id TEXT NOT NULL,
	16	displayname TEXT,
	17	avatar_url TEXT,
	18	UNIQUE(user_id)
	19	);

+74

-0

synapse/storage/databases/main/schema/full_schemas/16/push.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS rejections(
	16	event_id TEXT NOT NULL,
	17	reason TEXT NOT NULL,
	18	last_check TEXT NOT NULL,
	19	UNIQUE (event_id)
	20	);
	21
	22	-- Push notification endpoints that users have configured
	23	CREATE TABLE IF NOT EXISTS pushers (
	24	id BIGINT PRIMARY KEY,
	25	user_name TEXT NOT NULL,
	26	access_token BIGINT DEFAULT NULL,
	27	profile_tag VARCHAR(32) NOT NULL,
	28	kind VARCHAR(8) NOT NULL,
	29	app_id VARCHAR(64) NOT NULL,
	30	app_display_name VARCHAR(64) NOT NULL,
	31	device_display_name VARCHAR(128) NOT NULL,
	32	pushkey bytea NOT NULL,
	33	ts BIGINT NOT NULL,
	34	lang VARCHAR(8),
	35	data bytea,
	36	last_token TEXT,
	37	last_success BIGINT,
	38	failing_since BIGINT,
	39	UNIQUE (app_id, pushkey)
	40	);
	41
	42	CREATE TABLE IF NOT EXISTS push_rules (
	43	id BIGINT PRIMARY KEY,
	44	user_name TEXT NOT NULL,
	45	rule_id TEXT NOT NULL,
	46	priority_class SMALLINT NOT NULL,
	47	priority INTEGER NOT NULL DEFAULT 0,
	48	conditions TEXT NOT NULL,
	49	actions TEXT NOT NULL,
	50	UNIQUE(user_name, rule_id)
	51	);
	52
	53	CREATE INDEX push_rules_user_name on push_rules (user_name);
	54
	55	CREATE TABLE IF NOT EXISTS user_filters(
	56	user_id TEXT,
	57	filter_id BIGINT,
	58	filter_json bytea
	59	);
	60
	61	CREATE INDEX user_filters_by_user_id_filter_id ON user_filters(
	62	user_id, filter_id
	63	);
	64
	65	CREATE TABLE IF NOT EXISTS push_rules_enable (
	66	id BIGINT PRIMARY KEY,
	67	user_name TEXT NOT NULL,
	68	rule_id TEXT NOT NULL,
	69	enabled SMALLINT,
	70	UNIQUE(user_name, rule_id)
	71	);
	72
	73	CREATE INDEX push_rules_enable_user_name on push_rules_enable (user_name);

+22

-0

synapse/storage/databases/main/schema/full_schemas/16/redactions.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14	CREATE TABLE IF NOT EXISTS redactions (
	15	event_id TEXT NOT NULL,
	16	redacts TEXT NOT NULL,
	17	UNIQUE (event_id)
	18	);
	19
	20	CREATE INDEX redactions_event_id ON redactions (event_id);
	21	CREATE INDEX redactions_redacts ON redactions (redacts);

+29

-0

synapse/storage/databases/main/schema/full_schemas/16/room_aliases.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS room_aliases(
	16	room_alias TEXT NOT NULL,
	17	room_id TEXT NOT NULL,
	18	UNIQUE (room_alias)
	19	);
	20
	21	CREATE INDEX room_aliases_id ON room_aliases(room_id);
	22
	23	CREATE TABLE IF NOT EXISTS room_alias_servers(
	24	room_alias TEXT NOT NULL,
	25	server TEXT NOT NULL
	26	);
	27
	28	CREATE INDEX room_alias_servers_alias ON room_alias_servers(room_alias);

+40

-0

synapse/storage/databases/main/schema/full_schemas/16/state.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE IF NOT EXISTS state_groups(
	16	id BIGINT PRIMARY KEY,
	17	room_id TEXT NOT NULL,
	18	event_id TEXT NOT NULL
	19	);
	20
	21	CREATE TABLE IF NOT EXISTS state_groups_state(
	22	state_group BIGINT NOT NULL,
	23	room_id TEXT NOT NULL,
	24	type TEXT NOT NULL,
	25	state_key TEXT NOT NULL,
	26	event_id TEXT NOT NULL
	27	);
	28
	29	CREATE TABLE IF NOT EXISTS event_to_state_groups(
	30	event_id TEXT NOT NULL,
	31	state_group BIGINT NOT NULL,
	32	UNIQUE (event_id)
	33	);
	34
	35	CREATE INDEX state_groups_id ON state_groups(id);
	36
	37	CREATE INDEX state_groups_state_id ON state_groups_state(state_group);
	38	CREATE INDEX state_groups_state_tuple ON state_groups_state(room_id, type, state_key);
	39	CREATE INDEX event_to_state_groups_id ON event_to_state_groups(event_id);

+44

-0

synapse/storage/databases/main/schema/full_schemas/16/transactions.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14	-- Stores what transaction ids we have received and what our response was
	15	CREATE TABLE IF NOT EXISTS received_transactions(
	16	transaction_id TEXT,
	17	origin TEXT,
	18	ts BIGINT,
	19	response_code INTEGER,
	20	response_json bytea,
	21	has_been_referenced smallint default 0, -- Whether thishas been referenced by a prev_tx
	22	UNIQUE (transaction_id, origin)
	23	);
	24
	25	CREATE INDEX transactions_have_ref ON received_transactions(origin, has_been_referenced);-- WHERE has_been_referenced = 0;
	26
	27	-- For sent transactions only.
	28	CREATE TABLE IF NOT EXISTS transaction_id_to_pdu(
	29	transaction_id INTEGER,
	30	destination TEXT,
	31	pdu_id TEXT,
	32	pdu_origin TEXT,
	33	UNIQUE (transaction_id, destination)
	34	);
	35
	36	CREATE INDEX transaction_id_to_pdu_dest ON transaction_id_to_pdu(destination);
	37
	38	-- To track destination health
	39	CREATE TABLE IF NOT EXISTS destinations(
	40	destination TEXT PRIMARY KEY,
	41	retry_last_ts BIGINT,
	42	retry_interval INTEGER
	43	);

+42

-0

synapse/storage/databases/main/schema/full_schemas/16/users.sql less more

	0	/* Copyright 2014-2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14	CREATE TABLE IF NOT EXISTS users(
	15	name TEXT,
	16	password_hash TEXT,
	17	creation_ts BIGINT,
	18	admin SMALLINT DEFAULT 0 NOT NULL,
	19	UNIQUE(name)
	20	);
	21
	22	CREATE TABLE IF NOT EXISTS access_tokens(
	23	id BIGINT PRIMARY KEY,
	24	user_id TEXT NOT NULL,
	25	device_id TEXT,
	26	token TEXT NOT NULL,
	27	last_used BIGINT,
	28	UNIQUE(token)
	29	);
	30
	31	CREATE TABLE IF NOT EXISTS user_ips (
	32	user_id TEXT NOT NULL,
	33	access_token TEXT NOT NULL,
	34	device_id TEXT,
	35	ip TEXT NOT NULL,
	36	user_agent TEXT NOT NULL,
	37	last_seen BIGINT NOT NULL
	38	);
	39
	40	CREATE INDEX user_ips_user ON user_ips(user_id);
	41	CREATE INDEX user_ips_user_ip ON user_ips(user_id, access_token, ip);

+1983

-0

synapse/storage/databases/main/schema/full_schemas/54/full.sql.postgres less more

	0
	1
	2
	3
	4
	5	CREATE TABLE access_tokens (
	6	id bigint NOT NULL,
	7	user_id text NOT NULL,
	8	device_id text,
	9	token text NOT NULL,
	10	last_used bigint
	11	);
	12
	13
	14
	15	CREATE TABLE account_data (
	16	user_id text NOT NULL,
	17	account_data_type text NOT NULL,
	18	stream_id bigint NOT NULL,
	19	content text NOT NULL
	20	);
	21
	22
	23
	24	CREATE TABLE account_data_max_stream_id (
	25	lock character(1) DEFAULT 'X'::bpchar NOT NULL,
	26	stream_id bigint NOT NULL,
	27	CONSTRAINT private_user_data_max_stream_id_lock_check CHECK ((lock = 'X'::bpchar))
	28	);
	29
	30
	31
	32	CREATE TABLE account_validity (
	33	user_id text NOT NULL,
	34	expiration_ts_ms bigint NOT NULL,
	35	email_sent boolean NOT NULL,
	36	renewal_token text
	37	);
	38
	39
	40
	41	CREATE TABLE application_services_state (
	42	as_id text NOT NULL,
	43	state character varying(5),
	44	last_txn integer
	45	);
	46
	47
	48
	49	CREATE TABLE application_services_txns (
	50	as_id text NOT NULL,
	51	txn_id integer NOT NULL,
	52	event_ids text NOT NULL
	53	);
	54
	55
	56
	57	CREATE TABLE appservice_room_list (
	58	appservice_id text NOT NULL,
	59	network_id text NOT NULL,
	60	room_id text NOT NULL
	61	);
	62
	63
	64
	65	CREATE TABLE appservice_stream_position (
	66	lock character(1) DEFAULT 'X'::bpchar NOT NULL,
	67	stream_ordering bigint,
	68	CONSTRAINT appservice_stream_position_lock_check CHECK ((lock = 'X'::bpchar))
	69	);
	70
	71
	72	CREATE TABLE blocked_rooms (
	73	room_id text NOT NULL,
	74	user_id text NOT NULL
	75	);
	76
	77
	78
	79	CREATE TABLE cache_invalidation_stream (
	80	stream_id bigint,
	81	cache_func text,
	82	keys text[],
	83	invalidation_ts bigint
	84	);
	85
	86
	87
	88	CREATE TABLE current_state_delta_stream (
	89	stream_id bigint NOT NULL,
	90	room_id text NOT NULL,
	91	type text NOT NULL,
	92	state_key text NOT NULL,
	93	event_id text,
	94	prev_event_id text
	95	);
	96
	97
	98
	99	CREATE TABLE current_state_events (
	100	event_id text NOT NULL,
	101	room_id text NOT NULL,
	102	type text NOT NULL,
	103	state_key text NOT NULL
	104	);
	105
	106
	107
	108	CREATE TABLE deleted_pushers (
	109	stream_id bigint NOT NULL,
	110	app_id text NOT NULL,
	111	pushkey text NOT NULL,
	112	user_id text NOT NULL
	113	);
	114
	115
	116
	117	CREATE TABLE destinations (
	118	destination text NOT NULL,
	119	retry_last_ts bigint,
	120	retry_interval integer
	121	);
	122
	123
	124
	125	CREATE TABLE device_federation_inbox (
	126	origin text NOT NULL,
	127	message_id text NOT NULL,
	128	received_ts bigint NOT NULL
	129	);
	130
	131
	132
	133	CREATE TABLE device_federation_outbox (
	134	destination text NOT NULL,
	135	stream_id bigint NOT NULL,
	136	queued_ts bigint NOT NULL,
	137	messages_json text NOT NULL
	138	);
	139
	140
	141
	142	CREATE TABLE device_inbox (
	143	user_id text NOT NULL,
	144	device_id text NOT NULL,
	145	stream_id bigint NOT NULL,
	146	message_json text NOT NULL
	147	);
	148
	149
	150
	151	CREATE TABLE device_lists_outbound_last_success (
	152	destination text NOT NULL,
	153	user_id text NOT NULL,
	154	stream_id bigint NOT NULL
	155	);
	156
	157
	158
	159	CREATE TABLE device_lists_outbound_pokes (
	160	destination text NOT NULL,
	161	stream_id bigint NOT NULL,
	162	user_id text NOT NULL,
	163	device_id text NOT NULL,
	164	sent boolean NOT NULL,
	165	ts bigint NOT NULL
	166	);
	167
	168
	169
	170	CREATE TABLE device_lists_remote_cache (
	171	user_id text NOT NULL,
	172	device_id text NOT NULL,
	173	content text NOT NULL
	174	);
	175
	176
	177
	178	CREATE TABLE device_lists_remote_extremeties (
	179	user_id text NOT NULL,
	180	stream_id text NOT NULL
	181	);
	182
	183
	184
	185	CREATE TABLE device_lists_stream (
	186	stream_id bigint NOT NULL,
	187	user_id text NOT NULL,
	188	device_id text NOT NULL
	189	);
	190
	191
	192
	193	CREATE TABLE device_max_stream_id (
	194	stream_id bigint NOT NULL
	195	);
	196
	197
	198
	199	CREATE TABLE devices (
	200	user_id text NOT NULL,
	201	device_id text NOT NULL,
	202	display_name text
	203	);
	204
	205
	206
	207	CREATE TABLE e2e_device_keys_json (
	208	user_id text NOT NULL,
	209	device_id text NOT NULL,
	210	ts_added_ms bigint NOT NULL,
	211	key_json text NOT NULL
	212	);
	213
	214
	215
	216	CREATE TABLE e2e_one_time_keys_json (
	217	user_id text NOT NULL,
	218	device_id text NOT NULL,
	219	algorithm text NOT NULL,
	220	key_id text NOT NULL,
	221	ts_added_ms bigint NOT NULL,
	222	key_json text NOT NULL
	223	);
	224
	225
	226
	227	CREATE TABLE e2e_room_keys (
	228	user_id text NOT NULL,
	229	room_id text NOT NULL,
	230	session_id text NOT NULL,
	231	version bigint NOT NULL,
	232	first_message_index integer,
	233	forwarded_count integer,
	234	is_verified boolean,
	235	session_data text NOT NULL
	236	);
	237
	238
	239
	240	CREATE TABLE e2e_room_keys_versions (
	241	user_id text NOT NULL,
	242	version bigint NOT NULL,
	243	algorithm text NOT NULL,
	244	auth_data text NOT NULL,
	245	deleted smallint DEFAULT 0 NOT NULL
	246	);
	247
	248
	249
	250	CREATE TABLE erased_users (
	251	user_id text NOT NULL
	252	);
	253
	254
	255
	256	CREATE TABLE event_auth (
	257	event_id text NOT NULL,
	258	auth_id text NOT NULL,
	259	room_id text NOT NULL
	260	);
	261
	262
	263
	264	CREATE TABLE event_backward_extremities (
	265	event_id text NOT NULL,
	266	room_id text NOT NULL
	267	);
	268
	269
	270
	271	CREATE TABLE event_edges (
	272	event_id text NOT NULL,
	273	prev_event_id text NOT NULL,
	274	room_id text NOT NULL,
	275	is_state boolean NOT NULL
	276	);
	277
	278
	279
	280	CREATE TABLE event_forward_extremities (
	281	event_id text NOT NULL,
	282	room_id text NOT NULL
	283	);
	284
	285
	286
	287	CREATE TABLE event_json (
	288	event_id text NOT NULL,
	289	room_id text NOT NULL,
	290	internal_metadata text NOT NULL,
	291	json text NOT NULL,
	292	format_version integer
	293	);
	294
	295
	296
	297	CREATE TABLE event_push_actions (
	298	room_id text NOT NULL,
	299	event_id text NOT NULL,
	300	user_id text NOT NULL,
	301	profile_tag character varying(32),
	302	actions text NOT NULL,
	303	topological_ordering bigint,
	304	stream_ordering bigint,
	305	notif smallint,
	306	highlight smallint
	307	);
	308
	309
	310
	311	CREATE TABLE event_push_actions_staging (
	312	event_id text NOT NULL,
	313	user_id text NOT NULL,
	314	actions text NOT NULL,
	315	notif smallint NOT NULL,
	316	highlight smallint NOT NULL
	317	);
	318
	319
	320
	321	CREATE TABLE event_push_summary (
	322	user_id text NOT NULL,
	323	room_id text NOT NULL,
	324	notif_count bigint NOT NULL,
	325	stream_ordering bigint NOT NULL
	326	);
	327
	328
	329
	330	CREATE TABLE event_push_summary_stream_ordering (
	331	lock character(1) DEFAULT 'X'::bpchar NOT NULL,
	332	stream_ordering bigint NOT NULL,
	333	CONSTRAINT event_push_summary_stream_ordering_lock_check CHECK ((lock = 'X'::bpchar))
	334	);
	335
	336
	337
	338	CREATE TABLE event_reference_hashes (
	339	event_id text,
	340	algorithm text,
	341	hash bytea
	342	);
	343
	344
	345
	346	CREATE TABLE event_relations (
	347	event_id text NOT NULL,
	348	relates_to_id text NOT NULL,
	349	relation_type text NOT NULL,
	350	aggregation_key text
	351	);
	352
	353
	354
	355	CREATE TABLE event_reports (
	356	id bigint NOT NULL,
	357	received_ts bigint NOT NULL,
	358	room_id text NOT NULL,
	359	event_id text NOT NULL,
	360	user_id text NOT NULL,
	361	reason text,
	362	content text
	363	);
	364
	365
	366
	367	CREATE TABLE event_search (
	368	event_id text,
	369	room_id text,
	370	sender text,
	371	key text,
	372	vector tsvector,
	373	origin_server_ts bigint,
	374	stream_ordering bigint
	375	);
	376
	377
	378
	379	CREATE TABLE event_to_state_groups (
	380	event_id text NOT NULL,
	381	state_group bigint NOT NULL
	382	);
	383
	384
	385
	386	CREATE TABLE events (
	387	stream_ordering integer NOT NULL,
	388	topological_ordering bigint NOT NULL,
	389	event_id text NOT NULL,
	390	type text NOT NULL,
	391	room_id text NOT NULL,
	392	content text,
	393	unrecognized_keys text,
	394	processed boolean NOT NULL,
	395	outlier boolean NOT NULL,
	396	depth bigint DEFAULT 0 NOT NULL,
	397	origin_server_ts bigint,
	398	received_ts bigint,
	399	sender text,
	400	contains_url boolean
	401	);
	402
	403
	404
	405	CREATE TABLE ex_outlier_stream (
	406	event_stream_ordering bigint NOT NULL,
	407	event_id text NOT NULL,
	408	state_group bigint NOT NULL
	409	);
	410
	411
	412
	413	CREATE TABLE federation_stream_position (
	414	type text NOT NULL,
	415	stream_id integer NOT NULL
	416	);
	417
	418
	419
	420	CREATE TABLE group_attestations_remote (
	421	group_id text NOT NULL,
	422	user_id text NOT NULL,
	423	valid_until_ms bigint NOT NULL,
	424	attestation_json text NOT NULL
	425	);
	426
	427
	428
	429	CREATE TABLE group_attestations_renewals (
	430	group_id text NOT NULL,
	431	user_id text NOT NULL,
	432	valid_until_ms bigint NOT NULL
	433	);
	434
	435
	436
	437	CREATE TABLE group_invites (
	438	group_id text NOT NULL,
	439	user_id text NOT NULL
	440	);
	441
	442
	443
	444	CREATE TABLE group_roles (
	445	group_id text NOT NULL,
	446	role_id text NOT NULL,
	447	profile text NOT NULL,
	448	is_public boolean NOT NULL
	449	);
	450
	451
	452
	453	CREATE TABLE group_room_categories (
	454	group_id text NOT NULL,
	455	category_id text NOT NULL,
	456	profile text NOT NULL,
	457	is_public boolean NOT NULL
	458	);
	459
	460
	461
	462	CREATE TABLE group_rooms (
	463	group_id text NOT NULL,
	464	room_id text NOT NULL,
	465	is_public boolean NOT NULL
	466	);
	467
	468
	469
	470	CREATE TABLE group_summary_roles (
	471	group_id text NOT NULL,
	472	role_id text NOT NULL,
	473	role_order bigint NOT NULL,
	474	CONSTRAINT group_summary_roles_role_order_check CHECK ((role_order > 0))
	475	);
	476
	477
	478
	479	CREATE TABLE group_summary_room_categories (
	480	group_id text NOT NULL,
	481	category_id text NOT NULL,
	482	cat_order bigint NOT NULL,
	483	CONSTRAINT group_summary_room_categories_cat_order_check CHECK ((cat_order > 0))
	484	);
	485
	486
	487
	488	CREATE TABLE group_summary_rooms (
	489	group_id text NOT NULL,
	490	room_id text NOT NULL,
	491	category_id text NOT NULL,
	492	room_order bigint NOT NULL,
	493	is_public boolean NOT NULL,
	494	CONSTRAINT group_summary_rooms_room_order_check CHECK ((room_order > 0))
	495	);
	496
	497
	498
	499	CREATE TABLE group_summary_users (
	500	group_id text NOT NULL,
	501	user_id text NOT NULL,
	502	role_id text NOT NULL,
	503	user_order bigint NOT NULL,
	504	is_public boolean NOT NULL
	505	);
	506
	507
	508
	509	CREATE TABLE group_users (
	510	group_id text NOT NULL,
	511	user_id text NOT NULL,
	512	is_admin boolean NOT NULL,
	513	is_public boolean NOT NULL
	514	);
	515
	516
	517
	518	CREATE TABLE groups (
	519	group_id text NOT NULL,
	520	name text,
	521	avatar_url text,
	522	short_description text,
	523	long_description text,
	524	is_public boolean NOT NULL,
	525	join_policy text DEFAULT 'invite'::text NOT NULL
	526	);
	527
	528
	529
	530	CREATE TABLE guest_access (
	531	event_id text NOT NULL,
	532	room_id text NOT NULL,
	533	guest_access text NOT NULL
	534	);
	535
	536
	537
	538	CREATE TABLE history_visibility (
	539	event_id text NOT NULL,
	540	room_id text NOT NULL,
	541	history_visibility text NOT NULL
	542	);
	543
	544
	545
	546	CREATE TABLE local_group_membership (
	547	group_id text NOT NULL,
	548	user_id text NOT NULL,
	549	is_admin boolean NOT NULL,
	550	membership text NOT NULL,
	551	is_publicised boolean NOT NULL,
	552	content text NOT NULL
	553	);
	554
	555
	556
	557	CREATE TABLE local_group_updates (
	558	stream_id bigint NOT NULL,
	559	group_id text NOT NULL,
	560	user_id text NOT NULL,
	561	type text NOT NULL,
	562	content text NOT NULL
	563	);
	564
	565
	566
	567	CREATE TABLE local_invites (
	568	stream_id bigint NOT NULL,
	569	inviter text NOT NULL,
	570	invitee text NOT NULL,
	571	event_id text NOT NULL,
	572	room_id text NOT NULL,
	573	locally_rejected text,
	574	replaced_by text
	575	);
	576
	577
	578
	579	CREATE TABLE local_media_repository (
	580	media_id text,
	581	media_type text,
	582	media_length integer,
	583	created_ts bigint,
	584	upload_name text,
	585	user_id text,
	586	quarantined_by text,
	587	url_cache text,
	588	last_access_ts bigint
	589	);
	590
	591
	592
	593	CREATE TABLE local_media_repository_thumbnails (
	594	media_id text,
	595	thumbnail_width integer,
	596	thumbnail_height integer,
	597	thumbnail_type text,
	598	thumbnail_method text,
	599	thumbnail_length integer
	600	);
	601
	602
	603
	604	CREATE TABLE local_media_repository_url_cache (
	605	url text,
	606	response_code integer,
	607	etag text,
	608	expires_ts bigint,
	609	og text,
	610	media_id text,
	611	download_ts bigint
	612	);
	613
	614
	615
	616	CREATE TABLE monthly_active_users (
	617	user_id text NOT NULL,
	618	"timestamp" bigint NOT NULL
	619	);
	620
	621
	622
	623	CREATE TABLE open_id_tokens (
	624	token text NOT NULL,
	625	ts_valid_until_ms bigint NOT NULL,
	626	user_id text NOT NULL
	627	);
	628
	629
	630
	631	CREATE TABLE presence (
	632	user_id text NOT NULL,
	633	state character varying(20),
	634	status_msg text,
	635	mtime bigint
	636	);
	637
	638
	639
	640	CREATE TABLE presence_allow_inbound (
	641	observed_user_id text NOT NULL,
	642	observer_user_id text NOT NULL
	643	);
	644
	645
	646
	647	CREATE TABLE presence_stream (
	648	stream_id bigint,
	649	user_id text,
	650	state text,
	651	last_active_ts bigint,
	652	last_federation_update_ts bigint,
	653	last_user_sync_ts bigint,
	654	status_msg text,
	655	currently_active boolean
	656	);
	657
	658
	659
	660	CREATE TABLE profiles (
	661	user_id text NOT NULL,
	662	displayname text,
	663	avatar_url text
	664	);
	665
	666
	667
	668	CREATE TABLE public_room_list_stream (
	669	stream_id bigint NOT NULL,
	670	room_id text NOT NULL,
	671	visibility boolean NOT NULL,
	672	appservice_id text,
	673	network_id text
	674	);
	675
	676
	677
	678	CREATE TABLE push_rules (
	679	id bigint NOT NULL,
	680	user_name text NOT NULL,
	681	rule_id text NOT NULL,
	682	priority_class smallint NOT NULL,
	683	priority integer DEFAULT 0 NOT NULL,
	684	conditions text NOT NULL,
	685	actions text NOT NULL
	686	);
	687
	688
	689
	690	CREATE TABLE push_rules_enable (
	691	id bigint NOT NULL,
	692	user_name text NOT NULL,
	693	rule_id text NOT NULL,
	694	enabled smallint
	695	);
	696
	697
	698
	699	CREATE TABLE push_rules_stream (
	700	stream_id bigint NOT NULL,
	701	event_stream_ordering bigint NOT NULL,
	702	user_id text NOT NULL,
	703	rule_id text NOT NULL,
	704	op text NOT NULL,
	705	priority_class smallint,
	706	priority integer,
	707	conditions text,
	708	actions text
	709	);
	710
	711
	712
	713	CREATE TABLE pusher_throttle (
	714	pusher bigint NOT NULL,
	715	room_id text NOT NULL,
	716	last_sent_ts bigint,
	717	throttle_ms bigint
	718	);
	719
	720
	721
	722	CREATE TABLE pushers (
	723	id bigint NOT NULL,
	724	user_name text NOT NULL,
	725	access_token bigint,
	726	profile_tag text NOT NULL,
	727	kind text NOT NULL,
	728	app_id text NOT NULL,
	729	app_display_name text NOT NULL,
	730	device_display_name text NOT NULL,
	731	pushkey text NOT NULL,
	732	ts bigint NOT NULL,
	733	lang text,
	734	data text,
	735	last_stream_ordering integer,
	736	last_success bigint,
	737	failing_since bigint
	738	);
	739
	740
	741
	742	CREATE TABLE ratelimit_override (
	743	user_id text NOT NULL,
	744	messages_per_second bigint,
	745	burst_count bigint
	746	);
	747
	748
	749
	750	CREATE TABLE receipts_graph (
	751	room_id text NOT NULL,
	752	receipt_type text NOT NULL,
	753	user_id text NOT NULL,
	754	event_ids text NOT NULL,
	755	data text NOT NULL
	756	);
	757
	758
	759
	760	CREATE TABLE receipts_linearized (
	761	stream_id bigint NOT NULL,
	762	room_id text NOT NULL,
	763	receipt_type text NOT NULL,
	764	user_id text NOT NULL,
	765	event_id text NOT NULL,
	766	data text NOT NULL
	767	);
	768
	769
	770
	771	CREATE TABLE received_transactions (
	772	transaction_id text,
	773	origin text,
	774	ts bigint,
	775	response_code integer,
	776	response_json bytea,
	777	has_been_referenced smallint DEFAULT 0
	778	);
	779
	780
	781
	782	CREATE TABLE redactions (
	783	event_id text NOT NULL,
	784	redacts text NOT NULL
	785	);
	786
	787
	788
	789	CREATE TABLE rejections (
	790	event_id text NOT NULL,
	791	reason text NOT NULL,
	792	last_check text NOT NULL
	793	);
	794
	795
	796
	797	CREATE TABLE remote_media_cache (
	798	media_origin text,
	799	media_id text,
	800	media_type text,
	801	created_ts bigint,
	802	upload_name text,
	803	media_length integer,
	804	filesystem_id text,
	805	last_access_ts bigint,
	806	quarantined_by text
	807	);
	808
	809
	810
	811	CREATE TABLE remote_media_cache_thumbnails (
	812	media_origin text,
	813	media_id text,
	814	thumbnail_width integer,
	815	thumbnail_height integer,
	816	thumbnail_method text,
	817	thumbnail_type text,
	818	thumbnail_length integer,
	819	filesystem_id text
	820	);
	821
	822
	823
	824	CREATE TABLE remote_profile_cache (
	825	user_id text NOT NULL,
	826	displayname text,
	827	avatar_url text,
	828	last_check bigint NOT NULL
	829	);
	830
	831
	832
	833	CREATE TABLE room_account_data (
	834	user_id text NOT NULL,
	835	room_id text NOT NULL,
	836	account_data_type text NOT NULL,
	837	stream_id bigint NOT NULL,
	838	content text NOT NULL
	839	);
	840
	841
	842
	843	CREATE TABLE room_alias_servers (
	844	room_alias text NOT NULL,
	845	server text NOT NULL
	846	);
	847
	848
	849
	850	CREATE TABLE room_aliases (
	851	room_alias text NOT NULL,
	852	room_id text NOT NULL,
	853	creator text
	854	);
	855
	856
	857
	858	CREATE TABLE room_depth (
	859	room_id text NOT NULL,
	860	min_depth integer NOT NULL
	861	);
	862
	863
	864
	865	CREATE TABLE room_memberships (
	866	event_id text NOT NULL,
	867	user_id text NOT NULL,
	868	sender text NOT NULL,
	869	room_id text NOT NULL,
	870	membership text NOT NULL,
	871	forgotten integer DEFAULT 0,
	872	display_name text,
	873	avatar_url text
	874	);
	875
	876
	877
	878	CREATE TABLE room_names (
	879	event_id text NOT NULL,
	880	room_id text NOT NULL,
	881	name text NOT NULL
	882	);
	883
	884
	885
	886	CREATE TABLE room_state (
	887	room_id text NOT NULL,
	888	join_rules text,
	889	history_visibility text,
	890	encryption text,
	891	name text,
	892	topic text,
	893	avatar text,
	894	canonical_alias text
	895	);
	896
	897
	898
	899	CREATE TABLE room_stats (
	900	room_id text NOT NULL,
	901	ts bigint NOT NULL,
	902	bucket_size integer NOT NULL,
	903	current_state_events integer NOT NULL,
	904	joined_members integer NOT NULL,
	905	invited_members integer NOT NULL,
	906	left_members integer NOT NULL,
	907	banned_members integer NOT NULL,
	908	state_events integer NOT NULL
	909	);
	910
	911
	912
	913	CREATE TABLE room_stats_earliest_token (
	914	room_id text NOT NULL,
	915	token bigint NOT NULL
	916	);
	917
	918
	919
	920	CREATE TABLE room_tags (
	921	user_id text NOT NULL,
	922	room_id text NOT NULL,
	923	tag text NOT NULL,
	924	content text NOT NULL
	925	);
	926
	927
	928
	929	CREATE TABLE room_tags_revisions (
	930	user_id text NOT NULL,
	931	room_id text NOT NULL,
	932	stream_id bigint NOT NULL
	933	);
	934
	935
	936
	937	CREATE TABLE rooms (
	938	room_id text NOT NULL,
	939	is_public boolean,
	940	creator text
	941	);
	942
	943
	944
	945	CREATE TABLE server_keys_json (
	946	server_name text NOT NULL,
	947	key_id text NOT NULL,
	948	from_server text NOT NULL,
	949	ts_added_ms bigint NOT NULL,
	950	ts_valid_until_ms bigint NOT NULL,
	951	key_json bytea NOT NULL
	952	);
	953
	954
	955
	956	CREATE TABLE server_signature_keys (
	957	server_name text,
	958	key_id text,
	959	from_server text,
	960	ts_added_ms bigint,
	961	verify_key bytea,
	962	ts_valid_until_ms bigint
	963	);
	964
	965
	966
	967	CREATE TABLE state_events (
	968	event_id text NOT NULL,
	969	room_id text NOT NULL,
	970	type text NOT NULL,
	971	state_key text NOT NULL,
	972	prev_state text
	973	);
	974
	975
	976
	977	CREATE TABLE stats_stream_pos (
	978	lock character(1) DEFAULT 'X'::bpchar NOT NULL,
	979	stream_id bigint,
	980	CONSTRAINT stats_stream_pos_lock_check CHECK ((lock = 'X'::bpchar))
	981	);
	982
	983
	984
	985	CREATE TABLE stream_ordering_to_exterm (
	986	stream_ordering bigint NOT NULL,
	987	room_id text NOT NULL,
	988	event_id text NOT NULL
	989	);
	990
	991
	992
	993	CREATE TABLE threepid_guest_access_tokens (
	994	medium text,
	995	address text,
	996	guest_access_token text,
	997	first_inviter text
	998	);
	999
	1000
	1001
	1002	CREATE TABLE topics (
	1003	event_id text NOT NULL,
	1004	room_id text NOT NULL,
	1005	topic text NOT NULL
	1006	);
	1007
	1008
	1009
	1010	CREATE TABLE user_daily_visits (
	1011	user_id text NOT NULL,
	1012	device_id text,
	1013	"timestamp" bigint NOT NULL
	1014	);
	1015
	1016
	1017
	1018	CREATE TABLE user_directory (
	1019	user_id text NOT NULL,
	1020	room_id text,
	1021	display_name text,
	1022	avatar_url text
	1023	);
	1024
	1025
	1026
	1027	CREATE TABLE user_directory_search (
	1028	user_id text NOT NULL,
	1029	vector tsvector
	1030	);
	1031
	1032
	1033
	1034	CREATE TABLE user_directory_stream_pos (
	1035	lock character(1) DEFAULT 'X'::bpchar NOT NULL,
	1036	stream_id bigint,
	1037	CONSTRAINT user_directory_stream_pos_lock_check CHECK ((lock = 'X'::bpchar))
	1038	);
	1039
	1040
	1041
	1042	CREATE TABLE user_filters (
	1043	user_id text,
	1044	filter_id bigint,
	1045	filter_json bytea
	1046	);
	1047
	1048
	1049
	1050	CREATE TABLE user_ips (
	1051	user_id text NOT NULL,
	1052	access_token text NOT NULL,
	1053	device_id text,
	1054	ip text NOT NULL,
	1055	user_agent text NOT NULL,
	1056	last_seen bigint NOT NULL
	1057	);
	1058
	1059
	1060
	1061	CREATE TABLE user_stats (
	1062	user_id text NOT NULL,
	1063	ts bigint NOT NULL,
	1064	bucket_size integer NOT NULL,
	1065	public_rooms integer NOT NULL,
	1066	private_rooms integer NOT NULL
	1067	);
	1068
	1069
	1070
	1071	CREATE TABLE user_threepid_id_server (
	1072	user_id text NOT NULL,
	1073	medium text NOT NULL,
	1074	address text NOT NULL,
	1075	id_server text NOT NULL
	1076	);
	1077
	1078
	1079
	1080	CREATE TABLE user_threepids (
	1081	user_id text NOT NULL,
	1082	medium text NOT NULL,
	1083	address text NOT NULL,
	1084	validated_at bigint NOT NULL,
	1085	added_at bigint NOT NULL
	1086	);
	1087
	1088
	1089
	1090	CREATE TABLE users (
	1091	name text,
	1092	password_hash text,
	1093	creation_ts bigint,
	1094	admin smallint DEFAULT 0 NOT NULL,
	1095	upgrade_ts bigint,
	1096	is_guest smallint DEFAULT 0 NOT NULL,
	1097	appservice_id text,
	1098	consent_version text,
	1099	consent_server_notice_sent text,
	1100	user_type text
	1101	);
	1102
	1103
	1104
	1105	CREATE TABLE users_in_public_rooms (
	1106	user_id text NOT NULL,
	1107	room_id text NOT NULL
	1108	);
	1109
	1110
	1111
	1112	CREATE TABLE users_pending_deactivation (
	1113	user_id text NOT NULL
	1114	);
	1115
	1116
	1117
	1118	CREATE TABLE users_who_share_private_rooms (
	1119	user_id text NOT NULL,
	1120	other_user_id text NOT NULL,
	1121	room_id text NOT NULL
	1122	);
	1123
	1124
	1125
	1126	ALTER TABLE ONLY access_tokens
	1127	ADD CONSTRAINT access_tokens_pkey PRIMARY KEY (id);
	1128
	1129
	1130
	1131	ALTER TABLE ONLY access_tokens
	1132	ADD CONSTRAINT access_tokens_token_key UNIQUE (token);
	1133
	1134
	1135
	1136	ALTER TABLE ONLY account_data
	1137	ADD CONSTRAINT account_data_uniqueness UNIQUE (user_id, account_data_type);
	1138
	1139
	1140
	1141	ALTER TABLE ONLY account_validity
	1142	ADD CONSTRAINT account_validity_pkey PRIMARY KEY (user_id);
	1143
	1144
	1145
	1146	ALTER TABLE ONLY application_services_state
	1147	ADD CONSTRAINT application_services_state_pkey PRIMARY KEY (as_id);
	1148
	1149
	1150
	1151	ALTER TABLE ONLY application_services_txns
	1152	ADD CONSTRAINT application_services_txns_as_id_txn_id_key UNIQUE (as_id, txn_id);
	1153
	1154
	1155
	1156	ALTER TABLE ONLY appservice_stream_position
	1157	ADD CONSTRAINT appservice_stream_position_lock_key UNIQUE (lock);
	1158
	1159
	1160
	1161	ALTER TABLE ONLY current_state_events
	1162	ADD CONSTRAINT current_state_events_event_id_key UNIQUE (event_id);
	1163
	1164
	1165
	1166	ALTER TABLE ONLY current_state_events
	1167	ADD CONSTRAINT current_state_events_room_id_type_state_key_key UNIQUE (room_id, type, state_key);
	1168
	1169
	1170
	1171	ALTER TABLE ONLY destinations
	1172	ADD CONSTRAINT destinations_pkey PRIMARY KEY (destination);
	1173
	1174
	1175
	1176	ALTER TABLE ONLY devices
	1177	ADD CONSTRAINT device_uniqueness UNIQUE (user_id, device_id);
	1178
	1179
	1180
	1181	ALTER TABLE ONLY e2e_device_keys_json
	1182	ADD CONSTRAINT e2e_device_keys_json_uniqueness UNIQUE (user_id, device_id);
	1183
	1184
	1185
	1186	ALTER TABLE ONLY e2e_one_time_keys_json
	1187	ADD CONSTRAINT e2e_one_time_keys_json_uniqueness UNIQUE (user_id, device_id, algorithm, key_id);
	1188
	1189
	1190
	1191	ALTER TABLE ONLY event_backward_extremities
	1192	ADD CONSTRAINT event_backward_extremities_event_id_room_id_key UNIQUE (event_id, room_id);
	1193
	1194
	1195
	1196	ALTER TABLE ONLY event_edges
	1197	ADD CONSTRAINT event_edges_event_id_prev_event_id_room_id_is_state_key UNIQUE (event_id, prev_event_id, room_id, is_state);
	1198
	1199
	1200
	1201	ALTER TABLE ONLY event_forward_extremities
	1202	ADD CONSTRAINT event_forward_extremities_event_id_room_id_key UNIQUE (event_id, room_id);
	1203
	1204
	1205
	1206	ALTER TABLE ONLY event_push_actions
	1207	ADD CONSTRAINT event_id_user_id_profile_tag_uniqueness UNIQUE (room_id, event_id, user_id, profile_tag);
	1208
	1209
	1210
	1211	ALTER TABLE ONLY event_json
	1212	ADD CONSTRAINT event_json_event_id_key UNIQUE (event_id);
	1213
	1214
	1215
	1216	ALTER TABLE ONLY event_push_summary_stream_ordering
	1217	ADD CONSTRAINT event_push_summary_stream_ordering_lock_key UNIQUE (lock);
	1218
	1219
	1220
	1221	ALTER TABLE ONLY event_reference_hashes
	1222	ADD CONSTRAINT event_reference_hashes_event_id_algorithm_key UNIQUE (event_id, algorithm);
	1223
	1224
	1225
	1226	ALTER TABLE ONLY event_reports
	1227	ADD CONSTRAINT event_reports_pkey PRIMARY KEY (id);
	1228
	1229
	1230
	1231	ALTER TABLE ONLY event_to_state_groups
	1232	ADD CONSTRAINT event_to_state_groups_event_id_key UNIQUE (event_id);
	1233
	1234
	1235
	1236	ALTER TABLE ONLY events
	1237	ADD CONSTRAINT events_event_id_key UNIQUE (event_id);
	1238
	1239
	1240
	1241	ALTER TABLE ONLY events
	1242	ADD CONSTRAINT events_pkey PRIMARY KEY (stream_ordering);
	1243
	1244
	1245
	1246	ALTER TABLE ONLY ex_outlier_stream
	1247	ADD CONSTRAINT ex_outlier_stream_pkey PRIMARY KEY (event_stream_ordering);
	1248
	1249
	1250
	1251	ALTER TABLE ONLY group_roles
	1252	ADD CONSTRAINT group_roles_group_id_role_id_key UNIQUE (group_id, role_id);
	1253
	1254
	1255
	1256	ALTER TABLE ONLY group_room_categories
	1257	ADD CONSTRAINT group_room_categories_group_id_category_id_key UNIQUE (group_id, category_id);
	1258
	1259
	1260
	1261	ALTER TABLE ONLY group_summary_roles
	1262	ADD CONSTRAINT group_summary_roles_group_id_role_id_role_order_key UNIQUE (group_id, role_id, role_order);
	1263
	1264
	1265
	1266	ALTER TABLE ONLY group_summary_room_categories
	1267	ADD CONSTRAINT group_summary_room_categories_group_id_category_id_cat_orde_key UNIQUE (group_id, category_id, cat_order);
	1268
	1269
	1270
	1271	ALTER TABLE ONLY group_summary_rooms
	1272	ADD CONSTRAINT group_summary_rooms_group_id_category_id_room_id_room_order_key UNIQUE (group_id, category_id, room_id, room_order);
	1273
	1274
	1275
	1276	ALTER TABLE ONLY guest_access
	1277	ADD CONSTRAINT guest_access_event_id_key UNIQUE (event_id);
	1278
	1279
	1280
	1281	ALTER TABLE ONLY history_visibility
	1282	ADD CONSTRAINT history_visibility_event_id_key UNIQUE (event_id);
	1283
	1284
	1285
	1286	ALTER TABLE ONLY local_media_repository
	1287	ADD CONSTRAINT local_media_repository_media_id_key UNIQUE (media_id);
	1288
	1289
	1290
	1291	ALTER TABLE ONLY local_media_repository_thumbnails
	1292	ADD CONSTRAINT local_media_repository_thumbn_media_id_thumbnail_width_thum_key UNIQUE (media_id, thumbnail_width, thumbnail_height, thumbnail_type);
	1293
	1294
	1295
	1296	ALTER TABLE ONLY user_threepids
	1297	ADD CONSTRAINT medium_address UNIQUE (medium, address);
	1298
	1299
	1300
	1301	ALTER TABLE ONLY open_id_tokens
	1302	ADD CONSTRAINT open_id_tokens_pkey PRIMARY KEY (token);
	1303
	1304
	1305
	1306	ALTER TABLE ONLY presence_allow_inbound
	1307	ADD CONSTRAINT presence_allow_inbound_observed_user_id_observer_user_id_key UNIQUE (observed_user_id, observer_user_id);
	1308
	1309
	1310
	1311	ALTER TABLE ONLY presence
	1312	ADD CONSTRAINT presence_user_id_key UNIQUE (user_id);
	1313
	1314
	1315
	1316	ALTER TABLE ONLY account_data_max_stream_id
	1317	ADD CONSTRAINT private_user_data_max_stream_id_lock_key UNIQUE (lock);
	1318
	1319
	1320
	1321	ALTER TABLE ONLY profiles
	1322	ADD CONSTRAINT profiles_user_id_key UNIQUE (user_id);
	1323
	1324
	1325
	1326	ALTER TABLE ONLY push_rules_enable
	1327	ADD CONSTRAINT push_rules_enable_pkey PRIMARY KEY (id);
	1328
	1329
	1330
	1331	ALTER TABLE ONLY push_rules_enable
	1332	ADD CONSTRAINT push_rules_enable_user_name_rule_id_key UNIQUE (user_name, rule_id);
	1333
	1334
	1335
	1336	ALTER TABLE ONLY push_rules
	1337	ADD CONSTRAINT push_rules_pkey PRIMARY KEY (id);
	1338
	1339
	1340
	1341	ALTER TABLE ONLY push_rules
	1342	ADD CONSTRAINT push_rules_user_name_rule_id_key UNIQUE (user_name, rule_id);
	1343
	1344
	1345
	1346	ALTER TABLE ONLY pusher_throttle
	1347	ADD CONSTRAINT pusher_throttle_pkey PRIMARY KEY (pusher, room_id);
	1348
	1349
	1350
	1351	ALTER TABLE ONLY pushers
	1352	ADD CONSTRAINT pushers2_app_id_pushkey_user_name_key UNIQUE (app_id, pushkey, user_name);
	1353
	1354
	1355
	1356	ALTER TABLE ONLY pushers
	1357	ADD CONSTRAINT pushers2_pkey PRIMARY KEY (id);
	1358
	1359
	1360
	1361	ALTER TABLE ONLY receipts_graph
	1362	ADD CONSTRAINT receipts_graph_uniqueness UNIQUE (room_id, receipt_type, user_id);
	1363
	1364
	1365
	1366	ALTER TABLE ONLY receipts_linearized
	1367	ADD CONSTRAINT receipts_linearized_uniqueness UNIQUE (room_id, receipt_type, user_id);
	1368
	1369
	1370
	1371	ALTER TABLE ONLY received_transactions
	1372	ADD CONSTRAINT received_transactions_transaction_id_origin_key UNIQUE (transaction_id, origin);
	1373
	1374
	1375
	1376	ALTER TABLE ONLY redactions
	1377	ADD CONSTRAINT redactions_event_id_key UNIQUE (event_id);
	1378
	1379
	1380
	1381	ALTER TABLE ONLY rejections
	1382	ADD CONSTRAINT rejections_event_id_key UNIQUE (event_id);
	1383
	1384
	1385
	1386	ALTER TABLE ONLY remote_media_cache
	1387	ADD CONSTRAINT remote_media_cache_media_origin_media_id_key UNIQUE (media_origin, media_id);
	1388
	1389
	1390
	1391	ALTER TABLE ONLY remote_media_cache_thumbnails
	1392	ADD CONSTRAINT remote_media_cache_thumbnails_media_origin_media_id_thumbna_key UNIQUE (media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type);
	1393
	1394
	1395
	1396	ALTER TABLE ONLY room_account_data
	1397	ADD CONSTRAINT room_account_data_uniqueness UNIQUE (user_id, room_id, account_data_type);
	1398
	1399
	1400
	1401	ALTER TABLE ONLY room_aliases
	1402	ADD CONSTRAINT room_aliases_room_alias_key UNIQUE (room_alias);
	1403
	1404
	1405
	1406	ALTER TABLE ONLY room_depth
	1407	ADD CONSTRAINT room_depth_room_id_key UNIQUE (room_id);
	1408
	1409
	1410
	1411	ALTER TABLE ONLY room_memberships
	1412	ADD CONSTRAINT room_memberships_event_id_key UNIQUE (event_id);
	1413
	1414
	1415
	1416	ALTER TABLE ONLY room_names
	1417	ADD CONSTRAINT room_names_event_id_key UNIQUE (event_id);
	1418
	1419
	1420
	1421	ALTER TABLE ONLY room_tags_revisions
	1422	ADD CONSTRAINT room_tag_revisions_uniqueness UNIQUE (user_id, room_id);
	1423
	1424
	1425
	1426	ALTER TABLE ONLY room_tags
	1427	ADD CONSTRAINT room_tag_uniqueness UNIQUE (user_id, room_id, tag);
	1428
	1429
	1430
	1431	ALTER TABLE ONLY rooms
	1432	ADD CONSTRAINT rooms_pkey PRIMARY KEY (room_id);
	1433
	1434
	1435
	1436	ALTER TABLE ONLY server_keys_json
	1437	ADD CONSTRAINT server_keys_json_uniqueness UNIQUE (server_name, key_id, from_server);
	1438
	1439
	1440
	1441	ALTER TABLE ONLY server_signature_keys
	1442	ADD CONSTRAINT server_signature_keys_server_name_key_id_key UNIQUE (server_name, key_id);
	1443
	1444
	1445
	1446	ALTER TABLE ONLY state_events
	1447	ADD CONSTRAINT state_events_event_id_key UNIQUE (event_id);
	1448
	1449
	1450	ALTER TABLE ONLY stats_stream_pos
	1451	ADD CONSTRAINT stats_stream_pos_lock_key UNIQUE (lock);
	1452
	1453
	1454
	1455	ALTER TABLE ONLY topics
	1456	ADD CONSTRAINT topics_event_id_key UNIQUE (event_id);
	1457
	1458
	1459
	1460	ALTER TABLE ONLY user_directory_stream_pos
	1461	ADD CONSTRAINT user_directory_stream_pos_lock_key UNIQUE (lock);
	1462
	1463
	1464
	1465	ALTER TABLE ONLY users
	1466	ADD CONSTRAINT users_name_key UNIQUE (name);
	1467
	1468
	1469
	1470	CREATE INDEX access_tokens_device_id ON access_tokens USING btree (user_id, device_id);
	1471
	1472
	1473
	1474	CREATE INDEX account_data_stream_id ON account_data USING btree (user_id, stream_id);
	1475
	1476
	1477
	1478	CREATE INDEX application_services_txns_id ON application_services_txns USING btree (as_id);
	1479
	1480
	1481
	1482	CREATE UNIQUE INDEX appservice_room_list_idx ON appservice_room_list USING btree (appservice_id, network_id, room_id);
	1483
	1484
	1485
	1486	CREATE UNIQUE INDEX blocked_rooms_idx ON blocked_rooms USING btree (room_id);
	1487
	1488
	1489
	1490	CREATE INDEX cache_invalidation_stream_id ON cache_invalidation_stream USING btree (stream_id);
	1491
	1492
	1493
	1494	CREATE INDEX current_state_delta_stream_idx ON current_state_delta_stream USING btree (stream_id);
	1495
	1496
	1497
	1498	CREATE INDEX current_state_events_member_index ON current_state_events USING btree (state_key) WHERE (type = 'm.room.member'::text);
	1499
	1500
	1501
	1502	CREATE INDEX deleted_pushers_stream_id ON deleted_pushers USING btree (stream_id);
	1503
	1504
	1505
	1506	CREATE INDEX device_federation_inbox_sender_id ON device_federation_inbox USING btree (origin, message_id);
	1507
	1508
	1509
	1510	CREATE INDEX device_federation_outbox_destination_id ON device_federation_outbox USING btree (destination, stream_id);
	1511
	1512
	1513
	1514	CREATE INDEX device_federation_outbox_id ON device_federation_outbox USING btree (stream_id);
	1515
	1516
	1517
	1518	CREATE INDEX device_inbox_stream_id_user_id ON device_inbox USING btree (stream_id, user_id);
	1519
	1520
	1521
	1522	CREATE INDEX device_inbox_user_stream_id ON device_inbox USING btree (user_id, device_id, stream_id);
	1523
	1524
	1525
	1526	CREATE INDEX device_lists_outbound_last_success_idx ON device_lists_outbound_last_success USING btree (destination, user_id, stream_id);
	1527
	1528
	1529
	1530	CREATE INDEX device_lists_outbound_pokes_id ON device_lists_outbound_pokes USING btree (destination, stream_id);
	1531
	1532
	1533
	1534	CREATE INDEX device_lists_outbound_pokes_stream ON device_lists_outbound_pokes USING btree (stream_id);
	1535
	1536
	1537
	1538	CREATE INDEX device_lists_outbound_pokes_user ON device_lists_outbound_pokes USING btree (destination, user_id);
	1539
	1540
	1541
	1542	CREATE UNIQUE INDEX device_lists_remote_cache_unique_id ON device_lists_remote_cache USING btree (user_id, device_id);
	1543
	1544
	1545
	1546	CREATE UNIQUE INDEX device_lists_remote_extremeties_unique_idx ON device_lists_remote_extremeties USING btree (user_id);
	1547
	1548
	1549
	1550	CREATE INDEX device_lists_stream_id ON device_lists_stream USING btree (stream_id, user_id);
	1551
	1552
	1553
	1554	CREATE INDEX device_lists_stream_user_id ON device_lists_stream USING btree (user_id, device_id);
	1555
	1556
	1557
	1558	CREATE UNIQUE INDEX e2e_room_keys_idx ON e2e_room_keys USING btree (user_id, room_id, session_id);
	1559
	1560
	1561
	1562	CREATE UNIQUE INDEX e2e_room_keys_versions_idx ON e2e_room_keys_versions USING btree (user_id, version);
	1563
	1564
	1565
	1566	CREATE UNIQUE INDEX erased_users_user ON erased_users USING btree (user_id);
	1567
	1568
	1569
	1570	CREATE INDEX ev_b_extrem_id ON event_backward_extremities USING btree (event_id);
	1571
	1572
	1573
	1574	CREATE INDEX ev_b_extrem_room ON event_backward_extremities USING btree (room_id);
	1575
	1576
	1577
	1578	CREATE INDEX ev_edges_id ON event_edges USING btree (event_id);
	1579
	1580
	1581
	1582	CREATE INDEX ev_edges_prev_id ON event_edges USING btree (prev_event_id);
	1583
	1584
	1585
	1586	CREATE INDEX ev_extrem_id ON event_forward_extremities USING btree (event_id);
	1587
	1588
	1589
	1590	CREATE INDEX ev_extrem_room ON event_forward_extremities USING btree (room_id);
	1591
	1592
	1593
	1594	CREATE INDEX evauth_edges_id ON event_auth USING btree (event_id);
	1595
	1596
	1597
	1598	CREATE INDEX event_contains_url_index ON events USING btree (room_id, topological_ordering, stream_ordering) WHERE ((contains_url = true) AND (outlier = false));
	1599
	1600
	1601
	1602	CREATE INDEX event_json_room_id ON event_json USING btree (room_id);
	1603
	1604
	1605
	1606	CREATE INDEX event_push_actions_highlights_index ON event_push_actions USING btree (user_id, room_id, topological_ordering, stream_ordering) WHERE (highlight = 1);
	1607
	1608
	1609
	1610	CREATE INDEX event_push_actions_rm_tokens ON event_push_actions USING btree (user_id, room_id, topological_ordering, stream_ordering);
	1611
	1612
	1613
	1614	CREATE INDEX event_push_actions_room_id_user_id ON event_push_actions USING btree (room_id, user_id);
	1615
	1616
	1617
	1618	CREATE INDEX event_push_actions_staging_id ON event_push_actions_staging USING btree (event_id);
	1619
	1620
	1621
	1622	CREATE INDEX event_push_actions_stream_ordering ON event_push_actions USING btree (stream_ordering, user_id);
	1623
	1624
	1625
	1626	CREATE INDEX event_push_actions_u_highlight ON event_push_actions USING btree (user_id, stream_ordering);
	1627
	1628
	1629
	1630	CREATE INDEX event_push_summary_user_rm ON event_push_summary USING btree (user_id, room_id);
	1631
	1632
	1633
	1634	CREATE INDEX event_reference_hashes_id ON event_reference_hashes USING btree (event_id);
	1635
	1636
	1637
	1638	CREATE UNIQUE INDEX event_relations_id ON event_relations USING btree (event_id);
	1639
	1640
	1641
	1642	CREATE INDEX event_relations_relates ON event_relations USING btree (relates_to_id, relation_type, aggregation_key);
	1643
	1644
	1645
	1646	CREATE INDEX event_search_ev_ridx ON event_search USING btree (room_id);
	1647
	1648
	1649
	1650	CREATE UNIQUE INDEX event_search_event_id_idx ON event_search USING btree (event_id);
	1651
	1652
	1653
	1654	CREATE INDEX event_search_fts_idx ON event_search USING gin (vector);
	1655
	1656
	1657
	1658	CREATE INDEX event_to_state_groups_sg_index ON event_to_state_groups USING btree (state_group);
	1659
	1660
	1661
	1662	CREATE INDEX events_order_room ON events USING btree (room_id, topological_ordering, stream_ordering);
	1663
	1664
	1665
	1666	CREATE INDEX events_room_stream ON events USING btree (room_id, stream_ordering);
	1667
	1668
	1669
	1670	CREATE INDEX events_ts ON events USING btree (origin_server_ts, stream_ordering);
	1671
	1672
	1673
	1674	CREATE INDEX group_attestations_remote_g_idx ON group_attestations_remote USING btree (group_id, user_id);
	1675
	1676
	1677
	1678	CREATE INDEX group_attestations_remote_u_idx ON group_attestations_remote USING btree (user_id);
	1679
	1680
	1681
	1682	CREATE INDEX group_attestations_remote_v_idx ON group_attestations_remote USING btree (valid_until_ms);
	1683
	1684
	1685
	1686	CREATE INDEX group_attestations_renewals_g_idx ON group_attestations_renewals USING btree (group_id, user_id);
	1687
	1688
	1689
	1690	CREATE INDEX group_attestations_renewals_u_idx ON group_attestations_renewals USING btree (user_id);
	1691
	1692
	1693
	1694	CREATE INDEX group_attestations_renewals_v_idx ON group_attestations_renewals USING btree (valid_until_ms);
	1695
	1696
	1697
	1698	CREATE UNIQUE INDEX group_invites_g_idx ON group_invites USING btree (group_id, user_id);
	1699
	1700
	1701
	1702	CREATE INDEX group_invites_u_idx ON group_invites USING btree (user_id);
	1703
	1704
	1705
	1706	CREATE UNIQUE INDEX group_rooms_g_idx ON group_rooms USING btree (group_id, room_id);
	1707
	1708
	1709
	1710	CREATE INDEX group_rooms_r_idx ON group_rooms USING btree (room_id);
	1711
	1712
	1713
	1714	CREATE UNIQUE INDEX group_summary_rooms_g_idx ON group_summary_rooms USING btree (group_id, room_id, category_id);
	1715
	1716
	1717
	1718	CREATE INDEX group_summary_users_g_idx ON group_summary_users USING btree (group_id);
	1719
	1720
	1721
	1722	CREATE UNIQUE INDEX group_users_g_idx ON group_users USING btree (group_id, user_id);
	1723
	1724
	1725
	1726	CREATE INDEX group_users_u_idx ON group_users USING btree (user_id);
	1727
	1728
	1729
	1730	CREATE UNIQUE INDEX groups_idx ON groups USING btree (group_id);
	1731
	1732
	1733
	1734	CREATE INDEX local_group_membership_g_idx ON local_group_membership USING btree (group_id);
	1735
	1736
	1737
	1738	CREATE INDEX local_group_membership_u_idx ON local_group_membership USING btree (user_id, group_id);
	1739
	1740
	1741
	1742	CREATE INDEX local_invites_for_user_idx ON local_invites USING btree (invitee, locally_rejected, replaced_by, room_id);
	1743
	1744
	1745
	1746	CREATE INDEX local_invites_id ON local_invites USING btree (stream_id);
	1747
	1748
	1749
	1750	CREATE INDEX local_media_repository_thumbnails_media_id ON local_media_repository_thumbnails USING btree (media_id);
	1751
	1752
	1753
	1754	CREATE INDEX local_media_repository_url_cache_by_url_download_ts ON local_media_repository_url_cache USING btree (url, download_ts);
	1755
	1756
	1757
	1758	CREATE INDEX local_media_repository_url_cache_expires_idx ON local_media_repository_url_cache USING btree (expires_ts);
	1759
	1760
	1761
	1762	CREATE INDEX local_media_repository_url_cache_media_idx ON local_media_repository_url_cache USING btree (media_id);
	1763
	1764
	1765
	1766	CREATE INDEX local_media_repository_url_idx ON local_media_repository USING btree (created_ts) WHERE (url_cache IS NOT NULL);
	1767
	1768
	1769
	1770	CREATE INDEX monthly_active_users_time_stamp ON monthly_active_users USING btree ("timestamp");
	1771
	1772
	1773
	1774	CREATE UNIQUE INDEX monthly_active_users_users ON monthly_active_users USING btree (user_id);
	1775
	1776
	1777
	1778	CREATE INDEX open_id_tokens_ts_valid_until_ms ON open_id_tokens USING btree (ts_valid_until_ms);
	1779
	1780
	1781
	1782	CREATE INDEX presence_stream_id ON presence_stream USING btree (stream_id, user_id);
	1783
	1784
	1785
	1786	CREATE INDEX presence_stream_user_id ON presence_stream USING btree (user_id);
	1787
	1788
	1789
	1790	CREATE INDEX public_room_index ON rooms USING btree (is_public);
	1791
	1792
	1793
	1794	CREATE INDEX public_room_list_stream_idx ON public_room_list_stream USING btree (stream_id);
	1795
	1796
	1797
	1798	CREATE INDEX public_room_list_stream_rm_idx ON public_room_list_stream USING btree (room_id, stream_id);
	1799
	1800
	1801
	1802	CREATE INDEX push_rules_enable_user_name ON push_rules_enable USING btree (user_name);
	1803
	1804
	1805
	1806	CREATE INDEX push_rules_stream_id ON push_rules_stream USING btree (stream_id);
	1807
	1808
	1809
	1810	CREATE INDEX push_rules_stream_user_stream_id ON push_rules_stream USING btree (user_id, stream_id);
	1811
	1812
	1813
	1814	CREATE INDEX push_rules_user_name ON push_rules USING btree (user_name);
	1815
	1816
	1817
	1818	CREATE UNIQUE INDEX ratelimit_override_idx ON ratelimit_override USING btree (user_id);
	1819
	1820
	1821
	1822	CREATE INDEX receipts_linearized_id ON receipts_linearized USING btree (stream_id);
	1823
	1824
	1825
	1826	CREATE INDEX receipts_linearized_room_stream ON receipts_linearized USING btree (room_id, stream_id);
	1827
	1828
	1829
	1830	CREATE INDEX receipts_linearized_user ON receipts_linearized USING btree (user_id);
	1831
	1832
	1833
	1834	CREATE INDEX received_transactions_ts ON received_transactions USING btree (ts);
	1835
	1836
	1837
	1838	CREATE INDEX redactions_redacts ON redactions USING btree (redacts);
	1839
	1840
	1841
	1842	CREATE INDEX remote_profile_cache_time ON remote_profile_cache USING btree (last_check);
	1843
	1844
	1845
	1846	CREATE UNIQUE INDEX remote_profile_cache_user_id ON remote_profile_cache USING btree (user_id);
	1847
	1848
	1849
	1850	CREATE INDEX room_account_data_stream_id ON room_account_data USING btree (user_id, stream_id);
	1851
	1852
	1853
	1854	CREATE INDEX room_alias_servers_alias ON room_alias_servers USING btree (room_alias);
	1855
	1856
	1857
	1858	CREATE INDEX room_aliases_id ON room_aliases USING btree (room_id);
	1859
	1860
	1861
	1862	CREATE INDEX room_depth_room ON room_depth USING btree (room_id);
	1863
	1864
	1865
	1866	CREATE INDEX room_memberships_room_id ON room_memberships USING btree (room_id);
	1867
	1868
	1869
	1870	CREATE INDEX room_memberships_user_id ON room_memberships USING btree (user_id);
	1871
	1872
	1873
	1874	CREATE INDEX room_names_room_id ON room_names USING btree (room_id);
	1875
	1876
	1877
	1878	CREATE UNIQUE INDEX room_state_room ON room_state USING btree (room_id);
	1879
	1880
	1881
	1882	CREATE UNIQUE INDEX room_stats_earliest_token_idx ON room_stats_earliest_token USING btree (room_id);
	1883
	1884
	1885
	1886	CREATE UNIQUE INDEX room_stats_room_ts ON room_stats USING btree (room_id, ts);
	1887
	1888
	1889
	1890	CREATE INDEX stream_ordering_to_exterm_idx ON stream_ordering_to_exterm USING btree (stream_ordering);
	1891
	1892
	1893
	1894	CREATE INDEX stream_ordering_to_exterm_rm_idx ON stream_ordering_to_exterm USING btree (room_id, stream_ordering);
	1895
	1896
	1897
	1898	CREATE UNIQUE INDEX threepid_guest_access_tokens_index ON threepid_guest_access_tokens USING btree (medium, address);
	1899
	1900
	1901
	1902	CREATE INDEX topics_room_id ON topics USING btree (room_id);
	1903
	1904
	1905
	1906	CREATE INDEX user_daily_visits_ts_idx ON user_daily_visits USING btree ("timestamp");
	1907
	1908
	1909
	1910	CREATE INDEX user_daily_visits_uts_idx ON user_daily_visits USING btree (user_id, "timestamp");
	1911
	1912
	1913
	1914	CREATE INDEX user_directory_room_idx ON user_directory USING btree (room_id);
	1915
	1916
	1917
	1918	CREATE INDEX user_directory_search_fts_idx ON user_directory_search USING gin (vector);
	1919
	1920
	1921
	1922	CREATE UNIQUE INDEX user_directory_search_user_idx ON user_directory_search USING btree (user_id);
	1923
	1924
	1925
	1926	CREATE UNIQUE INDEX user_directory_user_idx ON user_directory USING btree (user_id);
	1927
	1928
	1929
	1930	CREATE INDEX user_filters_by_user_id_filter_id ON user_filters USING btree (user_id, filter_id);
	1931
	1932
	1933
	1934	CREATE INDEX user_ips_device_id ON user_ips USING btree (user_id, device_id, last_seen);
	1935
	1936
	1937
	1938	CREATE INDEX user_ips_last_seen ON user_ips USING btree (user_id, last_seen);
	1939
	1940
	1941
	1942	CREATE INDEX user_ips_last_seen_only ON user_ips USING btree (last_seen);
	1943
	1944
	1945
	1946	CREATE UNIQUE INDEX user_ips_user_token_ip_unique_index ON user_ips USING btree (user_id, access_token, ip);
	1947
	1948
	1949
	1950	CREATE UNIQUE INDEX user_stats_user_ts ON user_stats USING btree (user_id, ts);
	1951
	1952
	1953
	1954	CREATE UNIQUE INDEX user_threepid_id_server_idx ON user_threepid_id_server USING btree (user_id, medium, address, id_server);
	1955
	1956
	1957
	1958	CREATE INDEX user_threepids_medium_address ON user_threepids USING btree (medium, address);
	1959
	1960
	1961
	1962	CREATE INDEX user_threepids_user_id ON user_threepids USING btree (user_id);
	1963
	1964
	1965
	1966	CREATE INDEX users_creation_ts ON users USING btree (creation_ts);
	1967
	1968
	1969
	1970	CREATE UNIQUE INDEX users_in_public_rooms_u_idx ON users_in_public_rooms USING btree (user_id, room_id);
	1971
	1972
	1973
	1974	CREATE INDEX users_who_share_private_rooms_o_idx ON users_who_share_private_rooms USING btree (other_user_id);
	1975
	1976
	1977
	1978	CREATE INDEX users_who_share_private_rooms_r_idx ON users_who_share_private_rooms USING btree (room_id);
	1979
	1980
	1981
	1982	CREATE UNIQUE INDEX users_who_share_private_rooms_u_idx ON users_who_share_private_rooms USING btree (user_id, other_user_id, room_id);

+253

-0

synapse/storage/databases/main/schema/full_schemas/54/full.sql.sqlite less more

	0	CREATE TABLE application_services_state( as_id TEXT PRIMARY KEY, state VARCHAR(5), last_txn INTEGER );
	1	CREATE TABLE application_services_txns( as_id TEXT NOT NULL, txn_id INTEGER NOT NULL, event_ids TEXT NOT NULL, UNIQUE(as_id, txn_id) );
	2	CREATE INDEX application_services_txns_id ON application_services_txns ( as_id );
	3	CREATE TABLE presence( user_id TEXT NOT NULL, state VARCHAR(20), status_msg TEXT, mtime BIGINT, UNIQUE (user_id) );
	4	CREATE TABLE presence_allow_inbound( observed_user_id TEXT NOT NULL, observer_user_id TEXT NOT NULL, UNIQUE (observed_user_id, observer_user_id) );
	5	CREATE TABLE users( name TEXT, password_hash TEXT, creation_ts BIGINT, admin SMALLINT DEFAULT 0 NOT NULL, upgrade_ts BIGINT, is_guest SMALLINT DEFAULT 0 NOT NULL, appservice_id TEXT, consent_version TEXT, consent_server_notice_sent TEXT, user_type TEXT DEFAULT NULL, UNIQUE(name) );
	6	CREATE TABLE access_tokens( id BIGINT PRIMARY KEY, user_id TEXT NOT NULL, device_id TEXT, token TEXT NOT NULL, last_used BIGINT, UNIQUE(token) );
	7	CREATE TABLE user_ips ( user_id TEXT NOT NULL, access_token TEXT NOT NULL, device_id TEXT, ip TEXT NOT NULL, user_agent TEXT NOT NULL, last_seen BIGINT NOT NULL );
	8	CREATE TABLE profiles( user_id TEXT NOT NULL, displayname TEXT, avatar_url TEXT, UNIQUE(user_id) );
	9	CREATE TABLE received_transactions( transaction_id TEXT, origin TEXT, ts BIGINT, response_code INTEGER, response_json bytea, has_been_referenced smallint default 0, UNIQUE (transaction_id, origin) );
	10	CREATE TABLE destinations( destination TEXT PRIMARY KEY, retry_last_ts BIGINT, retry_interval INTEGER );
	11	CREATE TABLE events( stream_ordering INTEGER PRIMARY KEY, topological_ordering BIGINT NOT NULL, event_id TEXT NOT NULL, type TEXT NOT NULL, room_id TEXT NOT NULL, content TEXT, unrecognized_keys TEXT, processed BOOL NOT NULL, outlier BOOL NOT NULL, depth BIGINT DEFAULT 0 NOT NULL, origin_server_ts BIGINT, received_ts BIGINT, sender TEXT, contains_url BOOLEAN, UNIQUE (event_id) );
	12	CREATE INDEX events_order_room ON events ( room_id, topological_ordering, stream_ordering );
	13	CREATE TABLE event_json( event_id TEXT NOT NULL, room_id TEXT NOT NULL, internal_metadata TEXT NOT NULL, json TEXT NOT NULL, format_version INTEGER, UNIQUE (event_id) );
	14	CREATE INDEX event_json_room_id ON event_json(room_id);
	15	CREATE TABLE state_events( event_id TEXT NOT NULL, room_id TEXT NOT NULL, type TEXT NOT NULL, state_key TEXT NOT NULL, prev_state TEXT, UNIQUE (event_id) );
	16	CREATE TABLE current_state_events( event_id TEXT NOT NULL, room_id TEXT NOT NULL, type TEXT NOT NULL, state_key TEXT NOT NULL, UNIQUE (event_id), UNIQUE (room_id, type, state_key) );
	17	CREATE TABLE room_memberships( event_id TEXT NOT NULL, user_id TEXT NOT NULL, sender TEXT NOT NULL, room_id TEXT NOT NULL, membership TEXT NOT NULL, forgotten INTEGER DEFAULT 0, display_name TEXT, avatar_url TEXT, UNIQUE (event_id) );
	18	CREATE INDEX room_memberships_room_id ON room_memberships (room_id);
	19	CREATE INDEX room_memberships_user_id ON room_memberships (user_id);
	20	CREATE TABLE topics( event_id TEXT NOT NULL, room_id TEXT NOT NULL, topic TEXT NOT NULL, UNIQUE (event_id) );
	21	CREATE INDEX topics_room_id ON topics(room_id);
	22	CREATE TABLE room_names( event_id TEXT NOT NULL, room_id TEXT NOT NULL, name TEXT NOT NULL, UNIQUE (event_id) );
	23	CREATE INDEX room_names_room_id ON room_names(room_id);
	24	CREATE TABLE rooms( room_id TEXT PRIMARY KEY NOT NULL, is_public BOOL, creator TEXT );
	25	CREATE TABLE server_signature_keys( server_name TEXT, key_id TEXT, from_server TEXT, ts_added_ms BIGINT, verify_key bytea, ts_valid_until_ms BIGINT, UNIQUE (server_name, key_id) );
	26	CREATE TABLE rejections( event_id TEXT NOT NULL, reason TEXT NOT NULL, last_check TEXT NOT NULL, UNIQUE (event_id) );
	27	CREATE TABLE push_rules ( id BIGINT PRIMARY KEY, user_name TEXT NOT NULL, rule_id TEXT NOT NULL, priority_class SMALLINT NOT NULL, priority INTEGER NOT NULL DEFAULT 0, conditions TEXT NOT NULL, actions TEXT NOT NULL, UNIQUE(user_name, rule_id) );
	28	CREATE INDEX push_rules_user_name on push_rules (user_name);
	29	CREATE TABLE user_filters( user_id TEXT, filter_id BIGINT, filter_json bytea );
	30	CREATE INDEX user_filters_by_user_id_filter_id ON user_filters( user_id, filter_id );
	31	CREATE TABLE push_rules_enable ( id BIGINT PRIMARY KEY, user_name TEXT NOT NULL, rule_id TEXT NOT NULL, enabled SMALLINT, UNIQUE(user_name, rule_id) );
	32	CREATE INDEX push_rules_enable_user_name on push_rules_enable (user_name);
	33	CREATE TABLE event_forward_extremities( event_id TEXT NOT NULL, room_id TEXT NOT NULL, UNIQUE (event_id, room_id) );
	34	CREATE INDEX ev_extrem_room ON event_forward_extremities(room_id);
	35	CREATE INDEX ev_extrem_id ON event_forward_extremities(event_id);
	36	CREATE TABLE event_backward_extremities( event_id TEXT NOT NULL, room_id TEXT NOT NULL, UNIQUE (event_id, room_id) );
	37	CREATE INDEX ev_b_extrem_room ON event_backward_extremities(room_id);
	38	CREATE INDEX ev_b_extrem_id ON event_backward_extremities(event_id);
	39	CREATE TABLE event_edges( event_id TEXT NOT NULL, prev_event_id TEXT NOT NULL, room_id TEXT NOT NULL, is_state BOOL NOT NULL, UNIQUE (event_id, prev_event_id, room_id, is_state) );
	40	CREATE INDEX ev_edges_id ON event_edges(event_id);
	41	CREATE INDEX ev_edges_prev_id ON event_edges(prev_event_id);
	42	CREATE TABLE room_depth( room_id TEXT NOT NULL, min_depth INTEGER NOT NULL, UNIQUE (room_id) );
	43	CREATE INDEX room_depth_room ON room_depth(room_id);
	44	CREATE TABLE event_to_state_groups( event_id TEXT NOT NULL, state_group BIGINT NOT NULL, UNIQUE (event_id) );
	45	CREATE TABLE local_media_repository ( media_id TEXT, media_type TEXT, media_length INTEGER, created_ts BIGINT, upload_name TEXT, user_id TEXT, quarantined_by TEXT, url_cache TEXT, last_access_ts BIGINT, UNIQUE (media_id) );
	46	CREATE TABLE local_media_repository_thumbnails ( media_id TEXT, thumbnail_width INTEGER, thumbnail_height INTEGER, thumbnail_type TEXT, thumbnail_method TEXT, thumbnail_length INTEGER, UNIQUE ( media_id, thumbnail_width, thumbnail_height, thumbnail_type ) );
	47	CREATE INDEX local_media_repository_thumbnails_media_id ON local_media_repository_thumbnails (media_id);
	48	CREATE TABLE remote_media_cache ( media_origin TEXT, media_id TEXT, media_type TEXT, created_ts BIGINT, upload_name TEXT, media_length INTEGER, filesystem_id TEXT, last_access_ts BIGINT, quarantined_by TEXT, UNIQUE (media_origin, media_id) );
	49	CREATE TABLE remote_media_cache_thumbnails ( media_origin TEXT, media_id TEXT, thumbnail_width INTEGER, thumbnail_height INTEGER, thumbnail_method TEXT, thumbnail_type TEXT, thumbnail_length INTEGER, filesystem_id TEXT, UNIQUE ( media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type ) );
	50	CREATE TABLE redactions ( event_id TEXT NOT NULL, redacts TEXT NOT NULL, UNIQUE (event_id) );
	51	CREATE INDEX redactions_redacts ON redactions (redacts);
	52	CREATE TABLE room_aliases( room_alias TEXT NOT NULL, room_id TEXT NOT NULL, creator TEXT, UNIQUE (room_alias) );
	53	CREATE INDEX room_aliases_id ON room_aliases(room_id);
	54	CREATE TABLE room_alias_servers( room_alias TEXT NOT NULL, server TEXT NOT NULL );
	55	CREATE INDEX room_alias_servers_alias ON room_alias_servers(room_alias);
	56	CREATE TABLE event_reference_hashes ( event_id TEXT, algorithm TEXT, hash bytea, UNIQUE (event_id, algorithm) );
	57	CREATE INDEX event_reference_hashes_id ON event_reference_hashes(event_id);
	58	CREATE TABLE IF NOT EXISTS "server_keys_json" ( server_name TEXT NOT NULL, key_id TEXT NOT NULL, from_server TEXT NOT NULL, ts_added_ms BIGINT NOT NULL, ts_valid_until_ms BIGINT NOT NULL, key_json bytea NOT NULL, CONSTRAINT server_keys_json_uniqueness UNIQUE (server_name, key_id, from_server) );
	59	CREATE TABLE e2e_device_keys_json ( user_id TEXT NOT NULL, device_id TEXT NOT NULL, ts_added_ms BIGINT NOT NULL, key_json TEXT NOT NULL, CONSTRAINT e2e_device_keys_json_uniqueness UNIQUE (user_id, device_id) );
	60	CREATE TABLE e2e_one_time_keys_json ( user_id TEXT NOT NULL, device_id TEXT NOT NULL, algorithm TEXT NOT NULL, key_id TEXT NOT NULL, ts_added_ms BIGINT NOT NULL, key_json TEXT NOT NULL, CONSTRAINT e2e_one_time_keys_json_uniqueness UNIQUE (user_id, device_id, algorithm, key_id) );
	61	CREATE TABLE receipts_graph( room_id TEXT NOT NULL, receipt_type TEXT NOT NULL, user_id TEXT NOT NULL, event_ids TEXT NOT NULL, data TEXT NOT NULL, CONSTRAINT receipts_graph_uniqueness UNIQUE (room_id, receipt_type, user_id) );
	62	CREATE TABLE receipts_linearized ( stream_id BIGINT NOT NULL, room_id TEXT NOT NULL, receipt_type TEXT NOT NULL, user_id TEXT NOT NULL, event_id TEXT NOT NULL, data TEXT NOT NULL, CONSTRAINT receipts_linearized_uniqueness UNIQUE (room_id, receipt_type, user_id) );
	63	CREATE INDEX receipts_linearized_id ON receipts_linearized( stream_id );
	64	CREATE INDEX receipts_linearized_room_stream ON receipts_linearized( room_id, stream_id );
	65	CREATE TABLE IF NOT EXISTS "user_threepids" ( user_id TEXT NOT NULL, medium TEXT NOT NULL, address TEXT NOT NULL, validated_at BIGINT NOT NULL, added_at BIGINT NOT NULL, CONSTRAINT medium_address UNIQUE (medium, address) );
	66	CREATE INDEX user_threepids_user_id ON user_threepids(user_id);
	67	CREATE VIRTUAL TABLE event_search USING fts4 ( event_id, room_id, sender, key, value )
	68	/* event_search(event_id,room_id,sender,"key",value) */;
	69	CREATE TABLE IF NOT EXISTS 'event_search_content'(docid INTEGER PRIMARY KEY, 'c0event_id', 'c1room_id', 'c2sender', 'c3key', 'c4value');
	70	CREATE TABLE IF NOT EXISTS 'event_search_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
	71	CREATE TABLE IF NOT EXISTS 'event_search_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));
	72	CREATE TABLE IF NOT EXISTS 'event_search_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
	73	CREATE TABLE IF NOT EXISTS 'event_search_stat'(id INTEGER PRIMARY KEY, value BLOB);
	74	CREATE TABLE guest_access( event_id TEXT NOT NULL, room_id TEXT NOT NULL, guest_access TEXT NOT NULL, UNIQUE (event_id) );
	75	CREATE TABLE history_visibility( event_id TEXT NOT NULL, room_id TEXT NOT NULL, history_visibility TEXT NOT NULL, UNIQUE (event_id) );
	76	CREATE TABLE room_tags( user_id TEXT NOT NULL, room_id TEXT NOT NULL, tag TEXT NOT NULL, content TEXT NOT NULL, CONSTRAINT room_tag_uniqueness UNIQUE (user_id, room_id, tag) );
	77	CREATE TABLE room_tags_revisions ( user_id TEXT NOT NULL, room_id TEXT NOT NULL, stream_id BIGINT NOT NULL, CONSTRAINT room_tag_revisions_uniqueness UNIQUE (user_id, room_id) );
	78	CREATE TABLE IF NOT EXISTS "account_data_max_stream_id"( Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, stream_id BIGINT NOT NULL, CHECK (Lock='X') );
	79	CREATE TABLE account_data( user_id TEXT NOT NULL, account_data_type TEXT NOT NULL, stream_id BIGINT NOT NULL, content TEXT NOT NULL, CONSTRAINT account_data_uniqueness UNIQUE (user_id, account_data_type) );
	80	CREATE TABLE room_account_data( user_id TEXT NOT NULL, room_id TEXT NOT NULL, account_data_type TEXT NOT NULL, stream_id BIGINT NOT NULL, content TEXT NOT NULL, CONSTRAINT room_account_data_uniqueness UNIQUE (user_id, room_id, account_data_type) );
	81	CREATE INDEX account_data_stream_id on account_data(user_id, stream_id);
	82	CREATE INDEX room_account_data_stream_id on room_account_data(user_id, stream_id);
	83	CREATE INDEX events_ts ON events(origin_server_ts, stream_ordering);
	84	CREATE TABLE event_push_actions( room_id TEXT NOT NULL, event_id TEXT NOT NULL, user_id TEXT NOT NULL, profile_tag VARCHAR(32), actions TEXT NOT NULL, topological_ordering BIGINT, stream_ordering BIGINT, notif SMALLINT, highlight SMALLINT, CONSTRAINT event_id_user_id_profile_tag_uniqueness UNIQUE (room_id, event_id, user_id, profile_tag) );
	85	CREATE INDEX event_push_actions_room_id_user_id on event_push_actions(room_id, user_id);
	86	CREATE INDEX events_room_stream on events(room_id, stream_ordering);
	87	CREATE INDEX public_room_index on rooms(is_public);
	88	CREATE INDEX receipts_linearized_user ON receipts_linearized( user_id );
	89	CREATE INDEX event_push_actions_rm_tokens on event_push_actions( user_id, room_id, topological_ordering, stream_ordering );
	90	CREATE TABLE presence_stream( stream_id BIGINT, user_id TEXT, state TEXT, last_active_ts BIGINT, last_federation_update_ts BIGINT, last_user_sync_ts BIGINT, status_msg TEXT, currently_active BOOLEAN );
	91	CREATE INDEX presence_stream_id ON presence_stream(stream_id, user_id);
	92	CREATE INDEX presence_stream_user_id ON presence_stream(user_id);
	93	CREATE TABLE push_rules_stream( stream_id BIGINT NOT NULL, event_stream_ordering BIGINT NOT NULL, user_id TEXT NOT NULL, rule_id TEXT NOT NULL, op TEXT NOT NULL, priority_class SMALLINT, priority INTEGER, conditions TEXT, actions TEXT );
	94	CREATE INDEX push_rules_stream_id ON push_rules_stream(stream_id);
	95	CREATE INDEX push_rules_stream_user_stream_id on push_rules_stream(user_id, stream_id);
	96	CREATE TABLE ex_outlier_stream( event_stream_ordering BIGINT PRIMARY KEY NOT NULL, event_id TEXT NOT NULL, state_group BIGINT NOT NULL );
	97	CREATE TABLE threepid_guest_access_tokens( medium TEXT, address TEXT, guest_access_token TEXT, first_inviter TEXT );
	98	CREATE UNIQUE INDEX threepid_guest_access_tokens_index ON threepid_guest_access_tokens(medium, address);
	99	CREATE TABLE local_invites( stream_id BIGINT NOT NULL, inviter TEXT NOT NULL, invitee TEXT NOT NULL, event_id TEXT NOT NULL, room_id TEXT NOT NULL, locally_rejected TEXT, replaced_by TEXT );
	100	CREATE INDEX local_invites_id ON local_invites(stream_id);
	101	CREATE INDEX local_invites_for_user_idx ON local_invites(invitee, locally_rejected, replaced_by, room_id);
	102	CREATE INDEX event_push_actions_stream_ordering on event_push_actions( stream_ordering, user_id );
	103	CREATE TABLE open_id_tokens ( token TEXT NOT NULL PRIMARY KEY, ts_valid_until_ms bigint NOT NULL, user_id TEXT NOT NULL, UNIQUE (token) );
	104	CREATE INDEX open_id_tokens_ts_valid_until_ms ON open_id_tokens(ts_valid_until_ms);
	105	CREATE TABLE pusher_throttle( pusher BIGINT NOT NULL, room_id TEXT NOT NULL, last_sent_ts BIGINT, throttle_ms BIGINT, PRIMARY KEY (pusher, room_id) );
	106	CREATE TABLE event_reports( id BIGINT NOT NULL PRIMARY KEY, received_ts BIGINT NOT NULL, room_id TEXT NOT NULL, event_id TEXT NOT NULL, user_id TEXT NOT NULL, reason TEXT, content TEXT );
	107	CREATE TABLE devices ( user_id TEXT NOT NULL, device_id TEXT NOT NULL, display_name TEXT, CONSTRAINT device_uniqueness UNIQUE (user_id, device_id) );
	108	CREATE TABLE appservice_stream_position( Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, stream_ordering BIGINT, CHECK (Lock='X') );
	109	CREATE TABLE device_inbox ( user_id TEXT NOT NULL, device_id TEXT NOT NULL, stream_id BIGINT NOT NULL, message_json TEXT NOT NULL );
	110	CREATE INDEX device_inbox_user_stream_id ON device_inbox(user_id, device_id, stream_id);
	111	CREATE INDEX received_transactions_ts ON received_transactions(ts);
	112	CREATE TABLE device_federation_outbox ( destination TEXT NOT NULL, stream_id BIGINT NOT NULL, queued_ts BIGINT NOT NULL, messages_json TEXT NOT NULL );
	113	CREATE INDEX device_federation_outbox_destination_id ON device_federation_outbox(destination, stream_id);
	114	CREATE TABLE device_federation_inbox ( origin TEXT NOT NULL, message_id TEXT NOT NULL, received_ts BIGINT NOT NULL );
	115	CREATE INDEX device_federation_inbox_sender_id ON device_federation_inbox(origin, message_id);
	116	CREATE TABLE device_max_stream_id ( stream_id BIGINT NOT NULL );
	117	CREATE TABLE public_room_list_stream ( stream_id BIGINT NOT NULL, room_id TEXT NOT NULL, visibility BOOLEAN NOT NULL , appservice_id TEXT, network_id TEXT);
	118	CREATE INDEX public_room_list_stream_idx on public_room_list_stream( stream_id );
	119	CREATE INDEX public_room_list_stream_rm_idx on public_room_list_stream( room_id, stream_id );
	120	CREATE TABLE stream_ordering_to_exterm ( stream_ordering BIGINT NOT NULL, room_id TEXT NOT NULL, event_id TEXT NOT NULL );
	121	CREATE INDEX stream_ordering_to_exterm_idx on stream_ordering_to_exterm( stream_ordering );
	122	CREATE INDEX stream_ordering_to_exterm_rm_idx on stream_ordering_to_exterm( room_id, stream_ordering );
	123	CREATE TABLE IF NOT EXISTS "event_auth"( event_id TEXT NOT NULL, auth_id TEXT NOT NULL, room_id TEXT NOT NULL );
	124	CREATE INDEX evauth_edges_id ON event_auth(event_id);
	125	CREATE INDEX user_threepids_medium_address on user_threepids (medium, address);
	126	CREATE TABLE appservice_room_list( appservice_id TEXT NOT NULL, network_id TEXT NOT NULL, room_id TEXT NOT NULL );
	127	CREATE UNIQUE INDEX appservice_room_list_idx ON appservice_room_list( appservice_id, network_id, room_id );
	128	CREATE INDEX device_federation_outbox_id ON device_federation_outbox(stream_id);
	129	CREATE TABLE federation_stream_position( type TEXT NOT NULL, stream_id INTEGER NOT NULL );
	130	CREATE TABLE device_lists_remote_cache ( user_id TEXT NOT NULL, device_id TEXT NOT NULL, content TEXT NOT NULL );
	131	CREATE TABLE device_lists_remote_extremeties ( user_id TEXT NOT NULL, stream_id TEXT NOT NULL );
	132	CREATE TABLE device_lists_stream ( stream_id BIGINT NOT NULL, user_id TEXT NOT NULL, device_id TEXT NOT NULL );
	133	CREATE INDEX device_lists_stream_id ON device_lists_stream(stream_id, user_id);
	134	CREATE TABLE device_lists_outbound_pokes ( destination TEXT NOT NULL, stream_id BIGINT NOT NULL, user_id TEXT NOT NULL, device_id TEXT NOT NULL, sent BOOLEAN NOT NULL, ts BIGINT NOT NULL );
	135	CREATE INDEX device_lists_outbound_pokes_id ON device_lists_outbound_pokes(destination, stream_id);
	136	CREATE INDEX device_lists_outbound_pokes_user ON device_lists_outbound_pokes(destination, user_id);
	137	CREATE TABLE event_push_summary ( user_id TEXT NOT NULL, room_id TEXT NOT NULL, notif_count BIGINT NOT NULL, stream_ordering BIGINT NOT NULL );
	138	CREATE INDEX event_push_summary_user_rm ON event_push_summary(user_id, room_id);
	139	CREATE TABLE event_push_summary_stream_ordering ( Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, stream_ordering BIGINT NOT NULL, CHECK (Lock='X') );
	140	CREATE TABLE IF NOT EXISTS "pushers" ( id BIGINT PRIMARY KEY, user_name TEXT NOT NULL, access_token BIGINT DEFAULT NULL, profile_tag TEXT NOT NULL, kind TEXT NOT NULL, app_id TEXT NOT NULL, app_display_name TEXT NOT NULL, device_display_name TEXT NOT NULL, pushkey TEXT NOT NULL, ts BIGINT NOT NULL, lang TEXT, data TEXT, last_stream_ordering INTEGER, last_success BIGINT, failing_since BIGINT, UNIQUE (app_id, pushkey, user_name) );
	141	CREATE INDEX device_lists_outbound_pokes_stream ON device_lists_outbound_pokes(stream_id);
	142	CREATE TABLE ratelimit_override ( user_id TEXT NOT NULL, messages_per_second BIGINT, burst_count BIGINT );
	143	CREATE UNIQUE INDEX ratelimit_override_idx ON ratelimit_override(user_id);
	144	CREATE TABLE current_state_delta_stream ( stream_id BIGINT NOT NULL, room_id TEXT NOT NULL, type TEXT NOT NULL, state_key TEXT NOT NULL, event_id TEXT, prev_event_id TEXT );
	145	CREATE INDEX current_state_delta_stream_idx ON current_state_delta_stream(stream_id);
	146	CREATE TABLE device_lists_outbound_last_success ( destination TEXT NOT NULL, user_id TEXT NOT NULL, stream_id BIGINT NOT NULL );
	147	CREATE INDEX device_lists_outbound_last_success_idx ON device_lists_outbound_last_success( destination, user_id, stream_id );
	148	CREATE TABLE user_directory_stream_pos ( Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, stream_id BIGINT, CHECK (Lock='X') );
	149	CREATE VIRTUAL TABLE user_directory_search USING fts4 ( user_id, value )
	150	/* user_directory_search(user_id,value) */;
	151	CREATE TABLE IF NOT EXISTS 'user_directory_search_content'(docid INTEGER PRIMARY KEY, 'c0user_id', 'c1value');
	152	CREATE TABLE IF NOT EXISTS 'user_directory_search_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
	153	CREATE TABLE IF NOT EXISTS 'user_directory_search_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));
	154	CREATE TABLE IF NOT EXISTS 'user_directory_search_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
	155	CREATE TABLE IF NOT EXISTS 'user_directory_search_stat'(id INTEGER PRIMARY KEY, value BLOB);
	156	CREATE TABLE blocked_rooms ( room_id TEXT NOT NULL, user_id TEXT NOT NULL );
	157	CREATE UNIQUE INDEX blocked_rooms_idx ON blocked_rooms(room_id);
	158	CREATE TABLE IF NOT EXISTS "local_media_repository_url_cache"( url TEXT, response_code INTEGER, etag TEXT, expires_ts BIGINT, og TEXT, media_id TEXT, download_ts BIGINT );
	159	CREATE INDEX local_media_repository_url_cache_expires_idx ON local_media_repository_url_cache(expires_ts);
	160	CREATE INDEX local_media_repository_url_cache_by_url_download_ts ON local_media_repository_url_cache(url, download_ts);
	161	CREATE INDEX local_media_repository_url_cache_media_idx ON local_media_repository_url_cache(media_id);
	162	CREATE TABLE group_users ( group_id TEXT NOT NULL, user_id TEXT NOT NULL, is_admin BOOLEAN NOT NULL, is_public BOOLEAN NOT NULL );
	163	CREATE TABLE group_invites ( group_id TEXT NOT NULL, user_id TEXT NOT NULL );
	164	CREATE TABLE group_rooms ( group_id TEXT NOT NULL, room_id TEXT NOT NULL, is_public BOOLEAN NOT NULL );
	165	CREATE TABLE group_summary_rooms ( group_id TEXT NOT NULL, room_id TEXT NOT NULL, category_id TEXT NOT NULL, room_order BIGINT NOT NULL, is_public BOOLEAN NOT NULL, UNIQUE (group_id, category_id, room_id, room_order), CHECK (room_order > 0) );
	166	CREATE UNIQUE INDEX group_summary_rooms_g_idx ON group_summary_rooms(group_id, room_id, category_id);
	167	CREATE TABLE group_summary_room_categories ( group_id TEXT NOT NULL, category_id TEXT NOT NULL, cat_order BIGINT NOT NULL, UNIQUE (group_id, category_id, cat_order), CHECK (cat_order > 0) );
	168	CREATE TABLE group_room_categories ( group_id TEXT NOT NULL, category_id TEXT NOT NULL, profile TEXT NOT NULL, is_public BOOLEAN NOT NULL, UNIQUE (group_id, category_id) );
	169	CREATE TABLE group_summary_users ( group_id TEXT NOT NULL, user_id TEXT NOT NULL, role_id TEXT NOT NULL, user_order BIGINT NOT NULL, is_public BOOLEAN NOT NULL );
	170	CREATE INDEX group_summary_users_g_idx ON group_summary_users(group_id);
	171	CREATE TABLE group_summary_roles ( group_id TEXT NOT NULL, role_id TEXT NOT NULL, role_order BIGINT NOT NULL, UNIQUE (group_id, role_id, role_order), CHECK (role_order > 0) );
	172	CREATE TABLE group_roles ( group_id TEXT NOT NULL, role_id TEXT NOT NULL, profile TEXT NOT NULL, is_public BOOLEAN NOT NULL, UNIQUE (group_id, role_id) );
	173	CREATE TABLE group_attestations_renewals ( group_id TEXT NOT NULL, user_id TEXT NOT NULL, valid_until_ms BIGINT NOT NULL );
	174	CREATE INDEX group_attestations_renewals_g_idx ON group_attestations_renewals(group_id, user_id);
	175	CREATE INDEX group_attestations_renewals_u_idx ON group_attestations_renewals(user_id);
	176	CREATE INDEX group_attestations_renewals_v_idx ON group_attestations_renewals(valid_until_ms);
	177	CREATE TABLE group_attestations_remote ( group_id TEXT NOT NULL, user_id TEXT NOT NULL, valid_until_ms BIGINT NOT NULL, attestation_json TEXT NOT NULL );
	178	CREATE INDEX group_attestations_remote_g_idx ON group_attestations_remote(group_id, user_id);
	179	CREATE INDEX group_attestations_remote_u_idx ON group_attestations_remote(user_id);
	180	CREATE INDEX group_attestations_remote_v_idx ON group_attestations_remote(valid_until_ms);
	181	CREATE TABLE local_group_membership ( group_id TEXT NOT NULL, user_id TEXT NOT NULL, is_admin BOOLEAN NOT NULL, membership TEXT NOT NULL, is_publicised BOOLEAN NOT NULL, content TEXT NOT NULL );
	182	CREATE INDEX local_group_membership_u_idx ON local_group_membership(user_id, group_id);
	183	CREATE INDEX local_group_membership_g_idx ON local_group_membership(group_id);
	184	CREATE TABLE local_group_updates ( stream_id BIGINT NOT NULL, group_id TEXT NOT NULL, user_id TEXT NOT NULL, type TEXT NOT NULL, content TEXT NOT NULL );
	185	CREATE TABLE remote_profile_cache ( user_id TEXT NOT NULL, displayname TEXT, avatar_url TEXT, last_check BIGINT NOT NULL );
	186	CREATE UNIQUE INDEX remote_profile_cache_user_id ON remote_profile_cache(user_id);
	187	CREATE INDEX remote_profile_cache_time ON remote_profile_cache(last_check);
	188	CREATE TABLE IF NOT EXISTS "deleted_pushers" ( stream_id BIGINT NOT NULL, app_id TEXT NOT NULL, pushkey TEXT NOT NULL, user_id TEXT NOT NULL );
	189	CREATE INDEX deleted_pushers_stream_id ON deleted_pushers (stream_id);
	190	CREATE TABLE IF NOT EXISTS "groups" ( group_id TEXT NOT NULL, name TEXT, avatar_url TEXT, short_description TEXT, long_description TEXT, is_public BOOL NOT NULL , join_policy TEXT NOT NULL DEFAULT 'invite');
	191	CREATE UNIQUE INDEX groups_idx ON groups(group_id);
	192	CREATE TABLE IF NOT EXISTS "user_directory" ( user_id TEXT NOT NULL, room_id TEXT, display_name TEXT, avatar_url TEXT );
	193	CREATE INDEX user_directory_room_idx ON user_directory(room_id);
	194	CREATE UNIQUE INDEX user_directory_user_idx ON user_directory(user_id);
	195	CREATE TABLE event_push_actions_staging ( event_id TEXT NOT NULL, user_id TEXT NOT NULL, actions TEXT NOT NULL, notif SMALLINT NOT NULL, highlight SMALLINT NOT NULL );
	196	CREATE INDEX event_push_actions_staging_id ON event_push_actions_staging(event_id);
	197	CREATE TABLE users_pending_deactivation ( user_id TEXT NOT NULL );
	198	CREATE UNIQUE INDEX group_invites_g_idx ON group_invites(group_id, user_id);
	199	CREATE UNIQUE INDEX group_users_g_idx ON group_users(group_id, user_id);
	200	CREATE INDEX group_users_u_idx ON group_users(user_id);
	201	CREATE INDEX group_invites_u_idx ON group_invites(user_id);
	202	CREATE UNIQUE INDEX group_rooms_g_idx ON group_rooms(group_id, room_id);
	203	CREATE INDEX group_rooms_r_idx ON group_rooms(room_id);
	204	CREATE TABLE user_daily_visits ( user_id TEXT NOT NULL, device_id TEXT, timestamp BIGINT NOT NULL );
	205	CREATE INDEX user_daily_visits_uts_idx ON user_daily_visits(user_id, timestamp);
	206	CREATE INDEX user_daily_visits_ts_idx ON user_daily_visits(timestamp);
	207	CREATE TABLE erased_users ( user_id TEXT NOT NULL );
	208	CREATE UNIQUE INDEX erased_users_user ON erased_users(user_id);
	209	CREATE TABLE monthly_active_users ( user_id TEXT NOT NULL, timestamp BIGINT NOT NULL );
	210	CREATE UNIQUE INDEX monthly_active_users_users ON monthly_active_users(user_id);
	211	CREATE INDEX monthly_active_users_time_stamp ON monthly_active_users(timestamp);
	212	CREATE TABLE IF NOT EXISTS "e2e_room_keys_versions" ( user_id TEXT NOT NULL, version BIGINT NOT NULL, algorithm TEXT NOT NULL, auth_data TEXT NOT NULL, deleted SMALLINT DEFAULT 0 NOT NULL );
	213	CREATE UNIQUE INDEX e2e_room_keys_versions_idx ON e2e_room_keys_versions(user_id, version);
	214	CREATE TABLE IF NOT EXISTS "e2e_room_keys" ( user_id TEXT NOT NULL, room_id TEXT NOT NULL, session_id TEXT NOT NULL, version BIGINT NOT NULL, first_message_index INT, forwarded_count INT, is_verified BOOLEAN, session_data TEXT NOT NULL );
	215	CREATE UNIQUE INDEX e2e_room_keys_idx ON e2e_room_keys(user_id, room_id, session_id);
	216	CREATE TABLE users_who_share_private_rooms ( user_id TEXT NOT NULL, other_user_id TEXT NOT NULL, room_id TEXT NOT NULL );
	217	CREATE UNIQUE INDEX users_who_share_private_rooms_u_idx ON users_who_share_private_rooms(user_id, other_user_id, room_id);
	218	CREATE INDEX users_who_share_private_rooms_r_idx ON users_who_share_private_rooms(room_id);
	219	CREATE INDEX users_who_share_private_rooms_o_idx ON users_who_share_private_rooms(other_user_id);
	220	CREATE TABLE user_threepid_id_server ( user_id TEXT NOT NULL, medium TEXT NOT NULL, address TEXT NOT NULL, id_server TEXT NOT NULL );
	221	CREATE UNIQUE INDEX user_threepid_id_server_idx ON user_threepid_id_server( user_id, medium, address, id_server );
	222	CREATE TABLE users_in_public_rooms ( user_id TEXT NOT NULL, room_id TEXT NOT NULL );
	223	CREATE UNIQUE INDEX users_in_public_rooms_u_idx ON users_in_public_rooms(user_id, room_id);
	224	CREATE TABLE account_validity ( user_id TEXT PRIMARY KEY, expiration_ts_ms BIGINT NOT NULL, email_sent BOOLEAN NOT NULL, renewal_token TEXT );
	225	CREATE TABLE event_relations ( event_id TEXT NOT NULL, relates_to_id TEXT NOT NULL, relation_type TEXT NOT NULL, aggregation_key TEXT );
	226	CREATE UNIQUE INDEX event_relations_id ON event_relations(event_id);
	227	CREATE INDEX event_relations_relates ON event_relations(relates_to_id, relation_type, aggregation_key);
	228	CREATE TABLE stats_stream_pos ( Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, stream_id BIGINT, CHECK (Lock='X') );
	229	CREATE TABLE user_stats ( user_id TEXT NOT NULL, ts BIGINT NOT NULL, bucket_size INT NOT NULL, public_rooms INT NOT NULL, private_rooms INT NOT NULL );
	230	CREATE UNIQUE INDEX user_stats_user_ts ON user_stats(user_id, ts);
	231	CREATE TABLE room_stats ( room_id TEXT NOT NULL, ts BIGINT NOT NULL, bucket_size INT NOT NULL, current_state_events INT NOT NULL, joined_members INT NOT NULL, invited_members INT NOT NULL, left_members INT NOT NULL, banned_members INT NOT NULL, state_events INT NOT NULL );
	232	CREATE UNIQUE INDEX room_stats_room_ts ON room_stats(room_id, ts);
	233	CREATE TABLE room_state ( room_id TEXT NOT NULL, join_rules TEXT, history_visibility TEXT, encryption TEXT, name TEXT, topic TEXT, avatar TEXT, canonical_alias TEXT );
	234	CREATE UNIQUE INDEX room_state_room ON room_state(room_id);
	235	CREATE TABLE room_stats_earliest_token ( room_id TEXT NOT NULL, token BIGINT NOT NULL );
	236	CREATE UNIQUE INDEX room_stats_earliest_token_idx ON room_stats_earliest_token(room_id);
	237	CREATE INDEX access_tokens_device_id ON access_tokens (user_id, device_id);
	238	CREATE INDEX user_ips_device_id ON user_ips (user_id, device_id, last_seen);
	239	CREATE INDEX event_contains_url_index ON events (room_id, topological_ordering, stream_ordering);
	240	CREATE INDEX event_push_actions_u_highlight ON event_push_actions (user_id, stream_ordering);
	241	CREATE INDEX event_push_actions_highlights_index ON event_push_actions (user_id, room_id, topological_ordering, stream_ordering);
	242	CREATE INDEX current_state_events_member_index ON current_state_events (state_key);
	243	CREATE INDEX device_inbox_stream_id_user_id ON device_inbox (stream_id, user_id);
	244	CREATE INDEX device_lists_stream_user_id ON device_lists_stream (user_id, device_id);
	245	CREATE INDEX local_media_repository_url_idx ON local_media_repository (created_ts);
	246	CREATE INDEX user_ips_last_seen ON user_ips (user_id, last_seen);
	247	CREATE INDEX user_ips_last_seen_only ON user_ips (last_seen);
	248	CREATE INDEX users_creation_ts ON users (creation_ts);
	249	CREATE INDEX event_to_state_groups_sg_index ON event_to_state_groups (state_group);
	250	CREATE UNIQUE INDEX device_lists_remote_cache_unique_id ON device_lists_remote_cache (user_id, device_id);
	251	CREATE UNIQUE INDEX device_lists_remote_extremeties_unique_idx ON device_lists_remote_extremeties (user_id);
	252	CREATE UNIQUE INDEX user_ips_user_token_ip_unique_index ON user_ips (user_id, access_token, ip);

+8

-0

synapse/storage/databases/main/schema/full_schemas/54/stream_positions.sql less more

	0
	1	INSERT INTO appservice_stream_position (stream_ordering) SELECT COALESCE(MAX(stream_ordering), 0) FROM events;
	2	INSERT INTO federation_stream_position (type, stream_id) VALUES ('federation', -1);
	3	INSERT INTO federation_stream_position (type, stream_id) SELECT 'events', coalesce(max(stream_ordering), -1) FROM events;
	4	INSERT INTO user_directory_stream_pos (stream_id) VALUES (0);
	5	INSERT INTO stats_stream_pos (stream_id) VALUES (0);
	6	INSERT INTO event_push_summary_stream_ordering (stream_ordering) VALUES (0);
	7	-- device_max_stream_id is handled separately in 56/device_stream_id_insert.sql⏎

+21

-0

synapse/storage/databases/main/schema/full_schemas/README.md less more

	0	# Synapse Database Schemas
	1
	2	These schemas are used as a basis to create brand new Synapse databases, on both
	3	SQLite3 and Postgres.
	4
	5	## Building full schema dumps
	6
	7	If you want to recreate these schemas, they need to be made from a database that
	8	has had all background updates run.
	9
	10	To do so, use `scripts-dev/make_full_schema.sh`. This will produce new
	11	`full.sql.postgres ` and `full.sql.sqlite` files.
	12
	13	Ensure postgres is installed and your user has the ability to run bash commands
	14	such as `createdb`, then call
	15
	16	./scripts-dev/make_full_schema.sh -p postgres_username -o output_dir/
	17
	18	There are currently two folders with full-schema snapshots. `16` is a snapshot
	19	from 2015, for historical reference. The other contains the most recent full
	20	schema snapshot.

+711

-0

synapse/storage/databases/main/search.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2015, 2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16	import re
	17	from collections import namedtuple
	18	from typing import List, Optional
	19
	20	from synapse.api.errors import SynapseError
	21	from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
	22	from synapse.storage.database import DatabasePool
	23	from synapse.storage.databases.main.events_worker import EventRedactBehaviour
	24	from synapse.storage.engines import PostgresEngine, Sqlite3Engine
	25
	26	logger = logging.getLogger(__name__)
	27
	28	SearchEntry = namedtuple(
	29	"SearchEntry",
	30	["key", "value", "event_id", "room_id", "stream_ordering", "origin_server_ts"],
	31	)
	32
	33
	34	class SearchWorkerStore(SQLBaseStore):
	35	def store_search_entries_txn(self, txn, entries):
	36	"""Add entries to the search table
	37
	38	Args:
	39	txn (cursor):
	40	entries (iterable[SearchEntry]):
	41	entries to be added to the table
	42	"""
	43	if not self.hs.config.enable_search:
	44	return
	45	if isinstance(self.database_engine, PostgresEngine):
	46	sql = (
	47	"INSERT INTO event_search"
	48	" (event_id, room_id, key, vector, stream_ordering, origin_server_ts)"
	49	" VALUES (?,?,?,to_tsvector('english', ?),?,?)"
	50	)
	51
	52	args = (
	53	(
	54	entry.event_id,
	55	entry.room_id,
	56	entry.key,
	57	entry.value,
	58	entry.stream_ordering,
	59	entry.origin_server_ts,
	60	)
	61	for entry in entries
	62	)
	63
	64	txn.executemany(sql, args)
	65
	66	elif isinstance(self.database_engine, Sqlite3Engine):
	67	sql = (
	68	"INSERT INTO event_search (event_id, room_id, key, value)"
	69	" VALUES (?,?,?,?)"
	70	)
	71	args = (
	72	(entry.event_id, entry.room_id, entry.key, entry.value)
	73	for entry in entries
	74	)
	75
	76	txn.executemany(sql, args)
	77	else:
	78	# This should be unreachable.
	79	raise Exception("Unrecognized database engine")
	80
	81
	82	class SearchBackgroundUpdateStore(SearchWorkerStore):
	83
	84	EVENT_SEARCH_UPDATE_NAME = "event_search"
	85	EVENT_SEARCH_ORDER_UPDATE_NAME = "event_search_order"
	86	EVENT_SEARCH_USE_GIST_POSTGRES_NAME = "event_search_postgres_gist"
	87	EVENT_SEARCH_USE_GIN_POSTGRES_NAME = "event_search_postgres_gin"
	88
	89	def __init__(self, database: DatabasePool, db_conn, hs):
	90	super(SearchBackgroundUpdateStore, self).__init__(database, db_conn, hs)
	91
	92	if not hs.config.enable_search:
	93	return
	94
	95	self.db_pool.updates.register_background_update_handler(
	96	self.EVENT_SEARCH_UPDATE_NAME, self._background_reindex_search
	97	)
	98	self.db_pool.updates.register_background_update_handler(
	99	self.EVENT_SEARCH_ORDER_UPDATE_NAME, self._background_reindex_search_order
	100	)
	101
	102	# we used to have a background update to turn the GIN index into a
	103	# GIST one; we no longer do that (obviously) because we actually want
	104	# a GIN index. However, it's possible that some people might still have
	105	# the background update queued, so we register a handler to clear the
	106	# background update.
	107	self.db_pool.updates.register_noop_background_update(
	108	self.EVENT_SEARCH_USE_GIST_POSTGRES_NAME
	109	)
	110
	111	self.db_pool.updates.register_background_update_handler(
	112	self.EVENT_SEARCH_USE_GIN_POSTGRES_NAME, self._background_reindex_gin_search
	113	)
	114
	115	async def _background_reindex_search(self, progress, batch_size):
	116	# we work through the events table from highest stream id to lowest
	117	target_min_stream_id = progress["target_min_stream_id_inclusive"]
	118	max_stream_id = progress["max_stream_id_exclusive"]
	119	rows_inserted = progress.get("rows_inserted", 0)
	120
	121	TYPES = ["m.room.name", "m.room.message", "m.room.topic"]
	122
	123	def reindex_search_txn(txn):
	124	sql = (
	125	"SELECT stream_ordering, event_id, room_id, type, json, "
	126	" origin_server_ts FROM events"
	127	" JOIN event_json USING (room_id, event_id)"
	128	" WHERE ? <= stream_ordering AND stream_ordering < ?"
	129	" AND (%s)"
	130	" ORDER BY stream_ordering DESC"
	131	" LIMIT ?"
	132	) % (" OR ".join("type = '%s'" % (t,) for t in TYPES),)
	133
	134	txn.execute(sql, (target_min_stream_id, max_stream_id, batch_size))
	135
	136	# we could stream straight from the results into
	137	# store_search_entries_txn with a generator function, but that
	138	# would mean having two cursors open on the database at once.
	139	# Instead we just build a list of results.
	140	rows = self.db_pool.cursor_to_dict(txn)
	141	if not rows:
	142	return 0
	143
	144	min_stream_id = rows[-1]["stream_ordering"]
	145
	146	event_search_rows = []
	147	for row in rows:
	148	try:
	149	event_id = row["event_id"]
	150	room_id = row["room_id"]
	151	etype = row["type"]
	152	stream_ordering = row["stream_ordering"]
	153	origin_server_ts = row["origin_server_ts"]
	154	try:
	155	event_json = db_to_json(row["json"])
	156	content = event_json["content"]
	157	except Exception:
	158	continue
	159
	160	if etype == "m.room.message":
	161	key = "content.body"
	162	value = content["body"]
	163	elif etype == "m.room.topic":
	164	key = "content.topic"
	165	value = content["topic"]
	166	elif etype == "m.room.name":
	167	key = "content.name"
	168	value = content["name"]
	169	else:
	170	raise Exception("unexpected event type %s" % etype)
	171	except (KeyError, AttributeError):
	172	# If the event is missing a necessary field then
	173	# skip over it.
	174	continue
	175
	176	if not isinstance(value, str):
	177	# If the event body, name or topic isn't a string
	178	# then skip over it
	179	continue
	180
	181	event_search_rows.append(
	182	SearchEntry(
	183	key=key,
	184	value=value,
	185	event_id=event_id,
	186	room_id=room_id,
	187	stream_ordering=stream_ordering,
	188	origin_server_ts=origin_server_ts,
	189	)
	190	)
	191
	192	self.store_search_entries_txn(txn, event_search_rows)
	193
	194	progress = {
	195	"target_min_stream_id_inclusive": target_min_stream_id,
	196	"max_stream_id_exclusive": min_stream_id,
	197	"rows_inserted": rows_inserted + len(event_search_rows),
	198	}
	199
	200	self.db_pool.updates._background_update_progress_txn(
	201	txn, self.EVENT_SEARCH_UPDATE_NAME, progress
	202	)
	203
	204	return len(event_search_rows)
	205
	206	result = await self.db_pool.runInteraction(
	207	self.EVENT_SEARCH_UPDATE_NAME, reindex_search_txn
	208	)
	209
	210	if not result:
	211	await self.db_pool.updates._end_background_update(
	212	self.EVENT_SEARCH_UPDATE_NAME
	213	)
	214
	215	return result
	216
	217	async def _background_reindex_gin_search(self, progress, batch_size):
	218	"""This handles old synapses which used GIST indexes, if any;
	219	converting them back to be GIN as per the actual schema.
	220	"""
	221
	222	def create_index(conn):
	223	conn.rollback()
	224
	225	# we have to set autocommit, because postgres refuses to
	226	# CREATE INDEX CONCURRENTLY without it.
	227	conn.set_session(autocommit=True)
	228
	229	try:
	230	c = conn.cursor()
	231
	232	# if we skipped the conversion to GIST, we may already/still
	233	# have an event_search_fts_idx; unfortunately postgres 9.4
	234	# doesn't support CREATE INDEX IF EXISTS so we just catch the
	235	# exception and ignore it.
	236	import psycopg2
	237
	238	try:
	239	c.execute(
	240	"CREATE INDEX CONCURRENTLY event_search_fts_idx"
	241	" ON event_search USING GIN (vector)"
	242	)
	243	except psycopg2.ProgrammingError as e:
	244	logger.warning(
	245	"Ignoring error %r when trying to switch from GIST to GIN", e
	246	)
	247
	248	# we should now be able to delete the GIST index.
	249	c.execute("DROP INDEX IF EXISTS event_search_fts_idx_gist")
	250	finally:
	251	conn.set_session(autocommit=False)
	252
	253	if isinstance(self.database_engine, PostgresEngine):
	254	await self.db_pool.runWithConnection(create_index)
	255
	256	await self.db_pool.updates._end_background_update(
	257	self.EVENT_SEARCH_USE_GIN_POSTGRES_NAME
	258	)
	259	return 1
	260
	261	async def _background_reindex_search_order(self, progress, batch_size):
	262	target_min_stream_id = progress["target_min_stream_id_inclusive"]
	263	max_stream_id = progress["max_stream_id_exclusive"]
	264	rows_inserted = progress.get("rows_inserted", 0)
	265	have_added_index = progress["have_added_indexes"]
	266
	267	if not have_added_index:
	268
	269	def create_index(conn):
	270	conn.rollback()
	271	conn.set_session(autocommit=True)
	272	c = conn.cursor()
	273
	274	# We create with NULLS FIRST so that when we search backwards
	275	# we get the ones with non null origin_server_ts first
	276	c.execute(
	277	"CREATE INDEX CONCURRENTLY event_search_room_order ON event_search("
	278	"room_id, origin_server_ts NULLS FIRST, stream_ordering NULLS FIRST)"
	279	)
	280	c.execute(
	281	"CREATE INDEX CONCURRENTLY event_search_order ON event_search("
	282	"origin_server_ts NULLS FIRST, stream_ordering NULLS FIRST)"
	283	)
	284	conn.set_session(autocommit=False)
	285
	286	await self.db_pool.runWithConnection(create_index)
	287
	288	pg = dict(progress)
	289	pg["have_added_indexes"] = True
	290
	291	await self.db_pool.runInteraction(
	292	self.EVENT_SEARCH_ORDER_UPDATE_NAME,
	293	self.db_pool.updates._background_update_progress_txn,
	294	self.EVENT_SEARCH_ORDER_UPDATE_NAME,
	295	pg,
	296	)
	297
	298	def reindex_search_txn(txn):
	299	sql = (
	300	"UPDATE event_search AS es SET stream_ordering = e.stream_ordering,"
	301	" origin_server_ts = e.origin_server_ts"
	302	" FROM events AS e"
	303	" WHERE e.event_id = es.event_id"
	304	" AND ? <= e.stream_ordering AND e.stream_ordering < ?"
	305	" RETURNING es.stream_ordering"
	306	)
	307
	308	min_stream_id = max_stream_id - batch_size
	309	txn.execute(sql, (min_stream_id, max_stream_id))
	310	rows = txn.fetchall()
	311
	312	if min_stream_id < target_min_stream_id:
	313	# We've recached the end.
	314	return len(rows), False
	315
	316	progress = {
	317	"target_min_stream_id_inclusive": target_min_stream_id,
	318	"max_stream_id_exclusive": min_stream_id,
	319	"rows_inserted": rows_inserted + len(rows),
	320	"have_added_indexes": True,
	321	}
	322
	323	self.db_pool.updates._background_update_progress_txn(
	324	txn, self.EVENT_SEARCH_ORDER_UPDATE_NAME, progress
	325	)
	326
	327	return len(rows), True
	328
	329	num_rows, finished = await self.db_pool.runInteraction(
	330	self.EVENT_SEARCH_ORDER_UPDATE_NAME, reindex_search_txn
	331	)
	332
	333	if not finished:
	334	await self.db_pool.updates._end_background_update(
	335	self.EVENT_SEARCH_ORDER_UPDATE_NAME
	336	)
	337
	338	return num_rows
	339
	340
	341	class SearchStore(SearchBackgroundUpdateStore):
	342	def __init__(self, database: DatabasePool, db_conn, hs):
	343	super(SearchStore, self).__init__(database, db_conn, hs)
	344
	345	async def search_msgs(self, room_ids, search_term, keys):
	346	"""Performs a full text search over events with given keys.
	347
	348	Args:
	349	room_ids (list): List of room ids to search in
	350	search_term (str): Search term to search for
	351	keys (list): List of keys to search in, currently supports
	352	"content.body", "content.name", "content.topic"
	353
	354	Returns:
	355	list of dicts
	356	"""
	357	clauses = []
	358
	359	search_query = _parse_query(self.database_engine, search_term)
	360
	361	args = []
	362
	363	# Make sure we don't explode because the person is in too many rooms.
	364	# We filter the results below regardless.
	365	if len(room_ids) < 500:
	366	clause, args = make_in_list_sql_clause(
	367	self.database_engine, "room_id", room_ids
	368	)
	369	clauses = [clause]
	370
	371	local_clauses = []
	372	for key in keys:
	373	local_clauses.append("key = ?")
	374	args.append(key)
	375
	376	clauses.append("(%s)" % (" OR ".join(local_clauses),))
	377
	378	count_args = args
	379	count_clauses = clauses
	380
	381	if isinstance(self.database_engine, PostgresEngine):
	382	sql = (
	383	"SELECT ts_rank_cd(vector, to_tsquery('english', ?)) AS rank,"
	384	" room_id, event_id"
	385	" FROM event_search"
	386	" WHERE vector @@ to_tsquery('english', ?)"
	387	)
	388	args = [search_query, search_query] + args
	389
	390	count_sql = (
	391	"SELECT room_id, count(*) as count FROM event_search"
	392	" WHERE vector @@ to_tsquery('english', ?)"
	393	)
	394	count_args = [search_query] + count_args
	395	elif isinstance(self.database_engine, Sqlite3Engine):
	396	sql = (
	397	"SELECT rank(matchinfo(event_search)) as rank, room_id, event_id"
	398	" FROM event_search"
	399	" WHERE value MATCH ?"
	400	)
	401	args = [search_query] + args
	402
	403	count_sql = (
	404	"SELECT room_id, count(*) as count FROM event_search"
	405	" WHERE value MATCH ?"
	406	)
	407	count_args = [search_term] + count_args
	408	else:
	409	# This should be unreachable.
	410	raise Exception("Unrecognized database engine")
	411
	412	for clause in clauses:
	413	sql += " AND " + clause
	414
	415	for clause in count_clauses:
	416	count_sql += " AND " + clause
	417
	418	# We add an arbitrary limit here to ensure we don't try to pull the
	419	# entire table from the database.
	420	sql += " ORDER BY rank DESC LIMIT 500"
	421
	422	results = await self.db_pool.execute(
	423	"search_msgs", self.db_pool.cursor_to_dict, sql, *args
	424	)
	425
	426	results = list(filter(lambda row: row["room_id"] in room_ids, results))
	427
	428	# We set redact_behaviour to BLOCK here to prevent redacted events being returned in
	429	# search results (which is a data leak)
	430	events = await self.get_events_as_list(
	431	[r["event_id"] for r in results],
	432	redact_behaviour=EventRedactBehaviour.BLOCK,
	433	)
	434
	435	event_map = {ev.event_id: ev for ev in events}
	436
	437	highlights = None
	438	if isinstance(self.database_engine, PostgresEngine):
	439	highlights = await self._find_highlights_in_postgres(search_query, events)
	440
	441	count_sql += " GROUP BY room_id"
	442
	443	count_results = await self.db_pool.execute(
	444	"search_rooms_count", self.db_pool.cursor_to_dict, count_sql, *count_args
	445	)
	446
	447	count = sum(row["count"] for row in count_results if row["room_id"] in room_ids)
	448
	449	return {
	450	"results": [
	451	{"event": event_map[r["event_id"]], "rank": r["rank"]}
	452	for r in results
	453	if r["event_id"] in event_map
	454	],
	455	"highlights": highlights,
	456	"count": count,
	457	}
	458
	459	async def search_rooms(
	460	self,
	461	room_ids: List[str],
	462	search_term: str,
	463	keys: List[str],
	464	limit,
	465	pagination_token: Optional[str] = None,
	466	) -> List[dict]:
	467	"""Performs a full text search over events with given keys.
	468
	469	Args:
	470	room_ids: The room_ids to search in
	471	search_term: Search term to search for
	472	keys: List of keys to search in, currently supports "content.body",
	473	"content.name", "content.topic"
	474	pagination_token: A pagination token previously returned
	475
	476	Returns:
	477	Each match as a dictionary.
	478	"""
	479	clauses = []
	480
	481	search_query = _parse_query(self.database_engine, search_term)
	482
	483	args = []
	484
	485	# Make sure we don't explode because the person is in too many rooms.
	486	# We filter the results below regardless.
	487	if len(room_ids) < 500:
	488	clause, args = make_in_list_sql_clause(
	489	self.database_engine, "room_id", room_ids
	490	)
	491	clauses = [clause]
	492
	493	local_clauses = []
	494	for key in keys:
	495	local_clauses.append("key = ?")
	496	args.append(key)
	497
	498	clauses.append("(%s)" % (" OR ".join(local_clauses),))
	499
	500	# take copies of the current args and clauses lists, before adding
	501	# pagination clauses to main query.
	502	count_args = list(args)
	503	count_clauses = list(clauses)
	504
	505	if pagination_token:
	506	try:
	507	origin_server_ts, stream = pagination_token.split(",")
	508	origin_server_ts = int(origin_server_ts)
	509	stream = int(stream)
	510	except Exception:
	511	raise SynapseError(400, "Invalid pagination token")
	512
	513	clauses.append(
	514	"(origin_server_ts < ?"
	515	" OR (origin_server_ts = ? AND stream_ordering < ?))"
	516	)
	517	args.extend([origin_server_ts, origin_server_ts, stream])
	518
	519	if isinstance(self.database_engine, PostgresEngine):
	520	sql = (
	521	"SELECT ts_rank_cd(vector, to_tsquery('english', ?)) as rank,"
	522	" origin_server_ts, stream_ordering, room_id, event_id"
	523	" FROM event_search"
	524	" WHERE vector @@ to_tsquery('english', ?) AND "
	525	)
	526	args = [search_query, search_query] + args
	527
	528	count_sql = (
	529	"SELECT room_id, count(*) as count FROM event_search"
	530	" WHERE vector @@ to_tsquery('english', ?) AND "
	531	)
	532	count_args = [search_query] + count_args
	533	elif isinstance(self.database_engine, Sqlite3Engine):
	534	# We use CROSS JOIN here to ensure we use the right indexes.
	535	# https://sqlite.org/optoverview.html#crossjoin
	536	#
	537	# We want to use the full text search index on event_search to
	538	# extract all possible matches first, then lookup those matches
	539	# in the events table to get the topological ordering. We need
	540	# to use the indexes in this order because sqlite refuses to
	541	# MATCH unless it uses the full text search index
	542	sql = (
	543	"SELECT rank(matchinfo) as rank, room_id, event_id,"
	544	" origin_server_ts, stream_ordering"
	545	" FROM (SELECT key, event_id, matchinfo(event_search) as matchinfo"
	546	" FROM event_search"
	547	" WHERE value MATCH ?"
	548	" )"
	549	" CROSS JOIN events USING (event_id)"
	550	" WHERE "
	551	)
	552	args = [search_query] + args
	553
	554	count_sql = (
	555	"SELECT room_id, count(*) as count FROM event_search"
	556	" WHERE value MATCH ? AND "
	557	)
	558	count_args = [search_term] + count_args
	559	else:
	560	# This should be unreachable.
	561	raise Exception("Unrecognized database engine")
	562
	563	sql += " AND ".join(clauses)
	564	count_sql += " AND ".join(count_clauses)
	565
	566	# We add an arbitrary limit here to ensure we don't try to pull the
	567	# entire table from the database.
	568	if isinstance(self.database_engine, PostgresEngine):
	569	sql += (
	570	" ORDER BY origin_server_ts DESC NULLS LAST,"
	571	" stream_ordering DESC NULLS LAST LIMIT ?"
	572	)
	573	elif isinstance(self.database_engine, Sqlite3Engine):
	574	sql += " ORDER BY origin_server_ts DESC, stream_ordering DESC LIMIT ?"
	575	else:
	576	raise Exception("Unrecognized database engine")
	577
	578	args.append(limit)
	579
	580	results = await self.db_pool.execute(
	581	"search_rooms", self.db_pool.cursor_to_dict, sql, *args
	582	)
	583
	584	results = list(filter(lambda row: row["room_id"] in room_ids, results))
	585
	586	# We set redact_behaviour to BLOCK here to prevent redacted events being returned in
	587	# search results (which is a data leak)
	588	events = await self.get_events_as_list(
	589	[r["event_id"] for r in results],
	590	redact_behaviour=EventRedactBehaviour.BLOCK,
	591	)
	592
	593	event_map = {ev.event_id: ev for ev in events}
	594
	595	highlights = None
	596	if isinstance(self.database_engine, PostgresEngine):
	597	highlights = await self._find_highlights_in_postgres(search_query, events)
	598
	599	count_sql += " GROUP BY room_id"
	600
	601	count_results = await self.db_pool.execute(
	602	"search_rooms_count", self.db_pool.cursor_to_dict, count_sql, *count_args
	603	)
	604
	605	count = sum(row["count"] for row in count_results if row["room_id"] in room_ids)
	606
	607	return {
	608	"results": [
	609	{
	610	"event": event_map[r["event_id"]],
	611	"rank": r["rank"],
	612	"pagination_token": "%s,%s"
	613	% (r["origin_server_ts"], r["stream_ordering"]),
	614	}
	615	for r in results
	616	if r["event_id"] in event_map
	617	],
	618	"highlights": highlights,
	619	"count": count,
	620	}
	621
	622	def _find_highlights_in_postgres(self, search_query, events):
	623	"""Given a list of events and a search term, return a list of words
	624	that match from the content of the event.
	625
	626	This is used to give a list of words that clients can match against to
	627	highlight the matching parts.
	628
	629	Args:
	630	search_query (str)
	631	events (list): A list of events
	632
	633	Returns:
	634	deferred : A set of strings.
	635	"""
	636
	637	def f(txn):
	638	highlight_words = set()
	639	for event in events:
	640	# As a hack we simply join values of all possible keys. This is
	641	# fine since we're only using them to find possible highlights.
	642	values = []
	643	for key in ("body", "name", "topic"):
	644	v = event.content.get(key, None)
	645	if v:
	646	values.append(v)
	647
	648	if not values:
	649	continue
	650
	651	value = " ".join(values)
	652
	653	# We need to find some values for StartSel and StopSel that
	654	# aren't in the value so that we can pick results out.
	655	start_sel = "<"
	656	stop_sel = ">"
	657
	658	while start_sel in value:
	659	start_sel += "<"
	660	while stop_sel in value:
	661	stop_sel += ">"
	662
	663	query = "SELECT ts_headline(?, to_tsquery('english', ?), %s)" % (
	664	_to_postgres_options(
	665	{
	666	"StartSel": start_sel,
	667	"StopSel": stop_sel,
	668	"MaxFragments": "50",
	669	}
	670	)
	671	)
	672	txn.execute(query, (value, search_query))
	673	(headline,) = txn.fetchall()[0]
	674
	675	# Now we need to pick the possible highlights out of the haedline
	676	# result.
	677	matcher_regex = "%s(.*?)%s" % (
	678	re.escape(start_sel),
	679	re.escape(stop_sel),
	680	)
	681
	682	res = re.findall(matcher_regex, headline)
	683	highlight_words.update([r.lower() for r in res])
	684
	685	return highlight_words
	686
	687	return self.db_pool.runInteraction("_find_highlights", f)
	688
	689
	690	def _to_postgres_options(options_dict):
	691	return "'%s'" % (",".join("%s=%s" % (k, v) for k, v in options_dict.items()),)
	692
	693
	694	def _parse_query(database_engine, search_term):
	695	"""Takes a plain unicode string from the user and converts it into a form
	696	that can be passed to database.
	697	We use this so that we can add prefix matching, which isn't something
	698	that is supported by default.
	699	"""
	700
	701	# Pull out the individual words, discarding any non-word characters.
	702	results = re.findall(r"([\w\-]+)", search_term, re.UNICODE)
	703
	704	if isinstance(database_engine, PostgresEngine):
	705	return " & ".join(result + ":*" for result in results)
	706	elif isinstance(database_engine, Sqlite3Engine):
	707	return " & ".join(result + "*" for result in results)
	708	else:
	709	# This should be unreachable.
	710	raise Exception("Unrecognized database engine")

+68

-0

synapse/storage/databases/main/signatures.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from unpaddedbase64 import encode_base64
	16
	17	from synapse.storage._base import SQLBaseStore
	18	from synapse.util.caches.descriptors import cached, cachedList
	19
	20
	21	class SignatureWorkerStore(SQLBaseStore):
	22	@cached()
	23	def get_event_reference_hash(self, event_id):
	24	# This is a dummy function to allow get_event_reference_hashes
	25	# to use its cache
	26	raise NotImplementedError()
	27
	28	@cachedList(
	29	cached_method_name="get_event_reference_hash", list_name="event_ids", num_args=1
	30	)
	31	def get_event_reference_hashes(self, event_ids):
	32	def f(txn):
	33	return {
	34	event_id: self._get_event_reference_hashes_txn(txn, event_id)
	35	for event_id in event_ids
	36	}
	37
	38	return self.db_pool.runInteraction("get_event_reference_hashes", f)
	39
	40	async def add_event_hashes(self, event_ids):
	41	hashes = await self.get_event_reference_hashes(event_ids)
	42	hashes = {
	43	e_id: {k: encode_base64(v) for k, v in h.items() if k == "sha256"}
	44	for e_id, h in hashes.items()
	45	}
	46
	47	return list(hashes.items())
	48
	49	def _get_event_reference_hashes_txn(self, txn, event_id):
	50	"""Get all the hashes for a given PDU.
	51	Args:
	52	txn (cursor):
	53	event_id (str): Id for the Event.
	54	Returns:
	55	A dict[unicode, bytes] of algorithm -> hash.
	56	"""
	57	query = (
	58	"SELECT algorithm, hash"
	59	" FROM event_reference_hashes"
	60	" WHERE event_id = ?"
	61	)
	62	txn.execute(query, (event_id,))
	63	return {k: v for k, v in txn}
	64
	65
	66	class SignatureStore(SignatureWorkerStore):
	67	"""Persistence for event signatures and hashes"""

+509

-0

synapse/storage/databases/main/state.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2020 The Matrix.org Foundation C.I.C.
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15	import collections.abc
	16	import logging
	17	from collections import namedtuple
	18	from typing import Iterable, Optional, Set
	19
	20	from synapse.api.constants import EventTypes, Membership
	21	from synapse.api.errors import NotFoundError, UnsupportedRoomVersionError
	22	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, RoomVersion
	23	from synapse.events import EventBase
	24	from synapse.storage._base import SQLBaseStore
	25	from synapse.storage.database import DatabasePool
	26	from synapse.storage.databases.main.events_worker import EventsWorkerStore
	27	from synapse.storage.databases.main.roommember import RoomMemberWorkerStore
	28	from synapse.storage.state import StateFilter
	29	from synapse.util.caches import intern_string
	30	from synapse.util.caches.descriptors import cached, cachedList
	31
	32	logger = logging.getLogger(__name__)
	33
	34
	35	MAX_STATE_DELTA_HOPS = 100
	36
	37
	38	class _GetStateGroupDelta(
	39	namedtuple("_GetStateGroupDelta", ("prev_group", "delta_ids"))
	40	):
	41	"""Return type of get_state_group_delta that implements __len__, which lets
	42	us use the itrable flag when caching
	43	"""
	44
	45	__slots__ = []
	46
	47	def __len__(self):
	48	return len(self.delta_ids) if self.delta_ids else 0
	49
	50
	51	# this inherits from EventsWorkerStore because it calls self.get_events
	52	class StateGroupWorkerStore(EventsWorkerStore, SQLBaseStore):
	53	"""The parts of StateGroupStore that can be called from workers.
	54	"""
	55
	56	def __init__(self, database: DatabasePool, db_conn, hs):
	57	super(StateGroupWorkerStore, self).__init__(database, db_conn, hs)
	58
	59	async def get_room_version(self, room_id: str) -> RoomVersion:
	60	"""Get the room_version of a given room
	61
	62	Raises:
	63	NotFoundError: if the room is unknown
	64
	65	UnsupportedRoomVersionError: if the room uses an unknown room version.
	66	Typically this happens if support for the room's version has been
	67	removed from Synapse.
	68	"""
	69	room_version_id = await self.get_room_version_id(room_id)
	70	v = KNOWN_ROOM_VERSIONS.get(room_version_id)
	71
	72	if not v:
	73	raise UnsupportedRoomVersionError(
	74	"Room %s uses a room version %s which is no longer supported"
	75	% (room_id, room_version_id)
	76	)
	77
	78	return v
	79
	80	@cached(max_entries=10000)
	81	async def get_room_version_id(self, room_id: str) -> str:
	82	"""Get the room_version of a given room
	83
	84	Raises:
	85	NotFoundError: if the room is unknown
	86	"""
	87
	88	# First we try looking up room version from the database, but for old
	89	# rooms we might not have added the room version to it yet so we fall
	90	# back to previous behaviour and look in current state events.
	91
	92	# We really should have an entry in the rooms table for every room we
	93	# care about, but let's be a bit paranoid (at least while the background
	94	# update is happening) to avoid breaking existing rooms.
	95	version = await self.db_pool.simple_select_one_onecol(
	96	table="rooms",
	97	keyvalues={"room_id": room_id},
	98	retcol="room_version",
	99	desc="get_room_version",
	100	allow_none=True,
	101	)
	102
	103	if version is not None:
	104	return version
	105
	106	# Retrieve the room's create event
	107	create_event = await self.get_create_event_for_room(room_id)
	108	return create_event.content.get("room_version", "1")
	109
	110	async def get_room_predecessor(self, room_id: str) -> Optional[dict]:
	111	"""Get the predecessor of an upgraded room if it exists.
	112	Otherwise return None.
	113
	114	Args:
	115	room_id: The room ID.
	116
	117	Returns:
	118	A dictionary containing the structure of the predecessor
	119	field from the room's create event. The structure is subject to other servers,
	120	but it is expected to be:
	121	* room_id (str): The room ID of the predecessor room
	122	* event_id (str): The ID of the tombstone event in the predecessor room
	123
	124	None if a predecessor key is not found, or is not a dictionary.
	125
	126	Raises:
	127	NotFoundError if the given room is unknown
	128	"""
	129	# Retrieve the room's create event
	130	create_event = await self.get_create_event_for_room(room_id)
	131
	132	# Retrieve the predecessor key of the create event
	133	predecessor = create_event.content.get("predecessor", None)
	134
	135	# Ensure the key is a dictionary
	136	if not isinstance(predecessor, collections.abc.Mapping):
	137	return None
	138
	139	return predecessor
	140
	141	async def get_create_event_for_room(self, room_id: str) -> EventBase:
	142	"""Get the create state event for a room.
	143
	144	Args:
	145	room_id: The room ID.
	146
	147	Returns:
	148	The room creation event.
	149
	150	Raises:
	151	NotFoundError if the room is unknown
	152	"""
	153	state_ids = await self.get_current_state_ids(room_id)
	154	create_id = state_ids.get((EventTypes.Create, ""))
	155
	156	# If we can't find the create event, assume we've hit a dead end
	157	if not create_id:
	158	raise NotFoundError("Unknown room %s" % (room_id,))
	159
	160	# Retrieve the room's create event and return
	161	create_event = await self.get_event(create_id)
	162	return create_event
	163
	164	@cached(max_entries=100000, iterable=True)
	165	def get_current_state_ids(self, room_id):
	166	"""Get the current state event ids for a room based on the
	167	current_state_events table.
	168
	169	Args:
	170	room_id (str)
	171
	172	Returns:
	173	deferred: dict of (type, state_key) -> event_id
	174	"""
	175
	176	def _get_current_state_ids_txn(txn):
	177	txn.execute(
	178	"""SELECT type, state_key, event_id FROM current_state_events
	179	WHERE room_id = ?
	180	""",
	181	(room_id,),
	182	)
	183
	184	return {(intern_string(r[0]), intern_string(r[1])): r[2] for r in txn}
	185
	186	return self.db_pool.runInteraction(
	187	"get_current_state_ids", _get_current_state_ids_txn
	188	)
	189
	190	# FIXME: how should this be cached?
	191	def get_filtered_current_state_ids(
	192	self, room_id: str, state_filter: StateFilter = StateFilter.all()
	193	):
	194	"""Get the current state event of a given type for a room based on the
	195	current_state_events table. This may not be as up-to-date as the result
	196	of doing a fresh state resolution as per state_handler.get_current_state
	197
	198	Args:
	199	room_id
	200	state_filter: The state filter used to fetch state
	201	from the database.
	202
	203	Returns:
	204	defer.Deferred[StateMap[str]]: Map from type/state_key to event ID.
	205	"""
	206
	207	where_clause, where_args = state_filter.make_sql_filter_clause()
	208
	209	if not where_clause:
	210	# We delegate to the cached version
	211	return self.get_current_state_ids(room_id)
	212
	213	def _get_filtered_current_state_ids_txn(txn):
	214	results = {}
	215	sql = """
	216	SELECT type, state_key, event_id FROM current_state_events
	217	WHERE room_id = ?
	218	"""
	219
	220	if where_clause:
	221	sql += " AND (%s)" % (where_clause,)
	222
	223	args = [room_id]
	224	args.extend(where_args)
	225	txn.execute(sql, args)
	226	for row in txn:
	227	typ, state_key, event_id = row
	228	key = (intern_string(typ), intern_string(state_key))
	229	results[key] = event_id
	230
	231	return results
	232
	233	return self.db_pool.runInteraction(
	234	"get_filtered_current_state_ids", _get_filtered_current_state_ids_txn
	235	)
	236
	237	async def get_canonical_alias_for_room(self, room_id: str) -> Optional[str]:
	238	"""Get canonical alias for room, if any
	239
	240	Args:
	241	room_id: The room ID
	242
	243	Returns:
	244	The canonical alias, if any
	245	"""
	246
	247	state = await self.get_filtered_current_state_ids(
	248	room_id, StateFilter.from_types([(EventTypes.CanonicalAlias, "")])
	249	)
	250
	251	event_id = state.get((EventTypes.CanonicalAlias, ""))
	252	if not event_id:
	253	return
	254
	255	event = await self.get_event(event_id, allow_none=True)
	256	if not event:
	257	return
	258
	259	return event.content.get("canonical_alias")
	260
	261	@cached(max_entries=50000)
	262	def _get_state_group_for_event(self, event_id):
	263	return self.db_pool.simple_select_one_onecol(
	264	table="event_to_state_groups",
	265	keyvalues={"event_id": event_id},
	266	retcol="state_group",
	267	allow_none=True,
	268	desc="_get_state_group_for_event",
	269	)
	270
	271	@cachedList(
	272	cached_method_name="_get_state_group_for_event",
	273	list_name="event_ids",
	274	num_args=1,
	275	inlineCallbacks=True,
	276	)
	277	def _get_state_group_for_events(self, event_ids):
	278	"""Returns mapping event_id -> state_group
	279	"""
	280	rows = yield self.db_pool.simple_select_many_batch(
	281	table="event_to_state_groups",
	282	column="event_id",
	283	iterable=event_ids,
	284	keyvalues={},
	285	retcols=("event_id", "state_group"),
	286	desc="_get_state_group_for_events",
	287	)
	288
	289	return {row["event_id"]: row["state_group"] for row in rows}
	290
	291	async def get_referenced_state_groups(
	292	self, state_groups: Iterable[int]
	293	) -> Set[int]:
	294	"""Check if the state groups are referenced by events.
	295
	296	Args:
	297	state_groups
	298
	299	Returns:
	300	The subset of state groups that are referenced.
	301	"""
	302
	303	rows = await self.db_pool.simple_select_many_batch(
	304	table="event_to_state_groups",
	305	column="state_group",
	306	iterable=state_groups,
	307	keyvalues={},
	308	retcols=("DISTINCT state_group",),
	309	desc="get_referenced_state_groups",
	310	)
	311
	312	return {row["state_group"] for row in rows}
	313
	314
	315	class MainStateBackgroundUpdateStore(RoomMemberWorkerStore):
	316
	317	CURRENT_STATE_INDEX_UPDATE_NAME = "current_state_members_idx"
	318	EVENT_STATE_GROUP_INDEX_UPDATE_NAME = "event_to_state_groups_sg_index"
	319	DELETE_CURRENT_STATE_UPDATE_NAME = "delete_old_current_state_events"
	320
	321	def __init__(self, database: DatabasePool, db_conn, hs):
	322	super(MainStateBackgroundUpdateStore, self).__init__(database, db_conn, hs)
	323
	324	self.server_name = hs.hostname
	325
	326	self.db_pool.updates.register_background_index_update(
	327	self.CURRENT_STATE_INDEX_UPDATE_NAME,
	328	index_name="current_state_events_member_index",
	329	table="current_state_events",
	330	columns=["state_key"],
	331	where_clause="type='m.room.member'",
	332	)
	333	self.db_pool.updates.register_background_index_update(
	334	self.EVENT_STATE_GROUP_INDEX_UPDATE_NAME,
	335	index_name="event_to_state_groups_sg_index",
	336	table="event_to_state_groups",
	337	columns=["state_group"],
	338	)
	339	self.db_pool.updates.register_background_update_handler(
	340	self.DELETE_CURRENT_STATE_UPDATE_NAME, self._background_remove_left_rooms,
	341	)
	342
	343	async def _background_remove_left_rooms(self, progress, batch_size):
	344	"""Background update to delete rows from `current_state_events` and
	345	`event_forward_extremities` tables of rooms that the server is no
	346	longer joined to.
	347	"""
	348
	349	last_room_id = progress.get("last_room_id", "")
	350
	351	def _background_remove_left_rooms_txn(txn):
	352	# get a batch of room ids to consider
	353	sql = """
	354	SELECT DISTINCT room_id FROM current_state_events
	355	WHERE room_id > ? ORDER BY room_id LIMIT ?
	356	"""
	357
	358	txn.execute(sql, (last_room_id, batch_size))
	359	room_ids = [row[0] for row in txn]
	360	if not room_ids:
	361	return True, set()
	362
	363	###########################################################################
	364	#
	365	# exclude rooms where we have active members
	366
	367	sql = """
	368	SELECT room_id
	369	FROM local_current_membership
	370	WHERE
	371	room_id > ? AND room_id <= ?
	372	AND membership = 'join'
	373	GROUP BY room_id
	374	"""
	375
	376	txn.execute(sql, (last_room_id, room_ids[-1]))
	377	joined_room_ids = {row[0] for row in txn}
	378	to_delete = set(room_ids) - joined_room_ids
	379
	380	###########################################################################
	381	#
	382	# exclude rooms which we are in the process of constructing; these otherwise
	383	# qualify as "rooms with no local users", and would have their
	384	# forward extremities cleaned up.
	385
	386	# the following query will return a list of rooms which have forward
	387	# extremities that are not also the create event in the room - ie
	388	# those that are not being created currently.
	389
	390	sql = """
	391	SELECT DISTINCT efe.room_id
	392	FROM event_forward_extremities efe
	393	LEFT JOIN current_state_events cse ON
	394	cse.event_id = efe.event_id
	395	AND cse.type = 'm.room.create'
	396	AND cse.state_key = ''
	397	WHERE
	398	cse.event_id IS NULL
	399	AND efe.room_id > ? AND efe.room_id <= ?
	400	"""
	401
	402	txn.execute(sql, (last_room_id, room_ids[-1]))
	403
	404	# build a set of those rooms within `to_delete` that do not appear in
	405	# the above, leaving us with the rooms in `to_delete` that are being
	406	# created.
	407	creating_rooms = to_delete.difference(row[0] for row in txn)
	408	logger.info("skipping rooms which are being created: %s", creating_rooms)
	409
	410	# now remove the rooms being created from the list of those to delete.
	411	#
	412	# (we could have just taken the intersection of `to_delete` with the result
	413	# of the sql query, but it's useful to be able to log `creating_rooms`; and
	414	# having done so, it's quicker to remove the (few) creating rooms from
	415	# `to_delete` than it is to form the intersection with the (larger) list of
	416	# not-creating-rooms)
	417
	418	to_delete -= creating_rooms
	419
	420	###########################################################################
	421	#
	422	# now clear the state for the rooms
	423
	424	logger.info("Deleting current state left rooms: %r", to_delete)
	425
	426	# First we get all users that we still think were joined to the
	427	# room. This is so that we can mark those device lists as
	428	# potentially stale, since there may have been a period where the
	429	# server didn't share a room with the remote user and therefore may
	430	# have missed any device updates.
	431	rows = self.db_pool.simple_select_many_txn(
	432	txn,
	433	table="current_state_events",
	434	column="room_id",
	435	iterable=to_delete,
	436	keyvalues={"type": EventTypes.Member, "membership": Membership.JOIN},
	437	retcols=("state_key",),
	438	)
	439
	440	potentially_left_users = {row["state_key"] for row in rows}
	441
	442	# Now lets actually delete the rooms from the DB.
	443	self.db_pool.simple_delete_many_txn(
	444	txn,
	445	table="current_state_events",
	446	column="room_id",
	447	iterable=to_delete,
	448	keyvalues={},
	449	)
	450
	451	self.db_pool.simple_delete_many_txn(
	452	txn,
	453	table="event_forward_extremities",
	454	column="room_id",
	455	iterable=to_delete,
	456	keyvalues={},
	457	)
	458
	459	self.db_pool.updates._background_update_progress_txn(
	460	txn,
	461	self.DELETE_CURRENT_STATE_UPDATE_NAME,
	462	{"last_room_id": room_ids[-1]},
	463	)
	464
	465	return False, potentially_left_users
	466
	467	finished, potentially_left_users = await self.db_pool.runInteraction(
	468	"_background_remove_left_rooms", _background_remove_left_rooms_txn
	469	)
	470
	471	if finished:
	472	await self.db_pool.updates._end_background_update(
	473	self.DELETE_CURRENT_STATE_UPDATE_NAME
	474	)
	475
	476	# Now go and check if we still share a room with the remote users in
	477	# the deleted rooms. If not mark their device lists as stale.
	478	joined_users = await self.get_users_server_still_shares_room_with(
	479	potentially_left_users
	480	)
	481
	482	for user_id in potentially_left_users - joined_users:
	483	await self.mark_remote_user_device_list_as_unsubscribed(user_id)
	484
	485	return batch_size
	486
	487
	488	class StateStore(StateGroupWorkerStore, MainStateBackgroundUpdateStore):
	489	""" Keeps track of the state at a given event.
	490
	491	This is done by the concept of `state groups`. Every event is a assigned
	492	a state group (identified by an arbitrary string), which references a
	493	collection of state events. The current state of an event is then the
	494	collection of state events referenced by the event's state group.
	495
	496	Hence, every change in the current state causes a new state group to be
	497	generated. However, if no change happens (e.g., if we get a message event
	498	with only one parent it inherits the state group from its parent.)
	499
	500	There are three tables:
	501	* `state_groups`: Stores group name, first event with in the group and
	502	room id.
	503	* `event_to_state_groups`: Maps events to state groups.
	504	* `state_groups_state`: Maps state group to state events.
	505	"""
	506
	507	def __init__(self, database: DatabasePool, db_conn, hs):
	508	super(StateStore, self).__init__(database, db_conn, hs)

+121

-0

synapse/storage/databases/main/state_deltas.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2018 Vector Creations Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16
	17	from twisted.internet import defer
	18
	19	from synapse.storage._base import SQLBaseStore
	20
	21	logger = logging.getLogger(__name__)
	22
	23
	24	class StateDeltasStore(SQLBaseStore):
	25	def get_current_state_deltas(self, prev_stream_id: int, max_stream_id: int):
	26	"""Fetch a list of room state changes since the given stream id
	27
	28	Each entry in the result contains the following fields:
	29	- stream_id (int)
	30	- room_id (str)
	31	- type (str): event type
	32	- state_key (str):
	33	- event_id (str\|None): new event_id for this state key. None if the
	34	state has been deleted.
	35	- prev_event_id (str\|None): previous event_id for this state key. None
	36	if it's new state.
	37
	38	Args:
	39	prev_stream_id (int): point to get changes since (exclusive)
	40	max_stream_id (int): the point that we know has been correctly persisted
	41	- ie, an upper limit to return changes from.
	42
	43	Returns:
	44	Deferred[tuple[int, list[dict]]: A tuple consisting of:
	45	- the stream id which these results go up to
	46	- list of current_state_delta_stream rows. If it is empty, we are
	47	up to date.
	48	"""
	49	prev_stream_id = int(prev_stream_id)
	50
	51	# check we're not going backwards
	52	assert prev_stream_id <= max_stream_id
	53
	54	if not self._curr_state_delta_stream_cache.has_any_entity_changed(
	55	prev_stream_id
	56	):
	57	# if the CSDs haven't changed between prev_stream_id and now, we
	58	# know for certain that they haven't changed between prev_stream_id and
	59	# max_stream_id.
	60	return defer.succeed((max_stream_id, []))
	61
	62	def get_current_state_deltas_txn(txn):
	63	# First we calculate the max stream id that will give us less than
	64	# N results.
	65	# We arbitarily limit to 100 stream_id entries to ensure we don't
	66	# select toooo many.
	67	sql = """
	68	SELECT stream_id, count(*)
	69	FROM current_state_delta_stream
	70	WHERE stream_id > ? AND stream_id <= ?
	71	GROUP BY stream_id
	72	ORDER BY stream_id ASC
	73	LIMIT 100
	74	"""
	75	txn.execute(sql, (prev_stream_id, max_stream_id))
	76
	77	total = 0
	78
	79	for stream_id, count in txn:
	80	total += count
	81	if total > 100:
	82	# We arbitarily limit to 100 entries to ensure we don't
	83	# select toooo many.
	84	logger.debug(
	85	"Clipping current_state_delta_stream rows to stream_id %i",
	86	stream_id,
	87	)
	88	clipped_stream_id = stream_id
	89	break
	90	else:
	91	# if there's no problem, we may as well go right up to the max_stream_id
	92	clipped_stream_id = max_stream_id
	93
	94	# Now actually get the deltas
	95	sql = """
	96	SELECT stream_id, room_id, type, state_key, event_id, prev_event_id
	97	FROM current_state_delta_stream
	98	WHERE ? < stream_id AND stream_id <= ?
	99	ORDER BY stream_id ASC
	100	"""
	101	txn.execute(sql, (prev_stream_id, clipped_stream_id))
	102	return clipped_stream_id, self.db_pool.cursor_to_dict(txn)
	103
	104	return self.db_pool.runInteraction(
	105	"get_current_state_deltas", get_current_state_deltas_txn
	106	)
	107
	108	def _get_max_stream_id_in_current_state_deltas_txn(self, txn):
	109	return self.db_pool.simple_select_one_onecol_txn(
	110	txn,
	111	table="current_state_delta_stream",
	112	keyvalues={},
	113	retcol="COALESCE(MAX(stream_id), -1)",
	114	)
	115
	116	def get_max_stream_id_in_current_state_deltas(self):
	117	return self.db_pool.runInteraction(
	118	"get_max_stream_id_in_current_state_deltas",
	119	self._get_max_stream_id_in_current_state_deltas_txn,
	120	)

+886

-0

synapse/storage/databases/main/stats.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2018, 2019 New Vector Ltd
	2	# Copyright 2019 The Matrix.org Foundation C.I.C.
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import logging
	17	from itertools import chain
	18	from typing import Tuple
	19
	20	from twisted.internet.defer import DeferredLock
	21
	22	from synapse.api.constants import EventTypes, Membership
	23	from synapse.storage.database import DatabasePool
	24	from synapse.storage.databases.main.state_deltas import StateDeltasStore
	25	from synapse.storage.engines import PostgresEngine
	26	from synapse.util.caches.descriptors import cached
	27
	28	logger = logging.getLogger(__name__)
	29
	30	# these fields track absolutes (e.g. total number of rooms on the server)
	31	# You can think of these as Prometheus Gauges.
	32	# You can draw these stats on a line graph.
	33	# Example: number of users in a room
	34	ABSOLUTE_STATS_FIELDS = {
	35	"room": (
	36	"current_state_events",
	37	"joined_members",
	38	"invited_members",
	39	"left_members",
	40	"banned_members",
	41	"local_users_in_room",
	42	),
	43	"user": ("joined_rooms",),
	44	}
	45
	46	# these fields are per-timeslice and so should be reset to 0 upon a new slice
	47	# You can draw these stats on a histogram.
	48	# Example: number of events sent locally during a time slice
	49	PER_SLICE_FIELDS = {
	50	"room": ("total_events", "total_event_bytes"),
	51	"user": ("invites_sent", "rooms_created", "total_events", "total_event_bytes"),
	52	}
	53
	54	TYPE_TO_TABLE = {"room": ("room_stats", "room_id"), "user": ("user_stats", "user_id")}
	55
	56	# these are the tables (& ID columns) which contain our actual subjects
	57	TYPE_TO_ORIGIN_TABLE = {"room": ("rooms", "room_id"), "user": ("users", "name")}
	58
	59
	60	class StatsStore(StateDeltasStore):
	61	def __init__(self, database: DatabasePool, db_conn, hs):
	62	super(StatsStore, self).__init__(database, db_conn, hs)
	63
	64	self.server_name = hs.hostname
	65	self.clock = self.hs.get_clock()
	66	self.stats_enabled = hs.config.stats_enabled
	67	self.stats_bucket_size = hs.config.stats_bucket_size
	68
	69	self.stats_delta_processing_lock = DeferredLock()
	70
	71	self.db_pool.updates.register_background_update_handler(
	72	"populate_stats_process_rooms", self._populate_stats_process_rooms
	73	)
	74	self.db_pool.updates.register_background_update_handler(
	75	"populate_stats_process_rooms_2", self._populate_stats_process_rooms_2
	76	)
	77	self.db_pool.updates.register_background_update_handler(
	78	"populate_stats_process_users", self._populate_stats_process_users
	79	)
	80	# we no longer need to perform clean-up, but we will give ourselves
	81	# the potential to reintroduce it in the future – so documentation
	82	# will still encourage the use of this no-op handler.
	83	self.db_pool.updates.register_noop_background_update("populate_stats_cleanup")
	84	self.db_pool.updates.register_noop_background_update("populate_stats_prepare")
	85
	86	def quantise_stats_time(self, ts):
	87	"""
	88	Quantises a timestamp to be a multiple of the bucket size.
	89
	90	Args:
	91	ts (int): the timestamp to quantise, in milliseconds since the Unix
	92	Epoch
	93
	94	Returns:
	95	int: a timestamp which
	96	- is divisible by the bucket size;
	97	- is no later than `ts`; and
	98	- is the largest such timestamp.
	99	"""
	100	return (ts // self.stats_bucket_size) * self.stats_bucket_size
	101
	102	async def _populate_stats_process_users(self, progress, batch_size):
	103	"""
	104	This is a background update which regenerates statistics for users.
	105	"""
	106	if not self.stats_enabled:
	107	await self.db_pool.updates._end_background_update(
	108	"populate_stats_process_users"
	109	)
	110	return 1
	111
	112	last_user_id = progress.get("last_user_id", "")
	113
	114	def _get_next_batch(txn):
	115	sql = """
	116	SELECT DISTINCT name FROM users
	117	WHERE name > ?
	118	ORDER BY name ASC
	119	LIMIT ?
	120	"""
	121	txn.execute(sql, (last_user_id, batch_size))
	122	return [r for r, in txn]
	123
	124	users_to_work_on = await self.db_pool.runInteraction(
	125	"_populate_stats_process_users", _get_next_batch
	126	)
	127
	128	# No more rooms -- complete the transaction.
	129	if not users_to_work_on:
	130	await self.db_pool.updates._end_background_update(
	131	"populate_stats_process_users"
	132	)
	133	return 1
	134
	135	for user_id in users_to_work_on:
	136	await self._calculate_and_set_initial_state_for_user(user_id)
	137	progress["last_user_id"] = user_id
	138
	139	await self.db_pool.runInteraction(
	140	"populate_stats_process_users",
	141	self.db_pool.updates._background_update_progress_txn,
	142	"populate_stats_process_users",
	143	progress,
	144	)
	145
	146	return len(users_to_work_on)
	147
	148	async def _populate_stats_process_rooms(self, progress, batch_size):
	149	"""
	150	This was a background update which regenerated statistics for rooms.
	151
	152	It has been replaced by StatsStore._populate_stats_process_rooms_2. This background
	153	job has been scheduled to run as part of Synapse v1.0.0, and again now. To ensure
	154	someone upgrading from <v1.0.0, this background task has been turned into a no-op
	155	so that the potentially expensive task is not run twice.
	156
	157	Further context: https://github.com/matrix-org/synapse/pull/7977
	158	"""
	159	await self.db_pool.updates._end_background_update(
	160	"populate_stats_process_rooms"
	161	)
	162	return 1
	163
	164	async def _populate_stats_process_rooms_2(self, progress, batch_size):
	165	"""
	166	This is a background update which regenerates statistics for rooms.
	167
	168	It replaces StatsStore._populate_stats_process_rooms. See its docstring for the
	169	reasoning.
	170	"""
	171	if not self.stats_enabled:
	172	await self.db_pool.updates._end_background_update(
	173	"populate_stats_process_rooms_2"
	174	)
	175	return 1
	176
	177	last_room_id = progress.get("last_room_id", "")
	178
	179	def _get_next_batch(txn):
	180	sql = """
	181	SELECT DISTINCT room_id FROM current_state_events
	182	WHERE room_id > ?
	183	ORDER BY room_id ASC
	184	LIMIT ?
	185	"""
	186	txn.execute(sql, (last_room_id, batch_size))
	187	return [r for r, in txn]
	188
	189	rooms_to_work_on = await self.db_pool.runInteraction(
	190	"populate_stats_rooms_2_get_batch", _get_next_batch
	191	)
	192
	193	# No more rooms -- complete the transaction.
	194	if not rooms_to_work_on:
	195	await self.db_pool.updates._end_background_update(
	196	"populate_stats_process_rooms_2"
	197	)
	198	return 1
	199
	200	for room_id in rooms_to_work_on:
	201	await self._calculate_and_set_initial_state_for_room(room_id)
	202	progress["last_room_id"] = room_id
	203
	204	await self.db_pool.runInteraction(
	205	"_populate_stats_process_rooms_2",
	206	self.db_pool.updates._background_update_progress_txn,
	207	"populate_stats_process_rooms_2",
	208	progress,
	209	)
	210
	211	return len(rooms_to_work_on)
	212
	213	def get_stats_positions(self):
	214	"""
	215	Returns the stats processor positions.
	216	"""
	217	return self.db_pool.simple_select_one_onecol(
	218	table="stats_incremental_position",
	219	keyvalues={},
	220	retcol="stream_id",
	221	desc="stats_incremental_position",
	222	)
	223
	224	def update_room_state(self, room_id, fields):
	225	"""
	226	Args:
	227	room_id (str)
	228	fields (dict[str:Any])
	229	"""
	230
	231	# For whatever reason some of the fields may contain null bytes, which
	232	# postgres isn't a fan of, so we replace those fields with null.
	233	for col in (
	234	"join_rules",
	235	"history_visibility",
	236	"encryption",
	237	"name",
	238	"topic",
	239	"avatar",
	240	"canonical_alias",
	241	):
	242	field = fields.get(col)
	243	if field and "\0" in field:
	244	fields[col] = None
	245
	246	return self.db_pool.simple_upsert(
	247	table="room_stats_state",
	248	keyvalues={"room_id": room_id},
	249	values=fields,
	250	desc="update_room_state",
	251	)
	252
	253	def get_statistics_for_subject(self, stats_type, stats_id, start, size=100):
	254	"""
	255	Get statistics for a given subject.
	256
	257	Args:
	258	stats_type (str): The type of subject
	259	stats_id (str): The ID of the subject (e.g. room_id or user_id)
	260	start (int): Pagination start. Number of entries, not timestamp.
	261	size (int): How many entries to return.
	262
	263	Returns:
	264	Deferred[list[dict]], where the dict has the keys of
	265	ABSOLUTE_STATS_FIELDS[stats_type], and "bucket_size" and "end_ts".
	266	"""
	267	return self.db_pool.runInteraction(
	268	"get_statistics_for_subject",
	269	self._get_statistics_for_subject_txn,
	270	stats_type,
	271	stats_id,
	272	start,
	273	size,
	274	)
	275
	276	def _get_statistics_for_subject_txn(
	277	self, txn, stats_type, stats_id, start, size=100
	278	):
	279	"""
	280	Transaction-bound version of L{get_statistics_for_subject}.
	281	"""
	282
	283	table, id_col = TYPE_TO_TABLE[stats_type]
	284	selected_columns = list(
	285	ABSOLUTE_STATS_FIELDS[stats_type] + PER_SLICE_FIELDS[stats_type]
	286	)
	287
	288	slice_list = self.db_pool.simple_select_list_paginate_txn(
	289	txn,
	290	table + "_historical",
	291	"end_ts",
	292	start,
	293	size,
	294	retcols=selected_columns + ["bucket_size", "end_ts"],
	295	keyvalues={id_col: stats_id},
	296	order_direction="DESC",
	297	)
	298
	299	return slice_list
	300
	301	@cached()
	302	def get_earliest_token_for_stats(self, stats_type, id):
	303	"""
	304	Fetch the "earliest token". This is used by the room stats delta
	305	processor to ignore deltas that have been processed between the
	306	start of the background task and any particular room's stats
	307	being calculated.
	308
	309	Returns:
	310	Deferred[int]
	311	"""
	312	table, id_col = TYPE_TO_TABLE[stats_type]
	313
	314	return self.db_pool.simple_select_one_onecol(
	315	"%s_current" % (table,),
	316	keyvalues={id_col: id},
	317	retcol="completed_delta_stream_id",
	318	allow_none=True,
	319	)
	320
	321	def bulk_update_stats_delta(self, ts, updates, stream_id):
	322	"""Bulk update stats tables for a given stream_id and updates the stats
	323	incremental position.
	324
	325	Args:
	326	ts (int): Current timestamp in ms
	327	updates(dict[str, dict[str, dict[str, Counter]]]): The updates to
	328	commit as a mapping stats_type -> stats_id -> field -> delta.
	329	stream_id (int): Current position.
	330
	331	Returns:
	332	Deferred
	333	"""
	334
	335	def _bulk_update_stats_delta_txn(txn):
	336	for stats_type, stats_updates in updates.items():
	337	for stats_id, fields in stats_updates.items():
	338	logger.debug(
	339	"Updating %s stats for %s: %s", stats_type, stats_id, fields
	340	)
	341	self._update_stats_delta_txn(
	342	txn,
	343	ts=ts,
	344	stats_type=stats_type,
	345	stats_id=stats_id,
	346	fields=fields,
	347	complete_with_stream_id=stream_id,
	348	)
	349
	350	self.db_pool.simple_update_one_txn(
	351	txn,
	352	table="stats_incremental_position",
	353	keyvalues={},
	354	updatevalues={"stream_id": stream_id},
	355	)
	356
	357	return self.db_pool.runInteraction(
	358	"bulk_update_stats_delta", _bulk_update_stats_delta_txn
	359	)
	360
	361	def update_stats_delta(
	362	self,
	363	ts,
	364	stats_type,
	365	stats_id,
	366	fields,
	367	complete_with_stream_id,
	368	absolute_field_overrides=None,
	369	):
	370	"""
	371	Updates the statistics for a subject, with a delta (difference/relative
	372	change).
	373
	374	Args:
	375	ts (int): timestamp of the change
	376	stats_type (str): "room" or "user" – the kind of subject
	377	stats_id (str): the subject's ID (room ID or user ID)
	378	fields (dict[str, int]): Deltas of stats values.
	379	complete_with_stream_id (int, optional):
	380	If supplied, converts an incomplete row into a complete row,
	381	with the supplied stream_id marked as the stream_id where the
	382	row was completed.
	383	absolute_field_overrides (dict[str, int]): Current stats values
	384	(i.e. not deltas) of absolute fields.
	385	Does not work with per-slice fields.
	386	"""
	387
	388	return self.db_pool.runInteraction(
	389	"update_stats_delta",
	390	self._update_stats_delta_txn,
	391	ts,
	392	stats_type,
	393	stats_id,
	394	fields,
	395	complete_with_stream_id=complete_with_stream_id,
	396	absolute_field_overrides=absolute_field_overrides,
	397	)
	398
	399	def _update_stats_delta_txn(
	400	self,
	401	txn,
	402	ts,
	403	stats_type,
	404	stats_id,
	405	fields,
	406	complete_with_stream_id,
	407	absolute_field_overrides=None,
	408	):
	409	if absolute_field_overrides is None:
	410	absolute_field_overrides = {}
	411
	412	table, id_col = TYPE_TO_TABLE[stats_type]
	413
	414	quantised_ts = self.quantise_stats_time(int(ts))
	415	end_ts = quantised_ts + self.stats_bucket_size
	416
	417	# Lets be paranoid and check that all the given field names are known
	418	abs_field_names = ABSOLUTE_STATS_FIELDS[stats_type]
	419	slice_field_names = PER_SLICE_FIELDS[stats_type]
	420	for field in chain(fields.keys(), absolute_field_overrides.keys()):
	421	if field not in abs_field_names and field not in slice_field_names:
	422	# guard against potential SQL injection dodginess
	423	raise ValueError(
	424	"%s is not a recognised field"
	425	" for stats type %s" % (field, stats_type)
	426	)
	427
	428	# Per slice fields do not get added to the _current table
	429
	430	# This calculates the deltas (`field = field + ?` values)
	431	# for absolute fields,
	432	# * defaulting to 0 if not specified
	433	# (required for the INSERT part of upserting to work)
	434	# * omitting overrides specified in `absolute_field_overrides`
	435	deltas_of_absolute_fields = {
	436	key: fields.get(key, 0)
	437	for key in abs_field_names
	438	if key not in absolute_field_overrides
	439	}
	440
	441	# Keep the delta stream ID field up to date
	442	absolute_field_overrides = absolute_field_overrides.copy()
	443	absolute_field_overrides["completed_delta_stream_id"] = complete_with_stream_id
	444
	445	# first upsert the `_current` table
	446	self._upsert_with_additive_relatives_txn(
	447	txn=txn,
	448	table=table + "_current",
	449	keyvalues={id_col: stats_id},
	450	absolutes=absolute_field_overrides,
	451	additive_relatives=deltas_of_absolute_fields,
	452	)
	453
	454	per_slice_additive_relatives = {
	455	key: fields.get(key, 0) for key in slice_field_names
	456	}
	457	self._upsert_copy_from_table_with_additive_relatives_txn(
	458	txn=txn,
	459	into_table=table + "_historical",
	460	keyvalues={id_col: stats_id},
	461	extra_dst_insvalues={"bucket_size": self.stats_bucket_size},
	462	extra_dst_keyvalues={"end_ts": end_ts},
	463	additive_relatives=per_slice_additive_relatives,
	464	src_table=table + "_current",
	465	copy_columns=abs_field_names,
	466	)
	467
	468	def _upsert_with_additive_relatives_txn(
	469	self, txn, table, keyvalues, absolutes, additive_relatives
	470	):
	471	"""Used to update values in the stats tables.
	472
	473	This is basically a slightly convoluted upsert that adds to any
	474	existing rows.
	475
	476	Args:
	477	txn
	478	table (str): Table name
	479	keyvalues (dict[str, any]): Row-identifying key values
	480	absolutes (dict[str, any]): Absolute (set) fields
	481	additive_relatives (dict[str, int]): Fields that will be added onto
	482	if existing row present.
	483	"""
	484	if self.database_engine.can_native_upsert:
	485	absolute_updates = [
	486	"%(field)s = EXCLUDED.%(field)s" % {"field": field}
	487	for field in absolutes.keys()
	488	]
	489
	490	relative_updates = [
	491	"%(field)s = EXCLUDED.%(field)s + %(table)s.%(field)s"
	492	% {"table": table, "field": field}
	493	for field in additive_relatives.keys()
	494	]
	495
	496	insert_cols = []
	497	qargs = []
	498
	499	for (key, val) in chain(
	500	keyvalues.items(), absolutes.items(), additive_relatives.items()
	501	):
	502	insert_cols.append(key)
	503	qargs.append(val)
	504
	505	sql = """
	506	INSERT INTO %(table)s (%(insert_cols_cs)s)
	507	VALUES (%(insert_vals_qs)s)
	508	ON CONFLICT (%(key_columns)s) DO UPDATE SET %(updates)s
	509	""" % {
	510	"table": table,
	511	"insert_cols_cs": ", ".join(insert_cols),
	512	"insert_vals_qs": ", ".join(
	513	["?"] * (len(keyvalues) + len(absolutes) + len(additive_relatives))
	514	),
	515	"key_columns": ", ".join(keyvalues),
	516	"updates": ", ".join(chain(absolute_updates, relative_updates)),
	517	}
	518
	519	txn.execute(sql, qargs)
	520	else:
	521	self.database_engine.lock_table(txn, table)
	522	retcols = list(chain(absolutes.keys(), additive_relatives.keys()))
	523	current_row = self.db_pool.simple_select_one_txn(
	524	txn, table, keyvalues, retcols, allow_none=True
	525	)
	526	if current_row is None:
	527	merged_dict = {keyvalues, absolutes, **additive_relatives}
	528	self.db_pool.simple_insert_txn(txn, table, merged_dict)
	529	else:
	530	for (key, val) in additive_relatives.items():
	531	current_row[key] += val
	532	current_row.update(absolutes)
	533	self.db_pool.simple_update_one_txn(txn, table, keyvalues, current_row)
	534
	535	def _upsert_copy_from_table_with_additive_relatives_txn(
	536	self,
	537	txn,
	538	into_table,
	539	keyvalues,
	540	extra_dst_keyvalues,
	541	extra_dst_insvalues,
	542	additive_relatives,
	543	src_table,
	544	copy_columns,
	545	):
	546	"""Updates the historic stats table with latest updates.
	547
	548	This involves copying "absolute" fields from the `_current` table, and
	549	adding relative fields to any existing values.
	550
	551	Args:
	552	txn: Transaction
	553	into_table (str): The destination table to UPSERT the row into
	554	keyvalues (dict[str, any]): Row-identifying key values
	555	extra_dst_keyvalues (dict[str, any]): Additional keyvalues
	556	for `into_table`.
	557	extra_dst_insvalues (dict[str, any]): Additional values to insert
	558	on new row creation for `into_table`.
	559	additive_relatives (dict[str, any]): Fields that will be added onto
	560	if existing row present. (Must be disjoint from copy_columns.)
	561	src_table (str): The source table to copy from
	562	copy_columns (iterable[str]): The list of columns to copy
	563	"""
	564	if self.database_engine.can_native_upsert:
	565	ins_columns = chain(
	566	keyvalues,
	567	copy_columns,
	568	additive_relatives,
	569	extra_dst_keyvalues,
	570	extra_dst_insvalues,
	571	)
	572	sel_exprs = chain(
	573	keyvalues,
	574	copy_columns,
	575	(
	576	"?"
	577	for _ in chain(
	578	additive_relatives, extra_dst_keyvalues, extra_dst_insvalues
	579	)
	580	),
	581	)
	582	keyvalues_where = ("%s = ?" % f for f in keyvalues)
	583
	584	sets_cc = ("%s = EXCLUDED.%s" % (f, f) for f in copy_columns)
	585	sets_ar = (
	586	"%s = EXCLUDED.%s + %s.%s" % (f, f, into_table, f)
	587	for f in additive_relatives
	588	)
	589
	590	sql = """
	591	INSERT INTO %(into_table)s (%(ins_columns)s)
	592	SELECT %(sel_exprs)s
	593	FROM %(src_table)s
	594	WHERE %(keyvalues_where)s
	595	ON CONFLICT (%(keyvalues)s)
	596	DO UPDATE SET %(sets)s
	597	""" % {
	598	"into_table": into_table,
	599	"ins_columns": ", ".join(ins_columns),
	600	"sel_exprs": ", ".join(sel_exprs),
	601	"keyvalues_where": " AND ".join(keyvalues_where),
	602	"src_table": src_table,
	603	"keyvalues": ", ".join(
	604	chain(keyvalues.keys(), extra_dst_keyvalues.keys())
	605	),
	606	"sets": ", ".join(chain(sets_cc, sets_ar)),
	607	}
	608
	609	qargs = list(
	610	chain(
	611	additive_relatives.values(),
	612	extra_dst_keyvalues.values(),
	613	extra_dst_insvalues.values(),
	614	keyvalues.values(),
	615	)
	616	)
	617	txn.execute(sql, qargs)
	618	else:
	619	self.database_engine.lock_table(txn, into_table)
	620	src_row = self.db_pool.simple_select_one_txn(
	621	txn, src_table, keyvalues, copy_columns
	622	)
	623	all_dest_keyvalues = {keyvalues, extra_dst_keyvalues}
	624	dest_current_row = self.db_pool.simple_select_one_txn(
	625	txn,
	626	into_table,
	627	keyvalues=all_dest_keyvalues,
	628	retcols=list(chain(additive_relatives.keys(), copy_columns)),
	629	allow_none=True,
	630	)
	631
	632	if dest_current_row is None:
	633	merged_dict = {
	634	**keyvalues,
	635	**extra_dst_keyvalues,
	636	**extra_dst_insvalues,
	637	**src_row,
	638	**additive_relatives,
	639	}
	640	self.db_pool.simple_insert_txn(txn, into_table, merged_dict)
	641	else:
	642	for (key, val) in additive_relatives.items():
	643	src_row[key] = dest_current_row[key] + val
	644	self.db_pool.simple_update_txn(
	645	txn, into_table, all_dest_keyvalues, src_row
	646	)
	647
	648	def get_changes_room_total_events_and_bytes(self, min_pos, max_pos):
	649	"""Fetches the counts of events in the given range of stream IDs.
	650
	651	Args:
	652	min_pos (int)
	653	max_pos (int)
	654
	655	Returns:
	656	Deferred[dict[str, dict[str, int]]]: Mapping of room ID to field
	657	changes.
	658	"""
	659
	660	return self.db_pool.runInteraction(
	661	"stats_incremental_total_events_and_bytes",
	662	self.get_changes_room_total_events_and_bytes_txn,
	663	min_pos,
	664	max_pos,
	665	)
	666
	667	def get_changes_room_total_events_and_bytes_txn(self, txn, low_pos, high_pos):
	668	"""Gets the total_events and total_event_bytes counts for rooms and
	669	senders, in a range of stream_orderings (including backfilled events).
	670
	671	Args:
	672	txn
	673	low_pos (int): Low stream ordering
	674	high_pos (int): High stream ordering
	675
	676	Returns:
	677	tuple[dict[str, dict[str, int]], dict[str, dict[str, int]]]: The
	678	room and user deltas for total_events/total_event_bytes in the
	679	format of `stats_id` -> fields
	680	"""
	681
	682	if low_pos >= high_pos:
	683	# nothing to do here.
	684	return {}, {}
	685
	686	if isinstance(self.database_engine, PostgresEngine):
	687	new_bytes_expression = "OCTET_LENGTH(json)"
	688	else:
	689	new_bytes_expression = "LENGTH(CAST(json AS BLOB))"
	690
	691	sql = """
	692	SELECT events.room_id, COUNT(*) AS new_events, SUM(%s) AS new_bytes
	693	FROM events INNER JOIN event_json USING (event_id)
	694	WHERE (? < stream_ordering AND stream_ordering <= ?)
	695	OR (? <= stream_ordering AND stream_ordering <= ?)
	696	GROUP BY events.room_id
	697	""" % (
	698	new_bytes_expression,
	699	)
	700
	701	txn.execute(sql, (low_pos, high_pos, -high_pos, -low_pos))
	702
	703	room_deltas = {
	704	room_id: {"total_events": new_events, "total_event_bytes": new_bytes}
	705	for room_id, new_events, new_bytes in txn
	706	}
	707
	708	sql = """
	709	SELECT events.sender, COUNT(*) AS new_events, SUM(%s) AS new_bytes
	710	FROM events INNER JOIN event_json USING (event_id)
	711	WHERE (? < stream_ordering AND stream_ordering <= ?)
	712	OR (? <= stream_ordering AND stream_ordering <= ?)
	713	GROUP BY events.sender
	714	""" % (
	715	new_bytes_expression,
	716	)
	717
	718	txn.execute(sql, (low_pos, high_pos, -high_pos, -low_pos))
	719
	720	user_deltas = {
	721	user_id: {"total_events": new_events, "total_event_bytes": new_bytes}
	722	for user_id, new_events, new_bytes in txn
	723	if self.hs.is_mine_id(user_id)
	724	}
	725
	726	return room_deltas, user_deltas
	727
	728	async def _calculate_and_set_initial_state_for_room(
	729	self, room_id: str
	730	) -> Tuple[dict, dict, int]:
	731	"""Calculate and insert an entry into room_stats_current.
	732
	733	Args:
	734	room_id: The room ID under calculation.
	735
	736	Returns:
	737	A tuple of room state, membership counts and stream position.
	738	"""
	739
	740	def _fetch_current_state_stats(txn):
	741	pos = self.get_room_max_stream_ordering()
	742
	743	rows = self.db_pool.simple_select_many_txn(
	744	txn,
	745	table="current_state_events",
	746	column="type",
	747	iterable=[
	748	EventTypes.Create,
	749	EventTypes.JoinRules,
	750	EventTypes.RoomHistoryVisibility,
	751	EventTypes.RoomEncryption,
	752	EventTypes.Name,
	753	EventTypes.Topic,
	754	EventTypes.RoomAvatar,
	755	EventTypes.CanonicalAlias,
	756	],
	757	keyvalues={"room_id": room_id, "state_key": ""},
	758	retcols=["event_id"],
	759	)
	760
	761	event_ids = [row["event_id"] for row in rows]
	762
	763	txn.execute(
	764	"""
	765	SELECT membership, count(*) FROM current_state_events
	766	WHERE room_id = ? AND type = 'm.room.member'
	767	GROUP BY membership
	768	""",
	769	(room_id,),
	770	)
	771	membership_counts = {membership: cnt for membership, cnt in txn}
	772
	773	txn.execute(
	774	"""
	775	SELECT COALESCE(count(*), 0) FROM current_state_events
	776	WHERE room_id = ?
	777	""",
	778	(room_id,),
	779	)
	780
	781	(current_state_events_count,) = txn.fetchone()
	782
	783	users_in_room = self.get_users_in_room_txn(txn, room_id)
	784
	785	return (
	786	event_ids,
	787	membership_counts,
	788	current_state_events_count,
	789	users_in_room,
	790	pos,
	791	)
	792
	793	(
	794	event_ids,
	795	membership_counts,
	796	current_state_events_count,
	797	users_in_room,
	798	pos,
	799	) = await self.db_pool.runInteraction(
	800	"get_initial_state_for_room", _fetch_current_state_stats
	801	)
	802
	803	state_event_map = await self.get_events(event_ids, get_prev_content=False)
	804
	805	room_state = {
	806	"join_rules": None,
	807	"history_visibility": None,
	808	"encryption": None,
	809	"name": None,
	810	"topic": None,
	811	"avatar": None,
	812	"canonical_alias": None,
	813	"is_federatable": True,
	814	}
	815
	816	for event in state_event_map.values():
	817	if event.type == EventTypes.JoinRules:
	818	room_state["join_rules"] = event.content.get("join_rule")
	819	elif event.type == EventTypes.RoomHistoryVisibility:
	820	room_state["history_visibility"] = event.content.get(
	821	"history_visibility"
	822	)
	823	elif event.type == EventTypes.RoomEncryption:
	824	room_state["encryption"] = event.content.get("algorithm")
	825	elif event.type == EventTypes.Name:
	826	room_state["name"] = event.content.get("name")
	827	elif event.type == EventTypes.Topic:
	828	room_state["topic"] = event.content.get("topic")
	829	elif event.type == EventTypes.RoomAvatar:
	830	room_state["avatar"] = event.content.get("url")
	831	elif event.type == EventTypes.CanonicalAlias:
	832	room_state["canonical_alias"] = event.content.get("alias")
	833	elif event.type == EventTypes.Create:
	834	room_state["is_federatable"] = (
	835	event.content.get("m.federate", True) is True
	836	)
	837
	838	await self.update_room_state(room_id, room_state)
	839
	840	local_users_in_room = [u for u in users_in_room if self.hs.is_mine_id(u)]
	841
	842	await self.update_stats_delta(
	843	ts=self.clock.time_msec(),
	844	stats_type="room",
	845	stats_id=room_id,
	846	fields={},
	847	complete_with_stream_id=pos,
	848	absolute_field_overrides={
	849	"current_state_events": current_state_events_count,
	850	"joined_members": membership_counts.get(Membership.JOIN, 0),
	851	"invited_members": membership_counts.get(Membership.INVITE, 0),
	852	"left_members": membership_counts.get(Membership.LEAVE, 0),
	853	"banned_members": membership_counts.get(Membership.BAN, 0),
	854	"local_users_in_room": len(local_users_in_room),
	855	},
	856	)
	857
	858	async def _calculate_and_set_initial_state_for_user(self, user_id):
	859	def _calculate_and_set_initial_state_for_user_txn(txn):
	860	pos = self._get_max_stream_id_in_current_state_deltas_txn(txn)
	861
	862	txn.execute(
	863	"""
	864	SELECT COUNT(distinct room_id) FROM current_state_events
	865	WHERE type = 'm.room.member' AND state_key = ?
	866	AND membership = 'join'
	867	""",
	868	(user_id,),
	869	)
	870	(count,) = txn.fetchone()
	871	return count, pos
	872
	873	joined_rooms, pos = await self.db_pool.runInteraction(
	874	"calculate_and_set_initial_state_for_user",
	875	_calculate_and_set_initial_state_for_user_txn,
	876	)
	877
	878	await self.update_stats_delta(
	879	ts=self.clock.time_msec(),
	880	stats_type="user",
	881	stats_id=user_id,
	882	fields={},
	883	complete_with_stream_id=pos,
	884	absolute_field_overrides={"joined_rooms": joined_rooms},
	885	)

+1064

-0

synapse/storage/databases/main/stream.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2017 Vector Creations Ltd
	3	# Copyright 2018-2019 New Vector Ltd
	4	# Copyright 2019 The Matrix.org Foundation C.I.C.
	5	#
	6	# Licensed under the Apache License, Version 2.0 (the "License");
	7	# you may not use this file except in compliance with the License.
	8	# You may obtain a copy of the License at
	9	#
	10	# http://www.apache.org/licenses/LICENSE-2.0
	11	#
	12	# Unless required by applicable law or agreed to in writing, software
	13	# distributed under the License is distributed on an "AS IS" BASIS,
	14	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	15	# See the License for the specific language governing permissions and
	16	# limitations under the License.
	17
	18	""" This module is responsible for getting events from the DB for pagination
	19	and event streaming.
	20
	21	The order it returns events in depend on whether we are streaming forwards or
	22	are paginating backwards. We do this because we want to handle out of order
	23	messages nicely, while still returning them in the correct order when we
	24	paginate bacwards.
	25
	26	This is implemented by keeping two ordering columns: stream_ordering and
	27	topological_ordering. Stream ordering is basically insertion/received order
	28	(except for events from backfill requests). The topological_ordering is a
	29	weak ordering of events based on the pdu graph.
	30
	31	This means that we have to have two different types of tokens, depending on
	32	what sort order was used:
	33	- stream tokens are of the form: "s%d", which maps directly to the column
	34	- topological tokems: "t%d-%d", where the integers map to the topological
	35	and stream ordering columns respectively.
	36	"""
	37
	38	import abc
	39	import logging
	40	from collections import namedtuple
	41	from typing import Optional
	42
	43	from twisted.internet import defer
	44
	45	from synapse.logging.context import make_deferred_yieldable, run_in_background
	46	from synapse.storage._base import SQLBaseStore
	47	from synapse.storage.database import DatabasePool, make_in_list_sql_clause
	48	from synapse.storage.databases.main.events_worker import EventsWorkerStore
	49	from synapse.storage.engines import PostgresEngine
	50	from synapse.types import RoomStreamToken
	51	from synapse.util.caches.stream_change_cache import StreamChangeCache
	52
	53	logger = logging.getLogger(__name__)
	54
	55
	56	MAX_STREAM_SIZE = 1000
	57
	58
	59	_STREAM_TOKEN = "stream"
	60	_TOPOLOGICAL_TOKEN = "topological"
	61
	62
	63	# Used as return values for pagination APIs
	64	_EventDictReturn = namedtuple(
	65	"_EventDictReturn", ("event_id", "topological_ordering", "stream_ordering")
	66	)
	67
	68
	69	def generate_pagination_where_clause(
	70	direction, column_names, from_token, to_token, engine
	71	):
	72	"""Creates an SQL expression to bound the columns by the pagination
	73	tokens.
	74
	75	For example creates an SQL expression like:
	76
	77	(6, 7) >= (topological_ordering, stream_ordering)
	78	AND (5, 3) < (topological_ordering, stream_ordering)
	79
	80	would be generated for dir=b, from_token=(6, 7) and to_token=(5, 3).
	81
	82	Note that tokens are considered to be after the row they are in, e.g. if
	83	a row A has a token T, then we consider A to be before T. This convention
	84	is important when figuring out inequalities for the generated SQL, and
	85	produces the following result:
	86	- If paginating forwards then we exclude any rows matching the from
	87	token, but include those that match the to token.
	88	- If paginating backwards then we include any rows matching the from
	89	token, but include those that match the to token.
	90
	91	Args:
	92	direction (str): Whether we're paginating backwards("b") or
	93	forwards ("f").
	94	column_names (tuple[str, str]): The column names to bound. Must not
	95	be user defined as these get inserted directly into the SQL
	96	statement without escapes.
	97	from_token (tuple[int, int]\|None): The start point for the pagination.
	98	This is an exclusive minimum bound if direction is "f", and an
	99	inclusive maximum bound if direction is "b".
	100	to_token (tuple[int, int]\|None): The endpoint point for the pagination.
	101	This is an inclusive maximum bound if direction is "f", and an
	102	exclusive minimum bound if direction is "b".
	103	engine: The database engine to generate the clauses for
	104
	105	Returns:
	106	str: The sql expression
	107	"""
	108	assert direction in ("b", "f")
	109
	110	where_clause = []
	111	if from_token:
	112	where_clause.append(
	113	_make_generic_sql_bound(
	114	bound=">=" if direction == "b" else "<",
	115	column_names=column_names,
	116	values=from_token,
	117	engine=engine,
	118	)
	119	)
	120
	121	if to_token:
	122	where_clause.append(
	123	_make_generic_sql_bound(
	124	bound="<" if direction == "b" else ">=",
	125	column_names=column_names,
	126	values=to_token,
	127	engine=engine,
	128	)
	129	)
	130
	131	return " AND ".join(where_clause)
	132
	133
	134	def _make_generic_sql_bound(bound, column_names, values, engine):
	135	"""Create an SQL expression that bounds the given column names by the
	136	values, e.g. create the equivalent of `(1, 2) < (col1, col2)`.
	137
	138	Only works with two columns.
	139
	140	Older versions of SQLite don't support that syntax so we have to expand it
	141	out manually.
	142
	143	Args:
	144	bound (str): The comparison operator to use. One of ">", "<", ">=",
	145	"<=", where the values are on the left and columns on the right.
	146	names (tuple[str, str]): The column names. Must not be user defined
	147	as these get inserted directly into the SQL statement without
	148	escapes.
	149	values (tuple[int\|None, int]): The values to bound the columns by. If
	150	the first value is None then only creates a bound on the second
	151	column.
	152	engine: The database engine to generate the SQL for
	153
	154	Returns:
	155	str
	156	"""
	157
	158	assert bound in (">", "<", ">=", "<=")
	159
	160	name1, name2 = column_names
	161	val1, val2 = values
	162
	163	if val1 is None:
	164	val2 = int(val2)
	165	return "(%d %s %s)" % (val2, bound, name2)
	166
	167	val1 = int(val1)
	168	val2 = int(val2)
	169
	170	if isinstance(engine, PostgresEngine):
	171	# Postgres doesn't optimise ``(x < a) OR (x=a AND y<b)`` as well
	172	# as it optimises ``(x,y) < (a,b)`` on multicolumn indexes. So we
	173	# use the later form when running against postgres.
	174	return "((%d,%d) %s (%s,%s))" % (val1, val2, bound, name1, name2)
	175
	176	# We want to generate queries of e.g. the form:
	177	#
	178	# (val1 < name1 OR (val1 = name1 AND val2 <= name2))
	179	#
	180	# which is equivalent to (val1, val2) < (name1, name2)
	181
	182	return """(
	183	{val1:d} {strict_bound} {name1}
	184	OR ({val1:d} = {name1} AND {val2:d} {bound} {name2})
	185	)""".format(
	186	name1=name1,
	187	val1=val1,
	188	name2=name2,
	189	val2=val2,
	190	strict_bound=bound[0], # The first bound must always be strict equality here
	191	bound=bound,
	192	)
	193
	194
	195	def filter_to_clause(event_filter):
	196	# NB: This may create SQL clauses that don't optimise well (and we don't
	197	# have indices on all possible clauses). E.g. it may create
	198	# "room_id == X AND room_id != X", which postgres doesn't optimise.
	199
	200	if not event_filter:
	201	return "", []
	202
	203	clauses = []
	204	args = []
	205
	206	if event_filter.types:
	207	clauses.append("(%s)" % " OR ".join("type = ?" for _ in event_filter.types))
	208	args.extend(event_filter.types)
	209
	210	for typ in event_filter.not_types:
	211	clauses.append("type != ?")
	212	args.append(typ)
	213
	214	if event_filter.senders:
	215	clauses.append("(%s)" % " OR ".join("sender = ?" for _ in event_filter.senders))
	216	args.extend(event_filter.senders)
	217
	218	for sender in event_filter.not_senders:
	219	clauses.append("sender != ?")
	220	args.append(sender)
	221
	222	if event_filter.rooms:
	223	clauses.append("(%s)" % " OR ".join("room_id = ?" for _ in event_filter.rooms))
	224	args.extend(event_filter.rooms)
	225
	226	for room_id in event_filter.not_rooms:
	227	clauses.append("room_id != ?")
	228	args.append(room_id)
	229
	230	if event_filter.contains_url:
	231	clauses.append("contains_url = ?")
	232	args.append(event_filter.contains_url)
	233
	234	# We're only applying the "labels" filter on the database query, because applying the
	235	# "not_labels" filter via a SQL query is non-trivial. Instead, we let
	236	# event_filter.check_fields apply it, which is not as efficient but makes the
	237	# implementation simpler.
	238	if event_filter.labels:
	239	clauses.append("(%s)" % " OR ".join("label = ?" for _ in event_filter.labels))
	240	args.extend(event_filter.labels)
	241
	242	return " AND ".join(clauses), args
	243
	244
	245	class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
	246	"""This is an abstract base class where subclasses must implement
	247	`get_room_max_stream_ordering` and `get_room_min_stream_ordering`
	248	which can be called in the initializer.
	249	"""
	250
	251	__metaclass__ = abc.ABCMeta
	252
	253	def __init__(self, database: DatabasePool, db_conn, hs):
	254	super(StreamWorkerStore, self).__init__(database, db_conn, hs)
	255
	256	self._instance_name = hs.get_instance_name()
	257	self._send_federation = hs.should_send_federation()
	258	self._federation_shard_config = hs.config.worker.federation_shard_config
	259
	260	# If we're a process that sends federation we may need to reset the
	261	# `federation_stream_position` table to match the current sharding
	262	# config. We don't do this now as otherwise two processes could conflict
	263	# during startup which would cause one to die.
	264	self._need_to_reset_federation_stream_positions = self._send_federation
	265
	266	events_max = self.get_room_max_stream_ordering()
	267	event_cache_prefill, min_event_val = self.db_pool.get_cache_dict(
	268	db_conn,
	269	"events",
	270	entity_column="room_id",
	271	stream_column="stream_ordering",
	272	max_value=events_max,
	273	)
	274	self._events_stream_cache = StreamChangeCache(
	275	"EventsRoomStreamChangeCache",
	276	min_event_val,
	277	prefilled_cache=event_cache_prefill,
	278	)
	279	self._membership_stream_cache = StreamChangeCache(
	280	"MembershipStreamChangeCache", events_max
	281	)
	282
	283	self._stream_order_on_start = self.get_room_max_stream_ordering()
	284
	285	@abc.abstractmethod
	286	def get_room_max_stream_ordering(self):
	287	raise NotImplementedError()
	288
	289	@abc.abstractmethod
	290	def get_room_min_stream_ordering(self):
	291	raise NotImplementedError()
	292
	293	@defer.inlineCallbacks
	294	def get_room_events_stream_for_rooms(
	295	self, room_ids, from_key, to_key, limit=0, order="DESC"
	296	):
	297	"""Get new room events in stream ordering since `from_key`.
	298
	299	Args:
	300	room_id (str)
	301	from_key (str): Token from which no events are returned before
	302	to_key (str): Token from which no events are returned after. (This
	303	is typically the current stream token)
	304	limit (int): Maximum number of events to return
	305	order (str): Either "DESC" or "ASC". Determines which events are
	306	returned when the result is limited. If "DESC" then the most
	307	recent `limit` events are returned, otherwise returns the
	308	oldest `limit` events.
	309
	310	Returns:
	311	Deferred[dict[str,tuple[list[FrozenEvent], str]]]
	312	A map from room id to a tuple containing:
	313	- list of recent events in the room
	314	- stream ordering key for the start of the chunk of events returned.
	315	"""
	316	from_id = RoomStreamToken.parse_stream_token(from_key).stream
	317
	318	room_ids = yield self._events_stream_cache.get_entities_changed(
	319	room_ids, from_id
	320	)
	321
	322	if not room_ids:
	323	return {}
	324
	325	results = {}
	326	room_ids = list(room_ids)
	327	for rm_ids in (room_ids[i : i + 20] for i in range(0, len(room_ids), 20)):
	328	res = yield make_deferred_yieldable(
	329	defer.gatherResults(
	330	[
	331	run_in_background(
	332	self.get_room_events_stream_for_room,
	333	room_id,
	334	from_key,
	335	to_key,
	336	limit,
	337	order=order,
	338	)
	339	for room_id in rm_ids
	340	],
	341	consumeErrors=True,
	342	)
	343	)
	344	results.update(dict(zip(rm_ids, res)))
	345
	346	return results
	347
	348	def get_rooms_that_changed(self, room_ids, from_key):
	349	"""Given a list of rooms and a token, return rooms where there may have
	350	been changes.
	351
	352	Args:
	353	room_ids (list)
	354	from_key (str): The room_key portion of a StreamToken
	355	"""
	356	from_key = RoomStreamToken.parse_stream_token(from_key).stream
	357	return {
	358	room_id
	359	for room_id in room_ids
	360	if self._events_stream_cache.has_entity_changed(room_id, from_key)
	361	}
	362
	363	@defer.inlineCallbacks
	364	def get_room_events_stream_for_room(
	365	self, room_id, from_key, to_key, limit=0, order="DESC"
	366	):
	367
	368	"""Get new room events in stream ordering since `from_key`.
	369
	370	Args:
	371	room_id (str)
	372	from_key (str): Token from which no events are returned before
	373	to_key (str): Token from which no events are returned after. (This
	374	is typically the current stream token)
	375	limit (int): Maximum number of events to return
	376	order (str): Either "DESC" or "ASC". Determines which events are
	377	returned when the result is limited. If "DESC" then the most
	378	recent `limit` events are returned, otherwise returns the
	379	oldest `limit` events.
	380
	381	Returns:
	382	Deferred[tuple[list[FrozenEvent], str]]: Returns the list of
	383	events (in ascending order) and the token from the start of
	384	the chunk of events returned.
	385	"""
	386	if from_key == to_key:
	387	return [], from_key
	388
	389	from_id = RoomStreamToken.parse_stream_token(from_key).stream
	390	to_id = RoomStreamToken.parse_stream_token(to_key).stream
	391
	392	has_changed = yield self._events_stream_cache.has_entity_changed(
	393	room_id, from_id
	394	)
	395
	396	if not has_changed:
	397	return [], from_key
	398
	399	def f(txn):
	400	sql = (
	401	"SELECT event_id, stream_ordering FROM events WHERE"
	402	" room_id = ?"
	403	" AND not outlier"
	404	" AND stream_ordering > ? AND stream_ordering <= ?"
	405	" ORDER BY stream_ordering %s LIMIT ?"
	406	) % (order,)
	407	txn.execute(sql, (room_id, from_id, to_id, limit))
	408
	409	rows = [_EventDictReturn(row[0], None, row[1]) for row in txn]
	410	return rows
	411
	412	rows = yield self.db_pool.runInteraction("get_room_events_stream_for_room", f)
	413
	414	ret = yield self.get_events_as_list(
	415	[r.event_id for r in rows], get_prev_content=True
	416	)
	417
	418	self._set_before_and_after(ret, rows, topo_order=from_id is None)
	419
	420	if order.lower() == "desc":
	421	ret.reverse()
	422
	423	if rows:
	424	key = "s%d" % min(r.stream_ordering for r in rows)
	425	else:
	426	# Assume we didn't get anything because there was nothing to
	427	# get.
	428	key = from_key
	429
	430	return ret, key
	431
	432	@defer.inlineCallbacks
	433	def get_membership_changes_for_user(self, user_id, from_key, to_key):
	434	from_id = RoomStreamToken.parse_stream_token(from_key).stream
	435	to_id = RoomStreamToken.parse_stream_token(to_key).stream
	436
	437	if from_key == to_key:
	438	return []
	439
	440	if from_id:
	441	has_changed = self._membership_stream_cache.has_entity_changed(
	442	user_id, int(from_id)
	443	)
	444	if not has_changed:
	445	return []
	446
	447	def f(txn):
	448	sql = (
	449	"SELECT m.event_id, stream_ordering FROM events AS e,"
	450	" room_memberships AS m"
	451	" WHERE e.event_id = m.event_id"
	452	" AND m.user_id = ?"
	453	" AND e.stream_ordering > ? AND e.stream_ordering <= ?"
	454	" ORDER BY e.stream_ordering ASC"
	455	)
	456	txn.execute(sql, (user_id, from_id, to_id))
	457
	458	rows = [_EventDictReturn(row[0], None, row[1]) for row in txn]
	459
	460	return rows
	461
	462	rows = yield self.db_pool.runInteraction("get_membership_changes_for_user", f)
	463
	464	ret = yield self.get_events_as_list(
	465	[r.event_id for r in rows], get_prev_content=True
	466	)
	467
	468	self._set_before_and_after(ret, rows, topo_order=False)
	469
	470	return ret
	471
	472	@defer.inlineCallbacks
	473	def get_recent_events_for_room(self, room_id, limit, end_token):
	474	"""Get the most recent events in the room in topological ordering.
	475
	476	Args:
	477	room_id (str)
	478	limit (int)
	479	end_token (str): The stream token representing now.
	480
	481	Returns:
	482	Deferred[tuple[list[FrozenEvent], str]]: Returns a list of
	483	events and a token pointing to the start of the returned
	484	events.
	485	The events returned are in ascending order.
	486	"""
	487
	488	rows, token = yield self.get_recent_event_ids_for_room(
	489	room_id, limit, end_token
	490	)
	491
	492	events = yield self.get_events_as_list(
	493	[r.event_id for r in rows], get_prev_content=True
	494	)
	495
	496	self._set_before_and_after(events, rows)
	497
	498	return (events, token)
	499
	500	@defer.inlineCallbacks
	501	def get_recent_event_ids_for_room(self, room_id, limit, end_token):
	502	"""Get the most recent events in the room in topological ordering.
	503
	504	Args:
	505	room_id (str)
	506	limit (int)
	507	end_token (str): The stream token representing now.
	508
	509	Returns:
	510	Deferred[tuple[list[_EventDictReturn], str]]: Returns a list of
	511	_EventDictReturn and a token pointing to the start of the returned
	512	events.
	513	The events returned are in ascending order.
	514	"""
	515	# Allow a zero limit here, and no-op.
	516	if limit == 0:
	517	return [], end_token
	518
	519	end_token = RoomStreamToken.parse(end_token)
	520
	521	rows, token = yield self.db_pool.runInteraction(
	522	"get_recent_event_ids_for_room",
	523	self._paginate_room_events_txn,
	524	room_id,
	525	from_token=end_token,
	526	limit=limit,
	527	)
	528
	529	# We want to return the results in ascending order.
	530	rows.reverse()
	531
	532	return rows, token
	533
	534	def get_room_event_before_stream_ordering(self, room_id, stream_ordering):
	535	"""Gets details of the first event in a room at or before a stream ordering
	536
	537	Args:
	538	room_id (str):
	539	stream_ordering (int):
	540
	541	Returns:
	542	Deferred[(int, int, str)]:
	543	(stream ordering, topological ordering, event_id)
	544	"""
	545
	546	def _f(txn):
	547	sql = (
	548	"SELECT stream_ordering, topological_ordering, event_id"
	549	" FROM events"
	550	" WHERE room_id = ? AND stream_ordering <= ?"
	551	" AND NOT outlier"
	552	" ORDER BY stream_ordering DESC"
	553	" LIMIT 1"
	554	)
	555	txn.execute(sql, (room_id, stream_ordering))
	556	return txn.fetchone()
	557
	558	return self.db_pool.runInteraction("get_room_event_before_stream_ordering", _f)
	559
	560	async def get_room_events_max_id(self, room_id: Optional[str] = None) -> str:
	561	"""Returns the current token for rooms stream.
	562
	563	By default, it returns the current global stream token. Specifying a
	564	`room_id` causes it to return the current room specific topological
	565	token.
	566	"""
	567	token = self.get_room_max_stream_ordering()
	568	if room_id is None:
	569	return "s%d" % (token,)
	570	else:
	571	topo = await self.db_pool.runInteraction(
	572	"_get_max_topological_txn", self._get_max_topological_txn, room_id
	573	)
	574	return "t%d-%d" % (topo, token)
	575
	576	def get_stream_token_for_event(self, event_id):
	577	"""The stream token for an event
	578	Args:
	579	event_id(str): The id of the event to look up a stream token for.
	580	Raises:
	581	StoreError if the event wasn't in the database.
	582	Returns:
	583	A deferred "s%d" stream token.
	584	"""
	585	return self.db_pool.simple_select_one_onecol(
	586	table="events", keyvalues={"event_id": event_id}, retcol="stream_ordering"
	587	).addCallback(lambda row: "s%d" % (row,))
	588
	589	def get_topological_token_for_event(self, event_id):
	590	"""The stream token for an event
	591	Args:
	592	event_id(str): The id of the event to look up a stream token for.
	593	Raises:
	594	StoreError if the event wasn't in the database.
	595	Returns:
	596	A deferred "t%d-%d" topological token.
	597	"""
	598	return self.db_pool.simple_select_one(
	599	table="events",
	600	keyvalues={"event_id": event_id},
	601	retcols=("stream_ordering", "topological_ordering"),
	602	desc="get_topological_token_for_event",
	603	).addCallback(
	604	lambda row: "t%d-%d" % (row["topological_ordering"], row["stream_ordering"])
	605	)
	606
	607	def get_max_topological_token(self, room_id, stream_key):
	608	"""Get the max topological token in a room before the given stream
	609	ordering.
	610
	611	Args:
	612	room_id (str)
	613	stream_key (int)
	614
	615	Returns:
	616	Deferred[int]
	617	"""
	618	sql = (
	619	"SELECT coalesce(max(topological_ordering), 0) FROM events"
	620	" WHERE room_id = ? AND stream_ordering < ?"
	621	)
	622	return self.db_pool.execute(
	623	"get_max_topological_token", None, sql, room_id, stream_key
	624	).addCallback(lambda r: r[0][0] if r else 0)
	625
	626	def _get_max_topological_txn(self, txn, room_id):
	627	txn.execute(
	628	"SELECT MAX(topological_ordering) FROM events WHERE room_id = ?",
	629	(room_id,),
	630	)
	631
	632	rows = txn.fetchall()
	633	return rows[0][0] if rows else 0
	634
	635	@staticmethod
	636	def _set_before_and_after(events, rows, topo_order=True):
	637	"""Inserts ordering information to events' internal metadata from
	638	the DB rows.
	639
	640	Args:
	641	events (list[FrozenEvent])
	642	rows (list[_EventDictReturn])
	643	topo_order (bool): Whether the events were ordered topologically
	644	or by stream ordering. If true then all rows should have a non
	645	null topological_ordering.
	646	"""
	647	for event, row in zip(events, rows):
	648	stream = row.stream_ordering
	649	if topo_order and row.topological_ordering:
	650	topo = row.topological_ordering
	651	else:
	652	topo = None
	653	internal = event.internal_metadata
	654	internal.before = str(RoomStreamToken(topo, stream - 1))
	655	internal.after = str(RoomStreamToken(topo, stream))
	656	internal.order = (int(topo) if topo else 0, int(stream))
	657
	658	@defer.inlineCallbacks
	659	def get_events_around(
	660	self, room_id, event_id, before_limit, after_limit, event_filter=None
	661	):
	662	"""Retrieve events and pagination tokens around a given event in a
	663	room.
	664
	665	Args:
	666	room_id (str)
	667	event_id (str)
	668	before_limit (int)
	669	after_limit (int)
	670	event_filter (Filter\|None)
	671
	672	Returns:
	673	dict
	674	"""
	675
	676	results = yield self.db_pool.runInteraction(
	677	"get_events_around",
	678	self._get_events_around_txn,
	679	room_id,
	680	event_id,
	681	before_limit,
	682	after_limit,
	683	event_filter,
	684	)
	685
	686	events_before = yield self.get_events_as_list(
	687	list(results["before"]["event_ids"]), get_prev_content=True
	688	)
	689
	690	events_after = yield self.get_events_as_list(
	691	list(results["after"]["event_ids"]), get_prev_content=True
	692	)
	693
	694	return {
	695	"events_before": events_before,
	696	"events_after": events_after,
	697	"start": results["before"]["token"],
	698	"end": results["after"]["token"],
	699	}
	700
	701	def _get_events_around_txn(
	702	self, txn, room_id, event_id, before_limit, after_limit, event_filter
	703	):
	704	"""Retrieves event_ids and pagination tokens around a given event in a
	705	room.
	706
	707	Args:
	708	room_id (str)
	709	event_id (str)
	710	before_limit (int)
	711	after_limit (int)
	712	event_filter (Filter\|None)
	713
	714	Returns:
	715	dict
	716	"""
	717
	718	results = self.db_pool.simple_select_one_txn(
	719	txn,
	720	"events",
	721	keyvalues={"event_id": event_id, "room_id": room_id},
	722	retcols=["stream_ordering", "topological_ordering"],
	723	)
	724
	725	# Paginating backwards includes the event at the token, but paginating
	726	# forward doesn't.
	727	before_token = RoomStreamToken(
	728	results["topological_ordering"] - 1, results["stream_ordering"]
	729	)
	730
	731	after_token = RoomStreamToken(
	732	results["topological_ordering"], results["stream_ordering"]
	733	)
	734
	735	rows, start_token = self._paginate_room_events_txn(
	736	txn,
	737	room_id,
	738	before_token,
	739	direction="b",
	740	limit=before_limit,
	741	event_filter=event_filter,
	742	)
	743	events_before = [r.event_id for r in rows]
	744
	745	rows, end_token = self._paginate_room_events_txn(
	746	txn,
	747	room_id,
	748	after_token,
	749	direction="f",
	750	limit=after_limit,
	751	event_filter=event_filter,
	752	)
	753	events_after = [r.event_id for r in rows]
	754
	755	return {
	756	"before": {"event_ids": events_before, "token": start_token},
	757	"after": {"event_ids": events_after, "token": end_token},
	758	}
	759
	760	@defer.inlineCallbacks
	761	def get_all_new_events_stream(self, from_id, current_id, limit):
	762	"""Get all new events
	763
	764	Returns all events with from_id < stream_ordering <= current_id.
	765
	766	Args:
	767	from_id (int): the stream_ordering of the last event we processed
	768	current_id (int): the stream_ordering of the most recently processed event
	769	limit (int): the maximum number of events to return
	770
	771	Returns:
	772	Deferred[Tuple[int, list[FrozenEvent]]]: A tuple of (next_id, events), where
	773	`next_id` is the next value to pass as `from_id` (it will either be the
	774	stream_ordering of the last returned event, or, if fewer than `limit` events
	775	were found, `current_id`.
	776	"""
	777
	778	def get_all_new_events_stream_txn(txn):
	779	sql = (
	780	"SELECT e.stream_ordering, e.event_id"
	781	" FROM events AS e"
	782	" WHERE"
	783	" ? < e.stream_ordering AND e.stream_ordering <= ?"
	784	" ORDER BY e.stream_ordering ASC"
	785	" LIMIT ?"
	786	)
	787
	788	txn.execute(sql, (from_id, current_id, limit))
	789	rows = txn.fetchall()
	790
	791	upper_bound = current_id
	792	if len(rows) == limit:
	793	upper_bound = rows[-1][0]
	794
	795	return upper_bound, [row[1] for row in rows]
	796
	797	upper_bound, event_ids = yield self.db_pool.runInteraction(
	798	"get_all_new_events_stream", get_all_new_events_stream_txn
	799	)
	800
	801	events = yield self.get_events_as_list(event_ids)
	802
	803	return upper_bound, events
	804
	805	async def get_federation_out_pos(self, typ: str) -> int:
	806	if self._need_to_reset_federation_stream_positions:
	807	await self.db_pool.runInteraction(
	808	"_reset_federation_positions_txn", self._reset_federation_positions_txn
	809	)
	810	self._need_to_reset_federation_stream_positions = False
	811
	812	return await self.db_pool.simple_select_one_onecol(
	813	table="federation_stream_position",
	814	retcol="stream_id",
	815	keyvalues={"type": typ, "instance_name": self._instance_name},
	816	desc="get_federation_out_pos",
	817	)
	818
	819	async def update_federation_out_pos(self, typ, stream_id):
	820	if self._need_to_reset_federation_stream_positions:
	821	await self.db_pool.runInteraction(
	822	"_reset_federation_positions_txn", self._reset_federation_positions_txn
	823	)
	824	self._need_to_reset_federation_stream_positions = False
	825
	826	return await self.db_pool.simple_update_one(
	827	table="federation_stream_position",
	828	keyvalues={"type": typ, "instance_name": self._instance_name},
	829	updatevalues={"stream_id": stream_id},
	830	desc="update_federation_out_pos",
	831	)
	832
	833	def _reset_federation_positions_txn(self, txn):
	834	"""Fiddles with the `federation_stream_position` table to make it match
	835	the configured federation sender instances during start up.
	836	"""
	837
	838	# The federation sender instances may have changed, so we need to
	839	# massage the `federation_stream_position` table to have a row per type
	840	# per instance sending federation. If there is a mismatch we update the
	841	# table with the correct rows using the minimum stream ID seen. This
	842	# may result in resending of events/EDUs to remote servers, but that is
	843	# preferable to dropping them.
	844
	845	if not self._send_federation:
	846	return
	847
	848	# Pull out the configured instances. If we don't have a shard config then
	849	# we assume that we're the only instance sending.
	850	configured_instances = self._federation_shard_config.instances
	851	if not configured_instances:
	852	configured_instances = [self._instance_name]
	853	elif self._instance_name not in configured_instances:
	854	return
	855
	856	instances_in_table = self.db_pool.simple_select_onecol_txn(
	857	txn,
	858	table="federation_stream_position",
	859	keyvalues={},
	860	retcol="instance_name",
	861	)
	862
	863	if set(instances_in_table) == set(configured_instances):
	864	# Nothing to do
	865	return
	866
	867	sql = """
	868	SELECT type, MIN(stream_id) FROM federation_stream_position
	869	GROUP BY type
	870	"""
	871	txn.execute(sql)
	872	min_positions = dict(txn) # Map from type -> min position
	873
	874	# Ensure we do actually have some values here
	875	assert set(min_positions) == {"federation", "events"}
	876
	877	sql = """
	878	DELETE FROM federation_stream_position
	879	WHERE NOT (%s)
	880	"""
	881	clause, args = make_in_list_sql_clause(
	882	txn.database_engine, "instance_name", configured_instances
	883	)
	884	txn.execute(sql % (clause,), args)
	885
	886	for typ, stream_id in min_positions.items():
	887	self.db_pool.simple_upsert_txn(
	888	txn,
	889	table="federation_stream_position",
	890	keyvalues={"type": typ, "instance_name": self._instance_name},
	891	values={"stream_id": stream_id},
	892	)
	893
	894	def has_room_changed_since(self, room_id, stream_id):
	895	return self._events_stream_cache.has_entity_changed(room_id, stream_id)
	896
	897	def _paginate_room_events_txn(
	898	self,
	899	txn,
	900	room_id,
	901	from_token,
	902	to_token=None,
	903	direction="b",
	904	limit=-1,
	905	event_filter=None,
	906	):
	907	"""Returns list of events before or after a given token.
	908
	909	Args:
	910	txn
	911	room_id (str)
	912	from_token (RoomStreamToken): The token used to stream from
	913	to_token (RoomStreamToken\|None): A token which if given limits the
	914	results to only those before
	915	direction(char): Either 'b' or 'f' to indicate whether we are
	916	paginating forwards or backwards from `from_key`.
	917	limit (int): The maximum number of events to return.
	918	event_filter (Filter\|None): If provided filters the events to
	919	those that match the filter.
	920
	921	Returns:
	922	Deferred[tuple[list[_EventDictReturn], str]]: Returns the results
	923	as a list of _EventDictReturn and a token that points to the end
	924	of the result set. If no events are returned then the end of the
	925	stream has been reached (i.e. there are no events between
	926	`from_token` and `to_token`), or `limit` is zero.
	927	"""
	928
	929	assert int(limit) >= 0
	930
	931	# Tokens really represent positions between elements, but we use
	932	# the convention of pointing to the event before the gap. Hence
	933	# we have a bit of asymmetry when it comes to equalities.
	934	args = [False, room_id]
	935	if direction == "b":
	936	order = "DESC"
	937	else:
	938	order = "ASC"
	939
	940	bounds = generate_pagination_where_clause(
	941	direction=direction,
	942	column_names=("topological_ordering", "stream_ordering"),
	943	from_token=from_token,
	944	to_token=to_token,
	945	engine=self.database_engine,
	946	)
	947
	948	filter_clause, filter_args = filter_to_clause(event_filter)
	949
	950	if filter_clause:
	951	bounds += " AND " + filter_clause
	952	args.extend(filter_args)
	953
	954	args.append(int(limit))
	955
	956	select_keywords = "SELECT"
	957	join_clause = ""
	958	if event_filter and event_filter.labels:
	959	# If we're not filtering on a label, then joining on event_labels will
	960	# return as many row for a single event as the number of labels it has. To
	961	# avoid this, only join if we're filtering on at least one label.
	962	join_clause = """
	963	LEFT JOIN event_labels
	964	USING (event_id, room_id, topological_ordering)
	965	"""
	966	if len(event_filter.labels) > 1:
	967	# Using DISTINCT in this SELECT query is quite expensive, because it
	968	# requires the engine to sort on the entire (not limited) result set,
	969	# i.e. the entire events table. We only need to use it when we're
	970	# filtering on more than two labels, because that's the only scenario
	971	# in which we can possibly to get multiple times the same event ID in
	972	# the results.
	973	select_keywords += "DISTINCT"
	974
	975	sql = """
	976	%(select_keywords)s event_id, topological_ordering, stream_ordering
	977	FROM events
	978	%(join_clause)s
	979	WHERE outlier = ? AND room_id = ? AND %(bounds)s
	980	ORDER BY topological_ordering %(order)s,
	981	stream_ordering %(order)s LIMIT ?
	982	""" % {
	983	"select_keywords": select_keywords,
	984	"join_clause": join_clause,
	985	"bounds": bounds,
	986	"order": order,
	987	}
	988
	989	txn.execute(sql, args)
	990
	991	rows = [_EventDictReturn(row[0], row[1], row[2]) for row in txn]
	992
	993	if rows:
	994	topo = rows[-1].topological_ordering
	995	toke = rows[-1].stream_ordering
	996	if direction == "b":
	997	# Tokens are positions between events.
	998	# This token points after the last event in the chunk.
	999	# We need it to point to the event before it in the chunk
	1000	# when we are going backwards so we subtract one from the
	1001	# stream part.
	1002	toke -= 1
	1003	next_token = RoomStreamToken(topo, toke)
	1004	else:
	1005	# TODO (erikj): We should work out what to do here instead.
	1006	next_token = to_token if to_token else from_token
	1007
	1008	return rows, str(next_token)
	1009
	1010	@defer.inlineCallbacks
	1011	def paginate_room_events(
	1012	self, room_id, from_key, to_key=None, direction="b", limit=-1, event_filter=None
	1013	):
	1014	"""Returns list of events before or after a given token.
	1015
	1016	Args:
	1017	room_id (str)
	1018	from_key (str): The token used to stream from
	1019	to_key (str\|None): A token which if given limits the results to
	1020	only those before
	1021	direction(char): Either 'b' or 'f' to indicate whether we are
	1022	paginating forwards or backwards from `from_key`.
	1023	limit (int): The maximum number of events to return.
	1024	event_filter (Filter\|None): If provided filters the events to
	1025	those that match the filter.
	1026
	1027	Returns:
	1028	tuple[list[FrozenEvent], str]: Returns the results as a list of
	1029	events and a token that points to the end of the result set. If no
	1030	events are returned then the end of the stream has been reached
	1031	(i.e. there are no events between `from_key` and `to_key`).
	1032	"""
	1033
	1034	from_key = RoomStreamToken.parse(from_key)
	1035	if to_key:
	1036	to_key = RoomStreamToken.parse(to_key)
	1037
	1038	rows, token = yield self.db_pool.runInteraction(
	1039	"paginate_room_events",
	1040	self._paginate_room_events_txn,
	1041	room_id,
	1042	from_key,
	1043	to_key,
	1044	direction,
	1045	limit,
	1046	event_filter,
	1047	)
	1048
	1049	events = yield self.get_events_as_list(
	1050	[r.event_id for r in rows], get_prev_content=True
	1051	)
	1052
	1053	self._set_before_and_after(events, rows)
	1054
	1055	return (events, token)
	1056
	1057
	1058	class StreamStore(StreamWorkerStore):
	1059	def get_room_max_stream_ordering(self):
	1060	return self._stream_id_gen.get_current_token()
	1061
	1062	def get_room_min_stream_ordering(self):
	1063	return self._backfill_id_gen.get_current_token()

+291

-0

synapse/storage/databases/main/tags.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2018 New Vector Ltd
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import logging
	17	from typing import Dict, List, Tuple
	18
	19	from canonicaljson import json
	20
	21	from synapse.storage._base import db_to_json
	22	from synapse.storage.databases.main.account_data import AccountDataWorkerStore
	23	from synapse.types import JsonDict
	24	from synapse.util.caches.descriptors import cached
	25
	26	logger = logging.getLogger(__name__)
	27
	28
	29	class TagsWorkerStore(AccountDataWorkerStore):
	30	@cached()
	31	async def get_tags_for_user(self, user_id: str) -> Dict[str, Dict[str, JsonDict]]:
	32	"""Get all the tags for a user.
	33
	34
	35	Args:
	36	user_id: The user to get the tags for.
	37	Returns:
	38	A mapping from room_id strings to dicts mapping from tag strings to
	39	tag content.
	40	"""
	41
	42	rows = await self.db_pool.simple_select_list(
	43	"room_tags", {"user_id": user_id}, ["room_id", "tag", "content"]
	44	)
	45
	46	tags_by_room = {}
	47	for row in rows:
	48	room_tags = tags_by_room.setdefault(row["room_id"], {})
	49	room_tags[row["tag"]] = db_to_json(row["content"])
	50	return tags_by_room
	51
	52	async def get_all_updated_tags(
	53	self, instance_name: str, last_id: int, current_id: int, limit: int
	54	) -> Tuple[List[Tuple[int, tuple]], int, bool]:
	55	"""Get updates for tags replication stream.
	56
	57	Args:
	58	instance_name: The writer we want to fetch updates from. Unused
	59	here since there is only ever one writer.
	60	last_id: The token to fetch updates from. Exclusive.
	61	current_id: The token to fetch updates up to. Inclusive.
	62	limit: The requested limit for the number of rows to return. The
	63	function may return more or fewer rows.
	64
	65	Returns:
	66	A tuple consisting of: the updates, a token to use to fetch
	67	subsequent updates, and whether we returned fewer rows than exists
	68	between the requested tokens due to the limit.
	69
	70	The token returned can be used in a subsequent call to this
	71	function to get further updatees.
	72
	73	The updates are a list of 2-tuples of stream ID and the row data
	74	"""
	75
	76	if last_id == current_id:
	77	return [], current_id, False
	78
	79	def get_all_updated_tags_txn(txn):
	80	sql = (
	81	"SELECT stream_id, user_id, room_id"
	82	" FROM room_tags_revisions as r"
	83	" WHERE ? < stream_id AND stream_id <= ?"
	84	" ORDER BY stream_id ASC LIMIT ?"
	85	)
	86	txn.execute(sql, (last_id, current_id, limit))
	87	return txn.fetchall()
	88
	89	tag_ids = await self.db_pool.runInteraction(
	90	"get_all_updated_tags", get_all_updated_tags_txn
	91	)
	92
	93	def get_tag_content(txn, tag_ids):
	94	sql = "SELECT tag, content FROM room_tags WHERE user_id=? AND room_id=?"
	95	results = []
	96	for stream_id, user_id, room_id in tag_ids:
	97	txn.execute(sql, (user_id, room_id))
	98	tags = []
	99	for tag, content in txn:
	100	tags.append(json.dumps(tag) + ":" + content)
	101	tag_json = "{" + ",".join(tags) + "}"
	102	results.append((stream_id, (user_id, room_id, tag_json)))
	103
	104	return results
	105
	106	batch_size = 50
	107	results = []
	108	for i in range(0, len(tag_ids), batch_size):
	109	tags = await self.db_pool.runInteraction(
	110	"get_all_updated_tag_content",
	111	get_tag_content,
	112	tag_ids[i : i + batch_size],
	113	)
	114	results.extend(tags)
	115
	116	limited = False
	117	upto_token = current_id
	118	if len(results) >= limit:
	119	upto_token = results[-1][0]
	120	limited = True
	121
	122	return results, upto_token, limited
	123
	124	async def get_updated_tags(
	125	self, user_id: str, stream_id: int
	126	) -> Dict[str, List[str]]:
	127	"""Get all the tags for the rooms where the tags have changed since the
	128	given version
	129
	130	Args:
	131	user_id(str): The user to get the tags for.
	132	stream_id(int): The earliest update to get for the user.
	133
	134	Returns:
	135	A mapping from room_id strings to lists of tag strings for all the
	136	rooms that changed since the stream_id token.
	137	"""
	138
	139	def get_updated_tags_txn(txn):
	140	sql = (
	141	"SELECT room_id from room_tags_revisions"
	142	" WHERE user_id = ? AND stream_id > ?"
	143	)
	144	txn.execute(sql, (user_id, stream_id))
	145	room_ids = [row[0] for row in txn]
	146	return room_ids
	147
	148	changed = self._account_data_stream_cache.has_entity_changed(
	149	user_id, int(stream_id)
	150	)
	151	if not changed:
	152	return {}
	153
	154	room_ids = await self.db_pool.runInteraction(
	155	"get_updated_tags", get_updated_tags_txn
	156	)
	157
	158	results = {}
	159	if room_ids:
	160	tags_by_room = await self.get_tags_for_user(user_id)
	161	for room_id in room_ids:
	162	results[room_id] = tags_by_room.get(room_id, {})
	163
	164	return results
	165
	166	async def get_tags_for_room(
	167	self, user_id: str, room_id: str
	168	) -> Dict[str, JsonDict]:
	169	"""Get all the tags for the given room
	170
	171	Args:
	172	user_id: The user to get tags for
	173	room_id: The room to get tags for
	174
	175	Returns:
	176	A mapping of tags to tag content.
	177	"""
	178	rows = await self.db_pool.simple_select_list(
	179	table="room_tags",
	180	keyvalues={"user_id": user_id, "room_id": room_id},
	181	retcols=("tag", "content"),
	182	desc="get_tags_for_room",
	183	)
	184	return {row["tag"]: db_to_json(row["content"]) for row in rows}
	185
	186
	187	class TagsStore(TagsWorkerStore):
	188	async def add_tag_to_room(
	189	self, user_id: str, room_id: str, tag: str, content: JsonDict
	190	) -> int:
	191	"""Add a tag to a room for a user.
	192
	193	Args:
	194	user_id: The user to add a tag for.
	195	room_id: The room to add a tag for.
	196	tag: The tag name to add.
	197	content: A json object to associate with the tag.
	198
	199	Returns:
	200	The next account data ID.
	201	"""
	202	content_json = json.dumps(content)
	203
	204	def add_tag_txn(txn, next_id):
	205	self.db_pool.simple_upsert_txn(
	206	txn,
	207	table="room_tags",
	208	keyvalues={"user_id": user_id, "room_id": room_id, "tag": tag},
	209	values={"content": content_json},
	210	)
	211	self._update_revision_txn(txn, user_id, room_id, next_id)
	212
	213	with self._account_data_id_gen.get_next() as next_id:
	214	await self.db_pool.runInteraction("add_tag", add_tag_txn, next_id)
	215
	216	self.get_tags_for_user.invalidate((user_id,))
	217
	218	return self._account_data_id_gen.get_current_token()
	219
	220	async def remove_tag_from_room(self, user_id: str, room_id: str, tag: str) -> int:
	221	"""Remove a tag from a room for a user.
	222
	223	Returns:
	224	The next account data ID.
	225	"""
	226
	227	def remove_tag_txn(txn, next_id):
	228	sql = (
	229	"DELETE FROM room_tags "
	230	" WHERE user_id = ? AND room_id = ? AND tag = ?"
	231	)
	232	txn.execute(sql, (user_id, room_id, tag))
	233	self._update_revision_txn(txn, user_id, room_id, next_id)
	234
	235	with self._account_data_id_gen.get_next() as next_id:
	236	await self.db_pool.runInteraction("remove_tag", remove_tag_txn, next_id)
	237
	238	self.get_tags_for_user.invalidate((user_id,))
	239
	240	return self._account_data_id_gen.get_current_token()
	241
	242	def _update_revision_txn(
	243	self, txn, user_id: str, room_id: str, next_id: int
	244	) -> None:
	245	"""Update the latest revision of the tags for the given user and room.
	246
	247	Args:
	248	txn: The database cursor
	249	user_id: The ID of the user.
	250	room_id: The ID of the room.
	251	next_id: The the revision to advance to.
	252	"""
	253
	254	txn.call_after(
	255	self._account_data_stream_cache.entity_has_changed, user_id, next_id
	256	)
	257
	258	# Note: This is only here for backwards compat to allow admins to
	259	# roll back to a previous Synapse version. Next time we update the
	260	# database version we can remove this table.
	261	update_max_id_sql = (
	262	"UPDATE account_data_max_stream_id"
	263	" SET stream_id = ?"
	264	" WHERE stream_id < ?"
	265	)
	266	txn.execute(update_max_id_sql, (next_id, next_id))
	267
	268	update_sql = (
	269	"UPDATE room_tags_revisions"
	270	" SET stream_id = ?"
	271	" WHERE user_id = ?"
	272	" AND room_id = ?"
	273	)
	274	txn.execute(update_sql, (next_id, user_id, room_id))
	275
	276	if txn.rowcount == 0:
	277	insert_sql = (
	278	"INSERT INTO room_tags_revisions (user_id, room_id, stream_id)"
	279	" VALUES (?, ?, ?)"
	280	)
	281	try:
	282	txn.execute(insert_sql, (user_id, room_id, next_id))
	283	except self.database_engine.module.IntegrityError:
	284	# Ignore insertion errors. It doesn't matter if the row wasn't
	285	# inserted because if two updates happend concurrently the one
	286	# with the higher stream_id will not be reported to a client
	287	# unless the previous update has completed. It doesn't matter
	288	# which stream_id ends up in the table, as long as it is higher
	289	# than the id that the client has.
	290	pass

+266

-0

synapse/storage/databases/main/transactions.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16	from collections import namedtuple
	17
	18	from canonicaljson import encode_canonical_json
	19
	20	from synapse.metrics.background_process_metrics import run_as_background_process
	21	from synapse.storage._base import SQLBaseStore, db_to_json
	22	from synapse.storage.database import DatabasePool
	23	from synapse.util.caches.expiringcache import ExpiringCache
	24
	25	db_binary_type = memoryview
	26
	27	logger = logging.getLogger(__name__)
	28
	29
	30	_TransactionRow = namedtuple(
	31	"_TransactionRow",
	32	("id", "transaction_id", "destination", "ts", "response_code", "response_json"),
	33	)
	34
	35	_UpdateTransactionRow = namedtuple(
	36	"_TransactionRow", ("response_code", "response_json")
	37	)
	38
	39	SENTINEL = object()
	40
	41
	42	class TransactionStore(SQLBaseStore):
	43	"""A collection of queries for handling PDUs.
	44	"""
	45
	46	def __init__(self, database: DatabasePool, db_conn, hs):
	47	super(TransactionStore, self).__init__(database, db_conn, hs)
	48
	49	self._clock.looping_call(self._start_cleanup_transactions, 30 * 60 * 1000)
	50
	51	self._destination_retry_cache = ExpiringCache(
	52	cache_name="get_destination_retry_timings",
	53	clock=self._clock,
	54	expiry_ms=5 * 60 * 1000,
	55	)
	56
	57	def get_received_txn_response(self, transaction_id, origin):
	58	"""For an incoming transaction from a given origin, check if we have
	59	already responded to it. If so, return the response code and response
	60	body (as a dict).
	61
	62	Args:
	63	transaction_id (str)
	64	origin(str)
	65
	66	Returns:
	67	tuple: None if we have not previously responded to
	68	this transaction or a 2-tuple of (int, dict)
	69	"""
	70
	71	return self.db_pool.runInteraction(
	72	"get_received_txn_response",
	73	self._get_received_txn_response,
	74	transaction_id,
	75	origin,
	76	)
	77
	78	def _get_received_txn_response(self, txn, transaction_id, origin):
	79	result = self.db_pool.simple_select_one_txn(
	80	txn,
	81	table="received_transactions",
	82	keyvalues={"transaction_id": transaction_id, "origin": origin},
	83	retcols=(
	84	"transaction_id",
	85	"origin",
	86	"ts",
	87	"response_code",
	88	"response_json",
	89	"has_been_referenced",
	90	),
	91	allow_none=True,
	92	)
	93
	94	if result and result["response_code"]:
	95	return result["response_code"], db_to_json(result["response_json"])
	96
	97	else:
	98	return None
	99
	100	def set_received_txn_response(self, transaction_id, origin, code, response_dict):
	101	"""Persist the response we returened for an incoming transaction, and
	102	should return for subsequent transactions with the same transaction_id
	103	and origin.
	104
	105	Args:
	106	txn
	107	transaction_id (str)
	108	origin (str)
	109	code (int)
	110	response_json (str)
	111	"""
	112
	113	return self.db_pool.simple_insert(
	114	table="received_transactions",
	115	values={
	116	"transaction_id": transaction_id,
	117	"origin": origin,
	118	"response_code": code,
	119	"response_json": db_binary_type(encode_canonical_json(response_dict)),
	120	"ts": self._clock.time_msec(),
	121	},
	122	or_ignore=True,
	123	desc="set_received_txn_response",
	124	)
	125
	126	async def get_destination_retry_timings(self, destination):
	127	"""Gets the current retry timings (if any) for a given destination.
	128
	129	Args:
	130	destination (str)
	131
	132	Returns:
	133	None if not retrying
	134	Otherwise a dict for the retry scheme
	135	"""
	136
	137	result = self._destination_retry_cache.get(destination, SENTINEL)
	138	if result is not SENTINEL:
	139	return result
	140
	141	result = await self.db_pool.runInteraction(
	142	"get_destination_retry_timings",
	143	self._get_destination_retry_timings,
	144	destination,
	145	)
	146
	147	# We don't hugely care about race conditions between getting and
	148	# invalidating the cache, since we time out fairly quickly anyway.
	149	self._destination_retry_cache[destination] = result
	150	return result
	151
	152	def _get_destination_retry_timings(self, txn, destination):
	153	result = self.db_pool.simple_select_one_txn(
	154	txn,
	155	table="destinations",
	156	keyvalues={"destination": destination},
	157	retcols=("destination", "failure_ts", "retry_last_ts", "retry_interval"),
	158	allow_none=True,
	159	)
	160
	161	if result and result["retry_last_ts"] > 0:
	162	return result
	163	else:
	164	return None
	165
	166	def set_destination_retry_timings(
	167	self, destination, failure_ts, retry_last_ts, retry_interval
	168	):
	169	"""Sets the current retry timings for a given destination.
	170	Both timings should be zero if retrying is no longer occuring.
	171
	172	Args:
	173	destination (str)
	174	failure_ts (int\|None) - when the server started failing (ms since epoch)
	175	retry_last_ts (int) - time of last retry attempt in unix epoch ms
	176	retry_interval (int) - how long until next retry in ms
	177	"""
	178
	179	self._destination_retry_cache.pop(destination, None)
	180	return self.db_pool.runInteraction(
	181	"set_destination_retry_timings",
	182	self._set_destination_retry_timings,
	183	destination,
	184	failure_ts,
	185	retry_last_ts,
	186	retry_interval,
	187	)
	188
	189	def _set_destination_retry_timings(
	190	self, txn, destination, failure_ts, retry_last_ts, retry_interval
	191	):
	192
	193	if self.database_engine.can_native_upsert:
	194	# Upsert retry time interval if retry_interval is zero (i.e. we're
	195	# resetting it) or greater than the existing retry interval.
	196
	197	sql = """
	198	INSERT INTO destinations (
	199	destination, failure_ts, retry_last_ts, retry_interval
	200	)
	201	VALUES (?, ?, ?, ?)
	202	ON CONFLICT (destination) DO UPDATE SET
	203	failure_ts = EXCLUDED.failure_ts,
	204	retry_last_ts = EXCLUDED.retry_last_ts,
	205	retry_interval = EXCLUDED.retry_interval
	206	WHERE
	207	EXCLUDED.retry_interval = 0
	208	OR destinations.retry_interval < EXCLUDED.retry_interval
	209	"""
	210
	211	txn.execute(sql, (destination, failure_ts, retry_last_ts, retry_interval))
	212
	213	return
	214
	215	self.database_engine.lock_table(txn, "destinations")
	216
	217	# We need to be careful here as the data may have changed from under us
	218	# due to a worker setting the timings.
	219
	220	prev_row = self.db_pool.simple_select_one_txn(
	221	txn,
	222	table="destinations",
	223	keyvalues={"destination": destination},
	224	retcols=("failure_ts", "retry_last_ts", "retry_interval"),
	225	allow_none=True,
	226	)
	227
	228	if not prev_row:
	229	self.db_pool.simple_insert_txn(
	230	txn,
	231	table="destinations",
	232	values={
	233	"destination": destination,
	234	"failure_ts": failure_ts,
	235	"retry_last_ts": retry_last_ts,
	236	"retry_interval": retry_interval,
	237	},
	238	)
	239	elif retry_interval == 0 or prev_row["retry_interval"] < retry_interval:
	240	self.db_pool.simple_update_one_txn(
	241	txn,
	242	"destinations",
	243	keyvalues={"destination": destination},
	244	updatevalues={
	245	"failure_ts": failure_ts,
	246	"retry_last_ts": retry_last_ts,
	247	"retry_interval": retry_interval,
	248	},
	249	)
	250
	251	def _start_cleanup_transactions(self):
	252	return run_as_background_process(
	253	"cleanup_transactions", self._cleanup_transactions
	254	)
	255
	256	def _cleanup_transactions(self):
	257	now = self._clock.time_msec()
	258	month_ago = now - 30 * 24 * 60 * 60 * 1000
	259
	260	def _cleanup_transactions_txn(txn):
	261	txn.execute("DELETE FROM received_transactions WHERE ts < ?", (month_ago,))
	262
	263	return self.db_pool.runInteraction(
	264	"_cleanup_transactions", _cleanup_transactions_txn
	265	)

+300

-0

synapse/storage/databases/main/ui_auth.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	from typing import Any, Dict, Optional, Union
	15
	16	import attr
	17	from canonicaljson import json
	18
	19	from synapse.api.errors import StoreError
	20	from synapse.storage._base import SQLBaseStore, db_to_json
	21	from synapse.types import JsonDict
	22	from synapse.util import stringutils as stringutils
	23
	24
	25	@attr.s
	26	class UIAuthSessionData:
	27	session_id = attr.ib(type=str)
	28	# The dictionary from the client root level, not the 'auth' key.
	29	clientdict = attr.ib(type=JsonDict)
	30	# The URI and method the session was intiatied with. These are checked at
	31	# each stage of the authentication to ensure that the asked for operation
	32	# has not changed.
	33	uri = attr.ib(type=str)
	34	method = attr.ib(type=str)
	35	# A string description of the operation that the current authentication is
	36	# authorising.
	37	description = attr.ib(type=str)
	38
	39
	40	class UIAuthWorkerStore(SQLBaseStore):
	41	"""
	42	Manage user interactive authentication sessions.
	43	"""
	44
	45	async def create_ui_auth_session(
	46	self, clientdict: JsonDict, uri: str, method: str, description: str,
	47	) -> UIAuthSessionData:
	48	"""
	49	Creates a new user interactive authentication session.
	50
	51	The session can be used to track the stages necessary to authenticate a
	52	user across multiple HTTP requests.
	53
	54	Args:
	55	clientdict:
	56	The dictionary from the client root level, not the 'auth' key.
	57	uri:
	58	The URI this session was initiated with, this is checked at each
	59	stage of the authentication to ensure that the asked for
	60	operation has not changed.
	61	method:
	62	The method this session was initiated with, this is checked at each
	63	stage of the authentication to ensure that the asked for
	64	operation has not changed.
	65	description:
	66	A string description of the operation that the current
	67	authentication is authorising.
	68	Returns:
	69	The newly created session.
	70	Raises:
	71	StoreError if a unique session ID cannot be generated.
	72	"""
	73	# The clientdict gets stored as JSON.
	74	clientdict_json = json.dumps(clientdict)
	75
	76	# autogen a session ID and try to create it. We may clash, so just
	77	# try a few times till one goes through, giving up eventually.
	78	attempts = 0
	79	while attempts < 5:
	80	session_id = stringutils.random_string(24)
	81
	82	try:
	83	await self.db_pool.simple_insert(
	84	table="ui_auth_sessions",
	85	values={
	86	"session_id": session_id,
	87	"clientdict": clientdict_json,
	88	"uri": uri,
	89	"method": method,
	90	"description": description,
	91	"serverdict": "{}",
	92	"creation_time": self.hs.get_clock().time_msec(),
	93	},
	94	desc="create_ui_auth_session",
	95	)
	96	return UIAuthSessionData(
	97	session_id, clientdict, uri, method, description
	98	)
	99	except self.db_pool.engine.module.IntegrityError:
	100	attempts += 1
	101	raise StoreError(500, "Couldn't generate a session ID.")
	102
	103	async def get_ui_auth_session(self, session_id: str) -> UIAuthSessionData:
	104	"""Retrieve a UI auth session.
	105
	106	Args:
	107	session_id: The ID of the session.
	108	Returns:
	109	A dict containing the device information.
	110	Raises:
	111	StoreError if the session is not found.
	112	"""
	113	result = await self.db_pool.simple_select_one(
	114	table="ui_auth_sessions",
	115	keyvalues={"session_id": session_id},
	116	retcols=("clientdict", "uri", "method", "description"),
	117	desc="get_ui_auth_session",
	118	)
	119
	120	result["clientdict"] = db_to_json(result["clientdict"])
	121
	122	return UIAuthSessionData(session_id, **result)
	123
	124	async def mark_ui_auth_stage_complete(
	125	self, session_id: str, stage_type: str, result: Union[str, bool, JsonDict],
	126	):
	127	"""
	128	Mark a session stage as completed.
	129
	130	Args:
	131	session_id: The ID of the corresponding session.
	132	stage_type: The completed stage type.
	133	result: The result of the stage verification.
	134	Raises:
	135	StoreError if the session cannot be found.
	136	"""
	137	# Add (or update) the results of the current stage to the database.
	138	#
	139	# Note that we need to allow for the same stage to complete multiple
	140	# times here so that registration is idempotent.
	141	try:
	142	await self.db_pool.simple_upsert(
	143	table="ui_auth_sessions_credentials",
	144	keyvalues={"session_id": session_id, "stage_type": stage_type},
	145	values={"result": json.dumps(result)},
	146	desc="mark_ui_auth_stage_complete",
	147	)
	148	except self.db_pool.engine.module.IntegrityError:
	149	raise StoreError(400, "Unknown session ID: %s" % (session_id,))
	150
	151	async def get_completed_ui_auth_stages(
	152	self, session_id: str
	153	) -> Dict[str, Union[str, bool, JsonDict]]:
	154	"""
	155	Retrieve the completed stages of a UI authentication session.
	156
	157	Args:
	158	session_id: The ID of the session.
	159	Returns:
	160	The completed stages mapped to the result of the verification of
	161	that auth-type.
	162	"""
	163	results = {}
	164	for row in await self.db_pool.simple_select_list(
	165	table="ui_auth_sessions_credentials",
	166	keyvalues={"session_id": session_id},
	167	retcols=("stage_type", "result"),
	168	desc="get_completed_ui_auth_stages",
	169	):
	170	results[row["stage_type"]] = db_to_json(row["result"])
	171
	172	return results
	173
	174	async def set_ui_auth_clientdict(
	175	self, session_id: str, clientdict: JsonDict
	176	) -> None:
	177	"""
	178	Store an updated clientdict for a given session ID.
	179
	180	Args:
	181	session_id: The ID of this session as returned from check_auth
	182	clientdict:
	183	The dictionary from the client root level, not the 'auth' key.
	184	"""
	185	# The clientdict gets stored as JSON.
	186	clientdict_json = json.dumps(clientdict)
	187
	188	await self.db_pool.simple_update_one(
	189	table="ui_auth_sessions",
	190	keyvalues={"session_id": session_id},
	191	updatevalues={"clientdict": clientdict_json},
	192	desc="set_ui_auth_client_dict",
	193	)
	194
	195	async def set_ui_auth_session_data(self, session_id: str, key: str, value: Any):
	196	"""
	197	Store a key-value pair into the sessions data associated with this
	198	request. This data is stored server-side and cannot be modified by
	199	the client.
	200
	201	Args:
	202	session_id: The ID of this session as returned from check_auth
	203	key: The key to store the data under
	204	value: The data to store
	205	Raises:
	206	StoreError if the session cannot be found.
	207	"""
	208	await self.db_pool.runInteraction(
	209	"set_ui_auth_session_data",
	210	self._set_ui_auth_session_data_txn,
	211	session_id,
	212	key,
	213	value,
	214	)
	215
	216	def _set_ui_auth_session_data_txn(self, txn, session_id: str, key: str, value: Any):
	217	# Get the current value.
	218	result = self.db_pool.simple_select_one_txn(
	219	txn,
	220	table="ui_auth_sessions",
	221	keyvalues={"session_id": session_id},
	222	retcols=("serverdict",),
	223	)
	224
	225	# Update it and add it back to the database.
	226	serverdict = db_to_json(result["serverdict"])
	227	serverdict[key] = value
	228
	229	self.db_pool.simple_update_one_txn(
	230	txn,
	231	table="ui_auth_sessions",
	232	keyvalues={"session_id": session_id},
	233	updatevalues={"serverdict": json.dumps(serverdict)},
	234	)
	235
	236	async def get_ui_auth_session_data(
	237	self, session_id: str, key: str, default: Optional[Any] = None
	238	) -> Any:
	239	"""
	240	Retrieve data stored with set_session_data
	241
	242	Args:
	243	session_id: The ID of this session as returned from check_auth
	244	key: The key to store the data under
	245	default: Value to return if the key has not been set
	246	Raises:
	247	StoreError if the session cannot be found.
	248	"""
	249	result = await self.db_pool.simple_select_one(
	250	table="ui_auth_sessions",
	251	keyvalues={"session_id": session_id},
	252	retcols=("serverdict",),
	253	desc="get_ui_auth_session_data",
	254	)
	255
	256	serverdict = db_to_json(result["serverdict"])
	257
	258	return serverdict.get(key, default)
	259
	260
	261	class UIAuthStore(UIAuthWorkerStore):
	262	def delete_old_ui_auth_sessions(self, expiration_time: int):
	263	"""
	264	Remove sessions which were last used earlier than the expiration time.
	265
	266	Args:
	267	expiration_time: The latest time that is still considered valid.
	268	This is an epoch time in milliseconds.
	269
	270	"""
	271	return self.db_pool.runInteraction(
	272	"delete_old_ui_auth_sessions",
	273	self._delete_old_ui_auth_sessions_txn,
	274	expiration_time,
	275	)
	276
	277	def _delete_old_ui_auth_sessions_txn(self, txn, expiration_time: int):
	278	# Get the expired sessions.
	279	sql = "SELECT session_id FROM ui_auth_sessions WHERE creation_time <= ?"
	280	txn.execute(sql, [expiration_time])
	281	session_ids = [r[0] for r in txn.fetchall()]
	282
	283	# Delete the corresponding completed credentials.
	284	self.db_pool.simple_delete_many_txn(
	285	txn,
	286	table="ui_auth_sessions_credentials",
	287	column="session_id",
	288	iterable=session_ids,
	289	keyvalues={},
	290	)
	291
	292	# Finally, delete the sessions.
	293	self.db_pool.simple_delete_many_txn(
	294	txn,
	295	table="ui_auth_sessions",
	296	column="session_id",
	297	iterable=session_ids,
	298	keyvalues={},
	299	)

+809

-0

synapse/storage/databases/main/user_directory.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2017 Vector Creations Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16	import re
	17
	18	from synapse.api.constants import EventTypes, JoinRules
	19	from synapse.storage.database import DatabasePool
	20	from synapse.storage.databases.main.state import StateFilter
	21	from synapse.storage.databases.main.state_deltas import StateDeltasStore
	22	from synapse.storage.engines import PostgresEngine, Sqlite3Engine
	23	from synapse.types import get_domain_from_id, get_localpart_from_id
	24	from synapse.util.caches.descriptors import cached
	25
	26	logger = logging.getLogger(__name__)
	27
	28
	29	TEMP_TABLE = "_temp_populate_user_directory"
	30
	31
	32	class UserDirectoryBackgroundUpdateStore(StateDeltasStore):
	33
	34	# How many records do we calculate before sending it to
	35	# add_users_who_share_private_rooms?
	36	SHARE_PRIVATE_WORKING_SET = 500
	37
	38	def __init__(self, database: DatabasePool, db_conn, hs):
	39	super(UserDirectoryBackgroundUpdateStore, self).__init__(database, db_conn, hs)
	40
	41	self.server_name = hs.hostname
	42
	43	self.db_pool.updates.register_background_update_handler(
	44	"populate_user_directory_createtables",
	45	self._populate_user_directory_createtables,
	46	)
	47	self.db_pool.updates.register_background_update_handler(
	48	"populate_user_directory_process_rooms",
	49	self._populate_user_directory_process_rooms,
	50	)
	51	self.db_pool.updates.register_background_update_handler(
	52	"populate_user_directory_process_users",
	53	self._populate_user_directory_process_users,
	54	)
	55	self.db_pool.updates.register_background_update_handler(
	56	"populate_user_directory_cleanup", self._populate_user_directory_cleanup
	57	)
	58
	59	async def _populate_user_directory_createtables(self, progress, batch_size):
	60
	61	# Get all the rooms that we want to process.
	62	def _make_staging_area(txn):
	63	sql = (
	64	"CREATE TABLE IF NOT EXISTS "
	65	+ TEMP_TABLE
	66	+ "_rooms(room_id TEXT NOT NULL, events BIGINT NOT NULL)"
	67	)
	68	txn.execute(sql)
	69
	70	sql = (
	71	"CREATE TABLE IF NOT EXISTS "
	72	+ TEMP_TABLE
	73	+ "_position(position TEXT NOT NULL)"
	74	)
	75	txn.execute(sql)
	76
	77	# Get rooms we want to process from the database
	78	sql = """
	79	SELECT room_id, count(*) FROM current_state_events
	80	GROUP BY room_id
	81	"""
	82	txn.execute(sql)
	83	rooms = [{"room_id": x[0], "events": x[1]} for x in txn.fetchall()]
	84	self.db_pool.simple_insert_many_txn(txn, TEMP_TABLE + "_rooms", rooms)
	85	del rooms
	86
	87	# If search all users is on, get all the users we want to add.
	88	if self.hs.config.user_directory_search_all_users:
	89	sql = (
	90	"CREATE TABLE IF NOT EXISTS "
	91	+ TEMP_TABLE
	92	+ "_users(user_id TEXT NOT NULL)"
	93	)
	94	txn.execute(sql)
	95
	96	txn.execute("SELECT name FROM users")
	97	users = [{"user_id": x[0]} for x in txn.fetchall()]
	98
	99	self.db_pool.simple_insert_many_txn(txn, TEMP_TABLE + "_users", users)
	100
	101	new_pos = await self.get_max_stream_id_in_current_state_deltas()
	102	await self.db_pool.runInteraction(
	103	"populate_user_directory_temp_build", _make_staging_area
	104	)
	105	await self.db_pool.simple_insert(
	106	TEMP_TABLE + "_position", {"position": new_pos}
	107	)
	108
	109	await self.db_pool.updates._end_background_update(
	110	"populate_user_directory_createtables"
	111	)
	112	return 1
	113
	114	async def _populate_user_directory_cleanup(self, progress, batch_size):
	115	"""
	116	Update the user directory stream position, then clean up the old tables.
	117	"""
	118	position = await self.db_pool.simple_select_one_onecol(
	119	TEMP_TABLE + "_position", None, "position"
	120	)
	121	await self.update_user_directory_stream_pos(position)
	122
	123	def _delete_staging_area(txn):
	124	txn.execute("DROP TABLE IF EXISTS " + TEMP_TABLE + "_rooms")
	125	txn.execute("DROP TABLE IF EXISTS " + TEMP_TABLE + "_users")
	126	txn.execute("DROP TABLE IF EXISTS " + TEMP_TABLE + "_position")
	127
	128	await self.db_pool.runInteraction(
	129	"populate_user_directory_cleanup", _delete_staging_area
	130	)
	131
	132	await self.db_pool.updates._end_background_update(
	133	"populate_user_directory_cleanup"
	134	)
	135	return 1
	136
	137	async def _populate_user_directory_process_rooms(self, progress, batch_size):
	138	"""
	139	Args:
	140	progress (dict)
	141	batch_size (int): Maximum number of state events to process
	142	per cycle.
	143	"""
	144	state = self.hs.get_state_handler()
	145
	146	# If we don't have progress filed, delete everything.
	147	if not progress:
	148	await self.delete_all_from_user_dir()
	149
	150	def _get_next_batch(txn):
	151	# Only fetch 250 rooms, so we don't fetch too many at once, even
	152	# if those 250 rooms have less than batch_size state events.
	153	sql = """
	154	SELECT room_id, events FROM %s
	155	ORDER BY events DESC
	156	LIMIT 250
	157	""" % (
	158	TEMP_TABLE + "_rooms",
	159	)
	160	txn.execute(sql)
	161	rooms_to_work_on = txn.fetchall()
	162
	163	if not rooms_to_work_on:
	164	return None
	165
	166	# Get how many are left to process, so we can give status on how
	167	# far we are in processing
	168	txn.execute("SELECT COUNT(*) FROM " + TEMP_TABLE + "_rooms")
	169	progress["remaining"] = txn.fetchone()[0]
	170
	171	return rooms_to_work_on
	172
	173	rooms_to_work_on = await self.db_pool.runInteraction(
	174	"populate_user_directory_temp_read", _get_next_batch
	175	)
	176
	177	# No more rooms -- complete the transaction.
	178	if not rooms_to_work_on:
	179	await self.db_pool.updates._end_background_update(
	180	"populate_user_directory_process_rooms"
	181	)
	182	return 1
	183
	184	logger.debug(
	185	"Processing the next %d rooms of %d remaining"
	186	% (len(rooms_to_work_on), progress["remaining"])
	187	)
	188
	189	processed_event_count = 0
	190
	191	for room_id, event_count in rooms_to_work_on:
	192	is_in_room = await self.is_host_joined(room_id, self.server_name)
	193
	194	if is_in_room:
	195	is_public = await self.is_room_world_readable_or_publicly_joinable(
	196	room_id
	197	)
	198
	199	users_with_profile = await state.get_current_users_in_room(room_id)
	200	user_ids = set(users_with_profile)
	201
	202	# Update each user in the user directory.
	203	for user_id, profile in users_with_profile.items():
	204	await self.update_profile_in_user_dir(
	205	user_id, profile.display_name, profile.avatar_url
	206	)
	207
	208	to_insert = set()
	209
	210	if is_public:
	211	for user_id in user_ids:
	212	if self.get_if_app_services_interested_in_user(user_id):
	213	continue
	214
	215	to_insert.add(user_id)
	216
	217	if to_insert:
	218	await self.add_users_in_public_rooms(room_id, to_insert)
	219	to_insert.clear()
	220	else:
	221	for user_id in user_ids:
	222	if not self.hs.is_mine_id(user_id):
	223	continue
	224
	225	if self.get_if_app_services_interested_in_user(user_id):
	226	continue
	227
	228	for other_user_id in user_ids:
	229	if user_id == other_user_id:
	230	continue
	231
	232	user_set = (user_id, other_user_id)
	233	to_insert.add(user_set)
	234
	235	# If it gets too big, stop and write to the database
	236	# to prevent storing too much in RAM.
	237	if len(to_insert) >= self.SHARE_PRIVATE_WORKING_SET:
	238	await self.add_users_who_share_private_room(
	239	room_id, to_insert
	240	)
	241	to_insert.clear()
	242
	243	if to_insert:
	244	await self.add_users_who_share_private_room(room_id, to_insert)
	245	to_insert.clear()
	246
	247	# We've finished a room. Delete it from the table.
	248	await self.db_pool.simple_delete_one(
	249	TEMP_TABLE + "_rooms", {"room_id": room_id}
	250	)
	251	# Update the remaining counter.
	252	progress["remaining"] -= 1
	253	await self.db_pool.runInteraction(
	254	"populate_user_directory",
	255	self.db_pool.updates._background_update_progress_txn,
	256	"populate_user_directory_process_rooms",
	257	progress,
	258	)
	259
	260	processed_event_count += event_count
	261
	262	if processed_event_count > batch_size:
	263	# Don't process any more rooms, we've hit our batch size.
	264	return processed_event_count
	265
	266	return processed_event_count
	267
	268	async def _populate_user_directory_process_users(self, progress, batch_size):
	269	"""
	270	If search_all_users is enabled, add all of the users to the user directory.
	271	"""
	272	if not self.hs.config.user_directory_search_all_users:
	273	await self.db_pool.updates._end_background_update(
	274	"populate_user_directory_process_users"
	275	)
	276	return 1
	277
	278	def _get_next_batch(txn):
	279	sql = "SELECT user_id FROM %s LIMIT %s" % (
	280	TEMP_TABLE + "_users",
	281	str(batch_size),
	282	)
	283	txn.execute(sql)
	284	users_to_work_on = txn.fetchall()
	285
	286	if not users_to_work_on:
	287	return None
	288
	289	users_to_work_on = [x[0] for x in users_to_work_on]
	290
	291	# Get how many are left to process, so we can give status on how
	292	# far we are in processing
	293	sql = "SELECT COUNT(*) FROM " + TEMP_TABLE + "_users"
	294	txn.execute(sql)
	295	progress["remaining"] = txn.fetchone()[0]
	296
	297	return users_to_work_on
	298
	299	users_to_work_on = await self.db_pool.runInteraction(
	300	"populate_user_directory_temp_read", _get_next_batch
	301	)
	302
	303	# No more users -- complete the transaction.
	304	if not users_to_work_on:
	305	await self.db_pool.updates._end_background_update(
	306	"populate_user_directory_process_users"
	307	)
	308	return 1
	309
	310	logger.debug(
	311	"Processing the next %d users of %d remaining"
	312	% (len(users_to_work_on), progress["remaining"])
	313	)
	314
	315	for user_id in users_to_work_on:
	316	profile = await self.get_profileinfo(get_localpart_from_id(user_id))
	317	await self.update_profile_in_user_dir(
	318	user_id, profile.display_name, profile.avatar_url
	319	)
	320
	321	# We've finished processing a user. Delete it from the table.
	322	await self.db_pool.simple_delete_one(
	323	TEMP_TABLE + "_users", {"user_id": user_id}
	324	)
	325	# Update the remaining counter.
	326	progress["remaining"] -= 1
	327	await self.db_pool.runInteraction(
	328	"populate_user_directory",
	329	self.db_pool.updates._background_update_progress_txn,
	330	"populate_user_directory_process_users",
	331	progress,
	332	)
	333
	334	return len(users_to_work_on)
	335
	336	async def is_room_world_readable_or_publicly_joinable(self, room_id):
	337	"""Check if the room is either world_readable or publically joinable
	338	"""
	339
	340	# Create a state filter that only queries join and history state event
	341	types_to_filter = (
	342	(EventTypes.JoinRules, ""),
	343	(EventTypes.RoomHistoryVisibility, ""),
	344	)
	345
	346	current_state_ids = await self.get_filtered_current_state_ids(
	347	room_id, StateFilter.from_types(types_to_filter)
	348	)
	349
	350	join_rules_id = current_state_ids.get((EventTypes.JoinRules, ""))
	351	if join_rules_id:
	352	join_rule_ev = await self.get_event(join_rules_id, allow_none=True)
	353	if join_rule_ev:
	354	if join_rule_ev.content.get("join_rule") == JoinRules.PUBLIC:
	355	return True
	356
	357	hist_vis_id = current_state_ids.get((EventTypes.RoomHistoryVisibility, ""))
	358	if hist_vis_id:
	359	hist_vis_ev = await self.get_event(hist_vis_id, allow_none=True)
	360	if hist_vis_ev:
	361	if hist_vis_ev.content.get("history_visibility") == "world_readable":
	362	return True
	363
	364	return False
	365
	366	def update_profile_in_user_dir(self, user_id, display_name, avatar_url):
	367	"""
	368	Update or add a user's profile in the user directory.
	369	"""
	370
	371	def _update_profile_in_user_dir_txn(txn):
	372	new_entry = self.db_pool.simple_upsert_txn(
	373	txn,
	374	table="user_directory",
	375	keyvalues={"user_id": user_id},
	376	values={"display_name": display_name, "avatar_url": avatar_url},
	377	lock=False, # We're only inserter
	378	)
	379
	380	if isinstance(self.database_engine, PostgresEngine):
	381	# We weight the localpart most highly, then display name and finally
	382	# server name
	383	if self.database_engine.can_native_upsert:
	384	sql = """
	385	INSERT INTO user_directory_search(user_id, vector)
	386	VALUES (?,
	387	setweight(to_tsvector('english', ?), 'A')
	388	\|\| setweight(to_tsvector('english', ?), 'D')
	389	\|\| setweight(to_tsvector('english', COALESCE(?, '')), 'B')
	390	) ON CONFLICT (user_id) DO UPDATE SET vector=EXCLUDED.vector
	391	"""
	392	txn.execute(
	393	sql,
	394	(
	395	user_id,
	396	get_localpart_from_id(user_id),
	397	get_domain_from_id(user_id),
	398	display_name,
	399	),
	400	)
	401	else:
	402	# TODO: Remove this code after we've bumped the minimum version
	403	# of postgres to always support upserts, so we can get rid of
	404	# `new_entry` usage
	405	if new_entry is True:
	406	sql = """
	407	INSERT INTO user_directory_search(user_id, vector)
	408	VALUES (?,
	409	setweight(to_tsvector('english', ?), 'A')
	410	\|\| setweight(to_tsvector('english', ?), 'D')
	411	\|\| setweight(to_tsvector('english', COALESCE(?, '')), 'B')
	412	)
	413	"""
	414	txn.execute(
	415	sql,
	416	(
	417	user_id,
	418	get_localpart_from_id(user_id),
	419	get_domain_from_id(user_id),
	420	display_name,
	421	),
	422	)
	423	elif new_entry is False:
	424	sql = """
	425	UPDATE user_directory_search
	426	SET vector = setweight(to_tsvector('english', ?), 'A')
	427	\|\| setweight(to_tsvector('english', ?), 'D')
	428	\|\| setweight(to_tsvector('english', COALESCE(?, '')), 'B')
	429	WHERE user_id = ?
	430	"""
	431	txn.execute(
	432	sql,
	433	(
	434	get_localpart_from_id(user_id),
	435	get_domain_from_id(user_id),
	436	display_name,
	437	user_id,
	438	),
	439	)
	440	else:
	441	raise RuntimeError(
	442	"upsert returned None when 'can_native_upsert' is False"
	443	)
	444	elif isinstance(self.database_engine, Sqlite3Engine):
	445	value = "%s %s" % (user_id, display_name) if display_name else user_id
	446	self.db_pool.simple_upsert_txn(
	447	txn,
	448	table="user_directory_search",
	449	keyvalues={"user_id": user_id},
	450	values={"value": value},
	451	lock=False, # We're only inserter
	452	)
	453	else:
	454	# This should be unreachable.
	455	raise Exception("Unrecognized database engine")
	456
	457	txn.call_after(self.get_user_in_directory.invalidate, (user_id,))
	458
	459	return self.db_pool.runInteraction(
	460	"update_profile_in_user_dir", _update_profile_in_user_dir_txn
	461	)
	462
	463	def add_users_who_share_private_room(self, room_id, user_id_tuples):
	464	"""Insert entries into the users_who_share_private_rooms table. The first
	465	user should be a local user.
	466
	467	Args:
	468	room_id (str)
	469	user_id_tuples([(str, str)]): iterable of 2-tuple of user IDs.
	470	"""
	471
	472	def _add_users_who_share_room_txn(txn):
	473	self.db_pool.simple_upsert_many_txn(
	474	txn,
	475	table="users_who_share_private_rooms",
	476	key_names=["user_id", "other_user_id", "room_id"],
	477	key_values=[
	478	(user_id, other_user_id, room_id)
	479	for user_id, other_user_id in user_id_tuples
	480	],
	481	value_names=(),
	482	value_values=None,
	483	)
	484
	485	return self.db_pool.runInteraction(
	486	"add_users_who_share_room", _add_users_who_share_room_txn
	487	)
	488
	489	def add_users_in_public_rooms(self, room_id, user_ids):
	490	"""Insert entries into the users_who_share_private_rooms table. The first
	491	user should be a local user.
	492
	493	Args:
	494	room_id (str)
	495	user_ids (list[str])
	496	"""
	497
	498	def _add_users_in_public_rooms_txn(txn):
	499
	500	self.db_pool.simple_upsert_many_txn(
	501	txn,
	502	table="users_in_public_rooms",
	503	key_names=["user_id", "room_id"],
	504	key_values=[(user_id, room_id) for user_id in user_ids],
	505	value_names=(),
	506	value_values=None,
	507	)
	508
	509	return self.db_pool.runInteraction(
	510	"add_users_in_public_rooms", _add_users_in_public_rooms_txn
	511	)
	512
	513	def delete_all_from_user_dir(self):
	514	"""Delete the entire user directory
	515	"""
	516
	517	def _delete_all_from_user_dir_txn(txn):
	518	txn.execute("DELETE FROM user_directory")
	519	txn.execute("DELETE FROM user_directory_search")
	520	txn.execute("DELETE FROM users_in_public_rooms")
	521	txn.execute("DELETE FROM users_who_share_private_rooms")
	522	txn.call_after(self.get_user_in_directory.invalidate_all)
	523
	524	return self.db_pool.runInteraction(
	525	"delete_all_from_user_dir", _delete_all_from_user_dir_txn
	526	)
	527
	528	@cached()
	529	def get_user_in_directory(self, user_id):
	530	return self.db_pool.simple_select_one(
	531	table="user_directory",
	532	keyvalues={"user_id": user_id},
	533	retcols=("display_name", "avatar_url"),
	534	allow_none=True,
	535	desc="get_user_in_directory",
	536	)
	537
	538	def update_user_directory_stream_pos(self, stream_id):
	539	return self.db_pool.simple_update_one(
	540	table="user_directory_stream_pos",
	541	keyvalues={},
	542	updatevalues={"stream_id": stream_id},
	543	desc="update_user_directory_stream_pos",
	544	)
	545
	546
	547	class UserDirectoryStore(UserDirectoryBackgroundUpdateStore):
	548
	549	# How many records do we calculate before sending it to
	550	# add_users_who_share_private_rooms?
	551	SHARE_PRIVATE_WORKING_SET = 500
	552
	553	def __init__(self, database: DatabasePool, db_conn, hs):
	554	super(UserDirectoryStore, self).__init__(database, db_conn, hs)
	555
	556	def remove_from_user_dir(self, user_id):
	557	def _remove_from_user_dir_txn(txn):
	558	self.db_pool.simple_delete_txn(
	559	txn, table="user_directory", keyvalues={"user_id": user_id}
	560	)
	561	self.db_pool.simple_delete_txn(
	562	txn, table="user_directory_search", keyvalues={"user_id": user_id}
	563	)
	564	self.db_pool.simple_delete_txn(
	565	txn, table="users_in_public_rooms", keyvalues={"user_id": user_id}
	566	)
	567	self.db_pool.simple_delete_txn(
	568	txn,
	569	table="users_who_share_private_rooms",
	570	keyvalues={"user_id": user_id},
	571	)
	572	self.db_pool.simple_delete_txn(
	573	txn,
	574	table="users_who_share_private_rooms",
	575	keyvalues={"other_user_id": user_id},
	576	)
	577	txn.call_after(self.get_user_in_directory.invalidate, (user_id,))
	578
	579	return self.db_pool.runInteraction(
	580	"remove_from_user_dir", _remove_from_user_dir_txn
	581	)
	582
	583	async def get_users_in_dir_due_to_room(self, room_id):
	584	"""Get all user_ids that are in the room directory because they're
	585	in the given room_id
	586	"""
	587	user_ids_share_pub = await self.db_pool.simple_select_onecol(
	588	table="users_in_public_rooms",
	589	keyvalues={"room_id": room_id},
	590	retcol="user_id",
	591	desc="get_users_in_dir_due_to_room",
	592	)
	593
	594	user_ids_share_priv = await self.db_pool.simple_select_onecol(
	595	table="users_who_share_private_rooms",
	596	keyvalues={"room_id": room_id},
	597	retcol="other_user_id",
	598	desc="get_users_in_dir_due_to_room",
	599	)
	600
	601	user_ids = set(user_ids_share_pub)
	602	user_ids.update(user_ids_share_priv)
	603
	604	return user_ids
	605
	606	def remove_user_who_share_room(self, user_id, room_id):
	607	"""
	608	Deletes entries in the users_who_share_*_rooms table. The first
	609	user should be a local user.
	610
	611	Args:
	612	user_id (str)
	613	room_id (str)
	614	"""
	615
	616	def _remove_user_who_share_room_txn(txn):
	617	self.db_pool.simple_delete_txn(
	618	txn,
	619	table="users_who_share_private_rooms",
	620	keyvalues={"user_id": user_id, "room_id": room_id},
	621	)
	622	self.db_pool.simple_delete_txn(
	623	txn,
	624	table="users_who_share_private_rooms",
	625	keyvalues={"other_user_id": user_id, "room_id": room_id},
	626	)
	627	self.db_pool.simple_delete_txn(
	628	txn,
	629	table="users_in_public_rooms",
	630	keyvalues={"user_id": user_id, "room_id": room_id},
	631	)
	632
	633	return self.db_pool.runInteraction(
	634	"remove_user_who_share_room", _remove_user_who_share_room_txn
	635	)
	636
	637	async def get_user_dir_rooms_user_is_in(self, user_id):
	638	"""
	639	Returns the rooms that a user is in.
	640
	641	Args:
	642	user_id(str): Must be a local user
	643
	644	Returns:
	645	list: user_id
	646	"""
	647	rows = await self.db_pool.simple_select_onecol(
	648	table="users_who_share_private_rooms",
	649	keyvalues={"user_id": user_id},
	650	retcol="room_id",
	651	desc="get_rooms_user_is_in",
	652	)
	653
	654	pub_rows = await self.db_pool.simple_select_onecol(
	655	table="users_in_public_rooms",
	656	keyvalues={"user_id": user_id},
	657	retcol="room_id",
	658	desc="get_rooms_user_is_in",
	659	)
	660
	661	users = set(pub_rows)
	662	users.update(rows)
	663	return list(users)
	664
	665	def get_user_directory_stream_pos(self):
	666	return self.db_pool.simple_select_one_onecol(
	667	table="user_directory_stream_pos",
	668	keyvalues={},
	669	retcol="stream_id",
	670	desc="get_user_directory_stream_pos",
	671	)
	672
	673	async def search_user_dir(self, user_id, search_term, limit):
	674	"""Searches for users in directory
	675
	676	Returns:
	677	dict of the form::
	678
	679	{
	680	"limited": <bool>, # whether there were more results or not
	681	"results": [ # Ordered by best match first
	682	{
	683	"user_id": <user_id>,
	684	"display_name": <display_name>,
	685	"avatar_url": <avatar_url>
	686	}
	687	]
	688	}
	689	"""
	690
	691	if self.hs.config.user_directory_search_all_users:
	692	join_args = (user_id,)
	693	where_clause = "user_id != ?"
	694	else:
	695	join_args = (user_id,)
	696	where_clause = """
	697	(
	698	EXISTS (select 1 from users_in_public_rooms WHERE user_id = t.user_id)
	699	OR EXISTS (
	700	SELECT 1 FROM users_who_share_private_rooms
	701	WHERE user_id = ? AND other_user_id = t.user_id
	702	)
	703	)
	704	"""
	705
	706	if isinstance(self.database_engine, PostgresEngine):
	707	full_query, exact_query, prefix_query = _parse_query_postgres(search_term)
	708
	709	# We order by rank and then if they have profile info
	710	# The ranking algorithm is hand tweaked for "best" results. Broadly
	711	# the idea is we give a higher weight to exact matches.
	712	# The array of numbers are the weights for the various part of the
	713	# search: (domain, _, display name, localpart)
	714	sql = """
	715	SELECT d.user_id AS user_id, display_name, avatar_url
	716	FROM user_directory_search as t
	717	INNER JOIN user_directory AS d USING (user_id)
	718	WHERE
	719	%s
	720	AND vector @@ to_tsquery('english', ?)
	721	ORDER BY
	722	(CASE WHEN d.user_id IS NOT NULL THEN 4.0 ELSE 1.0 END)
	723	* (CASE WHEN display_name IS NOT NULL THEN 1.2 ELSE 1.0 END)
	724	* (CASE WHEN avatar_url IS NOT NULL THEN 1.2 ELSE 1.0 END)
	725	* (
	726	3 * ts_rank_cd(
	727	'{0.1, 0.1, 0.9, 1.0}',
	728	vector,
	729	to_tsquery('english', ?),
	730	8
	731	)
	732	+ ts_rank_cd(
	733	'{0.1, 0.1, 0.9, 1.0}',
	734	vector,
	735	to_tsquery('english', ?),
	736	8
	737	)
	738	)
	739	DESC,
	740	display_name IS NULL,
	741	avatar_url IS NULL
	742	LIMIT ?
	743	""" % (
	744	where_clause,
	745	)
	746	args = join_args + (full_query, exact_query, prefix_query, limit + 1)
	747	elif isinstance(self.database_engine, Sqlite3Engine):
	748	search_query = _parse_query_sqlite(search_term)
	749
	750	sql = """
	751	SELECT d.user_id AS user_id, display_name, avatar_url
	752	FROM user_directory_search as t
	753	INNER JOIN user_directory AS d USING (user_id)
	754	WHERE
	755	%s
	756	AND value MATCH ?
	757	ORDER BY
	758	rank(matchinfo(user_directory_search)) DESC,
	759	display_name IS NULL,
	760	avatar_url IS NULL
	761	LIMIT ?
	762	""" % (
	763	where_clause,
	764	)
	765	args = join_args + (search_query, limit + 1)
	766	else:
	767	# This should be unreachable.
	768	raise Exception("Unrecognized database engine")
	769
	770	results = await self.db_pool.execute(
	771	"search_user_dir", self.db_pool.cursor_to_dict, sql, *args
	772	)
	773
	774	limited = len(results) > limit
	775
	776	return {"limited": limited, "results": results}
	777
	778
	779	def _parse_query_sqlite(search_term):
	780	"""Takes a plain unicode string from the user and converts it into a form
	781	that can be passed to database.
	782	We use this so that we can add prefix matching, which isn't something
	783	that is supported by default.
	784
	785	We specifically add both a prefix and non prefix matching term so that
	786	exact matches get ranked higher.
	787	"""
	788
	789	# Pull out the individual words, discarding any non-word characters.
	790	results = re.findall(r"([\w\-]+)", search_term, re.UNICODE)
	791	return " & ".join("(%s* OR %s)" % (result, result) for result in results)
	792
	793
	794	def _parse_query_postgres(search_term):
	795	"""Takes a plain unicode string from the user and converts it into a form
	796	that can be passed to database.
	797	We use this so that we can add prefix matching, which isn't something
	798	that is supported by default.
	799	"""
	800
	801	# Pull out the individual words, discarding any non-word characters.
	802	results = re.findall(r"([\w\-]+)", search_term, re.UNICODE)
	803
	804	both = " & ".join("(%s:* \| %s)" % (result, result) for result in results)
	805	exact = " & ".join("%s" % (result,) for result in results)
	806	prefix = " & ".join("%s:*" % (result,) for result in results)
	807
	808	return both, exact, prefix

+113

-0

synapse/storage/databases/main/user_erasure_store.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2018 New Vector Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import operator
	16
	17	from synapse.storage._base import SQLBaseStore
	18	from synapse.util.caches.descriptors import cached, cachedList
	19
	20
	21	class UserErasureWorkerStore(SQLBaseStore):
	22	@cached()
	23	def is_user_erased(self, user_id):
	24	"""
	25	Check if the given user id has requested erasure
	26
	27	Args:
	28	user_id (str): full user id to check
	29
	30	Returns:
	31	Deferred[bool]: True if the user has requested erasure
	32	"""
	33	return self.db_pool.simple_select_onecol(
	34	table="erased_users",
	35	keyvalues={"user_id": user_id},
	36	retcol="1",
	37	desc="is_user_erased",
	38	).addCallback(operator.truth)
	39
	40	@cachedList(
	41	cached_method_name="is_user_erased", list_name="user_ids", inlineCallbacks=True
	42	)
	43	def are_users_erased(self, user_ids):
	44	"""
	45	Checks which users in a list have requested erasure
	46
	47	Args:
	48	user_ids (iterable[str]): full user id to check
	49
	50	Returns:
	51	Deferred[dict[str, bool]]:
	52	for each user, whether the user has requested erasure.
	53	"""
	54	# this serves the dual purpose of (a) making sure we can do len and
	55	# iterate it multiple times, and (b) avoiding duplicates.
	56	user_ids = tuple(set(user_ids))
	57
	58	rows = yield self.db_pool.simple_select_many_batch(
	59	table="erased_users",
	60	column="user_id",
	61	iterable=user_ids,
	62	retcols=("user_id",),
	63	desc="are_users_erased",
	64	)
	65	erased_users = {row["user_id"] for row in rows}
	66
	67	res = {u: u in erased_users for u in user_ids}
	68	return res
	69
	70
	71	class UserErasureStore(UserErasureWorkerStore):
	72	def mark_user_erased(self, user_id: str) -> None:
	73	"""Indicate that user_id wishes their message history to be erased.
	74
	75	Args:
	76	user_id: full user_id to be erased
	77	"""
	78
	79	def f(txn):
	80	# first check if they are already in the list
	81	txn.execute("SELECT 1 FROM erased_users WHERE user_id = ?", (user_id,))
	82	if txn.fetchone():
	83	return
	84
	85	# they are not already there: do the insert.
	86	txn.execute("INSERT INTO erased_users (user_id) VALUES (?)", (user_id,))
	87
	88	self._invalidate_cache_and_stream(txn, self.is_user_erased, (user_id,))
	89
	90	return self.db_pool.runInteraction("mark_user_erased", f)
	91
	92	def mark_user_not_erased(self, user_id: str) -> None:
	93	"""Indicate that user_id is no longer erased.
	94
	95	Args:
	96	user_id: full user_id to be un-erased
	97	"""
	98
	99	def f(txn):
	100	# first check if they are already in the list
	101	txn.execute("SELECT 1 FROM erased_users WHERE user_id = ?", (user_id,))
	102	if not txn.fetchone():
	103	return
	104
	105	# They are there, delete them.
	106	self.simple_delete_one_txn(
	107	txn, "erased_users", keyvalues={"user_id": user_id}
	108	)
	109
	110	self._invalidate_cache_and_stream(txn, self.is_user_erased, (user_id,))
	111
	112	return self.db_pool.runInteraction("mark_user_not_erased", f)

+16

-0

synapse/storage/databases/state/__init__.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2019 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from synapse.storage.databases.state.store import StateGroupDataStore # noqa: F401

+370

-0

synapse/storage/databases/state/bg_updates.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16
	17	from synapse.storage._base import SQLBaseStore
	18	from synapse.storage.database import DatabasePool
	19	from synapse.storage.engines import PostgresEngine
	20	from synapse.storage.state import StateFilter
	21
	22	logger = logging.getLogger(__name__)
	23
	24
	25	MAX_STATE_DELTA_HOPS = 100
	26
	27
	28	class StateGroupBackgroundUpdateStore(SQLBaseStore):
	29	"""Defines functions related to state groups needed to run the state backgroud
	30	updates.
	31	"""
	32
	33	def _count_state_group_hops_txn(self, txn, state_group):
	34	"""Given a state group, count how many hops there are in the tree.
	35
	36	This is used to ensure the delta chains don't get too long.
	37	"""
	38	if isinstance(self.database_engine, PostgresEngine):
	39	sql = """
	40	WITH RECURSIVE state(state_group) AS (
	41	VALUES(?::bigint)
	42	UNION ALL
	43	SELECT prev_state_group FROM state_group_edges e, state s
	44	WHERE s.state_group = e.state_group
	45	)
	46	SELECT count(*) FROM state;
	47	"""
	48
	49	txn.execute(sql, (state_group,))
	50	row = txn.fetchone()
	51	if row and row[0]:
	52	return row[0]
	53	else:
	54	return 0
	55	else:
	56	# We don't use WITH RECURSIVE on sqlite3 as there are distributions
	57	# that ship with an sqlite3 version that doesn't support it (e.g. wheezy)
	58	next_group = state_group
	59	count = 0
	60
	61	while next_group:
	62	next_group = self.db_pool.simple_select_one_onecol_txn(
	63	txn,
	64	table="state_group_edges",
	65	keyvalues={"state_group": next_group},
	66	retcol="prev_state_group",
	67	allow_none=True,
	68	)
	69	if next_group:
	70	count += 1
	71
	72	return count
	73
	74	def _get_state_groups_from_groups_txn(
	75	self, txn, groups, state_filter=StateFilter.all()
	76	):
	77	results = {group: {} for group in groups}
	78
	79	where_clause, where_args = state_filter.make_sql_filter_clause()
	80
	81	# Unless the filter clause is empty, we're going to append it after an
	82	# existing where clause
	83	if where_clause:
	84	where_clause = " AND (%s)" % (where_clause,)
	85
	86	if isinstance(self.database_engine, PostgresEngine):
	87	# Temporarily disable sequential scans in this transaction. This is
	88	# a temporary hack until we can add the right indices in
	89	txn.execute("SET LOCAL enable_seqscan=off")
	90
	91	# The below query walks the state_group tree so that the "state"
	92	# table includes all state_groups in the tree. It then joins
	93	# against `state_groups_state` to fetch the latest state.
	94	# It assumes that previous state groups are always numerically
	95	# lesser.
	96	# The PARTITION is used to get the event_id in the greatest state
	97	# group for the given type, state_key.
	98	# This may return multiple rows per (type, state_key), but last_value
	99	# should be the same.
	100	sql = """
	101	WITH RECURSIVE state(state_group) AS (
	102	VALUES(?::bigint)
	103	UNION ALL
	104	SELECT prev_state_group FROM state_group_edges e, state s
	105	WHERE s.state_group = e.state_group
	106	)
	107	SELECT DISTINCT ON (type, state_key)
	108	type, state_key, event_id
	109	FROM state_groups_state
	110	WHERE state_group IN (
	111	SELECT state_group FROM state
	112	) %s
	113	ORDER BY type, state_key, state_group DESC
	114	"""
	115
	116	for group in groups:
	117	args = [group]
	118	args.extend(where_args)
	119
	120	txn.execute(sql % (where_clause,), args)
	121	for row in txn:
	122	typ, state_key, event_id = row
	123	key = (typ, state_key)
	124	results[group][key] = event_id
	125	else:
	126	max_entries_returned = state_filter.max_entries_returned()
	127
	128	# We don't use WITH RECURSIVE on sqlite3 as there are distributions
	129	# that ship with an sqlite3 version that doesn't support it (e.g. wheezy)
	130	for group in groups:
	131	next_group = group
	132
	133	while next_group:
	134	# We did this before by getting the list of group ids, and
	135	# then passing that list to sqlite to get latest event for
	136	# each (type, state_key). However, that was terribly slow
	137	# without the right indices (which we can't add until
	138	# after we finish deduping state, which requires this func)
	139	args = [next_group]
	140	args.extend(where_args)
	141
	142	txn.execute(
	143	"SELECT type, state_key, event_id FROM state_groups_state"
	144	" WHERE state_group = ? " + where_clause,
	145	args,
	146	)
	147	results[group].update(
	148	((typ, state_key), event_id)
	149	for typ, state_key, event_id in txn
	150	if (typ, state_key) not in results[group]
	151	)
	152
	153	# If the number of entries in the (type,state_key)->event_id dict
	154	# matches the number of (type,state_keys) types we were searching
	155	# for, then we must have found them all, so no need to go walk
	156	# further down the tree... UNLESS our types filter contained
	157	# wildcards (i.e. Nones) in which case we have to do an exhaustive
	158	# search
	159	if (
	160	max_entries_returned is not None
	161	and len(results[group]) == max_entries_returned
	162	):
	163	break
	164
	165	next_group = self.db_pool.simple_select_one_onecol_txn(
	166	txn,
	167	table="state_group_edges",
	168	keyvalues={"state_group": next_group},
	169	retcol="prev_state_group",
	170	allow_none=True,
	171	)
	172
	173	return results
	174
	175
	176	class StateBackgroundUpdateStore(StateGroupBackgroundUpdateStore):
	177
	178	STATE_GROUP_DEDUPLICATION_UPDATE_NAME = "state_group_state_deduplication"
	179	STATE_GROUP_INDEX_UPDATE_NAME = "state_group_state_type_index"
	180	STATE_GROUPS_ROOM_INDEX_UPDATE_NAME = "state_groups_room_id_idx"
	181
	182	def __init__(self, database: DatabasePool, db_conn, hs):
	183	super(StateBackgroundUpdateStore, self).__init__(database, db_conn, hs)
	184	self.db_pool.updates.register_background_update_handler(
	185	self.STATE_GROUP_DEDUPLICATION_UPDATE_NAME,
	186	self._background_deduplicate_state,
	187	)
	188	self.db_pool.updates.register_background_update_handler(
	189	self.STATE_GROUP_INDEX_UPDATE_NAME, self._background_index_state
	190	)
	191	self.db_pool.updates.register_background_index_update(
	192	self.STATE_GROUPS_ROOM_INDEX_UPDATE_NAME,
	193	index_name="state_groups_room_id_idx",
	194	table="state_groups",
	195	columns=["room_id"],
	196	)
	197
	198	async def _background_deduplicate_state(self, progress, batch_size):
	199	"""This background update will slowly deduplicate state by reencoding
	200	them as deltas.
	201	"""
	202	last_state_group = progress.get("last_state_group", 0)
	203	rows_inserted = progress.get("rows_inserted", 0)
	204	max_group = progress.get("max_group", None)
	205
	206	BATCH_SIZE_SCALE_FACTOR = 100
	207
	208	batch_size = max(1, int(batch_size / BATCH_SIZE_SCALE_FACTOR))
	209
	210	if max_group is None:
	211	rows = await self.db_pool.execute(
	212	"_background_deduplicate_state",
	213	None,
	214	"SELECT coalesce(max(id), 0) FROM state_groups",
	215	)
	216	max_group = rows[0][0]
	217
	218	def reindex_txn(txn):
	219	new_last_state_group = last_state_group
	220	for count in range(batch_size):
	221	txn.execute(
	222	"SELECT id, room_id FROM state_groups"
	223	" WHERE ? < id AND id <= ?"
	224	" ORDER BY id ASC"
	225	" LIMIT 1",
	226	(new_last_state_group, max_group),
	227	)
	228	row = txn.fetchone()
	229	if row:
	230	state_group, room_id = row
	231
	232	if not row or not state_group:
	233	return True, count
	234
	235	txn.execute(
	236	"SELECT state_group FROM state_group_edges"
	237	" WHERE state_group = ?",
	238	(state_group,),
	239	)
	240
	241	# If we reach a point where we've already started inserting
	242	# edges we should stop.
	243	if txn.fetchall():
	244	return True, count
	245
	246	txn.execute(
	247	"SELECT coalesce(max(id), 0) FROM state_groups"
	248	" WHERE id < ? AND room_id = ?",
	249	(state_group, room_id),
	250	)
	251	(prev_group,) = txn.fetchone()
	252	new_last_state_group = state_group
	253
	254	if prev_group:
	255	potential_hops = self._count_state_group_hops_txn(txn, prev_group)
	256	if potential_hops >= MAX_STATE_DELTA_HOPS:
	257	# We want to ensure chains are at most this long,#
	258	# otherwise read performance degrades.
	259	continue
	260
	261	prev_state = self._get_state_groups_from_groups_txn(
	262	txn, [prev_group]
	263	)
	264	prev_state = prev_state[prev_group]
	265
	266	curr_state = self._get_state_groups_from_groups_txn(
	267	txn, [state_group]
	268	)
	269	curr_state = curr_state[state_group]
	270
	271	if not set(prev_state.keys()) - set(curr_state.keys()):
	272	# We can only do a delta if the current has a strict super set
	273	# of keys
	274
	275	delta_state = {
	276	key: value
	277	for key, value in curr_state.items()
	278	if prev_state.get(key, None) != value
	279	}
	280
	281	self.db_pool.simple_delete_txn(
	282	txn,
	283	table="state_group_edges",
	284	keyvalues={"state_group": state_group},
	285	)
	286
	287	self.db_pool.simple_insert_txn(
	288	txn,
	289	table="state_group_edges",
	290	values={
	291	"state_group": state_group,
	292	"prev_state_group": prev_group,
	293	},
	294	)
	295
	296	self.db_pool.simple_delete_txn(
	297	txn,
	298	table="state_groups_state",
	299	keyvalues={"state_group": state_group},
	300	)
	301
	302	self.db_pool.simple_insert_many_txn(
	303	txn,
	304	table="state_groups_state",
	305	values=[
	306	{
	307	"state_group": state_group,
	308	"room_id": room_id,
	309	"type": key[0],
	310	"state_key": key[1],
	311	"event_id": state_id,
	312	}
	313	for key, state_id in delta_state.items()
	314	],
	315	)
	316
	317	progress = {
	318	"last_state_group": state_group,
	319	"rows_inserted": rows_inserted + batch_size,
	320	"max_group": max_group,
	321	}
	322
	323	self.db_pool.updates._background_update_progress_txn(
	324	txn, self.STATE_GROUP_DEDUPLICATION_UPDATE_NAME, progress
	325	)
	326
	327	return False, batch_size
	328
	329	finished, result = await self.db_pool.runInteraction(
	330	self.STATE_GROUP_DEDUPLICATION_UPDATE_NAME, reindex_txn
	331	)
	332
	333	if finished:
	334	await self.db_pool.updates._end_background_update(
	335	self.STATE_GROUP_DEDUPLICATION_UPDATE_NAME
	336	)
	337
	338	return result * BATCH_SIZE_SCALE_FACTOR
	339
	340	async def _background_index_state(self, progress, batch_size):
	341	def reindex_txn(conn):
	342	conn.rollback()
	343	if isinstance(self.database_engine, PostgresEngine):
	344	# postgres insists on autocommit for the index
	345	conn.set_session(autocommit=True)
	346	try:
	347	txn = conn.cursor()
	348	txn.execute(
	349	"CREATE INDEX CONCURRENTLY state_groups_state_type_idx"
	350	" ON state_groups_state(state_group, type, state_key)"
	351	)
	352	txn.execute("DROP INDEX IF EXISTS state_groups_state_id")
	353	finally:
	354	conn.set_session(autocommit=False)
	355	else:
	356	txn = conn.cursor()
	357	txn.execute(
	358	"CREATE INDEX state_groups_state_type_idx"
	359	" ON state_groups_state(state_group, type, state_key)"
	360	)
	361	txn.execute("DROP INDEX IF EXISTS state_groups_state_id")
	362
	363	await self.db_pool.runWithConnection(reindex_txn)
	364
	365	await self.db_pool.updates._end_background_update(
	366	self.STATE_GROUP_INDEX_UPDATE_NAME
	367	)
	368
	369	return 1

+16

-0

synapse/storage/databases/state/schema/delta/23/drop_state_index.sql less more

	0	/* Copyright 2015, 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	DROP INDEX IF EXISTS state_groups_state_tuple;

+33

-0

synapse/storage/databases/state/schema/delta/30/state_stream.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	/* We used to create a table called current_state_resets, but this is no
	17	* longer used and is removed in delta 54.
	18	*/
	19
	20	/* The outlier events that have aquired a state group typically through
	21	* backfill. This is tracked separately to the events table, as assigning a
	22	* state group change the position of the existing event in the stream
	23	* ordering.
	24	* However since a stream_ordering is assigned in persist_event for the
	25	* (event, state) pair, we can use that stream_ordering to identify when
	26	* the new state was assigned for the event.
	27	*/
	28	CREATE TABLE IF NOT EXISTS ex_outlier_stream(
	29	event_stream_ordering BIGINT PRIMARY KEY NOT NULL,
	30	event_id TEXT NOT NULL,
	31	state_group BIGINT NOT NULL
	32	);

+19

-0

synapse/storage/databases/state/schema/delta/32/remove_state_indices.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15
	16	-- The following indices are redundant, other indices are equivalent or
	17	-- supersets
	18	DROP INDEX IF EXISTS state_groups_id; -- Duplicate of PRIMARY KEY

+17

-0

synapse/storage/databases/state/schema/delta/35/add_state_index.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT into background_updates (update_name, progress_json, depends_on)
	16	VALUES ('state_group_state_type_index', '{}', 'state_group_state_deduplication');

+22

-0

synapse/storage/databases/state/schema/delta/35/state.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE state_group_edges(
	16	state_group BIGINT NOT NULL,
	17	prev_state_group BIGINT NOT NULL
	18	);
	19
	20	CREATE INDEX state_group_edges_idx ON state_group_edges(state_group);
	21	CREATE INDEX state_group_edges_prev_idx ON state_group_edges(prev_state_group);

+17

-0

synapse/storage/databases/state/schema/delta/35/state_dedupe.sql less more

	0	/* Copyright 2016 OpenMarket Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT into background_updates (update_name, progress_json)
	16	VALUES ('state_group_state_deduplication', '{}');

+34

-0

synapse/storage/databases/state/schema/delta/47/state_group_seq.py less more

	0	# Copyright 2018 New Vector Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14	from synapse.storage.engines import PostgresEngine
	15
	16
	17	def run_create(cur, database_engine, args, *kwargs):
	18	if isinstance(database_engine, PostgresEngine):
	19	# if we already have some state groups, we want to start making new
	20	# ones with a higher id.
	21	cur.execute("SELECT max(id) FROM state_groups")
	22	row = cur.fetchone()
	23
	24	if row[0] is None:
	25	start_val = 1
	26	else:
	27	start_val = row[0] + 1
	28
	29	cur.execute("CREATE SEQUENCE state_group_id_seq START WITH %s", (start_val,))
	30
	31
	32	def run_upgrade(args, *kwargs):
	33	pass

+17

-0

synapse/storage/databases/state/schema/delta/56/state_group_room_idx.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT INTO background_updates (update_name, progress_json) VALUES
	16	('state_groups_room_id_idx', '{}');

+37

-0

synapse/storage/databases/state/schema/full_schemas/54/full.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE TABLE state_groups (
	16	id BIGINT PRIMARY KEY,
	17	room_id TEXT NOT NULL,
	18	event_id TEXT NOT NULL
	19	);
	20
	21	CREATE TABLE state_groups_state (
	22	state_group BIGINT NOT NULL,
	23	room_id TEXT NOT NULL,
	24	type TEXT NOT NULL,
	25	state_key TEXT NOT NULL,
	26	event_id TEXT NOT NULL
	27	);
	28
	29	CREATE TABLE state_group_edges (
	30	state_group BIGINT NOT NULL,
	31	prev_state_group BIGINT NOT NULL
	32	);
	33
	34	CREATE INDEX state_group_edges_idx ON state_group_edges (state_group);
	35	CREATE INDEX state_group_edges_prev_idx ON state_group_edges (prev_state_group);
	36	CREATE INDEX state_groups_state_type_idx ON state_groups_state (state_group, type, state_key);

+21

-0

synapse/storage/databases/state/schema/full_schemas/54/sequence.sql.postgres less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	CREATE SEQUENCE state_group_id_seq
	16	START WITH 1
	17	INCREMENT BY 1
	18	NO MINVALUE
	19	NO MAXVALUE
	20	CACHE 1;

+644

-0

synapse/storage/databases/state/store.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16	from collections import namedtuple
	17	from typing import Dict, Iterable, List, Set, Tuple
	18
	19	from twisted.internet import defer
	20
	21	from synapse.api.constants import EventTypes
	22	from synapse.storage._base import SQLBaseStore
	23	from synapse.storage.database import DatabasePool
	24	from synapse.storage.databases.state.bg_updates import StateBackgroundUpdateStore
	25	from synapse.storage.state import StateFilter
	26	from synapse.storage.types import Cursor
	27	from synapse.storage.util.sequence import build_sequence_generator
	28	from synapse.types import StateMap
	29	from synapse.util.caches.descriptors import cached
	30	from synapse.util.caches.dictionary_cache import DictionaryCache
	31
	32	logger = logging.getLogger(__name__)
	33
	34
	35	MAX_STATE_DELTA_HOPS = 100
	36
	37
	38	class _GetStateGroupDelta(
	39	namedtuple("_GetStateGroupDelta", ("prev_group", "delta_ids"))
	40	):
	41	"""Return type of get_state_group_delta that implements __len__, which lets
	42	us use the itrable flag when caching
	43	"""
	44
	45	__slots__ = []
	46
	47	def __len__(self):
	48	return len(self.delta_ids) if self.delta_ids else 0
	49
	50
	51	class StateGroupDataStore(StateBackgroundUpdateStore, SQLBaseStore):
	52	"""A data store for fetching/storing state groups.
	53	"""
	54
	55	def __init__(self, database: DatabasePool, db_conn, hs):
	56	super(StateGroupDataStore, self).__init__(database, db_conn, hs)
	57
	58	# Originally the state store used a single DictionaryCache to cache the
	59	# event IDs for the state types in a given state group to avoid hammering
	60	# on the state_group* tables.
	61	#
	62	# The point of using a DictionaryCache is that it can cache a subset
	63	# of the state events for a given state group (i.e. a subset of the keys for a
	64	# given dict which is an entry in the cache for a given state group ID).
	65	#
	66	# However, this poses problems when performing complicated queries
	67	# on the store - for instance: "give me all the state for this group, but
	68	# limit members to this subset of users", as DictionaryCache's API isn't
	69	# rich enough to say "please cache any of these fields, apart from this subset".
	70	# This is problematic when lazy loading members, which requires this behaviour,
	71	# as without it the cache has no choice but to speculatively load all
	72	# state events for the group, which negates the efficiency being sought.
	73	#
	74	# Rather than overcomplicating DictionaryCache's API, we instead split the
	75	# state_group_cache into two halves - one for tracking non-member events,
	76	# and the other for tracking member_events. This means that lazy loading
	77	# queries can be made in a cache-friendly manner by querying both caches
	78	# separately and then merging the result. So for the example above, you
	79	# would query the members cache for a specific subset of state keys
	80	# (which DictionaryCache will handle efficiently and fine) and the non-members
	81	# cache for all state (which DictionaryCache will similarly handle fine)
	82	# and then just merge the results together.
	83	#
	84	# We size the non-members cache to be smaller than the members cache as the
	85	# vast majority of state in Matrix (today) is member events.
	86
	87	self._state_group_cache = DictionaryCache(
	88	"stateGroupCache",
	89	# TODO: this hasn't been tuned yet
	90	50000,
	91	)
	92	self._state_group_members_cache = DictionaryCache(
	93	"stateGroupMembersCache", 500000,
	94	)
	95
	96	def get_max_state_group_txn(txn: Cursor):
	97	txn.execute("SELECT COALESCE(max(id), 0) FROM state_groups")
	98	return txn.fetchone()[0]
	99
	100	self._state_group_seq_gen = build_sequence_generator(
	101	self.database_engine, get_max_state_group_txn, "state_group_id_seq"
	102	)
	103
	104	@cached(max_entries=10000, iterable=True)
	105	def get_state_group_delta(self, state_group):
	106	"""Given a state group try to return a previous group and a delta between
	107	the old and the new.
	108
	109	Returns:
	110	(prev_group, delta_ids), where both may be None.
	111	"""
	112
	113	def _get_state_group_delta_txn(txn):
	114	prev_group = self.db_pool.simple_select_one_onecol_txn(
	115	txn,
	116	table="state_group_edges",
	117	keyvalues={"state_group": state_group},
	118	retcol="prev_state_group",
	119	allow_none=True,
	120	)
	121
	122	if not prev_group:
	123	return _GetStateGroupDelta(None, None)
	124
	125	delta_ids = self.db_pool.simple_select_list_txn(
	126	txn,
	127	table="state_groups_state",
	128	keyvalues={"state_group": state_group},
	129	retcols=("type", "state_key", "event_id"),
	130	)
	131
	132	return _GetStateGroupDelta(
	133	prev_group,
	134	{(row["type"], row["state_key"]): row["event_id"] for row in delta_ids},
	135	)
	136
	137	return self.db_pool.runInteraction(
	138	"get_state_group_delta", _get_state_group_delta_txn
	139	)
	140
	141	async def _get_state_groups_from_groups(
	142	self, groups: List[int], state_filter: StateFilter
	143	) -> Dict[int, StateMap[str]]:
	144	"""Returns the state groups for a given set of groups from the
	145	database, filtering on types of state events.
	146
	147	Args:
	148	groups: list of state group IDs to query
	149	state_filter: The state filter used to fetch state
	150	from the database.
	151	Returns:
	152	Dict of state group to state map.
	153	"""
	154	results = {}
	155
	156	chunks = [groups[i : i + 100] for i in range(0, len(groups), 100)]
	157	for chunk in chunks:
	158	res = await self.db_pool.runInteraction(
	159	"_get_state_groups_from_groups",
	160	self._get_state_groups_from_groups_txn,
	161	chunk,
	162	state_filter,
	163	)
	164	results.update(res)
	165
	166	return results
	167
	168	def _get_state_for_group_using_cache(self, cache, group, state_filter):
	169	"""Checks if group is in cache. See `_get_state_for_groups`
	170
	171	Args:
	172	cache(DictionaryCache): the state group cache to use
	173	group(int): The state group to lookup
	174	state_filter (StateFilter): The state filter used to fetch state
	175	from the database.
	176
	177	Returns 2-tuple (`state_dict`, `got_all`).
	178	`got_all` is a bool indicating if we successfully retrieved all
	179	requests state from the cache, if False we need to query the DB for the
	180	missing state.
	181	"""
	182	is_all, known_absent, state_dict_ids = cache.get(group)
	183
	184	if is_all or state_filter.is_full():
	185	# Either we have everything or want everything, either way
	186	# `is_all` tells us whether we've gotten everything.
	187	return state_filter.filter_state(state_dict_ids), is_all
	188
	189	# tracks whether any of our requested types are missing from the cache
	190	missing_types = False
	191
	192	if state_filter.has_wildcards():
	193	# We don't know if we fetched all the state keys for the types in
	194	# the filter that are wildcards, so we have to assume that we may
	195	# have missed some.
	196	missing_types = True
	197	else:
	198	# There aren't any wild cards, so `concrete_types()` returns the
	199	# complete list of event types we're wanting.
	200	for key in state_filter.concrete_types():
	201	if key not in state_dict_ids and key not in known_absent:
	202	missing_types = True
	203	break
	204
	205	return state_filter.filter_state(state_dict_ids), not missing_types
	206
	207	async def _get_state_for_groups(
	208	self, groups: Iterable[int], state_filter: StateFilter = StateFilter.all()
	209	) -> Dict[int, StateMap[str]]:
	210	"""Gets the state at each of a list of state groups, optionally
	211	filtering by type/state_key
	212
	213	Args:
	214	groups: list of state groups for which we want
	215	to get the state.
	216	state_filter: The state filter used to fetch state
	217	from the database.
	218	Returns:
	219	Dict of state group to state map.
	220	"""
	221
	222	member_filter, non_member_filter = state_filter.get_member_split()
	223
	224	# Now we look them up in the member and non-member caches
	225	(
	226	non_member_state,
	227	incomplete_groups_nm,
	228	) = self._get_state_for_groups_using_cache(
	229	groups, self._state_group_cache, state_filter=non_member_filter
	230	)
	231
	232	(member_state, incomplete_groups_m,) = self._get_state_for_groups_using_cache(
	233	groups, self._state_group_members_cache, state_filter=member_filter
	234	)
	235
	236	state = dict(non_member_state)
	237	for group in groups:
	238	state[group].update(member_state[group])
	239
	240	# Now fetch any missing groups from the database
	241
	242	incomplete_groups = incomplete_groups_m \| incomplete_groups_nm
	243
	244	if not incomplete_groups:
	245	return state
	246
	247	cache_sequence_nm = self._state_group_cache.sequence
	248	cache_sequence_m = self._state_group_members_cache.sequence
	249
	250	# Help the cache hit ratio by expanding the filter a bit
	251	db_state_filter = state_filter.return_expanded()
	252
	253	group_to_state_dict = await self._get_state_groups_from_groups(
	254	list(incomplete_groups), state_filter=db_state_filter
	255	)
	256
	257	# Now lets update the caches
	258	self._insert_into_cache(
	259	group_to_state_dict,
	260	db_state_filter,
	261	cache_seq_num_members=cache_sequence_m,
	262	cache_seq_num_non_members=cache_sequence_nm,
	263	)
	264
	265	# And finally update the result dict, by filtering out any extra
	266	# stuff we pulled out of the database.
	267	for group, group_state_dict in group_to_state_dict.items():
	268	# We just replace any existing entries, as we will have loaded
	269	# everything we need from the database anyway.
	270	state[group] = state_filter.filter_state(group_state_dict)
	271
	272	return state
	273
	274	def _get_state_for_groups_using_cache(
	275	self, groups: Iterable[int], cache: DictionaryCache, state_filter: StateFilter
	276	) -> Tuple[Dict[int, StateMap[str]], Set[int]]:
	277	"""Gets the state at each of a list of state groups, optionally
	278	filtering by type/state_key, querying from a specific cache.
	279
	280	Args:
	281	groups: list of state groups for which we want to get the state.
	282	cache: the cache of group ids to state dicts which
	283	we will pass through - either the normal state cache or the
	284	specific members state cache.
	285	state_filter: The state filter used to fetch state from the
	286	database.
	287
	288	Returns:
	289	Tuple of dict of state_group_id to state map of entries in the
	290	cache, and the state group ids either missing from the cache or
	291	incomplete.
	292	"""
	293	results = {}
	294	incomplete_groups = set()
	295	for group in set(groups):
	296	state_dict_ids, got_all = self._get_state_for_group_using_cache(
	297	cache, group, state_filter
	298	)
	299	results[group] = state_dict_ids
	300
	301	if not got_all:
	302	incomplete_groups.add(group)
	303
	304	return results, incomplete_groups
	305
	306	def _insert_into_cache(
	307	self,
	308	group_to_state_dict,
	309	state_filter,
	310	cache_seq_num_members,
	311	cache_seq_num_non_members,
	312	):
	313	"""Inserts results from querying the database into the relevant cache.
	314
	315	Args:
	316	group_to_state_dict (dict): The new entries pulled from database.
	317	Map from state group to state dict
	318	state_filter (StateFilter): The state filter used to fetch state
	319	from the database.
	320	cache_seq_num_members (int): Sequence number of member cache since
	321	last lookup in cache
	322	cache_seq_num_non_members (int): Sequence number of member cache since
	323	last lookup in cache
	324	"""
	325
	326	# We need to work out which types we've fetched from the DB for the
	327	# member vs non-member caches. This should be as accurate as possible,
	328	# but can be an underestimate (e.g. when we have wild cards)
	329
	330	member_filter, non_member_filter = state_filter.get_member_split()
	331	if member_filter.is_full():
	332	# We fetched all member events
	333	member_types = None
	334	else:
	335	# `concrete_types()` will only return a subset when there are wild
	336	# cards in the filter, but that's fine.
	337	member_types = member_filter.concrete_types()
	338
	339	if non_member_filter.is_full():
	340	# We fetched all non member events
	341	non_member_types = None
	342	else:
	343	non_member_types = non_member_filter.concrete_types()
	344
	345	for group, group_state_dict in group_to_state_dict.items():
	346	state_dict_members = {}
	347	state_dict_non_members = {}
	348
	349	for k, v in group_state_dict.items():
	350	if k[0] == EventTypes.Member:
	351	state_dict_members[k] = v
	352	else:
	353	state_dict_non_members[k] = v
	354
	355	self._state_group_members_cache.update(
	356	cache_seq_num_members,
	357	key=group,
	358	value=state_dict_members,
	359	fetched_keys=member_types,
	360	)
	361
	362	self._state_group_cache.update(
	363	cache_seq_num_non_members,
	364	key=group,
	365	value=state_dict_non_members,
	366	fetched_keys=non_member_types,
	367	)
	368
	369	def store_state_group(
	370	self, event_id, room_id, prev_group, delta_ids, current_state_ids
	371	):
	372	"""Store a new set of state, returning a newly assigned state group.
	373
	374	Args:
	375	event_id (str): The event ID for which the state was calculated
	376	room_id (str)
	377	prev_group (int\|None): A previous state group for the room, optional.
	378	delta_ids (dict\|None): The delta between state at `prev_group` and
	379	`current_state_ids`, if `prev_group` was given. Same format as
	380	`current_state_ids`.
	381	current_state_ids (dict): The state to store. Map of (type, state_key)
	382	to event_id.
	383
	384	Returns:
	385	Deferred[int]: The state group ID
	386	"""
	387
	388	def _store_state_group_txn(txn):
	389	if current_state_ids is None:
	390	# AFAIK, this can never happen
	391	raise Exception("current_state_ids cannot be None")
	392
	393	state_group = self._state_group_seq_gen.get_next_id_txn(txn)
	394
	395	self.db_pool.simple_insert_txn(
	396	txn,
	397	table="state_groups",
	398	values={"id": state_group, "room_id": room_id, "event_id": event_id},
	399	)
	400
	401	# We persist as a delta if we can, while also ensuring the chain
	402	# of deltas isn't tooo long, as otherwise read performance degrades.
	403	if prev_group:
	404	is_in_db = self.db_pool.simple_select_one_onecol_txn(
	405	txn,
	406	table="state_groups",
	407	keyvalues={"id": prev_group},
	408	retcol="id",
	409	allow_none=True,
	410	)
	411	if not is_in_db:
	412	raise Exception(
	413	"Trying to persist state with unpersisted prev_group: %r"
	414	% (prev_group,)
	415	)
	416
	417	potential_hops = self._count_state_group_hops_txn(txn, prev_group)
	418	if prev_group and potential_hops < MAX_STATE_DELTA_HOPS:
	419	self.db_pool.simple_insert_txn(
	420	txn,
	421	table="state_group_edges",
	422	values={"state_group": state_group, "prev_state_group": prev_group},
	423	)
	424
	425	self.db_pool.simple_insert_many_txn(
	426	txn,
	427	table="state_groups_state",
	428	values=[
	429	{
	430	"state_group": state_group,
	431	"room_id": room_id,
	432	"type": key[0],
	433	"state_key": key[1],
	434	"event_id": state_id,
	435	}
	436	for key, state_id in delta_ids.items()
	437	],
	438	)
	439	else:
	440	self.db_pool.simple_insert_many_txn(
	441	txn,
	442	table="state_groups_state",
	443	values=[
	444	{
	445	"state_group": state_group,
	446	"room_id": room_id,
	447	"type": key[0],
	448	"state_key": key[1],
	449	"event_id": state_id,
	450	}
	451	for key, state_id in current_state_ids.items()
	452	],
	453	)
	454
	455	# Prefill the state group caches with this group.
	456	# It's fine to use the sequence like this as the state group map
	457	# is immutable. (If the map wasn't immutable then this prefill could
	458	# race with another update)
	459
	460	current_member_state_ids = {
	461	s: ev
	462	for (s, ev) in current_state_ids.items()
	463	if s[0] == EventTypes.Member
	464	}
	465	txn.call_after(
	466	self._state_group_members_cache.update,
	467	self._state_group_members_cache.sequence,
	468	key=state_group,
	469	value=dict(current_member_state_ids),
	470	)
	471
	472	current_non_member_state_ids = {
	473	s: ev
	474	for (s, ev) in current_state_ids.items()
	475	if s[0] != EventTypes.Member
	476	}
	477	txn.call_after(
	478	self._state_group_cache.update,
	479	self._state_group_cache.sequence,
	480	key=state_group,
	481	value=dict(current_non_member_state_ids),
	482	)
	483
	484	return state_group
	485
	486	return self.db_pool.runInteraction("store_state_group", _store_state_group_txn)
	487
	488	def purge_unreferenced_state_groups(
	489	self, room_id: str, state_groups_to_delete
	490	) -> defer.Deferred:
	491	"""Deletes no longer referenced state groups and de-deltas any state
	492	groups that reference them.
	493
	494	Args:
	495	room_id: The room the state groups belong to (must all be in the
	496	same room).
	497	state_groups_to_delete (Collection[int]): Set of all state groups
	498	to delete.
	499	"""
	500
	501	return self.db_pool.runInteraction(
	502	"purge_unreferenced_state_groups",
	503	self._purge_unreferenced_state_groups,
	504	room_id,
	505	state_groups_to_delete,
	506	)
	507
	508	def _purge_unreferenced_state_groups(self, txn, room_id, state_groups_to_delete):
	509	logger.info(
	510	"[purge] found %i state groups to delete", len(state_groups_to_delete)
	511	)
	512
	513	rows = self.db_pool.simple_select_many_txn(
	514	txn,
	515	table="state_group_edges",
	516	column="prev_state_group",
	517	iterable=state_groups_to_delete,
	518	keyvalues={},
	519	retcols=("state_group",),
	520	)
	521
	522	remaining_state_groups = {
	523	row["state_group"]
	524	for row in rows
	525	if row["state_group"] not in state_groups_to_delete
	526	}
	527
	528	logger.info(
	529	"[purge] de-delta-ing %i remaining state groups",
	530	len(remaining_state_groups),
	531	)
	532
	533	# Now we turn the state groups that reference to-be-deleted state
	534	# groups to non delta versions.
	535	for sg in remaining_state_groups:
	536	logger.info("[purge] de-delta-ing remaining state group %s", sg)
	537	curr_state = self._get_state_groups_from_groups_txn(txn, [sg])
	538	curr_state = curr_state[sg]
	539
	540	self.db_pool.simple_delete_txn(
	541	txn, table="state_groups_state", keyvalues={"state_group": sg}
	542	)
	543
	544	self.db_pool.simple_delete_txn(
	545	txn, table="state_group_edges", keyvalues={"state_group": sg}
	546	)
	547
	548	self.db_pool.simple_insert_many_txn(
	549	txn,
	550	table="state_groups_state",
	551	values=[
	552	{
	553	"state_group": sg,
	554	"room_id": room_id,
	555	"type": key[0],
	556	"state_key": key[1],
	557	"event_id": state_id,
	558	}
	559	for key, state_id in curr_state.items()
	560	],
	561	)
	562
	563	logger.info("[purge] removing redundant state groups")
	564	txn.executemany(
	565	"DELETE FROM state_groups_state WHERE state_group = ?",
	566	((sg,) for sg in state_groups_to_delete),
	567	)
	568	txn.executemany(
	569	"DELETE FROM state_groups WHERE id = ?",
	570	((sg,) for sg in state_groups_to_delete),
	571	)
	572
	573	async def get_previous_state_groups(
	574	self, state_groups: Iterable[int]
	575	) -> Dict[int, int]:
	576	"""Fetch the previous groups of the given state groups.
	577
	578	Args:
	579	state_groups
	580
	581	Returns:
	582	A mapping from state group to previous state group.
	583	"""
	584
	585	rows = await self.db_pool.simple_select_many_batch(
	586	table="state_group_edges",
	587	column="prev_state_group",
	588	iterable=state_groups,
	589	keyvalues={},
	590	retcols=("prev_state_group", "state_group"),
	591	desc="get_previous_state_groups",
	592	)
	593
	594	return {row["state_group"]: row["prev_state_group"] for row in rows}
	595
	596	def purge_room_state(self, room_id, state_groups_to_delete):
	597	"""Deletes all record of a room from state tables
	598
	599	Args:
	600	room_id (str):
	601	state_groups_to_delete (list[int]): State groups to delete
	602	"""
	603
	604	return self.db_pool.runInteraction(
	605	"purge_room_state",
	606	self._purge_room_state_txn,
	607	room_id,
	608	state_groups_to_delete,
	609	)
	610
	611	def _purge_room_state_txn(self, txn, room_id, state_groups_to_delete):
	612	# first we have to delete the state groups states
	613	logger.info("[purge] removing %s from state_groups_state", room_id)
	614
	615	self.db_pool.simple_delete_many_txn(
	616	txn,
	617	table="state_groups_state",
	618	column="state_group",
	619	iterable=state_groups_to_delete,
	620	keyvalues={},
	621	)
	622
	623	# ... and the state group edges
	624	logger.info("[purge] removing %s from state_group_edges", room_id)
	625
	626	self.db_pool.simple_delete_many_txn(
	627	txn,
	628	table="state_group_edges",
	629	column="state_group",
	630	iterable=state_groups_to_delete,
	631	keyvalues={},
	632	)
	633
	634	# ... and the state groups
	635	logger.info("[purge] removing %s from state_groups", room_id)
	636
	637	self.db_pool.simple_delete_many_txn(
	638	txn,
	639	table="state_groups",
	640	column="id",
	641	iterable=state_groups_to_delete,
	642	keyvalues={},
	643	)

+23

-27

synapse/storage/persist_events.py less more

24	24	from twisted.internet import defer
25	25
26	26	from synapse.api.constants import EventTypes, Membership
27		from synapse.events import FrozenEvent
	27	from synapse.events import EventBase
28	28	from synapse.events.snapshot import EventContext
29	29	from synapse.logging.context import PreserveLoggingContext, make_deferred_yieldable
30	30	from synapse.metrics.background_process_metrics import run_as_background_process
31		from synapse.storage.data_stores import DataStores
32		from synapse.storage.data_stores.main.events import DeltaState
	31	from synapse.storage.databases import Databases
	32	from synapse.storage.databases.main.events import DeltaState
33	33	from synapse.types import StateMap
34	34	from synapse.util.async_helpers import ObservableDeferred
35	35	from synapse.util.metrics import Measure

178	178	current state and forward extremity changes.
179	179	"""
180	180
181		def __init__(self, hs, stores: DataStores):
	181	def __init__(self, hs, stores: Databases):
182	182	# We ultimately want to split out the state store from the main store,
183	183	# so we use separate variables here even though they point to the same
184	184	# store for now.

191	191	self._event_persist_queue = _EventPeristenceQueue()
192	192	self._state_resolution_handler = hs.get_state_resolution_handler()
193	193
194		@defer.inlineCallbacks
195		def persist_events(
	194	async def persist_events(
196	195	self,
197		events_and_contexts: List[Tuple[FrozenEvent, EventContext]],
	196	events_and_contexts: List[Tuple[EventBase, EventContext]],
198	197	backfilled: bool = False,
199		):
	198	) -> int:
200	199	"""
201	200	Write events to the database
202	201	Args:

206	205	which might update the current state etc.
207	206
208	207	Returns:
209		Deferred[int]: the stream ordering of the latest persisted event
	208	the stream ordering of the latest persisted event
210	209	"""
211	210	partitioned = {}
212	211	for event, ctx in events_and_contexts:

222	221	for room_id in partitioned:
223	222	self._maybe_start_persisting(room_id)
224	223
225		yield make_deferred_yieldable(
	224	await make_deferred_yieldable(
226	225	defer.gatherResults(deferreds, consumeErrors=True)
227	226	)
228	227
229		max_persisted_id = yield self.main_store.get_current_events_token()
230
231		return max_persisted_id
232
233		@defer.inlineCallbacks
234		def persist_event(
235		self, event: FrozenEvent, context: EventContext, backfilled: bool = False
236		):
	228	return self.main_store.get_current_events_token()
	229
	230	async def persist_event(
	231	self, event: EventBase, context: EventContext, backfilled: bool = False
	232	) -> Tuple[int, int]:
237	233	"""
238	234	Returns:
239		Deferred[Tuple[int, int]]: the stream ordering of ``event``,
240		and the stream ordering of the latest persisted event
	235	The stream ordering of `event`, and the stream ordering of the
	236	latest persisted event
241	237	"""
242	238	deferred = self._event_persist_queue.add_to_queue(
243	239	event.room_id, [(event, context)], backfilled=backfilled

245	241
246	242	self._maybe_start_persisting(event.room_id)
247	243
248		yield make_deferred_yieldable(deferred)
249
250		max_persisted_id = yield self.main_store.get_current_events_token()
	244	await make_deferred_yieldable(deferred)
	245
	246	max_persisted_id = self.main_store.get_current_events_token()
251	247	return (event.internal_metadata.stream_ordering, max_persisted_id)
252	248
253	249	def _maybe_start_persisting(self, room_id: str):

261	257
262	258	async def _persist_events(
263	259	self,
264		events_and_contexts: List[Tuple[FrozenEvent, EventContext]],
	260	events_and_contexts: List[Tuple[EventBase, EventContext]],
265	261	backfilled: bool = False,
266	262	):
267	263	"""Calculates the change to current state and forward extremities, and

438	434	async def _calculate_new_extremities(
439	435	self,
440	436	room_id: str,
441		event_contexts: List[Tuple[FrozenEvent, EventContext]],
	437	event_contexts: List[Tuple[EventBase, EventContext]],
442	438	latest_event_ids: List[str],
443	439	):
444	440	"""Calculates the new forward extremities for a room given events to

496	492	async def _get_new_state_after_events(
497	493	self,
498	494	room_id: str,
499		events_context: List[Tuple[FrozenEvent, EventContext]],
	495	events_context: List[Tuple[EventBase, EventContext]],
500	496	old_latest_event_ids: Iterable[str],
501	497	new_latest_event_ids: Iterable[str],
502	498	) -> Tuple[Optional[StateMap[str]], Optional[StateMap[str]]]:

682	678	async def _is_server_still_joined(
683	679	self,
684	680	room_id: str,
685		ev_ctx_rm: List[Tuple[FrozenEvent, EventContext]],
	681	ev_ctx_rm: List[Tuple[EventBase, EventContext]],
686	682	delta: DeltaState,
687	683	current_state: Optional[StateMap[str]],
688	684	potentially_left_users: Set[str],

+23

-25

synapse/storage/prepare_database.py less more

46	46	pass
47	47
48	48
49		def prepare_database(db_conn, database_engine, config, data_stores=["main", "state"]):
50		"""Prepares a database for usage. Will either create all necessary tables
	49	def prepare_database(db_conn, database_engine, config, databases=["main", "state"]):
	50	"""Prepares a physical database for usage. Will either create all necessary tables
51	51	or upgrade from an older schema version.
52	52
53	53	If `config` is None then prepare_database will assert that no upgrade is

59	59	config (synapse.config.homeserver.HomeServerConfig\|None):
60	60	application config, or None if we are connecting to an existing
61	61	database which we expect to be configured already
62		data_stores (list[str]): The name of the data stores that will be used
63		with this database. Defaults to all data stores.
	62	databases (list[str]): The name of the databases that will be used
	63	with this physical database. Defaults to all databases.
64	64	"""
65	65
66	66	try:

86	86	upgraded,
87	87	database_engine,
88	88	config,
89		data_stores=data_stores,
	89	databases=databases,
90	90	)
91	91	else:
92		_setup_new_database(cur, database_engine, data_stores=data_stores)
	92	_setup_new_database(cur, database_engine, databases=databases)
93	93
94	94	# check if any of our configured dynamic modules want a database
95	95	if config is not None:

102	102	raise
103	103
104	104
105		def _setup_new_database(cur, database_engine, data_stores):
106		"""Sets up the database by finding a base set of "full schemas" and then
107		applying any necessary deltas, including schemas from the given data
	105	def _setup_new_database(cur, database_engine, databases):
	106	"""Sets up the physical database by finding a base set of "full schemas" and
	107	then applying any necessary deltas, including schemas from the given data
108	108	stores.
109	109
110	110	The "full_schemas" directory has subdirectories named after versions. This

137	137	Args:
138	138	cur (Cursor): a database cursor
139	139	database_engine (DatabaseEngine)
140		data_stores (list[str]): The names of the data stores to instantiate
141		on the given database.
	140	databases (list[str]): The names of the databases to instantiate
	141	on the given physical database.
142	142	"""
143	143
144	144	# We're about to set up a brand new database so we check that its

175	175	directories.extend(
176	176	os.path.join(
177	177	dir_path,
178		"data_stores",
179		data_store,
	178	"databases",
	179	database,
180	180	"schema",
181	181	"full_schemas",
182	182	str(max_current_ver),
183	183	)
184		for data_store in data_stores
	184	for database in databases
185	185	)
186	186
187	187	directory_entries = []

218	218	upgraded=False,
219	219	database_engine=database_engine,
220	220	config=None,
221		data_stores=data_stores,
	221	databases=databases,
222	222	is_empty=True,
223	223	)
224	224

230	230	upgraded,
231	231	database_engine,
232	232	config,
233		data_stores,
	233	databases,
234	234	is_empty=False,
235	235	):
236		"""Upgrades an existing database.
	236	"""Upgrades an existing physical database.
237	237
238	238	Delta files can either be SQL stored in *.sql files, or python modules
239	239	in *.py.

284	284	config (synapse.config.homeserver.HomeServerConfig\|None):
285	285	None if we are initialising a blank database, otherwise the application
286	286	config
287		data_stores (list[str]): The names of the data stores to instantiate
288		on the given database.
	287	databases (list[str]): The names of the databases to instantiate
	288	on the given physical database.
289	289	is_empty (bool): Is this a blank database? I.e. do we need to run the
290	290	upgrade portions of the delta scripts.
291	291	"""

302	302
303	303	# some of the deltas assume that config.server_name is set correctly, so now
304	304	# is a good time to run the sanity check.
305		if not is_empty and "main" in data_stores:
306		from synapse.storage.data_stores.main import check_database_before_upgrade
	305	if not is_empty and "main" in databases:
	306	from synapse.storage.databases.main import check_database_before_upgrade
307	307
308	308	check_database_before_upgrade(cur, database_engine, config)
309	309

329	329	# First we find the directories to search in
330	330	delta_dir = os.path.join(dir_path, "schema", "delta", str(v))
331	331	directories = [delta_dir]
332		for data_store in data_stores:
	332	for database in databases:
333	333	directories.append(
334		os.path.join(
335		dir_path, "data_stores", data_store, "schema", "delta", str(v)
336		)
	334	os.path.join(dir_path, "databases", database, "schema", "delta", str(v))
337	335	)
338	336
339	337	# Used to check if we have any duplicate file names

+18

-20

synapse/storage/purge_events.py less more

14	14
15	15	import itertools
16	16	import logging
17
18		from twisted.internet import defer
	17	from typing import Set
19	18
20	19	logger = logging.getLogger(__name__)
21	20

27	26	def __init__(self, hs, stores):
28	27	self.stores = stores
29	28
30		@defer.inlineCallbacks
31		def purge_room(self, room_id: str):
	29	async def purge_room(self, room_id: str):
32	30	"""Deletes all record of a room
33	31	"""
34	32
35		state_groups_to_delete = yield self.stores.main.purge_room(room_id)
36		yield self.stores.state.purge_room_state(room_id, state_groups_to_delete)
	33	state_groups_to_delete = await self.stores.main.purge_room(room_id)
	34	await self.stores.state.purge_room_state(room_id, state_groups_to_delete)
37	35
38		@defer.inlineCallbacks
39		def purge_history(self, room_id, token, delete_local_events):
	36	async def purge_history(
	37	self, room_id: str, token: str, delete_local_events: bool
	38	) -> None:
40	39	"""Deletes room history before a certain point
41	40
42	41	Args:
43		room_id (str):
	42	room_id: The room ID
44	43
45		token (str): A topological token to delete events before
	44	token: A topological token to delete events before
46	45
47		delete_local_events (bool):
	46	delete_local_events:
48	47	if True, we will delete local events as well as remote ones
49	48	(instead of just marking them as outliers and deleting their
50	49	state groups).
51	50	"""
52		state_groups = yield self.stores.main.purge_history(
	51	state_groups = await self.stores.main.purge_history(
53	52	room_id, token, delete_local_events
54	53	)
55	54
56	55	logger.info("[purge] finding state groups that can be deleted")
57	56
58		sg_to_delete = yield self._find_unreferenced_groups(state_groups)
	57	sg_to_delete = await self._find_unreferenced_groups(state_groups)
59	58
60		yield self.stores.state.purge_unreferenced_state_groups(room_id, sg_to_delete)
	59	await self.stores.state.purge_unreferenced_state_groups(room_id, sg_to_delete)
61	60
62		@defer.inlineCallbacks
63		def _find_unreferenced_groups(self, state_groups):
	61	async def _find_unreferenced_groups(self, state_groups: Set[int]) -> Set[int]:
64	62	"""Used when purging history to figure out which state groups can be
65	63	deleted.
66	64
67	65	Args:
68		state_groups (set[int]): Set of state groups referenced by events
	66	state_groups: Set of state groups referenced by events
69	67	that are going to be deleted.
70	68
71	69	Returns:
72		Deferred[set[int]] The set of state groups that can be deleted.
	70	The set of state groups that can be deleted.
73	71	"""
74	72	# Graph of state group -> previous group
75	73	graph = {}

92	90	current_search = set(itertools.islice(next_to_search, 100))
93	91	next_to_search -= current_search
94	92
95		referenced = yield self.stores.main.get_referenced_state_groups(
	93	referenced = await self.stores.main.get_referenced_state_groups(
96	94	current_search
97	95	)
98	96	referenced_groups \|= referenced

101	99	# groups that are referenced.
102	100	current_search -= referenced
103	101
104		edges = yield self.stores.state.get_previous_state_groups(current_search)
	102	edges = await self.stores.state.get_previous_state_groups(current_search)
105	103
106	104	prevs = set(edges.values())
107	105	# We don't bother re-handling groups we've already seen

+143

-131

synapse/storage/state.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
16		from typing import Iterable, List, TypeVar
	16	from typing import Awaitable, Dict, Iterable, List, Optional, Set, Tuple, TypeVar
17	17
18	18	import attr
19	19
20		from twisted.internet import defer
21
22	20	from synapse.api.constants import EventTypes
	21	from synapse.events import EventBase
23	22	from synapse.types import StateMap
24	23
25	24	logger = logging.getLogger(__name__)

33	32	"""A filter used when querying for state.
34	33
35	34	Attributes:
36		types (dict[str, set[str]\|None]): Map from type to set of state keys (or
37		None). This specifies which state_keys for the given type to fetch
38		from the DB. If None then all events with that type are fetched. If
39		the set is empty then no events with that type are fetched.
40		include_others (bool): Whether to fetch events with types that do not
	35	types: Map from type to set of state keys (or None). This specifies
	36	which state_keys for the given type to fetch from the DB. If None
	37	then all events with that type are fetched. If the set is empty
	38	then no events with that type are fetched.
	39	include_others: Whether to fetch events with types that do not
41	40	appear in `types`.
42	41	"""
43	42
44		types = attr.ib()
45		include_others = attr.ib(default=False)
	43	types = attr.ib(type=Dict[str, Optional[Set[str]]])
	44	include_others = attr.ib(default=False, type=bool)
46	45
47	46	def __attrs_post_init__(self):
48	47	# If `include_others` is set we canonicalise the filter by removing

51	50	self.types = {k: v for k, v in self.types.items() if v is not None}
52	51
53	52	@staticmethod
54		def all():
	53	def all() -> "StateFilter":
55	54	"""Creates a filter that fetches everything.
56	55
57	56	Returns:
58		StateFilter
	57	The new state filter.
59	58	"""
60	59	return StateFilter(types={}, include_others=True)
61	60
62	61	@staticmethod
63		def none():
	62	def none() -> "StateFilter":
64	63	"""Creates a filter that fetches nothing.
65	64
66	65	Returns:
67		StateFilter
	66	The new state filter.
68	67	"""
69	68	return StateFilter(types={}, include_others=False)
70	69
71	70	@staticmethod
72		def from_types(types):
	71	def from_types(types: Iterable[Tuple[str, Optional[str]]]) -> "StateFilter":
73	72	"""Creates a filter that only fetches the given types
74	73
75	74	Args:
76		types (Iterable[tuple[str, str\|None]]): A list of type and state
77		keys to fetch. A state_key of None fetches everything for
78		that type
79
80		Returns:
81		StateFilter
82		"""
83		type_dict = {}
	75	types: A list of type and state keys to fetch. A state_key of None
	76	fetches everything for that type
	77
	78	Returns:
	79	The new state filter.
	80	"""
	81	type_dict = {} # type: Dict[str, Optional[Set[str]]]
84	82	for typ, s in types:
85	83	if typ in type_dict:
86	84	if type_dict[typ] is None:

90	88	type_dict[typ] = None
91	89	continue
92	90
93		type_dict.setdefault(typ, set()).add(s)
	91	type_dict.setdefault(typ, set()).add(s) # type: ignore
94	92
95	93	return StateFilter(types=type_dict)
96	94
97	95	@staticmethod
98		def from_lazy_load_member_list(members):
	96	def from_lazy_load_member_list(members: Iterable[str]) -> "StateFilter":
99	97	"""Creates a filter that returns all non-member events, plus the member
100	98	events for the given users
101	99
102	100	Args:
103		members (iterable[str]): Set of user IDs
104
105		Returns:
106		StateFilter
	101	members: Set of user IDs
	102
	103	Returns:
	104	The new state filter
107	105	"""
108	106	return StateFilter(types={EventTypes.Member: set(members)}, include_others=True)
109	107
110		def return_expanded(self):
	108	def return_expanded(self) -> "StateFilter":
111	109	"""Creates a new StateFilter where type wild cards have been removed
112	110	(except for memberships). The returned filter is a superset of the
113	111	current one, i.e. anything that passes the current filter will pass

129	127	return all non-member events
130	128
131	129	Returns:
132		StateFilter
	130	The new state filter.
133	131	"""
134	132
135	133	if self.is_full():

166	164	include_others=True,
167	165	)
168	166
169		def make_sql_filter_clause(self):
	167	def make_sql_filter_clause(self) -> Tuple[str, List[str]]:
170	168	"""Converts the filter to an SQL clause.
171	169
172	170	For example:

178	176
179	177
180	178	Returns:
181		tuple[str, list]: The SQL string (may be empty) and arguments. An
182		empty SQL string is returned when the filter matches everything
183		(i.e. is "full").
	179	The SQL string (may be empty) and arguments. An empty SQL string is
	180	returned when the filter matches everything (i.e. is "full").
184	181	"""
185	182
186	183	where_clause = ""
187		where_args = []
	184	where_args = [] # type: List[str]
188	185
189	186	if self.is_full():
190	187	return where_clause, where_args

220	217
221	218	return where_clause, where_args
222	219
223		def max_entries_returned(self):
	220	def max_entries_returned(self) -> Optional[int]:
224	221	"""Returns the maximum number of entries this filter will return if
225	222	known, otherwise returns None.
226	223

259	256
260	257	return filtered_state
261	258
262		def is_full(self):
	259	def is_full(self) -> bool:
263	260	"""Whether this filter fetches everything or not
264	261
265	262	Returns:
266		bool
	263	True if the filter fetches everything.
267	264	"""
268	265	return self.include_others and not self.types
269	266
270		def has_wildcards(self):
	267	def has_wildcards(self) -> bool:
271	268	"""Whether the filter includes wildcards or is attempting to fetch
272	269	specific state.
273	270
274	271	Returns:
275		bool
	272	True if the filter includes wildcards.
276	273	"""
277	274
278	275	return self.include_others or any(
279	276	state_keys is None for state_keys in self.types.values()
280	277	)
281	278
282		def concrete_types(self):
	279	def concrete_types(self) -> List[Tuple[str, str]]:
283	280	"""Returns a list of concrete type/state_keys (i.e. not None) that
284	281	will be fetched. This will be a complete list if `has_wildcards`
285	282	returns False, but otherwise will be a subset (or even empty).
286	283
287	284	Returns:
288		list[tuple[str,str]]
	285	A list of type/state_keys tuples.
289	286	"""
290	287	return [
291	288	(t, s)

294	291	for s in state_keys
295	292	]
296	293
297		def get_member_split(self):
	294	def get_member_split(self) -> Tuple["StateFilter", "StateFilter"]:
298	295	"""Return the filter split into two: one which assumes it's exclusively
299	296	matching against member state, and one which assumes it's matching
300	297	against non member state.

306	303	state caches).
307	304
308	305	Returns:
309		tuple[StateFilter, StateFilter]: The member and non member filters
	306	The member and non member filters
310	307	"""
311	308
312	309	if EventTypes.Member in self.types:

339	336	"""Given a state group try to return a previous group and a delta between
340	337	the old and the new.
341	338
	339	Args:
	340	state_group: The state group used to retrieve state deltas.
	341
342	342	Returns:
343	343	Deferred[Tuple[Optional[int], Optional[StateMap[str]]]]:
344	344	(prev_group, delta_ids)

346	346
347	347	return self.stores.state.get_state_group_delta(state_group)
348	348
349		@defer.inlineCallbacks
350		def get_state_groups_ids(self, _room_id, event_ids):
	349	async def get_state_groups_ids(
	350	self, _room_id: str, event_ids: Iterable[str]
	351	) -> Dict[int, StateMap[str]]:
351	352	"""Get the event IDs of all the state for the state groups for the given events
352	353
353	354	Args:
354		_room_id (str): id of the room for these events
355		event_ids (iterable[str]): ids of the events
356
357		Returns:
358		Deferred[dict[int, StateMap[str]]]:
359		dict of state_group_id -> (dict of (type, state_key) -> event id)
	355	_room_id: id of the room for these events
	356	event_ids: ids of the events
	357
	358	Returns:
	359	dict of state_group_id -> (dict of (type, state_key) -> event id)
360	360	"""
361	361	if not event_ids:
362	362	return {}
363	363
364		event_to_groups = yield self.stores.main._get_state_group_for_events(event_ids)
	364	event_to_groups = await self.stores.main._get_state_group_for_events(event_ids)
365	365
366	366	groups = set(event_to_groups.values())
367		group_to_state = yield self.stores.state._get_state_for_groups(groups)
	367	group_to_state = await self.stores.state._get_state_for_groups(groups)
368	368
369	369	return group_to_state
370	370
371		@defer.inlineCallbacks
372		def get_state_ids_for_group(self, state_group):
	371	async def get_state_ids_for_group(self, state_group: int) -> StateMap[str]:
373	372	"""Get the event IDs of all the state in the given state group
374	373
375	374	Args:
376		state_group (int)
377
378		Returns:
379		Deferred[dict]: Resolves to a map of (type, state_key) -> event_id
380		"""
381		group_to_state = yield self._get_state_for_groups((state_group,))
	375	state_group: A state group for which we want to get the state IDs.
	376
	377	Returns:
	378	Resolves to a map of (type, state_key) -> event_id
	379	"""
	380	group_to_state = await self._get_state_for_groups((state_group,))
382	381
383	382	return group_to_state[state_group]
384	383
385		@defer.inlineCallbacks
386		def get_state_groups(self, room_id, event_ids):
	384	async def get_state_groups(
	385	self, room_id: str, event_ids: Iterable[str]
	386	) -> Dict[int, List[EventBase]]:
387	387	""" Get the state groups for the given list of event_ids
388		Returns:
389		Deferred[dict[int, list[EventBase]]]:
390		dict of state_group_id -> list of state events.
	388
	389	Args:
	390	room_id: ID of the room for these events.
	391	event_ids: The event IDs to retrieve state for.
	392
	393	Returns:
	394	dict of state_group_id -> list of state events.
391	395	"""
392	396	if not event_ids:
393	397	return {}
394	398
395		group_to_ids = yield self.get_state_groups_ids(room_id, event_ids)
396
397		state_event_map = yield self.stores.main.get_events(
	399	group_to_ids = await self.get_state_groups_ids(room_id, event_ids)
	400
	401	state_event_map = await self.stores.main.get_events(
398	402	[
399	403	ev_id
400	404	for group_ids in group_to_ids.values()

414	418
415	419	def _get_state_groups_from_groups(
416	420	self, groups: List[int], state_filter: StateFilter
417		):
	421	) -> Awaitable[Dict[int, StateMap[str]]]:
418	422	"""Returns the state groups for a given set of groups, filtering on
419	423	types of state events.
420	424

422	426	groups: list of state group IDs to query
423	427	state_filter: The state filter used to fetch state
424	428	from the database.
425		Returns:
426		Deferred[Dict[int, StateMap[str]]]: Dict of state group to state map.
	429
	430	Returns:
	431	Dict of state group to state map.
427	432	"""
428	433
429	434	return self.stores.state._get_state_groups_from_groups(groups, state_filter)
430	435
431		@defer.inlineCallbacks
432		def get_state_for_events(self, event_ids, state_filter=StateFilter.all()):
	436	async def get_state_for_events(
	437	self, event_ids: List[str], state_filter: StateFilter = StateFilter.all()
	438	):
433	439	"""Given a list of event_ids and type tuples, return a list of state
434	440	dicts for each event.
435		Args:
436		event_ids (list[string])
437		state_filter (StateFilter): The state filter used to fetch state
438		from the database.
439		Returns:
440		deferred: A dict of (event_id) -> (type, state_key) -> [state_events]
441		"""
442		event_to_groups = yield self.stores.main._get_state_group_for_events(event_ids)
	441
	442	Args:
	443	event_ids: The events to fetch the state of.
	444	state_filter: The state filter used to fetch state.
	445
	446	Returns:
	447	A dict of (event_id) -> (type, state_key) -> [state_events]
	448	"""
	449	event_to_groups = await self.stores.main._get_state_group_for_events(event_ids)
443	450
444	451	groups = set(event_to_groups.values())
445		group_to_state = yield self.stores.state._get_state_for_groups(
	452	group_to_state = await self.stores.state._get_state_for_groups(
446	453	groups, state_filter
447	454	)
448	455
449		state_event_map = yield self.stores.main.get_events(
	456	state_event_map = await self.stores.main.get_events(
450	457	[ev_id for sd in group_to_state.values() for ev_id in sd.values()],
451	458	get_prev_content=False,
452	459	)

462	469
463	470	return {event: event_to_state[event] for event in event_ids}
464	471
465		@defer.inlineCallbacks
466		def get_state_ids_for_events(self, event_ids, state_filter=StateFilter.all()):
	472	async def get_state_ids_for_events(
	473	self, event_ids: List[str], state_filter: StateFilter = StateFilter.all()
	474	):
467	475	"""
468	476	Get the state dicts corresponding to a list of events, containing the event_ids
469	477	of the state events (as opposed to the events themselves)
470	478
471	479	Args:
472		event_ids(list(str)): events whose state should be returned
473		state_filter (StateFilter): The state filter used to fetch state
474		from the database.
475
476		Returns:
477		A deferred dict from event_id -> (type, state_key) -> event_id
478		"""
479		event_to_groups = yield self.stores.main._get_state_group_for_events(event_ids)
	480	event_ids: events whose state should be returned
	481	state_filter: The state filter used to fetch state from the database.
	482
	483	Returns:
	484	A dict from event_id -> (type, state_key) -> event_id
	485	"""
	486	event_to_groups = await self.stores.main._get_state_group_for_events(event_ids)
480	487
481	488	groups = set(event_to_groups.values())
482		group_to_state = yield self.stores.state._get_state_for_groups(
	489	group_to_state = await self.stores.state._get_state_for_groups(
483	490	groups, state_filter
484	491	)
485	492

490	497
491	498	return {event: event_to_state[event] for event in event_ids}
492	499
493		@defer.inlineCallbacks
494		def get_state_for_event(self, event_id, state_filter=StateFilter.all()):
	500	async def get_state_for_event(
	501	self, event_id: str, state_filter: StateFilter = StateFilter.all()
	502	):
495	503	"""
496	504	Get the state dict corresponding to a particular event
497	505
498	506	Args:
499		event_id(str): event whose state should be returned
500		state_filter (StateFilter): The state filter used to fetch state
501		from the database.
	507	event_id: event whose state should be returned
	508	state_filter: The state filter used to fetch state from the database.
	509
	510	Returns:
	511	A dict from (type, state_key) -> state_event
	512	"""
	513	state_map = await self.get_state_for_events([event_id], state_filter)
	514	return state_map[event_id]
	515
	516	async def get_state_ids_for_event(
	517	self, event_id: str, state_filter: StateFilter = StateFilter.all()
	518	):
	519	"""
	520	Get the state dict corresponding to a particular event
	521
	522	Args:
	523	event_id: event whose state should be returned
	524	state_filter: The state filter used to fetch state from the database.
502	525
503	526	Returns:
504	527	A deferred dict from (type, state_key) -> state_event
505	528	"""
506		state_map = yield self.get_state_for_events([event_id], state_filter)
507		return state_map[event_id]
508
509		@defer.inlineCallbacks
510		def get_state_ids_for_event(self, event_id, state_filter=StateFilter.all()):
511		"""
512		Get the state dict corresponding to a particular event
513
514		Args:
515		event_id(str): event whose state should be returned
516		state_filter (StateFilter): The state filter used to fetch state
517		from the database.
518
519		Returns:
520		A deferred dict from (type, state_key) -> state_event
521		"""
522		state_map = yield self.get_state_ids_for_events([event_id], state_filter)
	529	state_map = await self.get_state_ids_for_events([event_id], state_filter)
523	530	return state_map[event_id]
524	531
525	532	def _get_state_for_groups(
526	533	self, groups: Iterable[int], state_filter: StateFilter = StateFilter.all()
527		):
	534	) -> Awaitable[Dict[int, StateMap[str]]]:
528	535	"""Gets the state at each of a list of state groups, optionally
529	536	filtering by type/state_key
530	537
531	538	Args:
532		groups (iterable[int]): list of state groups for which we want
533		to get the state.
534		state_filter (StateFilter): The state filter used to fetch state
	539	groups: list of state groups for which we want to get the state.
	540	state_filter: The state filter used to fetch state.
535	541	from the database.
536		Returns:
537		Deferred[dict[int, StateMap[str]]]: Dict of state group to state map.
	542
	543	Returns:
	544	Dict of state group to state map.
538	545	"""
539	546	return self.stores.state._get_state_for_groups(groups, state_filter)
540	547
541	548	def store_state_group(
542		self, event_id, room_id, prev_group, delta_ids, current_state_ids
	549	self,
	550	event_id: str,
	551	room_id: str,
	552	prev_group: Optional[int],
	553	delta_ids: Optional[dict],
	554	current_state_ids: dict,
543	555	):
544	556	"""Store a new set of state, returning a newly assigned state group.
545	557
546	558	Args:
547		event_id (str): The event ID for which the state was calculated
548		room_id (str)
549		prev_group (int\|None): A previous state group for the room, optional.
550		delta_ids (dict\|None): The delta between state at `prev_group` and
	559	event_id: The event ID for which the state was calculated.
	560	room_id: ID of the room for which the state was calculated.
	561	prev_group: A previous state group for the room, optional.
	562	delta_ids: The delta between state at `prev_group` and
551	563	`current_state_ids`, if `prev_group` was given. Same format as
552	564	`current_state_ids`.
553		current_state_ids (dict): The state to store. Map of (type, state_key)
	565	current_state_ids: The state to store. Map of (type, state_key)
554	566	to event_id.
555	567
556	568	Returns:

+2

-2

synapse/storage/util/id_generators.py less more

19	19
20	20	from typing_extensions import Deque
21	21
22		from synapse.storage.database import Database, LoggingTransaction
	22	from synapse.storage.database import DatabasePool, LoggingTransaction
23	23	from synapse.storage.util.sequence import PostgresSequenceGenerator
24	24
25	25

238	238	def __init__(
239	239	self,
240	240	db_conn,
241		db: Database,
	241	db: DatabasePool,
242	242	instance_name: str,
243	243	table: str,
244	244	instance_column: str,

+9

-13

synapse/streams/events.py less more

13	13	# limitations under the License.
14	14
15	15	from typing import Any, Dict
16
17		from twisted.internet import defer
18	16
19	17	from synapse.handlers.account_data import AccountDataEventSource
20	18	from synapse.handlers.presence import PresenceEventSource

39	37	} # type: Dict[str, Any]
40	38	self.store = hs.get_datastore()
41	39
42		@defer.inlineCallbacks
43		def get_current_token(self):
	40	def get_current_token(self) -> StreamToken:
44	41	push_rules_key, _ = self.store.get_push_rules_stream_token()
45	42	to_device_key = self.store.get_to_device_stream_token()
46	43	device_list_key = self.store.get_device_stream_token()
47	44	groups_key = self.store.get_group_stream_token()
48	45
49	46	token = StreamToken(
50		room_key=(yield self.sources["room"].get_current_key()),
51		presence_key=(yield self.sources["presence"].get_current_key()),
52		typing_key=(yield self.sources["typing"].get_current_key()),
53		receipt_key=(yield self.sources["receipt"].get_current_key()),
54		account_data_key=(yield self.sources["account_data"].get_current_key()),
	47	room_key=self.sources["room"].get_current_key(),
	48	presence_key=self.sources["presence"].get_current_key(),
	49	typing_key=self.sources["typing"].get_current_key(),
	50	receipt_key=self.sources["receipt"].get_current_key(),
	51	account_data_key=self.sources["account_data"].get_current_key(),
55	52	push_rules_key=push_rules_key,
56	53	to_device_key=to_device_key,
57	54	device_list_key=device_list_key,

59	56	)
60	57	return token
61	58
62		@defer.inlineCallbacks
63		def get_current_token_for_pagination(self):
	59	def get_current_token_for_pagination(self) -> StreamToken:
64	60	"""Get the current token for a given room to be used to paginate
65	61	events.
66	62

68	64	than `room`, since they are not used during pagination.
69	65
70	66	Returns:
71		Deferred[StreamToken]
	67	The current token for pagination.
72	68	"""
73	69	token = StreamToken(
74		room_key=(yield self.sources["room"].get_current_key()),
	70	room_key=self.sources["room"].get_current_key(),
75	71	presence_key=0,
76	72	typing_key=0,
77	73	receipt_key=0,

+16

-7

synapse/types.py less more

12	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
	15	import abc
15	16	import re
16	17	import string
17	18	import sys
18	19	from collections import namedtuple
19		from typing import Any, Dict, Tuple, TypeVar
	20	from typing import Any, Dict, Tuple, Type, TypeVar
20	21
21	22	import attr
22	23	from signedjson.key import decode_verify_key_bytes

32	33
33	34	T_co = TypeVar("T_co", covariant=True)
34	35
35		class Collection(Iterable[T_co], Container[T_co], Sized):
	36	class Collection(Iterable[T_co], Container[T_co], Sized): # type: ignore
36	37	__slots__ = ()
37	38
38	39

140	141	return string[1:idx]
141	142
142	143
	144	DS = TypeVar("DS", bound="DomainSpecificString")
	145
	146
143	147	class DomainSpecificString(namedtuple("DomainSpecificString", ("localpart", "domain"))):
144	148	"""Common base class among ID/name strings that have a local part and a
145	149	domain name, prefixed with a sigil.

149	153	'localpart' : The local part of the name (without the leading sigil)
150	154	'domain' : The domain part of the name
151	155	"""
	156
	157	__metaclass__ = abc.ABCMeta
	158
	159	SIGIL = abc.abstractproperty() # type: str # type: ignore
152	160
153	161	# Deny iteration because it will bite you if you try to create a singleton
154	162	# set by:

165	173	return self
166	174
167	175	@classmethod
168		def from_string(cls, s: str):
	176	def from_string(cls: Type[DS], s: str) -> DS:
169	177	"""Parse the string given by 's' into a structure object."""
170	178	if len(s) < 1 or s[0:1] != cls.SIGIL:
171	179	raise SynapseError(

189	197	# names on one HS
190	198	return cls(localpart=parts[0], domain=domain)
191	199
192		def to_string(self):
	200	def to_string(self) -> str:
193	201	"""Return a string encoding the fields of the structure object."""
194	202	return "%s%s:%s" % (self.SIGIL, self.localpart, self.domain)
195	203
196	204	@classmethod
197		def is_valid(cls, s):
	205	def is_valid(cls: Type[DS], s: str) -> bool:
198	206	try:
199	207	cls.from_string(s)
200	208	return True

234	242	SIGIL = "+"
235	243
236	244	@classmethod
237		def from_string(cls, s):
238		group_id = super(GroupID, cls).from_string(s)
	245	def from_string(cls: Type[DS], s: str) -> DS:
	246	group_id = super().from_string(s) # type: DS # type: ignore
	247
239	248	if not group_id.localpart:
240	249	raise SynapseError(400, "Group ID cannot be empty", Codes.INVALID_PARAM)
241	250

+4

-0

synapse/util/__init__.py less more

16	16	import re
17	17
18	18	import attr
	19	from canonicaljson import json
19	20
20	21	from twisted.internet import defer, task
21	22
22	23	from synapse.logging import context
23	24
24	25	logger = logging.getLogger(__name__)
	26
	27	# Create a custom encoder to reduce the whitespace produced by JSON encoding.
	28	json_encoder = json.JSONEncoder(separators=(",", ":"))
25	29
26	30
27	31	def unwrapFirstError(failure):

+1

-1

synapse/util/caches/descriptors.py less more

191	191	callbacks = [callback] if callback else []
192	192	self.check_thread()
193	193	observable = ObservableDeferred(value, consumeErrors=True)
194		observer = defer.maybeDeferred(observable.observe)
	194	observer = observable.observe()
195	195	entry = CacheEntry(deferred=observable, callbacks=callbacks)
196	196
197	197	existing_entry = self._pending_deferred_cache.pop(key, None)

+137

-0

synapse/util/daemonize.py less more

	0	# -- coding: utf-8 --
	1	# Copyright (c) 2012, 2013, 2014 Ilya Otyutskiy <ilya.otyutskiy@icloud.com>
	2	# Copyright 2020 The Matrix.org Foundation C.I.C.
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import atexit
	17	import fcntl
	18	import logging
	19	import os
	20	import signal
	21	import sys
	22
	23
	24	def daemonize_process(pid_file: str, logger: logging.Logger, chdir: str = "/") -> None:
	25	"""daemonize the current process
	26
	27	This calls fork(), and has the main process exit. When it returns we will be
	28	running in the child process.
	29	"""
	30
	31	# If pidfile already exists, we should read pid from there; to overwrite it, if
	32	# locking will fail, because locking attempt somehow purges the file contents.
	33	if os.path.isfile(pid_file):
	34	with open(pid_file, "r") as pid_fh:
	35	old_pid = pid_fh.read()
	36
	37	# Create a lockfile so that only one instance of this daemon is running at any time.
	38	try:
	39	lock_fh = open(pid_file, "w")
	40	except IOError:
	41	print("Unable to create the pidfile.")
	42	sys.exit(1)
	43
	44	try:
	45	# Try to get an exclusive lock on the file. This will fail if another process
	46	# has the file locked.
	47	fcntl.flock(lock_fh, fcntl.LOCK_EX \| fcntl.LOCK_NB)
	48	except IOError:
	49	print("Unable to lock on the pidfile.")
	50	# We need to overwrite the pidfile if we got here.
	51	#
	52	# XXX better to avoid overwriting it, surely. this looks racey as the pid file
	53	# could be created between us trying to read it and us trying to lock it.
	54	with open(pid_file, "w") as pid_fh:
	55	pid_fh.write(old_pid)
	56	sys.exit(1)
	57
	58	# Fork, creating a new process for the child.
	59	process_id = os.fork()
	60
	61	if process_id != 0:
	62	# parent process: exit.
	63
	64	# we use os._exit to avoid running the atexit handlers. In particular, that
	65	# means we don't flush the logs. This is important because if we are using
	66	# a MemoryHandler, we could have logs buffered which are now buffered in both
	67	# the main and the child process, so if we let the main process flush the logs,
	68	# we'll get two copies.
	69	os._exit(0)
	70
	71	# This is the child process. Continue.
	72
	73	# Stop listening for signals that the parent process receives.
	74	# This is done by getting a new process id.
	75	# setpgrp() is an alternative to setsid().
	76	# setsid puts the process in a new parent group and detaches its controlling
	77	# terminal.
	78
	79	os.setsid()
	80
	81	# point stdin, stdout, stderr at /dev/null
	82	devnull = "/dev/null"
	83	if hasattr(os, "devnull"):
	84	# Python has set os.devnull on this system, use it instead as it might be
	85	# different than /dev/null.
	86	devnull = os.devnull
	87
	88	devnull_fd = os.open(devnull, os.O_RDWR)
	89	os.dup2(devnull_fd, 0)
	90	os.dup2(devnull_fd, 1)
	91	os.dup2(devnull_fd, 2)
	92	os.close(devnull_fd)
	93
	94	# now that we have redirected stderr to /dev/null, any uncaught exceptions will
	95	# get sent to /dev/null, so make sure we log them.
	96	#
	97	# (we don't normally expect reactor.run to raise any exceptions, but this will
	98	# also catch any other uncaught exceptions before we get that far.)
	99
	100	def excepthook(type_, value, traceback):
	101	logger.critical("Unhanded exception", exc_info=(type_, value, traceback))
	102
	103	sys.excepthook = excepthook
	104
	105	# Set umask to default to safe file permissions when running as a root daemon. 027
	106	# is an octal number which we are typing as 0o27 for Python3 compatibility.
	107	os.umask(0o27)
	108
	109	# Change to a known directory. If this isn't done, starting a daemon in a
	110	# subdirectory that needs to be deleted results in "directory busy" errors.
	111	os.chdir(chdir)
	112
	113	try:
	114	lock_fh.write("%s" % (os.getpid()))
	115	lock_fh.flush()
	116	except IOError:
	117	logger.error("Unable to write pid to the pidfile.")
	118	print("Unable to write pid to the pidfile.")
	119	sys.exit(1)
	120
	121	# write a log line on SIGTERM.
	122	def sigterm(signum, frame):
	123	logger.warning("Caught signal %s. Stopping daemon." % signum)
	124	sys.exit(0)
	125
	126	signal.signal(signal.SIGTERM, sigterm)
	127
	128	# Cleanup pid file at exit.
	129	def exit():
	130	logger.warning("Stopping daemon.")
	131	os.remove(pid_file)
	132	sys.exit(0)
	133
	134	atexit.register(exit)
	135
	136	logger.warning("Starting daemon.")

+5

-2

synapse/util/frozenutils.py less more

62	62	)
63	63
64	64
65		# A JSONEncoder which is capable of encoding frozendicts without barfing
66		frozendict_json_encoder = json.JSONEncoder(default=_handle_frozendict)
	65	# A JSONEncoder which is capable of encoding frozendicts without barfing.
	66	# Additionally reduce the whitespace produced by JSON encoding.
	67	frozendict_json_encoder = json.JSONEncoder(
	68	default=_handle_frozendict, separators=(",", ":"),
	69	)

+28

-22

synapse/util/metrics.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		import inspect
16	15	import logging
17	16	from functools import wraps
	17	from typing import Any, Callable, Optional, TypeVar, cast
18	18
19	19	from prometheus_client import Counter
20
21		from twisted.internet import defer
22	20
23	21	from synapse.logging.context import LoggingContext, current_context
24	22	from synapse.metrics import InFlightGauge

59	57	sub_metrics=["real_time_max", "real_time_sum"],
60	58	)
61	59
	60	T = TypeVar("T", bound=Callable[..., Any])
62	61
63		def measure_func(name=None):
64		def wrapper(func):
	62
	63	def measure_func(name: Optional[str] = None) -> Callable[[T], T]:
	64	"""
	65	Used to decorate an async function with a `Measure` context manager.
	66
	67	Usage:
	68
	69	@measure_func()
	70	async def foo(...):
	71	...
	72
	73	Which is analogous to:
	74
	75	async def foo(...):
	76	with Measure(...):
	77	...
	78
	79	"""
	80
	81	def wrapper(func: T) -> T:
65	82	block_name = func.__name__ if name is None else name
66	83
67		if inspect.iscoroutinefunction(func):
	84	@wraps(func)
	85	async def measured_func(self, args, *kwargs):
	86	with Measure(self.clock, block_name):
	87	r = await func(self, args, *kwargs)
	88	return r
68	89
69		@wraps(func)
70		async def measured_func(self, args, *kwargs):
71		with Measure(self.clock, block_name):
72		r = await func(self, args, *kwargs)
73		return r
74
75		else:
76
77		@wraps(func)
78		@defer.inlineCallbacks
79		def measured_func(self, args, *kwargs):
80		with Measure(self.clock, block_name):
81		r = yield func(self, args, *kwargs)
82		return r
83
84		return measured_func
	90	return cast(T, measured_func)
85	91
86	92	return wrapper
87	93

+6

-10

synapse/util/retryutils.py less more

14	14	import logging
15	15	import random
16	16
17		from twisted.internet import defer
18
19	17	import synapse.logging.context
20	18	from synapse.api.errors import CodeMessageException
21	19

53	51	self.destination = destination
54	52
55	53
56		@defer.inlineCallbacks
57		def get_retry_limiter(destination, clock, store, ignore_backoff=False, **kwargs):
	54	async def get_retry_limiter(destination, clock, store, ignore_backoff=False, **kwargs):
58	55	"""For a given destination check if we have previously failed to
59	56	send a request there and are waiting before retrying the destination.
60	57	If we are not ready to retry the destination, this will raise a

72	69	Example usage:
73	70
74	71	try:
75		limiter = yield get_retry_limiter(destination, clock, store)
	72	limiter = await get_retry_limiter(destination, clock, store)
76	73	with limiter:
77		response = yield do_request()
	74	response = await do_request()
78	75	except NotRetryingDestination:
79	76	# We aren't ready to retry that destination.
80	77	raise

82	79	failure_ts = None
83	80	retry_last_ts, retry_interval = (0, 0)
84	81
85		retry_timings = yield store.get_destination_retry_timings(destination)
	82	retry_timings = await store.get_destination_retry_timings(destination)
86	83
87	84	if retry_timings:
88	85	failure_ts = retry_timings["failure_ts"]

221	218	if self.failure_ts is None:
222	219	self.failure_ts = retry_last_ts
223	220
224		@defer.inlineCallbacks
225		def store_retry_timings():
	221	async def store_retry_timings():
226	222	try:
227		yield self.store.set_destination_retry_timings(
	223	await self.store.set_destination_retry_timings(
228	224	self.destination,
229	225	self.failure_ts,
230	226	retry_last_ts,

+13

-17

synapse/visibility.py less more

15	15	import logging
16	16	import operator
17	17
18		from twisted.internet import defer
19
20	18	from synapse.api.constants import EventTypes, Membership
21	19	from synapse.events.utils import prune_event
22	20	from synapse.storage import Storage

38	36	)
39	37
40	38
41		@defer.inlineCallbacks
42		def filter_events_for_client(
	39	async def filter_events_for_client(
43	40	storage: Storage,
44	41	user_id,
45	42	events,

66	63	also be called to check whether a user can see the state at a given point.
67	64
68	65	Returns:
69		Deferred[list[synapse.events.EventBase]]
	66	list[synapse.events.EventBase]
70	67	"""
71	68	# Filter out events that have been soft failed so that we don't relay them
72	69	# to clients.
73	70	events = [e for e in events if not e.internal_metadata.is_soft_failed()]
74	71
75	72	types = ((EventTypes.RoomHistoryVisibility, ""), (EventTypes.Member, user_id))
76		event_id_to_state = yield storage.state.get_state_for_events(
	73	event_id_to_state = await storage.state.get_state_for_events(
77	74	frozenset(e.event_id for e in events),
78	75	state_filter=StateFilter.from_types(types),
79	76	)
80	77
81		ignore_dict_content = yield storage.main.get_global_account_data_by_type_for_user(
	78	ignore_dict_content = await storage.main.get_global_account_data_by_type_for_user(
82	79	"m.ignored_user_list", user_id
83	80	)
84	81

89	86	else []
90	87	)
91	88
92		erased_senders = yield storage.main.are_users_erased((e.sender for e in events))
	89	erased_senders = await storage.main.are_users_erased((e.sender for e in events))
93	90
94	91	if filter_send_to_client:
95	92	room_ids = {e.room_id for e in events}

98	95	for room_id in room_ids:
99	96	retention_policies[
100	97	room_id
101		] = yield storage.main.get_retention_policy_for_room(room_id)
	98	] = await storage.main.get_retention_policy_for_room(room_id)
102	99
103	100	def allowed(event):
104	101	"""

253	250	return list(filtered_events)
254	251
255	252
256		@defer.inlineCallbacks
257		def filter_events_for_server(
	253	async def filter_events_for_server(
258	254	storage: Storage,
259	255	server_name,
260	256	events,

276	272	backfill or not.
277	273
278	274	Returns
279		Deferred[list[FrozenEvent]]
	275	list[FrozenEvent]
280	276	"""
281	277
282	278	def is_sender_erased(event, erased_senders):

320	316	# Lets check to see if all the events have a history visibility
321	317	# of "shared" or "world_readable". If that's the case then we don't
322	318	# need to check membership (as we know the server is in the room).
323		event_to_state_ids = yield storage.state.get_state_ids_for_events(
	319	event_to_state_ids = await storage.state.get_state_ids_for_events(
324	320	frozenset(e.event_id for e in events),
325	321	state_filter=StateFilter.from_types(
326	322	types=((EventTypes.RoomHistoryVisibility, ""),)

338	334	if not visibility_ids:
339	335	all_open = True
340	336	else:
341		event_map = yield storage.main.get_events(visibility_ids)
	337	event_map = await storage.main.get_events(visibility_ids)
342	338	all_open = all(
343	339	e.content.get("history_visibility") in (None, "shared", "world_readable")
344	340	for e in event_map.values()
345	341	)
346	342
347	343	if not check_history_visibility_only:
348		erased_senders = yield storage.main.are_users_erased((e.sender for e in events))
	344	erased_senders = await storage.main.are_users_erased((e.sender for e in events))
349	345	else:
350	346	# We don't want to check whether users are erased, which is equivalent
351	347	# to no users having been erased.

374	370
375	371	# first, for each event we're wanting to return, get the event_ids
376	372	# of the history vis and membership state at those events.
377		event_to_state_ids = yield storage.state.get_state_ids_for_events(
	373	event_to_state_ids = await storage.state.get_state_ids_for_events(
378	374	frozenset(e.event_id for e in events),
379	375	state_filter=StateFilter.from_types(
380	376	types=((EventTypes.RoomHistoryVisibility, ""), (EventTypes.Member, None))

404	400	return False
405	401	return state_key[idx + 1 :] == server_name
406	402
407		event_map = yield storage.main.get_events(
	403	event_map = await storage.main.get_events(
408	404	[e_id for e_id, key in event_id_to_state_key.items() if include(key[0], key[1])]
409	405	)
410	406

+3

-3

synmark/__init__.py less more

46	46	stor = hs.get_datastore()
47	47
48	48	# Run the database background updates.
49		if hasattr(stor.db.updates, "do_next_background_update"):
50		while not await stor.db.updates.has_completed_background_updates():
51		await stor.db.updates.do_next_background_update(1)
	49	if hasattr(stor.db_pool.updates, "do_next_background_update"):
	50	while not await stor.db_pool.updates.has_completed_background_updates():
	51	await stor.db_pool.updates.do_next_background_update(1)
52	52
53	53	def cleanup():
54	54	for i in cleanup_tasks:

+56

-41

tests/api/test_auth.py less more

61	61	# this is overridden for the appservice tests
62	62	self.store.get_app_service_by_token = Mock(return_value=None)
63	63
	64	self.store.insert_client_ip = Mock(return_value=defer.succeed(None))
64	65	self.store.is_support_user = Mock(return_value=defer.succeed(False))
65	66
66	67	@defer.inlineCallbacks
67	68	def test_get_user_by_req_user_valid_token(self):
68	69	user_info = {"name": self.test_user, "token_id": "ditto", "device_id": "device"}
69		self.store.get_user_by_access_token = Mock(return_value=user_info)
	70	self.store.get_user_by_access_token = Mock(
	71	return_value=defer.succeed(user_info)
	72	)
70	73
71	74	request = Mock(args={})
72	75	request.args[b"access_token"] = [self.test_token]

75	78	self.assertEquals(requester.user.to_string(), self.test_user)
76	79
77	80	def test_get_user_by_req_user_bad_token(self):
78		self.store.get_user_by_access_token = Mock(return_value=None)
79
80		request = Mock(args={})
81		request.args[b"access_token"] = [self.test_token]
82		request.requestHeaders.getRawHeaders = mock_getRawHeaders()
83		d = self.auth.get_user_by_req(request)
	81	self.store.get_user_by_access_token = Mock(return_value=defer.succeed(None))
	82
	83	request = Mock(args={})
	84	request.args[b"access_token"] = [self.test_token]
	85	request.requestHeaders.getRawHeaders = mock_getRawHeaders()
	86	d = defer.ensureDeferred(self.auth.get_user_by_req(request))
84	87	f = self.failureResultOf(d, InvalidClientTokenError).value
85	88	self.assertEqual(f.code, 401)
86	89	self.assertEqual(f.errcode, "M_UNKNOWN_TOKEN")
87	90
88	91	def test_get_user_by_req_user_missing_token(self):
89	92	user_info = {"name": self.test_user, "token_id": "ditto"}
90		self.store.get_user_by_access_token = Mock(return_value=user_info)
91
92		request = Mock(args={})
93		request.requestHeaders.getRawHeaders = mock_getRawHeaders()
94		d = self.auth.get_user_by_req(request)
	93	self.store.get_user_by_access_token = Mock(
	94	return_value=defer.succeed(user_info)
	95	)
	96
	97	request = Mock(args={})
	98	request.requestHeaders.getRawHeaders = mock_getRawHeaders()
	99	d = defer.ensureDeferred(self.auth.get_user_by_req(request))
95	100	f = self.failureResultOf(d, MissingClientTokenError).value
96	101	self.assertEqual(f.code, 401)
97	102	self.assertEqual(f.errcode, "M_MISSING_TOKEN")

102	107	token="foobar", url="a_url", sender=self.test_user, ip_range_whitelist=None
103	108	)
104	109	self.store.get_app_service_by_token = Mock(return_value=app_service)
105		self.store.get_user_by_access_token = Mock(return_value=None)
	110	self.store.get_user_by_access_token = Mock(return_value=defer.succeed(None))
106	111
107	112	request = Mock(args={})
108	113	request.getClientIP.return_value = "127.0.0.1"

122	127	ip_range_whitelist=IPSet(["192.168/16"]),
123	128	)
124	129	self.store.get_app_service_by_token = Mock(return_value=app_service)
125		self.store.get_user_by_access_token = Mock(return_value=None)
	130	self.store.get_user_by_access_token = Mock(return_value=defer.succeed(None))
126	131
127	132	request = Mock(args={})
128	133	request.getClientIP.return_value = "192.168.10.10"

141	146	ip_range_whitelist=IPSet(["192.168/16"]),
142	147	)
143	148	self.store.get_app_service_by_token = Mock(return_value=app_service)
144		self.store.get_user_by_access_token = Mock(return_value=None)
	149	self.store.get_user_by_access_token = Mock(return_value=defer.succeed(None))
145	150
146	151	request = Mock(args={})
147	152	request.getClientIP.return_value = "131.111.8.42"
148	153	request.args[b"access_token"] = [self.test_token]
149	154	request.requestHeaders.getRawHeaders = mock_getRawHeaders()
150		d = self.auth.get_user_by_req(request)
	155	d = defer.ensureDeferred(self.auth.get_user_by_req(request))
151	156	f = self.failureResultOf(d, InvalidClientTokenError).value
152	157	self.assertEqual(f.code, 401)
153	158	self.assertEqual(f.errcode, "M_UNKNOWN_TOKEN")
154	159
155	160	def test_get_user_by_req_appservice_bad_token(self):
156	161	self.store.get_app_service_by_token = Mock(return_value=None)
157		self.store.get_user_by_access_token = Mock(return_value=None)
158
159		request = Mock(args={})
160		request.args[b"access_token"] = [self.test_token]
161		request.requestHeaders.getRawHeaders = mock_getRawHeaders()
162		d = self.auth.get_user_by_req(request)
	162	self.store.get_user_by_access_token = Mock(return_value=defer.succeed(None))
	163
	164	request = Mock(args={})
	165	request.args[b"access_token"] = [self.test_token]
	166	request.requestHeaders.getRawHeaders = mock_getRawHeaders()
	167	d = defer.ensureDeferred(self.auth.get_user_by_req(request))
163	168	f = self.failureResultOf(d, InvalidClientTokenError).value
164	169	self.assertEqual(f.code, 401)
165	170	self.assertEqual(f.errcode, "M_UNKNOWN_TOKEN")

167	172	def test_get_user_by_req_appservice_missing_token(self):
168	173	app_service = Mock(token="foobar", url="a_url", sender=self.test_user)
169	174	self.store.get_app_service_by_token = Mock(return_value=app_service)
170		self.store.get_user_by_access_token = Mock(return_value=None)
171
172		request = Mock(args={})
173		request.requestHeaders.getRawHeaders = mock_getRawHeaders()
174		d = self.auth.get_user_by_req(request)
	175	self.store.get_user_by_access_token = Mock(return_value=defer.succeed(None))
	176
	177	request = Mock(args={})
	178	request.requestHeaders.getRawHeaders = mock_getRawHeaders()
	179	d = defer.ensureDeferred(self.auth.get_user_by_req(request))
175	180	f = self.failureResultOf(d, MissingClientTokenError).value
176	181	self.assertEqual(f.code, 401)
177	182	self.assertEqual(f.errcode, "M_MISSING_TOKEN")

184	189	)
185	190	app_service.is_interested_in_user = Mock(return_value=True)
186	191	self.store.get_app_service_by_token = Mock(return_value=app_service)
187		self.store.get_user_by_access_token = Mock(return_value=None)
	192	# This just needs to return a truth-y value.
	193	self.store.get_user_by_id = Mock(
	194	return_value=defer.succeed({"is_guest": False})
	195	)
	196	self.store.get_user_by_access_token = Mock(return_value=defer.succeed(None))
188	197
189	198	request = Mock(args={})
190	199	request.getClientIP.return_value = "127.0.0.1"

203	212	)
204	213	app_service.is_interested_in_user = Mock(return_value=False)
205	214	self.store.get_app_service_by_token = Mock(return_value=app_service)
206		self.store.get_user_by_access_token = Mock(return_value=None)
	215	self.store.get_user_by_access_token = Mock(return_value=defer.succeed(None))
207	216
208	217	request = Mock(args={})
209	218	request.getClientIP.return_value = "127.0.0.1"
210	219	request.args[b"access_token"] = [self.test_token]
211	220	request.args[b"user_id"] = [masquerading_user_id]
212	221	request.requestHeaders.getRawHeaders = mock_getRawHeaders()
213		d = self.auth.get_user_by_req(request)
	222	d = defer.ensureDeferred(self.auth.get_user_by_req(request))
214	223	self.failureResultOf(d, AuthError)
215	224
216	225	@defer.inlineCallbacks
217	226	def test_get_user_from_macaroon(self):
218	227	self.store.get_user_by_access_token = Mock(
219		return_value={"name": "@baldrick:matrix.org", "device_id": "device"}
	228	return_value=defer.succeed(
	229	{"name": "@baldrick:matrix.org", "device_id": "device"}
	230	)
220	231	)
221	232
222	233	user_id = "@baldrick:matrix.org"

240	251
241	252	@defer.inlineCallbacks
242	253	def test_get_guest_user_from_macaroon(self):
243		self.store.get_user_by_id = Mock(return_value={"is_guest": True})
244		self.store.get_user_by_access_token = Mock(return_value=None)
	254	self.store.get_user_by_id = Mock(return_value=defer.succeed({"is_guest": True}))
	255	self.store.get_user_by_access_token = Mock(return_value=defer.succeed(None))
245	256
246	257	user_id = "@baldrick:matrix.org"
247	258	macaroon = pymacaroons.Macaroon(

281	292
282	293	def get_user(tok):
283	294	if token != tok:
284		return None
285		return {
286		"name": USER_ID,
287		"is_guest": False,
288		"token_id": 1234,
289		"device_id": "DEVICE",
290		}
	295	return defer.succeed(None)
	296	return defer.succeed(
	297	{
	298	"name": USER_ID,
	299	"is_guest": False,
	300	"token_id": 1234,
	301	"device_id": "DEVICE",
	302	}
	303	)
291	304
292	305	self.store.get_user_by_access_token = get_user
293		self.store.get_user_by_id = Mock(return_value={"is_guest": False})
	306	self.store.get_user_by_id = Mock(
	307	return_value=defer.succeed({"is_guest": False})
	308	)
294	309
295	310	# check the token works
296	311	request = Mock(args={})

+24

-12

tests/api/test_filtering.py less more

374	374	event = MockEvent(sender="@foo:bar", type="m.profile")
375	375	events = [event]
376	376
377		user_filter = yield self.filtering.get_user_filter(
378		user_localpart=user_localpart, filter_id=filter_id
	377	user_filter = yield defer.ensureDeferred(
	378	self.filtering.get_user_filter(
	379	user_localpart=user_localpart, filter_id=filter_id
	380	)
379	381	)
380	382
381	383	results = user_filter.filter_presence(events=events)

395	397	)
396	398	events = [event]
397	399
398		user_filter = yield self.filtering.get_user_filter(
399		user_localpart=user_localpart + "2", filter_id=filter_id
	400	user_filter = yield defer.ensureDeferred(
	401	self.filtering.get_user_filter(
	402	user_localpart=user_localpart + "2", filter_id=filter_id
	403	)
400	404	)
401	405
402	406	results = user_filter.filter_presence(events=events)

411	415	event = MockEvent(sender="@foo:bar", type="m.room.topic", room_id="!foo:bar")
412	416	events = [event]
413	417
414		user_filter = yield self.filtering.get_user_filter(
415		user_localpart=user_localpart, filter_id=filter_id
	418	user_filter = yield defer.ensureDeferred(
	419	self.filtering.get_user_filter(
	420	user_localpart=user_localpart, filter_id=filter_id
	421	)
416	422	)
417	423
418	424	results = user_filter.filter_room_state(events=events)

429	435	)
430	436	events = [event]
431	437
432		user_filter = yield self.filtering.get_user_filter(
433		user_localpart=user_localpart, filter_id=filter_id
	438	user_filter = yield defer.ensureDeferred(
	439	self.filtering.get_user_filter(
	440	user_localpart=user_localpart, filter_id=filter_id
	441	)
434	442	)
435	443
436	444	results = user_filter.filter_room_state(events)

464	472	self.assertEquals(
465	473	user_filter_json,
466	474	(
467		yield self.datastore.get_user_filter(
468		user_localpart=user_localpart, filter_id=0
	475	yield defer.ensureDeferred(
	476	self.datastore.get_user_filter(
	477	user_localpart=user_localpart, filter_id=0
	478	)
469	479	)
470	480	),
471	481	)

478	488	user_localpart=user_localpart, user_filter=user_filter_json
479	489	)
480	490
481		filter = yield self.filtering.get_user_filter(
482		user_localpart=user_localpart, filter_id=filter_id
	491	filter = yield defer.ensureDeferred(
	492	self.filtering.get_user_filter(
	493	user_localpart=user_localpart, filter_id=filter_id
	494	)
483	495	)
484	496
485	497	self.assertEquals(filter.get_filter_json(), user_filter_json)

+62

-29

tests/appservice/test_appservice.py less more

49	49	def test_regex_user_id_prefix_match(self):
50	50	self.service.namespaces[ApplicationService.NS_USERS].append(_regex("@irc_.*"))
51	51	self.event.sender = "@irc_foobar:matrix.org"
52		self.assertTrue((yield self.service.is_interested(self.event)))
	52	self.assertTrue(
	53	(yield defer.ensureDeferred(self.service.is_interested(self.event)))
	54	)
53	55
54	56	@defer.inlineCallbacks
55	57	def test_regex_user_id_prefix_no_match(self):
56	58	self.service.namespaces[ApplicationService.NS_USERS].append(_regex("@irc_.*"))
57	59	self.event.sender = "@someone_else:matrix.org"
58		self.assertFalse((yield self.service.is_interested(self.event)))
	60	self.assertFalse(
	61	(yield defer.ensureDeferred(self.service.is_interested(self.event)))
	62	)
59	63
60	64	@defer.inlineCallbacks
61	65	def test_regex_room_member_is_checked(self):

63	67	self.event.sender = "@someone_else:matrix.org"
64	68	self.event.type = "m.room.member"
65	69	self.event.state_key = "@irc_foobar:matrix.org"
66		self.assertTrue((yield self.service.is_interested(self.event)))
	70	self.assertTrue(
	71	(yield defer.ensureDeferred(self.service.is_interested(self.event)))
	72	)
67	73
68	74	@defer.inlineCallbacks
69	75	def test_regex_room_id_match(self):

71	77	_regex("!some_prefix.*some_suffix:matrix.org")
72	78	)
73	79	self.event.room_id = "!some_prefixs0m3th1nGsome_suffix:matrix.org"
74		self.assertTrue((yield self.service.is_interested(self.event)))
	80	self.assertTrue(
	81	(yield defer.ensureDeferred(self.service.is_interested(self.event)))
	82	)
75	83
76	84	@defer.inlineCallbacks
77	85	def test_regex_room_id_no_match(self):

79	87	_regex("!some_prefix.*some_suffix:matrix.org")
80	88	)
81	89	self.event.room_id = "!XqBunHwQIXUiqCaoxq:matrix.org"
82		self.assertFalse((yield self.service.is_interested(self.event)))
	90	self.assertFalse(
	91	(yield defer.ensureDeferred(self.service.is_interested(self.event)))
	92	)
83	93
84	94	@defer.inlineCallbacks
85	95	def test_regex_alias_match(self):
86	96	self.service.namespaces[ApplicationService.NS_ALIASES].append(
87	97	_regex("#irc_.*:matrix.org")
88	98	)
89		self.store.get_aliases_for_room.return_value = [
90		"#irc_foobar:matrix.org",
91		"#athing:matrix.org",
92		]
93		self.store.get_users_in_room.return_value = []
94		self.assertTrue((yield self.service.is_interested(self.event, self.store)))
	99	self.store.get_aliases_for_room.return_value = defer.succeed(
	100	["#irc_foobar:matrix.org", "#athing:matrix.org"]
	101	)
	102	self.store.get_users_in_room.return_value = defer.succeed([])
	103	self.assertTrue(
	104	(
	105	yield defer.ensureDeferred(
	106	self.service.is_interested(self.event, self.store)
	107	)
	108	)
	109	)
95	110
96	111	def test_non_exclusive_alias(self):
97	112	self.service.namespaces[ApplicationService.NS_ALIASES].append(

134	149	self.service.namespaces[ApplicationService.NS_ALIASES].append(
135	150	_regex("#irc_.*:matrix.org")
136	151	)
137		self.store.get_aliases_for_room.return_value = [
138		"#xmpp_foobar:matrix.org",
139		"#athing:matrix.org",
140		]
141		self.store.get_users_in_room.return_value = []
142		self.assertFalse((yield self.service.is_interested(self.event, self.store)))
	152	self.store.get_aliases_for_room.return_value = defer.succeed(
	153	["#xmpp_foobar:matrix.org", "#athing:matrix.org"]
	154	)
	155	self.store.get_users_in_room.return_value = defer.succeed([])
	156	self.assertFalse(
	157	(
	158	yield defer.ensureDeferred(
	159	self.service.is_interested(self.event, self.store)
	160	)
	161	)
	162	)
143	163
144	164	@defer.inlineCallbacks
145	165	def test_regex_multiple_matches(self):

148	168	)
149	169	self.service.namespaces[ApplicationService.NS_USERS].append(_regex("@irc_.*"))
150	170	self.event.sender = "@irc_foobar:matrix.org"
151		self.store.get_aliases_for_room.return_value = ["#irc_barfoo:matrix.org"]
152		self.store.get_users_in_room.return_value = []
153		self.assertTrue((yield self.service.is_interested(self.event, self.store)))
	171	self.store.get_aliases_for_room.return_value = defer.succeed(
	172	["#irc_barfoo:matrix.org"]
	173	)
	174	self.store.get_users_in_room.return_value = defer.succeed([])
	175	self.assertTrue(
	176	(
	177	yield defer.ensureDeferred(
	178	self.service.is_interested(self.event, self.store)
	179	)
	180	)
	181	)
154	182
155	183	@defer.inlineCallbacks
156	184	def test_interested_in_self(self):

160	188	self.event.type = "m.room.member"
161	189	self.event.content = {"membership": "invite"}
162	190	self.event.state_key = self.service.sender
163		self.assertTrue((yield self.service.is_interested(self.event)))
	191	self.assertTrue(
	192	(yield defer.ensureDeferred(self.service.is_interested(self.event)))
	193	)
164	194
165	195	@defer.inlineCallbacks
166	196	def test_member_list_match(self):
167	197	self.service.namespaces[ApplicationService.NS_USERS].append(_regex("@irc_.*"))
168		self.store.get_users_in_room.return_value = [
169		"@alice:here",
170		"@irc_fo:here", # AS user
171		"@bob:here",
172		]
173		self.store.get_aliases_for_room.return_value = []
	198	# Note that @irc_fo:here is the AS user.
	199	self.store.get_users_in_room.return_value = defer.succeed(
	200	["@alice:here", "@irc_fo:here", "@bob:here"]
	201	)
	202	self.store.get_aliases_for_room.return_value = defer.succeed([])
174	203
175	204	self.event.sender = "@xmpp_foobar:matrix.org"
176	205	self.assertTrue(
177		(yield self.service.is_interested(event=self.event, store=self.store))
178		)
	206	(
	207	yield defer.ensureDeferred(
	208	self.service.is_interested(event=self.event, store=self.store)
	209	)
	210	)
	211	)

+11

-8

tests/appservice/test_scheduler.py less more

24	24	from synapse.logging.context import make_deferred_yieldable
25	25
26	26	from tests import unittest
	27	from tests.test_utils import make_awaitable
27	28
28	29	from ..utils import MockClock
29	30

51	52	self.store.get_appservice_state = Mock(
52	53	return_value=defer.succeed(ApplicationServiceState.UP)
53	54	)
54		txn.send = Mock(return_value=defer.succeed(True))
	55	txn.send = Mock(return_value=make_awaitable(True))
55	56	self.store.create_appservice_txn = Mock(return_value=defer.succeed(txn))
56	57
57	58	# actual call
58		self.txnctrl.send(service, events)
	59	self.successResultOf(defer.ensureDeferred(self.txnctrl.send(service, events)))
59	60
60	61	self.store.create_appservice_txn.assert_called_once_with(
61	62	service=service, events=events # txn made and saved

76	77	self.store.create_appservice_txn = Mock(return_value=defer.succeed(txn))
77	78
78	79	# actual call
79		self.txnctrl.send(service, events)
	80	self.successResultOf(defer.ensureDeferred(self.txnctrl.send(service, events)))
80	81
81	82	self.store.create_appservice_txn.assert_called_once_with(
82	83	service=service, events=events # txn made and saved

97	98	return_value=defer.succeed(ApplicationServiceState.UP)
98	99	)
99	100	self.store.set_appservice_state = Mock(return_value=defer.succeed(True))
100		txn.send = Mock(return_value=defer.succeed(False)) # fails to send
	101	txn.send = Mock(return_value=make_awaitable(False)) # fails to send
101	102	self.store.create_appservice_txn = Mock(return_value=defer.succeed(txn))
102	103
103	104	# actual call
104		self.txnctrl.send(service, events)
	105	self.successResultOf(defer.ensureDeferred(self.txnctrl.send(service, events)))
105	106
106	107	self.store.create_appservice_txn.assert_called_once_with(
107	108	service=service, events=events

143	144	self.recoverer.recover()
144	145	# shouldn't have called anything prior to waiting for exp backoff
145	146	self.assertEquals(0, self.store.get_oldest_unsent_txn.call_count)
146		txn.send = Mock(return_value=True)
	147	txn.send = Mock(return_value=make_awaitable(True))
	148	txn.complete.return_value = make_awaitable(None)
147	149	# wait for exp backoff
148	150	self.clock.advance_time(2)
149	151	self.assertEquals(1, txn.send.call_count)

168	170
169	171	self.recoverer.recover()
170	172	self.assertEquals(0, self.store.get_oldest_unsent_txn.call_count)
171		txn.send = Mock(return_value=False)
	173	txn.send = Mock(return_value=make_awaitable(False))
	174	txn.complete.return_value = make_awaitable(None)
172	175	self.clock.advance_time(2)
173	176	self.assertEquals(1, txn.send.call_count)
174	177	self.assertEquals(0, txn.complete.call_count)

181	184	self.assertEquals(3, txn.send.call_count)
182	185	self.assertEquals(0, txn.complete.call_count)
183	186	self.assertEquals(0, self.callback.call_count)
184		txn.send = Mock(return_value=True) # successfully send the txn
	187	txn.send = Mock(return_value=make_awaitable(True)) # successfully send the txn
185	188	pop_txn = True # returns the txn the first time, then no more.
186	189	self.clock.advance_time(16)
187	190	self.assertEquals(1, txn.send.call_count) # new mock reset call count

+22

-32

tests/crypto/test_keyring.py less more

39	39	from synapse.storage.keys import FetchKeyResult
40	40
41	41	from tests import unittest
	42	from tests.test_utils import make_awaitable
42	43
43	44
44	45	class MockPerspectiveServer(object):

101	102	}
102	103	persp_deferred = defer.Deferred()
103	104
104		@defer.inlineCallbacks
105		def get_perspectives(**kwargs):
	105	async def get_perspectives(**kwargs):
106	106	self.assertEquals(current_context().request, "11")
107	107	with PreserveLoggingContext():
108		yield persp_deferred
	108	await persp_deferred
109	109	return persp_resp
110	110
111	111	self.http_client.post_json.side_effect = get_perspectives

201	201	with a null `ts_valid_until_ms`
202	202	"""
203	203	mock_fetcher = keyring.KeyFetcher()
204		mock_fetcher.get_keys = Mock(return_value=defer.succeed({}))
	204	mock_fetcher.get_keys = Mock(return_value=make_awaitable({}))
205	205
206	206	kr = keyring.Keyring(
207	207	self.hs, key_fetchers=(StoreKeyFetcher(self.hs), mock_fetcher)

244	244	"""Two requests for the same key should be deduped."""
245	245	key1 = signedjson.key.generate_signing_key(1)
246	246
247		def get_keys(keys_to_fetch):
	247	async def get_keys(keys_to_fetch):
248	248	# there should only be one request object (with the max validity)
249	249	self.assertEqual(keys_to_fetch, {"server1": {get_key_id(key1): 1500}})
250	250
251		return defer.succeed(
252		{
253		"server1": {
254		get_key_id(key1): FetchKeyResult(get_verify_key(key1), 1200)
255		}
	251	return {
	252	"server1": {
	253	get_key_id(key1): FetchKeyResult(get_verify_key(key1), 1200)
256	254	}
257		)
	255	}
258	256
259	257	mock_fetcher = keyring.KeyFetcher()
260	258	mock_fetcher.get_keys = Mock(side_effect=get_keys)

281	279	"""If the first fetcher cannot provide a recent enough key, we fall back"""
282	280	key1 = signedjson.key.generate_signing_key(1)
283	281
284		def get_keys1(keys_to_fetch):
	282	async def get_keys1(keys_to_fetch):
285	283	self.assertEqual(keys_to_fetch, {"server1": {get_key_id(key1): 1500}})
286		return defer.succeed(
287		{
288		"server1": {
289		get_key_id(key1): FetchKeyResult(get_verify_key(key1), 800)
290		}
	284	return {
	285	"server1": {get_key_id(key1): FetchKeyResult(get_verify_key(key1), 800)}
	286	}
	287
	288	async def get_keys2(keys_to_fetch):
	289	self.assertEqual(keys_to_fetch, {"server1": {get_key_id(key1): 1500}})
	290	return {
	291	"server1": {
	292	get_key_id(key1): FetchKeyResult(get_verify_key(key1), 1200)
291	293	}
292		)
293
294		def get_keys2(keys_to_fetch):
295		self.assertEqual(keys_to_fetch, {"server1": {get_key_id(key1): 1500}})
296		return defer.succeed(
297		{
298		"server1": {
299		get_key_id(key1): FetchKeyResult(get_verify_key(key1), 1200)
300		}
301		}
302		)
	294	}
303	295
304	296	mock_fetcher1 = keyring.KeyFetcher()
305	297	mock_fetcher1.get_keys = Mock(side_effect=get_keys1)

354	346	}
355	347	signedjson.sign.sign_json(response, SERVER_NAME, testkey)
356	348
357		def get_json(destination, path, **kwargs):
	349	async def get_json(destination, path, **kwargs):
358	350	self.assertEqual(destination, SERVER_NAME)
359	351	self.assertEqual(path, "/_matrix/key/v2/server/key1")
360	352	return response

443	435	Tell the mock http client to expect a perspectives-server key query
444	436	"""
445	437
446		def post_json(destination, path, data, **kwargs):
	438	async def post_json(destination, path, data, **kwargs):
447	439	self.assertEqual(destination, self.mock_perspective_server.server_name)
448	440	self.assertEqual(path, "/_matrix/key/v2/query")
449	441

579	571	# remove the perspectives server's signature
580	572	response = build_response()
581	573	del response["signatures"][self.mock_perspective_server.server_name]
582		self.http_client.post_json.return_value = {"server_keys": [response]}
583	574	keys = get_key_from_perspectives(response)
584	575	self.assertEqual(keys, {}, "Expected empty dict with missing persp server sig")
585	576
586	577	# remove the origin server's signature
587	578	response = build_response()
588	579	del response["signatures"][SERVER_NAME]
589		self.http_client.post_json.return_value = {"server_keys": [response]}
590	580	keys = get_key_from_perspectives(response)
591	581	self.assertEqual(keys, {}, "Expected empty dict with missing origin server sig")
592	582

+116

-6

tests/federation/test_complexity.py less more

22	22	from synapse.types import UserID
23	23
24	24	from tests import unittest
	25	from tests.test_utils import make_awaitable
25	26
26	27
27	28	class RoomComplexityTests(unittest.FederatingHomeserverTestCase):

77	78	fed_transport = self.hs.get_federation_transport_client()
78	79
79	80	# Mock out some things, because we don't want to test the whole join
80		fed_transport.client.get_json = Mock(return_value=defer.succeed({"v1": 9999}))
81		handler.federation_handler.do_invite_join = Mock(
82		return_value=defer.succeed(("", 1))
	81	fed_transport.client.get_json = Mock(return_value=make_awaitable({"v1": 9999}))
	82	handler.federation_handler.do_invite_join = Mock(
	83	return_value=make_awaitable(("", 1))
	84	)
	85
	86	d = handler._remote_join(
	87	None,
	88	["other.example.com"],
	89	"roomid",
	90	UserID.from_string(u1),
	91	{"membership": "join"},
	92	)
	93
	94	self.pump()
	95
	96	# The request failed with a SynapseError saying the resource limit was
	97	# exceeded.
	98	f = self.get_failure(d, SynapseError)
	99	self.assertEqual(f.value.code, 400, f.value)
	100	self.assertEqual(f.value.errcode, Codes.RESOURCE_LIMIT_EXCEEDED)
	101
	102	def test_join_too_large_admin(self):
	103	# Check whether an admin can join if option "admins_can_join" is undefined,
	104	# this option defaults to false, so the join should fail.
	105
	106	u1 = self.register_user("u1", "pass", admin=True)
	107
	108	handler = self.hs.get_room_member_handler()
	109	fed_transport = self.hs.get_federation_transport_client()
	110
	111	# Mock out some things, because we don't want to test the whole join
	112	fed_transport.client.get_json = Mock(return_value=make_awaitable({"v1": 9999}))
	113	handler.federation_handler.do_invite_join = Mock(
	114	return_value=make_awaitable(("", 1))
83	115	)
84	116
85	117	d = handler._remote_join(

115	147	fed_transport = self.hs.get_federation_transport_client()
116	148
117	149	# Mock out some things, because we don't want to test the whole join
118		fed_transport.client.get_json = Mock(return_value=defer.succeed(None))
119		handler.federation_handler.do_invite_join = Mock(
120		return_value=defer.succeed(("", 1))
	150	fed_transport.client.get_json = Mock(return_value=make_awaitable(None))
	151	handler.federation_handler.do_invite_join = Mock(
	152	return_value=make_awaitable(("", 1))
121	153	)
122	154
123	155	# Artificially raise the complexity

140	172	f = self.get_failure(d, SynapseError)
141	173	self.assertEqual(f.value.code, 400)
142	174	self.assertEqual(f.value.errcode, Codes.RESOURCE_LIMIT_EXCEEDED)
	175
	176
	177	class RoomComplexityAdminTests(unittest.FederatingHomeserverTestCase):
	178	# Test the behavior of joining rooms which exceed the complexity if option
	179	# limit_remote_rooms.admins_can_join is True.
	180
	181	servlets = [
	182	admin.register_servlets,
	183	room.register_servlets,
	184	login.register_servlets,
	185	]
	186
	187	def default_config(self):
	188	config = super().default_config()
	189	config["limit_remote_rooms"] = {
	190	"enabled": True,
	191	"complexity": 0.05,
	192	"admins_can_join": True,
	193	}
	194	return config
	195
	196	def test_join_too_large_no_admin(self):
	197	# A user which is not an admin should not be able to join a remote room
	198	# which is too complex.
	199
	200	u1 = self.register_user("u1", "pass")
	201
	202	handler = self.hs.get_room_member_handler()
	203	fed_transport = self.hs.get_federation_transport_client()
	204
	205	# Mock out some things, because we don't want to test the whole join
	206	fed_transport.client.get_json = Mock(return_value=make_awaitable({"v1": 9999}))
	207	handler.federation_handler.do_invite_join = Mock(
	208	return_value=make_awaitable(("", 1))
	209	)
	210
	211	d = handler._remote_join(
	212	None,
	213	["other.example.com"],
	214	"roomid",
	215	UserID.from_string(u1),
	216	{"membership": "join"},
	217	)
	218
	219	self.pump()
	220
	221	# The request failed with a SynapseError saying the resource limit was
	222	# exceeded.
	223	f = self.get_failure(d, SynapseError)
	224	self.assertEqual(f.value.code, 400, f.value)
	225	self.assertEqual(f.value.errcode, Codes.RESOURCE_LIMIT_EXCEEDED)
	226
	227	def test_join_too_large_admin(self):
	228	# An admin should be able to join rooms where a complexity check fails.
	229
	230	u1 = self.register_user("u1", "pass", admin=True)
	231
	232	handler = self.hs.get_room_member_handler()
	233	fed_transport = self.hs.get_federation_transport_client()
	234
	235	# Mock out some things, because we don't want to test the whole join
	236	fed_transport.client.get_json = Mock(return_value=make_awaitable({"v1": 9999}))
	237	handler.federation_handler.do_invite_join = Mock(
	238	return_value=make_awaitable(("", 1))
	239	)
	240
	241	d = handler._remote_join(
	242	None,
	243	["other.example.com"],
	244	"roomid",
	245	UserID.from_string(u1),
	246	{"membership": "join"},
	247	)
	248
	249	self.pump()
	250
	251	# The request success since the user is an admin
	252	self.get_success(d)

+5

-5

tests/federation/test_federation_sender.py less more

46	46	mock_send_transaction = (
47	47	self.hs.get_federation_transport_client().send_transaction
48	48	)
49		mock_send_transaction.return_value = defer.succeed({})
	49	mock_send_transaction.return_value = make_awaitable({})
50	50
51	51	sender = self.hs.get_federation_sender()
52	52	receipt = ReadReceipt(
53	53	"room_id", "m.read", "user_id", ["event_id"], {"ts": 1234}
54	54	)
55		self.successResultOf(sender.send_read_receipt(receipt))
	55	self.successResultOf(defer.ensureDeferred(sender.send_read_receipt(receipt)))
56	56
57	57	self.pump()
58	58

86	86	mock_send_transaction = (
87	87	self.hs.get_federation_transport_client().send_transaction
88	88	)
89		mock_send_transaction.return_value = defer.succeed({})
	89	mock_send_transaction.return_value = make_awaitable({})
90	90
91	91	sender = self.hs.get_federation_sender()
92	92	receipt = ReadReceipt(
93	93	"room_id", "m.read", "user_id", ["event_id"], {"ts": 1234}
94	94	)
95		self.successResultOf(sender.send_read_receipt(receipt))
	95	self.successResultOf(defer.ensureDeferred(sender.send_read_receipt(receipt)))
96	96
97	97	self.pump()
98	98

124	124	receipt = ReadReceipt(
125	125	"room_id", "m.read", "user_id", ["other_id"], {"ts": 1234}
126	126	)
127		self.successResultOf(sender.send_read_receipt(receipt))
	127	self.successResultOf(defer.ensureDeferred(sender.send_read_receipt(receipt)))
128	128	self.pump()
129	129	mock_send_transaction.assert_not_called()
130	130

+4

-3

tests/handlers/test_appservice.py less more

18	18
19	19	from synapse.handlers.appservice import ApplicationServicesHandler
20	20
	21	from tests.test_utils import make_awaitable
21	22	from tests.utils import MockClock
22	23
23	24	from .. import unittest

116	117	self._mkservice_alias(is_interested_in_alias=False),
117	118	]
118	119
119		self.mock_as_api.query_alias.return_value = defer.succeed(True)
	120	self.mock_as_api.query_alias.return_value = make_awaitable(True)
120	121	self.mock_store.get_app_services.return_value = services
121		self.mock_store.get_association_from_room_alias.return_value = defer.succeed(
	122	self.mock_store.get_association_from_room_alias.return_value = make_awaitable(
122	123	Mock(room_id=room_id, servers=servers)
123	124	)
124	125

134	135
135	136	def _mkservice(self, is_interested):
136	137	service = Mock()
137		service.is_interested.return_value = defer.succeed(is_interested)
	138	service.is_interested.return_value = make_awaitable(is_interested)
138	139	service.token = "mock_service_token"
139	140	service.url = "mock_service_url"
140	141	return service

+2

-3

tests/handlers/test_directory.py less more

15	15
16	16	from mock import Mock
17	17
18		from twisted.internet import defer
19
20	18	import synapse
21	19	import synapse.api.errors
22	20	from synapse.api.constants import EventTypes

25	23	from synapse.types import RoomAlias, create_requester
26	24
27	25	from tests import unittest
	26	from tests.test_utils import make_awaitable
28	27
29	28
30	29	class DirectoryTestCase(unittest.HomeserverTestCase):

70	69	self.assertEquals({"room_id": "!8765qwer:test", "servers": ["test"]}, result)
71	70
72	71	def test_get_remote_association(self):
73		self.mock_federation.make_query.return_value = defer.succeed(
	72	self.mock_federation.make_query.return_value = make_awaitable(
74	73	{"room_id": "!8765qwer:test", "servers": ["test", "remote"]}
75	74	)
76	75

+2

-1

tests/handlers/test_profile.py less more

23	23	from synapse.types import UserID
24	24
25	25	from tests import unittest
	26	from tests.test_utils import make_awaitable
26	27	from tests.utils import setup_test_homeserver
27	28
28	29

137	138
138	139	@defer.inlineCallbacks
139	140	def test_get_other_name(self):
140		self.mock_federation.make_query.return_value = defer.succeed(
	141	self.mock_federation.make_query.return_value = make_awaitable(
141	142	{"displayname": "Alice"}
142	143	)
143	144

+6

-5

tests/handlers/test_register.py less more

21	21	from synapse.handlers.register import RegistrationHandler
22	22	from synapse.types import RoomAlias, UserID, create_requester
23	23
	24	from tests.test_utils import make_awaitable
24	25	from tests.unittest import override_config
25	26
26	27	from .. import unittest

186	187	room_alias_str = "#room:test"
187	188	self.hs.config.auto_join_rooms = [room_alias_str]
188	189
189		self.store.is_real_user = Mock(return_value=defer.succeed(False))
	190	self.store.is_real_user = Mock(return_value=make_awaitable(False))
190	191	user_id = self.get_success(self.handler.register_user(localpart="support"))
191	192	rooms = self.get_success(self.store.get_rooms_for_user(user_id))
192	193	self.assertEqual(len(rooms), 0)

198	199	def test_auto_create_auto_join_rooms_when_user_is_the_first_real_user(self):
199	200	room_alias_str = "#room:test"
200	201
201		self.store.count_real_users = Mock(return_value=defer.succeed(1))
202		self.store.is_real_user = Mock(return_value=defer.succeed(True))
	202	self.store.count_real_users = Mock(return_value=make_awaitable(1))
	203	self.store.is_real_user = Mock(return_value=make_awaitable(True))
203	204	user_id = self.get_success(self.handler.register_user(localpart="real"))
204	205	rooms = self.get_success(self.store.get_rooms_for_user(user_id))
205	206	directory_handler = self.hs.get_handlers().directory_handler

213	214	room_alias_str = "#room:test"
214	215	self.hs.config.auto_join_rooms = [room_alias_str]
215	216
216		self.store.count_real_users = Mock(return_value=defer.succeed(2))
217		self.store.is_real_user = Mock(return_value=defer.succeed(True))
	217	self.store.count_real_users = Mock(return_value=make_awaitable(2))
	218	self.store.is_real_user = Mock(return_value=make_awaitable(True))
218	219	user_id = self.get_success(self.handler.register_user(localpart="real"))
219	220	rooms = self.get_success(self.store.get_rooms_for_user(user_id))
220	221	self.assertEqual(len(rooms), 0)

+81

-47

tests/handlers/test_stats.py less more

14	14
15	15	from synapse.rest import admin
16	16	from synapse.rest.client.v1 import login, room
17		from synapse.storage.data_stores.main import stats
	17	from synapse.storage.databases.main import stats
18	18
19	19	from tests import unittest
20	20

41	41	Add the background updates we need to run.
42	42	"""
43	43	# Ugh, have to reset this flag
44		self.store.db.updates._all_done = False
45
46		self.get_success(
47		self.store.db.simple_insert(
	44	self.store.db_pool.updates._all_done = False
	45
	46	self.get_success(
	47	self.store.db_pool.simple_insert(
48	48	"background_updates",
49	49	{"update_name": "populate_stats_prepare", "progress_json": "{}"},
50	50	)
51	51	)
52	52	self.get_success(
53		self.store.db.simple_insert(
	53	self.store.db_pool.simple_insert(
54	54	"background_updates",
55	55	{
56		"update_name": "populate_stats_process_rooms",
	56	"update_name": "populate_stats_process_rooms_2",
57	57	"progress_json": "{}",
58	58	"depends_on": "populate_stats_prepare",
59	59	},
60	60	)
61	61	)
62	62	self.get_success(
63		self.store.db.simple_insert(
	63	self.store.db_pool.simple_insert(
64	64	"background_updates",
65	65	{
66	66	"update_name": "populate_stats_process_users",
67	67	"progress_json": "{}",
68		"depends_on": "populate_stats_process_rooms",
	68	"depends_on": "populate_stats_process_rooms_2",
69	69	},
70	70	)
71	71	)
72	72	self.get_success(
73		self.store.db.simple_insert(
	73	self.store.db_pool.simple_insert(
74	74	"background_updates",
75	75	{
76	76	"update_name": "populate_stats_cleanup",

81	81	)
82	82
83	83	def get_all_room_state(self):
84		return self.store.db.simple_select_list(
	84	return self.store.db_pool.simple_select_list(
85	85	"room_stats_state", None, retcols=("name", "topic", "canonical_alias")
86	86	)
87	87

95	95	end_ts = self.store.quantise_stats_time(self.reactor.seconds() * 1000)
96	96
97	97	return self.get_success(
98		self.store.db.simple_select_one(
	98	self.store.db_pool.simple_select_one(
99	99	table + "_historical",
100	100	{id_col: stat_id, end_ts: end_ts},
101	101	cols,

108	108	self._add_background_updates()
109	109
110	110	while not self.get_success(
111		self.store.db.updates.has_completed_background_updates()
	111	self.store.db_pool.updates.has_completed_background_updates()
112	112	):
113	113	self.get_success(
114		self.store.db.updates.do_next_background_update(100), by=0.1
	114	self.store.db_pool.updates.do_next_background_update(100), by=0.1
115	115	)
116	116
117	117	def test_initial_room(self):

145	145	self._add_background_updates()
146	146
147	147	while not self.get_success(
148		self.store.db.updates.has_completed_background_updates()
	148	self.store.db_pool.updates.has_completed_background_updates()
149	149	):
150	150	self.get_success(
151		self.store.db.updates.do_next_background_update(100), by=0.1
	151	self.store.db_pool.updates.do_next_background_update(100), by=0.1
152	152	)
153	153
154	154	r = self.get_success(self.get_all_room_state())

185	185	# the position that the deltas should begin at, once they take over.
186	186	self.hs.config.stats_enabled = True
187	187	self.handler.stats_enabled = True
188		self.store.db.updates._all_done = False
189		self.get_success(
190		self.store.db.simple_update_one(
	188	self.store.db_pool.updates._all_done = False
	189	self.get_success(
	190	self.store.db_pool.simple_update_one(
191	191	table="stats_incremental_position",
192	192	keyvalues={},
193	193	updatevalues={"stream_id": 0},

195	195	)
196	196
197	197	self.get_success(
198		self.store.db.simple_insert(
	198	self.store.db_pool.simple_insert(
199	199	"background_updates",
200	200	{"update_name": "populate_stats_prepare", "progress_json": "{}"},
201	201	)
202	202	)
203	203
204	204	while not self.get_success(
205		self.store.db.updates.has_completed_background_updates()
	205	self.store.db_pool.updates.has_completed_background_updates()
206	206	):
207	207	self.get_success(
208		self.store.db.updates.do_next_background_update(100), by=0.1
	208	self.store.db_pool.updates.do_next_background_update(100), by=0.1
209	209	)
210	210
211	211	# Now, before the table is actually ingested, add some more events.

216	216
217	217	# Now do the initial ingestion.
218	218	self.get_success(
219		self.store.db.simple_insert(
220		"background_updates",
221		{"update_name": "populate_stats_process_rooms", "progress_json": "{}"},
222		)
223		)
224		self.get_success(
225		self.store.db.simple_insert(
	219	self.store.db_pool.simple_insert(
	220	"background_updates",
	221	{
	222	"update_name": "populate_stats_process_rooms_2",
	223	"progress_json": "{}",
	224	},
	225	)
	226	)
	227	self.get_success(
	228	self.store.db_pool.simple_insert(
226	229	"background_updates",
227	230	{
228	231	"update_name": "populate_stats_cleanup",
229	232	"progress_json": "{}",
230		"depends_on": "populate_stats_process_rooms",
	233	"depends_on": "populate_stats_process_rooms_2",
231	234	},
232	235	)
233	236	)
234	237
235		self.store.db.updates._all_done = False
	238	self.store.db_pool.updates._all_done = False
236	239	while not self.get_success(
237		self.store.db.updates.has_completed_background_updates()
	240	self.store.db_pool.updates.has_completed_background_updates()
238	241	):
239	242	self.get_success(
240		self.store.db.updates.do_next_background_update(100), by=0.1
	243	self.store.db_pool.updates.do_next_background_update(100), by=0.1
241	244	)
242	245
243	246	self.reactor.advance(86401)

345	348
346	349	self.assertEqual(r1stats_post["total_events"] - r1stats_ante["total_events"], 1)
347	350
	351	def test_updating_profile_information_does_not_increase_joined_members_count(self):
	352	"""
	353	Check that the joined_members count does not increase when a user changes their
	354	profile information (which is done by sending another join membership event into
	355	the room.
	356	"""
	357	self._perform_background_initial_update()
	358
	359	# Create a user and room
	360	u1 = self.register_user("u1", "pass")
	361	u1token = self.login("u1", "pass")
	362	r1 = self.helper.create_room_as(u1, tok=u1token)
	363
	364	# Get the current room stats
	365	r1stats_ante = self._get_current_stats("room", r1)
	366
	367	# Send a profile update into the room
	368	new_profile = {"displayname": "bob"}
	369	self.helper.change_membership(
	370	r1, u1, u1, "join", extra_data=new_profile, tok=u1token
	371	)
	372
	373	# Get the new room stats
	374	r1stats_post = self._get_current_stats("room", r1)
	375
	376	# Ensure that the user count did not changed
	377	self.assertEqual(r1stats_post["joined_members"], r1stats_ante["joined_members"])
	378	self.assertEqual(
	379	r1stats_post["local_users_in_room"], r1stats_ante["local_users_in_room"]
	380	)
	381
348	382	def test_send_state_event_nonoverwriting(self):
349	383	"""
350	384	When we send a non-overwriting state event, it increments total_events AND current_state_events

668	702
669	703	# preparation stage of the initial background update
670	704	# Ugh, have to reset this flag
671		self.store.db.updates._all_done = False
672
673		self.get_success(
674		self.store.db.simple_delete(
	705	self.store.db_pool.updates._all_done = False
	706
	707	self.get_success(
	708	self.store.db_pool.simple_delete(
675	709	"room_stats_current", {"1": 1}, "test_delete_stats"
676	710	)
677	711	)
678	712	self.get_success(
679		self.store.db.simple_delete(
	713	self.store.db_pool.simple_delete(
680	714	"user_stats_current", {"1": 1}, "test_delete_stats"
681	715	)
682	716	)

688	722
689	723	# now do the background updates
690	724
691		self.store.db.updates._all_done = False
692		self.get_success(
693		self.store.db.simple_insert(
	725	self.store.db_pool.updates._all_done = False
	726	self.get_success(
	727	self.store.db_pool.simple_insert(
694	728	"background_updates",
695	729	{
696		"update_name": "populate_stats_process_rooms",
	730	"update_name": "populate_stats_process_rooms_2",
697	731	"progress_json": "{}",
698	732	"depends_on": "populate_stats_prepare",
699	733	},
700	734	)
701	735	)
702	736	self.get_success(
703		self.store.db.simple_insert(
	737	self.store.db_pool.simple_insert(
704	738	"background_updates",
705	739	{
706	740	"update_name": "populate_stats_process_users",
707	741	"progress_json": "{}",
708		"depends_on": "populate_stats_process_rooms",
	742	"depends_on": "populate_stats_process_rooms_2",
709	743	},
710	744	)
711	745	)
712	746	self.get_success(
713		self.store.db.simple_insert(
	747	self.store.db_pool.simple_insert(
714	748	"background_updates",
715	749	{
716	750	"update_name": "populate_stats_cleanup",

721	755	)
722	756
723	757	while not self.get_success(
724		self.store.db.updates.has_completed_background_updates()
	758	self.store.db_pool.updates.has_completed_background_updates()
725	759	):
726	760	self.get_success(
727		self.store.db.updates.do_next_background_update(100), by=0.1
	761	self.store.db_pool.updates.do_next_background_update(100), by=0.1
728	762	)
729	763
730	764	r1stats_complete = self._get_current_stats("room", r1)

+5

-4

tests/handlers/test_typing.py less more

23	23	from synapse.types import UserID
24	24
25	25	from tests import unittest
	26	from tests.test_utils import make_awaitable
26	27	from tests.unittest import override_config
27	28	from tests.utils import register_federation_servlets
28	29

114	115	retry_timings_res
115	116	)
116	117
117		self.datastore.get_device_updates_by_remote.return_value = defer.succeed(
	118	self.datastore.get_device_updates_by_remote.side_effect = lambda destination, from_stream_id, limit: make_awaitable(
118	119	(0, [])
119	120	)
120	121

125	126
126	127	self.room_members = []
127	128
128		def check_user_in_room(room_id, user_id):
	129	async def check_user_in_room(room_id, user_id):
129	130	if user_id not in [u.to_string() for u in self.room_members]:
130	131	raise AuthError(401, "User is not in the room")
131		return defer.succeed(None)
	132	return None
132	133
133	134	hs.get_auth().check_user_in_room = check_user_in_room
134	135

150	151	self.datastore.get_current_state_deltas.return_value = (0, None)
151	152
152	153	self.datastore.get_to_device_stream_token = lambda: 0
153		self.datastore.get_new_device_msgs_for_remote = lambda args, *kargs: defer.succeed(
	154	self.datastore.get_new_device_msgs_for_remote = lambda args, *kargs: make_awaitable(
154	155	([], 0)
155	156	)
156	157	self.datastore.delete_device_msgs_for_remote = lambda args, *kargs: None

+13

-13

tests/handlers/test_user_directory.py less more

338	338
339	339	def get_users_in_public_rooms(self):
340	340	r = self.get_success(
341		self.store.db.simple_select_list(
	341	self.store.db_pool.simple_select_list(
342	342	"users_in_public_rooms", None, ("user_id", "room_id")
343	343	)
344	344	)

349	349
350	350	def get_users_who_share_private_rooms(self):
351	351	return self.get_success(
352		self.store.db.simple_select_list(
	352	self.store.db_pool.simple_select_list(
353	353	"users_who_share_private_rooms",
354	354	None,
355	355	["user_id", "other_user_id", "room_id"],

361	361	Add the background updates we need to run.
362	362	"""
363	363	# Ugh, have to reset this flag
364		self.store.db.updates._all_done = False
365
366		self.get_success(
367		self.store.db.simple_insert(
	364	self.store.db_pool.updates._all_done = False
	365
	366	self.get_success(
	367	self.store.db_pool.simple_insert(
368	368	"background_updates",
369	369	{
370	370	"update_name": "populate_user_directory_createtables",

373	373	)
374	374	)
375	375	self.get_success(
376		self.store.db.simple_insert(
	376	self.store.db_pool.simple_insert(
377	377	"background_updates",
378	378	{
379	379	"update_name": "populate_user_directory_process_rooms",

383	383	)
384	384	)
385	385	self.get_success(
386		self.store.db.simple_insert(
	386	self.store.db_pool.simple_insert(
387	387	"background_updates",
388	388	{
389	389	"update_name": "populate_user_directory_process_users",

393	393	)
394	394	)
395	395	self.get_success(
396		self.store.db.simple_insert(
	396	self.store.db_pool.simple_insert(
397	397	"background_updates",
398	398	{
399	399	"update_name": "populate_user_directory_cleanup",

436	436	self._add_background_updates()
437	437
438	438	while not self.get_success(
439		self.store.db.updates.has_completed_background_updates()
	439	self.store.db_pool.updates.has_completed_background_updates()
440	440	):
441	441	self.get_success(
442		self.store.db.updates.do_next_background_update(100), by=0.1
	442	self.store.db_pool.updates.do_next_background_update(100), by=0.1
443	443	)
444	444
445	445	shares_private = self.get_users_who_share_private_rooms()

475	475	self._add_background_updates()
476	476
477	477	while not self.get_success(
478		self.store.db.updates.has_completed_background_updates()
	478	self.store.db_pool.updates.has_completed_background_updates()
479	479	):
480	480	self.get_success(
481		self.store.db.updates.do_next_background_update(100), by=0.1
	481	self.store.db_pool.updates.do_next_background_update(100), by=0.1
482	482	)
483	483
484	484	shares_private = self.get_users_who_share_private_rooms()

+36

-14

tests/http/test_fedclient.py less more

57	57	@defer.inlineCallbacks
58	58	def do_request():
59	59	with LoggingContext("one") as context:
60		fetch_d = self.cl.get_json("testserv:8008", "foo/bar")
	60	fetch_d = defer.ensureDeferred(
	61	self.cl.get_json("testserv:8008", "foo/bar")
	62	)
61	63
62	64	# Nothing happened yet
63	65	self.assertNoResult(fetch_d)

119	121	"""
120	122	If the DNS lookup returns an error, it will bubble up.
121	123	"""
122		d = self.cl.get_json("testserv2:8008", "foo/bar", timeout=10000)
	124	d = defer.ensureDeferred(
	125	self.cl.get_json("testserv2:8008", "foo/bar", timeout=10000)
	126	)
123	127	self.pump()
124	128
125	129	f = self.failureResultOf(d)

127	131	self.assertIsInstance(f.value.inner_exception, DNSLookupError)
128	132
129	133	def test_client_connection_refused(self):
130		d = self.cl.get_json("testserv:8008", "foo/bar", timeout=10000)
	134	d = defer.ensureDeferred(
	135	self.cl.get_json("testserv:8008", "foo/bar", timeout=10000)
	136	)
131	137
132	138	self.pump()
133	139

153	159	If the HTTP request is not connected and is timed out, it'll give a
154	160	ConnectingCancelledError or TimeoutError.
155	161	"""
156		d = self.cl.get_json("testserv:8008", "foo/bar", timeout=10000)
	162	d = defer.ensureDeferred(
	163	self.cl.get_json("testserv:8008", "foo/bar", timeout=10000)
	164	)
157	165
158	166	self.pump()
159	167

183	191	If the HTTP request is connected, but gets no response before being
184	192	timed out, it'll give a ResponseNeverReceived.
185	193	"""
186		d = self.cl.get_json("testserv:8008", "foo/bar", timeout=10000)
	194	d = defer.ensureDeferred(
	195	self.cl.get_json("testserv:8008", "foo/bar", timeout=10000)
	196	)
187	197
188	198	self.pump()
189	199

225	235	# Try making a GET request to a blacklisted IPv4 address
226	236	# ------------------------------------------------------
227	237	# Make the request
228		d = cl.get_json("internal:8008", "foo/bar", timeout=10000)
	238	d = defer.ensureDeferred(cl.get_json("internal:8008", "foo/bar", timeout=10000))
229	239
230	240	# Nothing happened yet
231	241	self.assertNoResult(d)

243	253	# Try making a POST request to a blacklisted IPv6 address
244	254	# -------------------------------------------------------
245	255	# Make the request
246		d = cl.post_json("internalv6:8008", "foo/bar", timeout=10000)
	256	d = defer.ensureDeferred(
	257	cl.post_json("internalv6:8008", "foo/bar", timeout=10000)
	258	)
247	259
248	260	# Nothing has happened yet
249	261	self.assertNoResult(d)

262	274	# Try making a GET request to a non-blacklisted IPv4 address
263	275	# ----------------------------------------------------------
264	276	# Make the request
265		d = cl.post_json("fine:8008", "foo/bar", timeout=10000)
	277	d = defer.ensureDeferred(cl.post_json("fine:8008", "foo/bar", timeout=10000))
266	278
267	279	# Nothing has happened yet
268	280	self.assertNoResult(d)

285	297	request = MatrixFederationRequest(
286	298	method="GET", destination="testserv:8008", path="foo/bar"
287	299	)
288		d = self.cl._send_request(request, timeout=10000)
	300	d = defer.ensureDeferred(self.cl._send_request(request, timeout=10000))
289	301
290	302	self.pump()
291	303

309	321	If the HTTP request is connected, but gets no response before being
310	322	timed out, it'll give a ResponseNeverReceived.
311	323	"""
312		d = self.cl.post_json("testserv:8008", "foo/bar", timeout=10000)
	324	d = defer.ensureDeferred(
	325	self.cl.post_json("testserv:8008", "foo/bar", timeout=10000)
	326	)
313	327
314	328	self.pump()
315	329

341	355	requiring a trailing slash. We need to retry the request with a
342	356	trailing slash. Workaround for Synapse <= v0.99.3, explained in #3622.
343	357	"""
344		d = self.cl.get_json("testserv:8008", "foo/bar", try_trailing_slash_on_400=True)
	358	d = defer.ensureDeferred(
	359	self.cl.get_json("testserv:8008", "foo/bar", try_trailing_slash_on_400=True)
	360	)
345	361
346	362	# Send the request
347	363	self.pump()

394	410
395	411	See test_client_requires_trailing_slashes() for context.
396	412	"""
397		d = self.cl.get_json("testserv:8008", "foo/bar", try_trailing_slash_on_400=True)
	413	d = defer.ensureDeferred(
	414	self.cl.get_json("testserv:8008", "foo/bar", try_trailing_slash_on_400=True)
	415	)
398	416
399	417	# Send the request
400	418	self.pump()

431	449	self.failureResultOf(d)
432	450
433	451	def test_client_sends_body(self):
434		self.cl.post_json("testserv:8008", "foo/bar", timeout=10000, data={"a": "b"})
	452	defer.ensureDeferred(
	453	self.cl.post_json(
	454	"testserv:8008", "foo/bar", timeout=10000, data={"a": "b"}
	455	)
	456	)
435	457
436	458	self.pump()
437	459

452	474
453	475	def test_closes_connection(self):
454	476	"""Check that the client closes unused HTTP connections"""
455		d = self.cl.get_json("testserv:8008", "foo/bar")
	477	d = defer.ensureDeferred(self.cl.get_json("testserv:8008", "foo/bar"))
456	478
457	479	self.pump()
458	480

+3

-3

tests/replication/_base.py less more

64	64
65	65	# Since we use sqlite in memory databases we need to make sure the
66	66	# databases objects are the same.
67		self.worker_hs.get_datastore().db = hs.get_datastore().db
	67	self.worker_hs.get_datastore().db_pool = hs.get_datastore().db_pool
68	68
69	69	self.test_handler = self._build_replication_data_handler()
70	70	self.worker_hs.replication_data_handler = self.test_handler

197	197	self.streamer = self.hs.get_replication_streamer()
198	198
199	199	store = self.hs.get_datastore()
200		self.database = store.db
	200	self.database_pool = store.db_pool
201	201
202	202	self.reactor.lookups["testserv"] = "1.2.3.4"
203	203

253	253	)
254	254
255	255	store = worker_hs.get_datastore()
256		store.db._db_pool = self.database._db_pool
	256	store.db_pool._db_pool = self.database_pool._db_pool
257	257
258	258	repl_handler = ReplicationCommandHandler(worker_hs)
259	259	client = ClientReplicationStreamProtocol(

+4

-2

tests/replication/slave/storage/test_events.py less more

365	365	state_handler = self.hs.get_state_handler()
366	366	context = self.get_success(state_handler.compute_event_context(event))
367	367
368		self.master_store.add_push_actions_to_staging(
369		event.event_id, {user_id: actions for user_id, actions in push_actions}
	368	self.get_success(
	369	self.master_store.add_push_actions_to_staging(
	370	event.event_id, {user_id: actions for user_id, actions in push_actions}
	371	)
370	372	)
371	373	return event, context

+6

-7

tests/replication/test_federation_sender_shard.py less more

15	15
16	16	from mock import Mock
17	17
18		from twisted.internet import defer
19
20	18	from synapse.api.constants import EventTypes, Membership
21	19	from synapse.events.builder import EventBuilderFactory
22	20	from synapse.rest.admin import register_servlets_for_client_rest_resource

24	22	from synapse.types import UserID
25	23
26	24	from tests.replication._base import BaseMultiWorkerStreamTestCase
	25	from tests.test_utils import make_awaitable
27	26
28	27	logger = logging.getLogger(__name__)
29	28

45	44	new event.
46	45	"""
47	46	mock_client = Mock(spec=["put_json"])
48		mock_client.put_json.side_effect = lambda _, *__: defer.succeed({})
	47	mock_client.put_json.side_effect = lambda _, *__: make_awaitable({})
49	48
50	49	self.make_worker_hs(
51	50	"synapse.app.federation_sender",

73	72	new events.
74	73	"""
75	74	mock_client1 = Mock(spec=["put_json"])
76		mock_client1.put_json.side_effect = lambda _, *__: defer.succeed({})
	75	mock_client1.put_json.side_effect = lambda _, *__: make_awaitable({})
77	76	self.make_worker_hs(
78	77	"synapse.app.federation_sender",
79	78	{

85	84	)
86	85
87	86	mock_client2 = Mock(spec=["put_json"])
88		mock_client2.put_json.side_effect = lambda _, *__: defer.succeed({})
	87	mock_client2.put_json.side_effect = lambda _, *__: make_awaitable({})
89	88	self.make_worker_hs(
90	89	"synapse.app.federation_sender",
91	90	{

136	135	new typing EDUs.
137	136	"""
138	137	mock_client1 = Mock(spec=["put_json"])
139		mock_client1.put_json.side_effect = lambda _, *__: defer.succeed({})
	138	mock_client1.put_json.side_effect = lambda _, *__: make_awaitable({})
140	139	self.make_worker_hs(
141	140	"synapse.app.federation_sender",
142	141	{

148	147	)
149	148
150	149	mock_client2 = Mock(spec=["put_json"])
151		mock_client2.put_json.side_effect = lambda _, *__: defer.succeed({})
	150	mock_client2.put_json.side_effect = lambda _, *__: make_awaitable({})
152	151	self.make_worker_hs(
153	152	"synapse.app.federation_sender",
154	153	{

+2

-2

tests/rest/admin/test_admin.py less more

177	177
178	178	self.fetches = []
179	179
180		def get_file(destination, path, output_stream, args=None, max_size=None):
	180	async def get_file(destination, path, output_stream, args=None, max_size=None):
181	181	"""
182	182	Returns tuple[int,dict,str,int] of file length, response headers,
183	183	absolute URI, and response code.

191	191	d = Deferred()
192	192	d.addCallback(write_to)
193	193	self.fetches.append((d, destination, path, args))
194		return make_deferred_yieldable(d)
	194	return await make_deferred_yieldable(d)
195	195
196	196	client = Mock()
197	197	client.get_file = get_file

+57

-4

tests/rest/admin/test_room.py less more

282	282	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
283	283	self.assertEqual(Codes.BAD_JSON, channel.json_body["errcode"])
284	284
	285	def test_purge_is_not_bool(self):
	286	"""
	287	If parameter `purge` is not boolean, return an error
	288	"""
	289	body = json.dumps({"purge": "NotBool"})
	290
	291	request, channel = self.make_request(
	292	"POST",
	293	self.url,
	294	content=body.encode(encoding="utf_8"),
	295	access_token=self.admin_user_tok,
	296	)
	297	self.render(request)
	298
	299	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
	300	self.assertEqual(Codes.BAD_JSON, channel.json_body["errcode"])
	301
285	302	def test_purge_room_and_block(self):
286	303	"""Test to purge a room and block it.
287	304	Members will not be moved to a new room and will not receive a message.

296	313	# Assert one user in room
297	314	self._is_member(room_id=self.room_id, user_id=self.other_user)
298	315
299		body = json.dumps({"block": True})
	316	body = json.dumps({"block": True, "purge": True})
300	317
301	318	request, channel = self.make_request(
302	319	"POST",

330	347	# Assert one user in room
331	348	self._is_member(room_id=self.room_id, user_id=self.other_user)
332	349
333		body = json.dumps({"block": False})
	350	body = json.dumps({"block": False, "purge": True})
334	351
335	352	request, channel = self.make_request(
336	353	"POST",

347	364	self.assertIn("local_aliases", channel.json_body)
348	365
349	366	self._is_purged(self.room_id)
	367	self._is_blocked(self.room_id, expect=False)
	368	self._has_no_members(self.room_id)
	369
	370	def test_block_room_and_not_purge(self):
	371	"""Test to block a room without purging it.
	372	Members will not be moved to a new room and will not receive a message.
	373	The room will not be purged.
	374	"""
	375	# Test that room is not purged
	376	with self.assertRaises(AssertionError):
	377	self._is_purged(self.room_id)
	378
	379	# Test that room is not blocked
	380	self._is_blocked(self.room_id, expect=False)
	381
	382	# Assert one user in room
	383	self._is_member(room_id=self.room_id, user_id=self.other_user)
	384
	385	body = json.dumps({"block": False, "purge": False})
	386
	387	request, channel = self.make_request(
	388	"POST",
	389	self.url.encode("ascii"),
	390	content=body.encode(encoding="utf_8"),
	391	access_token=self.admin_user_tok,
	392	)
	393	self.render(request)
	394
	395	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
	396	self.assertEqual(None, channel.json_body["new_room_id"])
	397	self.assertEqual(self.other_user, channel.json_body["kicked_users"][0])
	398	self.assertIn("failed_to_kick_users", channel.json_body)
	399	self.assertIn("local_aliases", channel.json_body)
	400
	401	with self.assertRaises(AssertionError):
	402	self._is_purged(self.room_id)
350	403	self._is_blocked(self.room_id, expect=False)
351	404	self._has_no_members(self.room_id)
352	405

512	565	"state_groups_state",
513	566	):
514	567	count = self.get_success(
515		self.store.db.simple_select_one_onecol(
	568	self.store.db_pool.simple_select_one_onecol(
516	569	table=table,
517	570	keyvalues={"room_id": room_id},
518	571	retcol="COUNT(*)",

613	666	"state_groups_state",
614	667	):
615	668	count = self.get_success(
616		self.store.db.simple_select_one_onecol(
	669	self.store.db_pool.simple_select_one_onecol(
617	670	table=table,
618	671	keyvalues={"room_id": room_id},
619	672	retcol="COUNT(*)",

+7

-3

tests/rest/admin/test_user.py less more

19	19
20	20	from mock import Mock
21	21
	22	from twisted.internet import defer
	23
22	24	import synapse.rest.admin
23	25	from synapse.api.constants import UserTypes
24	26	from synapse.api.errors import HttpResponseException, ResourceLimitError

334	336	store = self.hs.get_datastore()
335	337
336	338	# Set monthly active users to the limit
337		store.get_monthly_active_count = Mock(return_value=self.hs.config.max_mau_value)
	339	store.get_monthly_active_count = Mock(
	340	return_value=defer.succeed(self.hs.config.max_mau_value)
	341	)
338	342	# Check that the blocking of monthly active users is working as expected
339	343	# The registration of a new user fails due to the limit
340	344	self.get_failure(

587	591
588	592	# Set monthly active users to the limit
589	593	self.store.get_monthly_active_count = Mock(
590		return_value=self.hs.config.max_mau_value
	594	return_value=defer.succeed(self.hs.config.max_mau_value)
591	595	)
592	596	# Check that the blocking of monthly active users is working as expected
593	597	# The registration of a new user fails due to the limit

627	631
628	632	# Set monthly active users to the limit
629	633	self.store.get_monthly_active_count = Mock(
630		return_value=self.hs.config.max_mau_value
	634	return_value=defer.succeed(self.hs.config.max_mau_value)
631	635	)
632	636	# Check that the blocking of monthly active users is working as expected
633	637	# The registration of a new user fails due to the limit

+3

-1

tests/rest/client/test_retention.py less more

143	143	# Get the create event to, later, check that we can still access it.
144	144	message_handler = self.hs.get_message_handler()
145	145	create_event = self.get_success(
146		message_handler.get_room_data(self.user_id, room_id, EventTypes.Create)
	146	message_handler.get_room_data(
	147	self.user_id, room_id, EventTypes.Create, state_key="", is_guest=False
	148	)
147	149	)
148	150
149	151	# Send a first event to the room. This is the event we'll want to be purged at the

+2

-2

tests/rest/client/v1/test_profile.py less more

69	69	profile_handler=self.mock_handler,
70	70	)
71	71
72		def _get_user_by_req(request=None, allow_guest=False):
73		return defer.succeed(synapse.types.create_requester(myid))
	72	async def _get_user_by_req(request=None, allow_guest=False):
	73	return synapse.types.create_requester(myid)
74	74
75	75	hs.get_auth().get_user_by_req = _get_user_by_req
76	76

+2

-4

tests/rest/client/v1/test_rooms.py less more

22	22
23	23	from mock import Mock
24	24
25		from twisted.internet import defer
26
27	25	import synapse.rest.admin
28	26	from synapse.api.constants import EventContentFields, EventTypes, Membership
29	27	from synapse.handlers.pagination import PurgeStatus

50	48
51	49	self.hs.get_federation_handler = Mock(return_value=Mock())
52	50
53		def _insert_client_ip(args, *kwargs):
54		return defer.succeed(None)
	51	async def _insert_client_ip(args, *kwargs):
	52	return None
55	53
56	54	self.hs.get_datastore().insert_client_ip = _insert_client_ip
57	55

+3

-3

tests/rest/client/v1/test_typing.py less more

45	45
46	46	hs.get_handlers().federation_handler = Mock()
47	47
48		def get_user_by_access_token(token=None, allow_guest=False):
	48	async def get_user_by_access_token(token=None, allow_guest=False):
49	49	return {
50	50	"user": UserID.from_string(self.auth_user_id),
51	51	"token_id": 1,

54	54
55	55	hs.get_auth().get_user_by_access_token = get_user_by_access_token
56	56
57		def _insert_client_ip(args, *kwargs):
58		return defer.succeed(None)
	57	async def _insert_client_ip(args, *kwargs):
	58	return None
59	59
60	60	hs.get_datastore().insert_client_ip = _insert_client_ip
61	61

+23

-1

tests/rest/client/v1/utils.py less more

87	87	expect_code=expect_code,
88	88	)
89	89
90		def change_membership(self, room, src, targ, membership, tok=None, expect_code=200):
	90	def change_membership(
	91	self,
	92	room: str,
	93	src: str,
	94	targ: str,
	95	membership: str,
	96	extra_data: dict = {},
	97	tok: Optional[str] = None,
	98	expect_code: int = 200,
	99	) -> None:
	100	"""
	101	Send a membership state event into a room.
	102
	103	Args:
	104	room: The ID of the room to send to
	105	src: The mxid of the event sender
	106	targ: The mxid of the event's target. The state key
	107	membership: The type of membership event
	108	extra_data: Extra information to include in the content of the event
	109	tok: The user access token to use
	110	expect_code: The expected HTTP response code
	111	"""
91	112	temp_id = self.auth_user_id
92	113	self.auth_user_id = src
93	114

96	117	path = path + "?access_token=%s" % tok
97	118
98	119	data = {"membership": membership}
	120	data.update(extra_data)
99	121
100	122	request, channel = make_request(
101	123	self.hs.get_reactor(), "PUT", path, json.dumps(data).encode("utf8")

+1

-1

tests/rest/client/v2_alpha/test_register.py less more

115	115	self.assertEquals(channel.result["code"], b"200", channel.result)
116	116	self.assertDictContainsSubset(det_data, channel.json_body)
117	117
	118	@override_config({"enable_registration": False})
118	119	def test_POST_disabled_registration(self):
119		self.hs.config.enable_registration = False
120	120	request_data = json.dumps({"username": "kermit", "password": "monkey"})
121	121	self.auth_result = (None, {"username": "kermit", "password": "monkey"}, None)
122	122

+2

-2

tests/rest/key/v2/test_remote_key_resource.py less more

52	52	Tell the mock http client to expect an outgoing GET request for the given key
53	53	"""
54	54
55		def get_json(destination, path, ignore_backoff=False, **kwargs):
	55	async def get_json(destination, path, ignore_backoff=False, **kwargs):
56	56	self.assertTrue(ignore_backoff)
57	57	self.assertEqual(destination, server_name)
58	58	key_id = "%s:%s" % (signing_key.alg, signing_key.version)

176	176
177	177	# wire up outbound POST /key/v2/query requests from hs2 so that they
178	178	# will be forwarded to hs1
179		def post_json(destination, path, data):
	179	async def post_json(destination, path, data):
180	180	self.assertEqual(destination, self.hs.hostname)
181	181	self.assertEqual(
182	182	path, "/_matrix/key/v2/query",

+34

-0

tests/rest/test_health.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15
	16	from synapse.rest.health import HealthResource
	17
	18	from tests import unittest
	19
	20
	21	class HealthCheckTests(unittest.HomeserverTestCase):
	22	def setUp(self):
	23	super().setUp()
	24
	25	# replace the JsonResource with a HealthResource.
	26	self.resource = HealthResource()
	27
	28	def test_health(self):
	29	request, channel = self.make_request("GET", "/health", shorthand=False)
	30	self.render(request)
	31
	32	self.assertEqual(request.code, 200)
	33	self.assertEqual(channel.result["body"], b"OK")

+6

-3

tests/server_notices/test_resource_limits_server_notices.py less more

26	26	)
27	27
28	28	from tests import unittest
	29	from tests.test_utils import make_awaitable
29	30	from tests.unittest import override_config
30	31	from tests.utils import default_config
31	32

78	79	return_value=defer.succeed("!something:localhost")
79	80	)
80	81	self._rlsn._store.add_tag_to_room = Mock(return_value=defer.succeed(None))
81		self._rlsn._store.get_tags_for_room = Mock(return_value=defer.succeed({}))
	82	self._rlsn._store.get_tags_for_room = Mock(
	83	side_effect=lambda user_id, room_id: make_awaitable({})
	84	)
82	85
83	86	@override_config({"hs_disabled": True})
84	87	def test_maybe_send_server_notice_disabled_hs(self):

257	260	self.user_id = "@user_id:test"
258	261
259	262	def test_server_notice_only_sent_once(self):
260		self.store.get_monthly_active_count = Mock(return_value=1000)
	263	self.store.get_monthly_active_count = Mock(return_value=defer.succeed(1000))
261	264
262	265	self.store.user_last_seen_monthly_active = Mock(
263	266	return_value=defer.succeed(1000)

274	277	self.server_notices_manager.get_or_create_notice_room_for_user(self.user_id)
275	278	)
276	279
277		token = self.get_success(self.event_source.get_current_token())
	280	token = self.event_source.get_current_token()
278	281	events, _ = self.get_success(
279	282	self.store.get_recent_events_for_room(
280	283	room_id, limit=100, end_token=token.room_key

+8

-8

tests/storage/test__base.py less more

322	322
323	323	self.table_name = "table_" + hs.get_secrets().token_hex(6)
324	324	self.get_success(
325		self.storage.db.runInteraction(
	325	self.storage.db_pool.runInteraction(
326	326	"create",
327	327	lambda x, a: x.execute(a),
328	328	"CREATE TABLE %s (id INTEGER, username TEXT, value TEXT)"

330	330	)
331	331	)
332	332	self.get_success(
333		self.storage.db.runInteraction(
	333	self.storage.db_pool.runInteraction(
334	334	"index",
335	335	lambda x, a: x.execute(a),
336	336	"CREATE UNIQUE INDEX %sindex ON %s(id, username)"

353	353	value_values = [["hello"], ["there"]]
354	354
355	355	self.get_success(
356		self.storage.db.runInteraction(
	356	self.storage.db_pool.runInteraction(
357	357	"test",
358		self.storage.db.simple_upsert_many_txn,
	358	self.storage.db_pool.simple_upsert_many_txn,
359	359	self.table_name,
360	360	key_names,
361	361	key_values,

366	366
367	367	# Check results are what we expect
368	368	res = self.get_success(
369		self.storage.db.simple_select_list(
	369	self.storage.db_pool.simple_select_list(
370	370	self.table_name, None, ["id, username, value"]
371	371	)
372	372	)

380	380	value_values = [["bleb"]]
381	381
382	382	self.get_success(
383		self.storage.db.runInteraction(
	383	self.storage.db_pool.runInteraction(
384	384	"test",
385		self.storage.db.simple_upsert_many_txn,
	385	self.storage.db_pool.simple_upsert_many_txn,
386	386	self.table_name,
387	387	key_names,
388	388	key_values,

393	393
394	394	# Check results are what we expect
395	395	res = self.get_success(
396		self.storage.db.simple_select_list(
	396	self.storage.db_pool.simple_select_list(
397	397	self.table_name, None, ["id, username, value"]
398	398	)
399	399	)

+15

-15

tests/storage/test_appservice.py less more

23	23
24	24	from synapse.appservice import ApplicationService, ApplicationServiceState
25	25	from synapse.config._base import ConfigError
26		from synapse.storage.data_stores.main.appservice import (
	26	from synapse.storage.database import DatabasePool, make_conn
	27	from synapse.storage.databases.main.appservice import (
27	28	ApplicationServiceStore,
28	29	ApplicationServiceTransactionStore,
29	30	)
30		from synapse.storage.database import Database, make_conn
31	31
32	32	from tests import unittest
33	33	from tests.utils import setup_test_homeserver

177	177	@defer.inlineCallbacks
178	178	def test_get_appservice_state_none(self):
179	179	service = Mock(id="999")
180		state = yield self.store.get_appservice_state(service)
	180	state = yield defer.ensureDeferred(self.store.get_appservice_state(service))
181	181	self.assertEquals(None, state)
182	182
183	183	@defer.inlineCallbacks
184	184	def test_get_appservice_state_up(self):
185	185	yield self._set_state(self.as_list[0]["id"], ApplicationServiceState.UP)
186	186	service = Mock(id=self.as_list[0]["id"])
187		state = yield self.store.get_appservice_state(service)
	187	state = yield defer.ensureDeferred(self.store.get_appservice_state(service))
188	188	self.assertEquals(ApplicationServiceState.UP, state)
189	189
190	190	@defer.inlineCallbacks

193	193	yield self._set_state(self.as_list[1]["id"], ApplicationServiceState.DOWN)
194	194	yield self._set_state(self.as_list[2]["id"], ApplicationServiceState.DOWN)
195	195	service = Mock(id=self.as_list[1]["id"])
196		state = yield self.store.get_appservice_state(service)
	196	state = yield defer.ensureDeferred(self.store.get_appservice_state(service))
197	197	self.assertEquals(ApplicationServiceState.DOWN, state)
198	198
199	199	@defer.inlineCallbacks
200	200	def test_get_appservices_by_state_none(self):
201		services = yield self.store.get_appservices_by_state(
202		ApplicationServiceState.DOWN
	201	services = yield defer.ensureDeferred(
	202	self.store.get_appservices_by_state(ApplicationServiceState.DOWN)
203	203	)
204	204	self.assertEquals(0, len(services))
205	205

338	338	def test_get_oldest_unsent_txn_none(self):
339	339	service = Mock(id=self.as_list[0]["id"])
340	340
341		txn = yield self.store.get_oldest_unsent_txn(service)
	341	txn = yield defer.ensureDeferred(self.store.get_oldest_unsent_txn(service))
342	342	self.assertEquals(None, txn)
343	343
344	344	@defer.inlineCallbacks

348	348	other_events = [Mock(event_id="e5"), Mock(event_id="e6")]
349	349
350	350	# we aren't testing store._base stuff here, so mock this out
351		self.store.get_events_as_list = Mock(return_value=events)
	351	self.store.get_events_as_list = Mock(return_value=defer.succeed(events))
352	352
353	353	yield self._insert_txn(self.as_list[1]["id"], 9, other_events)
354	354	yield self._insert_txn(service.id, 10, events)
355	355	yield self._insert_txn(service.id, 11, other_events)
356	356	yield self._insert_txn(service.id, 12, other_events)
357	357
358		txn = yield self.store.get_oldest_unsent_txn(service)
	358	txn = yield defer.ensureDeferred(self.store.get_oldest_unsent_txn(service))
359	359	self.assertEquals(service, txn.service)
360	360	self.assertEquals(10, txn.id)
361	361	self.assertEquals(events, txn.events)

365	365	yield self._set_state(self.as_list[0]["id"], ApplicationServiceState.DOWN)
366	366	yield self._set_state(self.as_list[1]["id"], ApplicationServiceState.UP)
367	367
368		services = yield self.store.get_appservices_by_state(
369		ApplicationServiceState.DOWN
	368	services = yield defer.ensureDeferred(
	369	self.store.get_appservices_by_state(ApplicationServiceState.DOWN)
370	370	)
371	371	self.assertEquals(1, len(services))
372	372	self.assertEquals(self.as_list[0]["id"], services[0].id)

378	378	yield self._set_state(self.as_list[2]["id"], ApplicationServiceState.DOWN)
379	379	yield self._set_state(self.as_list[3]["id"], ApplicationServiceState.UP)
380	380
381		services = yield self.store.get_appservices_by_state(
382		ApplicationServiceState.DOWN
	381	services = yield defer.ensureDeferred(
	382	self.store.get_appservices_by_state(ApplicationServiceState.DOWN)
383	383	)
384	384	self.assertEquals(2, len(services))
385	385	self.assertEquals(

390	390
391	391	# required for ApplicationServiceTransactionStoreTestCase tests
392	392	class TestTransactionStore(ApplicationServiceTransactionStore, ApplicationServiceStore):
393		def __init__(self, database: Database, db_conn, hs):
	393	def __init__(self, database: DatabasePool, db_conn, hs):
394	394	super(TestTransactionStore, self).__init__(database, db_conn, hs)
395	395
396	396

+5

-3

tests/storage/test_background_update.py less more

8	8
9	9	class BackgroundUpdateTestCase(unittest.HomeserverTestCase):
10	10	def prepare(self, reactor, clock, homeserver):
11		self.updates = self.hs.get_datastore().db.updates # type: BackgroundUpdater
	11	self.updates = (
	12	self.hs.get_datastore().db_pool.updates
	13	) # type: BackgroundUpdater
12	14	# the base test class should have run the real bg updates for us
13	15	self.assertTrue(
14	16	self.get_success(self.updates.has_completed_background_updates())

28	30
29	31	store = self.hs.get_datastore()
30	32	self.get_success(
31		store.db.simple_insert(
	33	store.db_pool.simple_insert(
32	34	"background_updates",
33	35	values={"update_name": "test_update", "progress_json": '{"my_key": 1}'},
34	36	)

39	41	def update(progress, count):
40	42	yield self.clock.sleep((count * duration_ms) / 1000)
41	43	progress = {"my_key": progress["my_key"] + 1}
42		yield store.db.runInteraction(
	44	yield store.db_pool.runInteraction(
43	45	"update_progress",
44	46	self.updates._background_update_progress_txn,
45	47	"test_update",

+11

-11

tests/storage/test_base.py less more

20	20	from twisted.internet import defer
21	21
22	22	from synapse.storage._base import SQLBaseStore
23		from synapse.storage.database import Database
	23	from synapse.storage.database import DatabasePool
24	24	from synapse.storage.engines import create_engine
25	25
26	26	from tests import unittest

56	56	fake_engine = Mock(wraps=engine)
57	57	fake_engine.can_native_upsert = False
58	58
59		db = Database(Mock(), Mock(config=sqlite_config), fake_engine)
	59	db = DatabasePool(Mock(), Mock(config=sqlite_config), fake_engine)
60	60	db._db_pool = self.db_pool
61	61
62	62	self.datastore = SQLBaseStore(db, None, hs)

65	65	def test_insert_1col(self):
66	66	self.mock_txn.rowcount = 1
67	67
68		yield self.datastore.db.simple_insert(
	68	yield self.datastore.db_pool.simple_insert(
69	69	table="tablename", values={"columname": "Value"}
70	70	)
71	71

77	77	def test_insert_3cols(self):
78	78	self.mock_txn.rowcount = 1
79	79
80		yield self.datastore.db.simple_insert(
	80	yield self.datastore.db_pool.simple_insert(
81	81	table="tablename",
82	82	# Use OrderedDict() so we can assert on the SQL generated
83	83	values=OrderedDict([("colA", 1), ("colB", 2), ("colC", 3)]),

92	92	self.mock_txn.rowcount = 1
93	93	self.mock_txn.__iter__ = Mock(return_value=iter([("Value",)]))
94	94
95		value = yield self.datastore.db.simple_select_one_onecol(
	95	value = yield self.datastore.db_pool.simple_select_one_onecol(
96	96	table="tablename", keyvalues={"keycol": "TheKey"}, retcol="retcol"
97	97	)
98	98

106	106	self.mock_txn.rowcount = 1
107	107	self.mock_txn.fetchone.return_value = (1, 2, 3)
108	108
109		ret = yield self.datastore.db.simple_select_one(
	109	ret = yield self.datastore.db_pool.simple_select_one(
110	110	table="tablename",
111	111	keyvalues={"keycol": "TheKey"},
112	112	retcols=["colA", "colB", "colC"],

122	122	self.mock_txn.rowcount = 0
123	123	self.mock_txn.fetchone.return_value = None
124	124
125		ret = yield self.datastore.db.simple_select_one(
	125	ret = yield self.datastore.db_pool.simple_select_one(
126	126	table="tablename",
127	127	keyvalues={"keycol": "Not here"},
128	128	retcols=["colA"],

137	137	self.mock_txn.__iter__ = Mock(return_value=iter([(1,), (2,), (3,)]))
138	138	self.mock_txn.description = (("colA", None, None, None, None, None, None),)
139	139
140		ret = yield self.datastore.db.simple_select_list(
	140	ret = yield self.datastore.db_pool.simple_select_list(
141	141	table="tablename", keyvalues={"keycol": "A set"}, retcols=["colA"]
142	142	)
143	143

150	150	def test_update_one_1col(self):
151	151	self.mock_txn.rowcount = 1
152	152
153		yield self.datastore.db.simple_update_one(
	153	yield self.datastore.db_pool.simple_update_one(
154	154	table="tablename",
155	155	keyvalues={"keycol": "TheKey"},
156	156	updatevalues={"columnname": "New Value"},

165	165	def test_update_one_4cols(self):
166	166	self.mock_txn.rowcount = 1
167	167
168		yield self.datastore.db.simple_update_one(
	168	yield self.datastore.db_pool.simple_update_one(
169	169	table="tablename",
170	170	keyvalues=OrderedDict([("colA", 1), ("colB", 2)]),
171	171	updatevalues=OrderedDict([("colC", 3), ("colD", 4)]),

180	180	def test_delete_one(self):
181	181	self.mock_txn.rowcount = 1
182	182
183		yield self.datastore.db.simple_delete_one(
	183	yield self.datastore.db_pool.simple_delete_one(
184	184	table="tablename", keyvalues={"keycol": "Go away"}
185	185	)
186	186

+6

-6

tests/storage/test_cleanup_extrems.py less more

46	46	"""
47	47	# Make sure we don't clash with in progress updates.
48	48	self.assertTrue(
49		self.store.db.updates._all_done, "Background updates are still ongoing"
	49	self.store.db_pool.updates._all_done, "Background updates are still ongoing"
50	50	)
51	51
52	52	schema_path = os.path.join(
53	53	prepare_database.dir_path,
54		"data_stores",
	54	"databases",
55	55	"main",
56	56	"schema",
57	57	"delta",

63	63	prepare_database.executescript(txn, schema_path)
64	64
65	65	self.get_success(
66		self.store.db.runInteraction(
	66	self.store.db_pool.runInteraction(
67	67	"test_delete_forward_extremities", run_delta_file
68	68	)
69	69	)
70	70
71	71	# Ugh, have to reset this flag
72		self.store.db.updates._all_done = False
	72	self.store.db_pool.updates._all_done = False
73	73
74	74	while not self.get_success(
75		self.store.db.updates.has_completed_background_updates()
	75	self.store.db_pool.updates.has_completed_background_updates()
76	76	):
77	77	self.get_success(
78		self.store.db.updates.do_next_background_update(100), by=0.1
	78	self.store.db_pool.updates.do_next_background_update(100), by=0.1
79	79	)
80	80
81	81	def test_soft_failed_extremities_handled_correctly(self):

+13

-13

tests/storage/test_client_ips.py less more

85	85	self.pump(0)
86	86
87	87	result = self.get_success(
88		self.store.db.simple_select_list(
	88	self.store.db_pool.simple_select_list(
89	89	table="user_ips",
90	90	keyvalues={"user_id": user_id},
91	91	retcols=["access_token", "ip", "user_agent", "device_id", "last_seen"],

116	116	self.pump(0)
117	117
118	118	result = self.get_success(
119		self.store.db.simple_select_list(
	119	self.store.db_pool.simple_select_list(
120	120	table="user_ips",
121	121	keyvalues={"user_id": user_id},
122	122	retcols=["access_token", "ip", "user_agent", "device_id", "last_seen"],

203	203	def test_devices_last_seen_bg_update(self):
204	204	# First make sure we have completed all updates.
205	205	while not self.get_success(
206		self.store.db.updates.has_completed_background_updates()
	206	self.store.db_pool.updates.has_completed_background_updates()
207	207	):
208	208	self.get_success(
209		self.store.db.updates.do_next_background_update(100), by=0.1
	209	self.store.db_pool.updates.do_next_background_update(100), by=0.1
210	210	)
211	211
212	212	user_id = "@user:id"

224	224
225	225	# But clear the associated entry in devices table
226	226	self.get_success(
227		self.store.db.simple_update(
	227	self.store.db_pool.simple_update(
228	228	table="devices",
229	229	keyvalues={"user_id": user_id, "device_id": device_id},
230	230	updatevalues={"last_seen": None, "ip": None, "user_agent": None},

251	251
252	252	# Register the background update to run again.
253	253	self.get_success(
254		self.store.db.simple_insert(
	254	self.store.db_pool.simple_insert(
255	255	table="background_updates",
256	256	values={
257	257	"update_name": "devices_last_seen",

262	262	)
263	263
264	264	# ... and tell the DataStore that it hasn't finished all updates yet
265		self.store.db.updates._all_done = False
	265	self.store.db_pool.updates._all_done = False
266	266
267	267	# Now let's actually drive the updates to completion
268	268	while not self.get_success(
269		self.store.db.updates.has_completed_background_updates()
	269	self.store.db_pool.updates.has_completed_background_updates()
270	270	):
271	271	self.get_success(
272		self.store.db.updates.do_next_background_update(100), by=0.1
	272	self.store.db_pool.updates.do_next_background_update(100), by=0.1
273	273	)
274	274
275	275	# We should now get the correct result again

292	292	def test_old_user_ips_pruned(self):
293	293	# First make sure we have completed all updates.
294	294	while not self.get_success(
295		self.store.db.updates.has_completed_background_updates()
	295	self.store.db_pool.updates.has_completed_background_updates()
296	296	):
297	297	self.get_success(
298		self.store.db.updates.do_next_background_update(100), by=0.1
	298	self.store.db_pool.updates.do_next_background_update(100), by=0.1
299	299	)
300	300
301	301	user_id = "@user:id"

314	314
315	315	# We should see that in the DB
316	316	result = self.get_success(
317		self.store.db.simple_select_list(
	317	self.store.db_pool.simple_select_list(
318	318	table="user_ips",
319	319	keyvalues={"user_id": user_id},
320	320	retcols=["access_token", "ip", "user_agent", "device_id", "last_seen"],

340	340
341	341	# We should get no results.
342	342	result = self.get_success(
343		self.store.db.simple_select_list(
	343	self.store.db_pool.simple_select_list(
344	344	table="user_ips",
345	345	keyvalues={"user_id": user_id},
346	346	retcols=["access_token", "ip", "user_agent", "device_id", "last_seen"],

+29

-15

tests/storage/test_devices.py less more

33	33
34	34	@defer.inlineCallbacks
35	35	def test_store_new_device(self):
36		yield self.store.store_device("user_id", "device_id", "display_name")
	36	yield defer.ensureDeferred(
	37	self.store.store_device("user_id", "device_id", "display_name")
	38	)
37	39
38	40	res = yield self.store.get_device("user_id", "device_id")
39	41	self.assertDictContainsSubset(

47	49
48	50	@defer.inlineCallbacks
49	51	def test_get_devices_by_user(self):
50		yield self.store.store_device("user_id", "device1", "display_name 1")
51		yield self.store.store_device("user_id", "device2", "display_name 2")
52		yield self.store.store_device("user_id2", "device3", "display_name 3")
	52	yield defer.ensureDeferred(
	53	self.store.store_device("user_id", "device1", "display_name 1")
	54	)
	55	yield defer.ensureDeferred(
	56	self.store.store_device("user_id", "device2", "display_name 2")
	57	)
	58	yield defer.ensureDeferred(
	59	self.store.store_device("user_id2", "device3", "display_name 3")
	60	)
53	61
54		res = yield self.store.get_devices_by_user("user_id")
	62	res = yield defer.ensureDeferred(self.store.get_devices_by_user("user_id"))
55	63	self.assertEqual(2, len(res.keys()))
56	64	self.assertDictContainsSubset(
57	65	{

75	83	device_ids = ["device_id1", "device_id2"]
76	84
77	85	# Add two device updates with a single stream_id
78		yield self.store.add_device_change_to_streams(
79		"user_id", device_ids, ["somehost"]
	86	yield defer.ensureDeferred(
	87	self.store.add_device_change_to_streams("user_id", device_ids, ["somehost"])
80	88	)
81	89
82	90	# Get all device updates ever meant for this remote
83		now_stream_id, device_updates = yield self.store.get_device_updates_by_remote(
84		"somehost", -1, limit=100
	91	now_stream_id, device_updates = yield defer.ensureDeferred(
	92	self.store.get_device_updates_by_remote("somehost", -1, limit=100)
85	93	)
86	94
87	95	# Check original device_ids are contained within these updates

98	106
99	107	@defer.inlineCallbacks
100	108	def test_update_device(self):
101		yield self.store.store_device("user_id", "device_id", "display_name 1")
	109	yield defer.ensureDeferred(
	110	self.store.store_device("user_id", "device_id", "display_name 1")
	111	)
102	112
103	113	res = yield self.store.get_device("user_id", "device_id")
104	114	self.assertEqual("display_name 1", res["display_name"])
105	115
106	116	# do a no-op first
107		yield self.store.update_device("user_id", "device_id")
	117	yield defer.ensureDeferred(self.store.update_device("user_id", "device_id"))
108	118	res = yield self.store.get_device("user_id", "device_id")
109	119	self.assertEqual("display_name 1", res["display_name"])
110	120
111	121	# do the update
112		yield self.store.update_device(
113		"user_id", "device_id", new_display_name="display_name 2"
	122	yield defer.ensureDeferred(
	123	self.store.update_device(
	124	"user_id", "device_id", new_display_name="display_name 2"
	125	)
114	126	)
115	127
116	128	# check it worked

120	132	@defer.inlineCallbacks
121	133	def test_update_unknown_device(self):
122	134	with self.assertRaises(synapse.api.errors.StoreError) as cm:
123		yield self.store.update_device(
124		"user_id", "unknown_device_id", new_display_name="display_name 2"
	135	yield defer.ensureDeferred(
	136	self.store.update_device(
	137	"user_id", "unknown_device_id", new_display_name="display_name 2"
	138	)
125	139	)
126	140	self.assertEqual(404, cm.exception.code)

+23

-9

tests/storage/test_directory.py less more

33	33
34	34	@defer.inlineCallbacks
35	35	def test_room_to_alias(self):
36		yield self.store.create_room_alias_association(
37		room_alias=self.alias, room_id=self.room.to_string(), servers=["test"]
	36	yield defer.ensureDeferred(
	37	self.store.create_room_alias_association(
	38	room_alias=self.alias, room_id=self.room.to_string(), servers=["test"]
	39	)
38	40	)
39	41
40	42	self.assertEquals(

44	46
45	47	@defer.inlineCallbacks
46	48	def test_alias_to_room(self):
47		yield self.store.create_room_alias_association(
48		room_alias=self.alias, room_id=self.room.to_string(), servers=["test"]
	49	yield defer.ensureDeferred(
	50	self.store.create_room_alias_association(
	51	room_alias=self.alias, room_id=self.room.to_string(), servers=["test"]
	52	)
49	53	)
50	54
51	55	self.assertObjectHasAttributes(
52	56	{"room_id": self.room.to_string(), "servers": ["test"]},
53		(yield self.store.get_association_from_room_alias(self.alias)),
	57	(
	58	yield defer.ensureDeferred(
	59	self.store.get_association_from_room_alias(self.alias)
	60	)
	61	),
54	62	)
55	63
56	64	@defer.inlineCallbacks
57	65	def test_delete_alias(self):
58		yield self.store.create_room_alias_association(
59		room_alias=self.alias, room_id=self.room.to_string(), servers=["test"]
	66	yield defer.ensureDeferred(
	67	self.store.create_room_alias_association(
	68	room_alias=self.alias, room_id=self.room.to_string(), servers=["test"]
	69	)
60	70	)
61	71
62		room_id = yield self.store.delete_room_alias(self.alias)
	72	room_id = yield defer.ensureDeferred(self.store.delete_room_alias(self.alias))
63	73	self.assertEqual(self.room.to_string(), room_id)
64	74
65	75	self.assertIsNone(
66		(yield self.store.get_association_from_room_alias(self.alias))
	76	(
	77	yield defer.ensureDeferred(
	78	self.store.get_association_from_room_alias(self.alias)
	79	)
	80	)
67	81	)

+17

-11

tests/storage/test_end_to_end_keys.py less more

29	29	now = 1470174257070
30	30	json = {"key": "value"}
31	31
32		yield self.store.store_device("user", "device", None)
	32	yield defer.ensureDeferred(self.store.store_device("user", "device", None))
33	33
34	34	yield self.store.set_e2e_device_keys("user", "device", now, json)
35	35
36		res = yield self.store.get_e2e_device_keys((("user", "device"),))
	36	res = yield defer.ensureDeferred(
	37	self.store.get_e2e_device_keys((("user", "device"),))
	38	)
37	39	self.assertIn("user", res)
38	40	self.assertIn("device", res["user"])
39	41	dev = res["user"]["device"]

44	46	now = 1470174257070
45	47	json = {"key": "value"}
46	48
47		yield self.store.store_device("user", "device", None)
	49	yield defer.ensureDeferred(self.store.store_device("user", "device", None))
48	50
49	51	changed = yield self.store.set_e2e_device_keys("user", "device", now, json)
50	52	self.assertTrue(changed)

60	62	json = {"key": "value"}
61	63
62	64	yield self.store.set_e2e_device_keys("user", "device", now, json)
63		yield self.store.store_device("user", "device", "display_name")
	65	yield defer.ensureDeferred(
	66	self.store.store_device("user", "device", "display_name")
	67	)
64	68
65		res = yield self.store.get_e2e_device_keys((("user", "device"),))
	69	res = yield defer.ensureDeferred(
	70	self.store.get_e2e_device_keys((("user", "device"),))
	71	)
66	72	self.assertIn("user", res)
67	73	self.assertIn("device", res["user"])
68	74	dev = res["user"]["device"]

74	80	def test_multiple_devices(self):
75	81	now = 1470174257070
76	82
77		yield self.store.store_device("user1", "device1", None)
78		yield self.store.store_device("user1", "device2", None)
79		yield self.store.store_device("user2", "device1", None)
80		yield self.store.store_device("user2", "device2", None)
	83	yield defer.ensureDeferred(self.store.store_device("user1", "device1", None))
	84	yield defer.ensureDeferred(self.store.store_device("user1", "device2", None))
	85	yield defer.ensureDeferred(self.store.store_device("user2", "device1", None))
	86	yield defer.ensureDeferred(self.store.store_device("user2", "device2", None))
81	87
82	88	yield self.store.set_e2e_device_keys("user1", "device1", now, {"key": "json11"})
83	89	yield self.store.set_e2e_device_keys("user1", "device2", now, {"key": "json12"})
84	90	yield self.store.set_e2e_device_keys("user2", "device1", now, {"key": "json21"})
85	91	yield self.store.set_e2e_device_keys("user2", "device2", now, {"key": "json22"})
86	92
87		res = yield self.store.get_e2e_device_keys(
88		(("user1", "device1"), ("user2", "device2"))
	93	res = yield defer.ensureDeferred(
	94	self.store.get_e2e_device_keys((("user1", "device1"), ("user2", "device2")))
89	95	)
90	96	self.assertIn("user1", res)
91	97	self.assertIn("device1", res["user1"])

+13

-11

tests/storage/test_event_federation.py less more

55	55	)
56	56
57	57	for i in range(0, 20):
58		self.get_success(self.store.db.runInteraction("insert", insert_event, i))
	58	self.get_success(
	59	self.store.db_pool.runInteraction("insert", insert_event, i)
	60	)
59	61
60	62	# this should get the last ten
61	63	r = self.get_success(self.store.get_prev_events_for_room(room_id))

80	82
81	83	for i in range(0, 20):
82	84	self.get_success(
83		self.store.db.runInteraction("insert", insert_event, i, room1)
84		)
85		self.get_success(
86		self.store.db.runInteraction("insert", insert_event, i, room2)
87		)
88		self.get_success(
89		self.store.db.runInteraction("insert", insert_event, i, room3)
	85	self.store.db_pool.runInteraction("insert", insert_event, i, room1)
	86	)
	87	self.get_success(
	88	self.store.db_pool.runInteraction("insert", insert_event, i, room2)
	89	)
	90	self.get_success(
	91	self.store.db_pool.runInteraction("insert", insert_event, i, room3)
90	92	)
91	93
92	94	# Test simple case

163	165
164	166	depth = depth_map[event_id]
165	167
166		self.store.db.simple_insert_txn(
	168	self.store.db_pool.simple_insert_txn(
167	169	txn,
168	170	table="events",
169	171	values={

178	180	},
179	181	)
180	182
181		self.store.db.simple_insert_many_txn(
	183	self.store.db_pool.simple_insert_many_txn(
182	184	txn,
183	185	table="event_auth",
184	186	values=[

191	193	for event_id in auth_graph:
192	194	next_stream_ordering += 1
193	195	self.get_success(
194		self.store.db.runInteraction(
	196	self.store.db_pool.runInteraction(
195	197	"insert", insert_event, event_id, next_stream_ordering
196	198	)
197	199	)

+19

-13

tests/storage/test_event_push_actions.py less more

38	38
39	39	@defer.inlineCallbacks
40	40	def test_get_unread_push_actions_for_user_in_range_for_http(self):
41		yield self.store.get_unread_push_actions_for_user_in_range_for_http(
42		USER_ID, 0, 1000, 20
	41	yield defer.ensureDeferred(
	42	self.store.get_unread_push_actions_for_user_in_range_for_http(
	43	USER_ID, 0, 1000, 20
	44	)
43	45	)
44	46
45	47	@defer.inlineCallbacks
46	48	def test_get_unread_push_actions_for_user_in_range_for_email(self):
47		yield self.store.get_unread_push_actions_for_user_in_range_for_email(
48		USER_ID, 0, 1000, 20
	49	yield defer.ensureDeferred(
	50	self.store.get_unread_push_actions_for_user_in_range_for_email(
	51	USER_ID, 0, 1000, 20
	52	)
49	53	)
50	54
51	55	@defer.inlineCallbacks

55	59
56	60	@defer.inlineCallbacks
57	61	def _assert_counts(noitf_count, highlight_count):
58		counts = yield self.store.db.runInteraction(
	62	counts = yield self.store.db_pool.runInteraction(
59	63	"", self.store._get_unread_counts_by_pos_txn, room_id, user_id, 0
60	64	)
61	65	self.assertEquals(

71	75	event.internal_metadata.stream_ordering = stream
72	76	event.depth = stream
73	77
74		yield self.store.add_push_actions_to_staging(
75		event.event_id, {user_id: action}
76		)
77		yield self.store.db.runInteraction(
	78	yield defer.ensureDeferred(
	79	self.store.add_push_actions_to_staging(
	80	event.event_id, {user_id: action}
	81	)
	82	)
	83	yield self.store.db_pool.runInteraction(
78	84	"",
79	85	self.persist_events_store._set_push_actions_for_event_and_users_txn,
80	86	[(event, None)],

82	88	)
83	89
84	90	def _rotate(stream):
85		return self.store.db.runInteraction(
	91	return self.store.db_pool.runInteraction(
86	92	"", self.store._rotate_notifs_before_txn, stream
87	93	)
88	94
89	95	def _mark_read(stream, depth):
90		return self.store.db.runInteraction(
	96	return self.store.db_pool.runInteraction(
91	97	"",
92	98	self.store._remove_old_push_actions_before_txn,
93	99	room_id,

116	122	yield _inject_actions(6, PlAIN_NOTIF)
117	123	yield _rotate(7)
118	124
119		yield self.store.db.simple_delete(
	125	yield self.store.db_pool.simple_delete(
120	126	table="event_push_actions", keyvalues={"1": 1}, desc=""
121	127	)
122	128

135	141	@defer.inlineCallbacks
136	142	def test_find_first_stream_ordering_after_ts(self):
137	143	def add_event(so, ts):
138		return self.store.db.simple_insert(
	144	return self.store.db_pool.simple_insert(
139	145	"events",
140	146	{
141	147	"stream_ordering": so,

+7

-7

tests/storage/test_id_generators.py less more

13	13	# limitations under the License.
14	14
15	15
16		from synapse.storage.database import Database
	16	from synapse.storage.database import DatabasePool
17	17	from synapse.storage.util.id_generators import MultiWriterIdGenerator
18	18
19	19	from tests.unittest import HomeserverTestCase

26	26
27	27	def prepare(self, reactor, clock, hs):
28	28	self.store = hs.get_datastore()
29		self.db = self.store.db # type: Database
	29	self.db_pool = self.store.db_pool # type: DatabasePool
30	30
31		self.get_success(self.db.runInteraction("_setup_db", self._setup_db))
	31	self.get_success(self.db_pool.runInteraction("_setup_db", self._setup_db))
32	32
33	33	def _setup_db(self, txn):
34	34	txn.execute("CREATE SEQUENCE foobar_seq")

46	46	def _create(conn):
47	47	return MultiWriterIdGenerator(
48	48	conn,
49		self.db,
	49	self.db_pool,
50	50	instance_name=instance_name,
51	51	table="foobar",
52	52	instance_column="instance_name",

54	54	sequence_name="foobar_seq",
55	55	)
56	56
57		return self.get_success(self.db.runWithConnection(_create))
	57	return self.get_success(self.db_pool.runWithConnection(_create))
58	58
59	59	def _insert_rows(self, instance_name: str, number: int):
60	60	def _insert(txn):

64	64	(instance_name,),
65	65	)
66	66
67		self.get_success(self.db.runInteraction("test_single_instance", _insert))
	67	self.get_success(self.db_pool.runInteraction("test_single_instance", _insert))
68	68
69	69	def test_empty(self):
70	70	"""Test an ID generator against an empty database gives sensible

177	177	self.assertEqual(id_gen.get_positions(), {"master": 7})
178	178	self.assertEqual(id_gen.get_current_token("master"), 7)
179	179
180		self.get_success(self.db.runInteraction("test", _get_next_txn))
	180	self.get_success(self.db_pool.runInteraction("test", _get_next_txn))
181	181
182	182	self.assertEqual(id_gen.get_positions(), {"master": 8})
183	183	self.assertEqual(id_gen.get_current_token("master"), 8)

+22

-9

tests/storage/test_monthly_active_users.py less more

18	18	from synapse.api.constants import UserTypes
19	19
20	20	from tests import unittest
	21	from tests.test_utils import make_awaitable
21	22	from tests.unittest import default_config, override_config
22	23
23	24	FORTY_DAYS = 40 * 24 * 60 * 60

77	78	# XXX why are we doing this here? this function is only run at startup
78	79	# so it is odd to re-run it here.
79	80	self.get_success(
80		self.store.db.runInteraction(
	81	self.store.db_pool.runInteraction(
81	82	"initialise", self.store._initialise_reserved_users, threepids
82	83	)
83	84	)

203	204	self.store.user_add_threepid(user, "email", email, now, now)
204	205	)
205	206
206		d = self.store.db.runInteraction(
	207	d = self.store.db_pool.runInteraction(
207	208	"initialise", self.store._initialise_reserved_users, threepids
208	209	)
209	210	self.get_success(d)

229	230	)
230	231	self.get_success(d)
231	232
232		self.store.upsert_monthly_active_user = Mock()
	233	self.store.upsert_monthly_active_user = Mock(
	234	side_effect=lambda user_id: make_awaitable(None)
	235	)
233	236
234	237	d = self.store.populate_monthly_active_users(user_id)
235	238	self.get_success(d)

237	240	self.store.upsert_monthly_active_user.assert_not_called()
238	241
239	242	def test_populate_monthly_users_should_update(self):
240		self.store.upsert_monthly_active_user = Mock()
	243	self.store.upsert_monthly_active_user = Mock(
	244	side_effect=lambda user_id: make_awaitable(None)
	245	)
241	246
242	247	self.store.is_trial_user = Mock(return_value=defer.succeed(False))
243	248

250	255	self.store.upsert_monthly_active_user.assert_called_once()
251	256
252	257	def test_populate_monthly_users_should_not_update(self):
253		self.store.upsert_monthly_active_user = Mock()
	258	self.store.upsert_monthly_active_user = Mock(
	259	side_effect=lambda user_id: make_awaitable(None)
	260	)
254	261
255	262	self.store.is_trial_user = Mock(return_value=defer.succeed(False))
256	263	self.store.user_last_seen_monthly_active = Mock(

279	286	]
280	287
281	288	self.hs.config.mau_limits_reserved_threepids = threepids
282		d = self.store.db.runInteraction(
	289	d = self.store.db_pool.runInteraction(
283	290	"initialise", self.store._initialise_reserved_users, threepids
284	291	)
285	292	self.get_success(d)

292	299	self.get_success(self.store.register_user(user_id=user2, password_hash=None))
293	300
294	301	now = int(self.hs.get_clock().time_msec())
295		self.store.user_add_threepid(user1, "email", user1_email, now, now)
296		self.store.user_add_threepid(user2, "email", user2_email, now, now)
	302	self.get_success(
	303	self.store.user_add_threepid(user1, "email", user1_email, now, now)
	304	)
	305	self.get_success(
	306	self.store.user_add_threepid(user2, "email", user2_email, now, now)
	307	)
297	308
298	309	users = self.get_success(self.store.get_registered_reserved_users())
299	310	self.assertEqual(len(users), len(threepids))

332	343
333	344	@override_config({"limit_usage_by_mau": False, "mau_stats_only": False})
334	345	def test_no_users_when_not_tracking(self):
335		self.store.upsert_monthly_active_user = Mock()
	346	self.store.upsert_monthly_active_user = Mock(
	347	side_effect=lambda user_id: make_awaitable(None)
	348	)
336	349
337	350	self.get_success(self.store.populate_monthly_active_users("@user:sever"))
338	351

+6

-2

tests/storage/test_purge.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14
	15	from twisted.internet import defer
14	16
15	17	from synapse.rest.client.v1 import room
16	18

48	50	event = self.successResultOf(event)
49	51
50	52	# Purge everything before this topological token
51		purge = storage.purge_events.purge_history(self.room_id, event, True)
	53	purge = defer.ensureDeferred(
	54	storage.purge_events.purge_history(self.room_id, event, True)
	55	)
52	56	self.pump()
53	57	self.assertEqual(self.successResultOf(purge), None)
54	58

87	91	)
88	92
89	93	# Purge everything before this topological token
90		purge = storage.purge_history(self.room_id, event, True)
	94	purge = defer.ensureDeferred(storage.purge_history(self.room_id, event, True))
91	95	self.pump()
92	96	f = self.failureResultOf(purge)
93	97	self.assertIn("greater than forward", f.value.args[0])

+9

-3

tests/storage/test_redaction.py less more

236	236
237	237	@defer.inlineCallbacks
238	238	def build(self, prev_event_ids):
239		built_event = yield self._base_builder.build(prev_event_ids)
	239	built_event = yield defer.ensureDeferred(
	240	self._base_builder.build(prev_event_ids)
	241	)
240	242
241	243	built_event._event_id = self._event_id
242	244	built_event._dict["event_id"] = self._event_id

247	249	@property
248	250	def room_id(self):
249	251	return self._base_builder.room_id
	252
	253	@property
	254	def type(self):
	255	return self._base_builder.type
250	256
251	257	event_1, context_1 = self.get_success(
252	258	self.event_creation_handler.create_new_client_event(

340	346	)
341	347
342	348	event_json = self.get_success(
343		self.store.db.simple_select_one_onecol(
	349	self.store.db_pool.simple_select_one_onecol(
344	350	table="event_json",
345	351	keyvalues={"event_id": msg_event.event_id},
346	352	retcol="json",

358	364	self.reactor.advance(60 * 60 * 2)
359	365
360	366	event_json = self.get_success(
361		self.store.db.simple_select_one_onecol(
	367	self.store.db_pool.simple_select_one_onecol(
362	368	table="event_json",
363	369	keyvalues={"event_id": msg_event.event_id},
364	370	retcol="json",

+12

-6

tests/storage/test_registration.py less more

57	57	@defer.inlineCallbacks
58	58	def test_add_tokens(self):
59	59	yield self.store.register_user(self.user_id, self.pwhash)
60		yield self.store.add_access_token_to_user(
61		self.user_id, self.tokens[1], self.device_id, valid_until_ms=None
	60	yield defer.ensureDeferred(
	61	self.store.add_access_token_to_user(
	62	self.user_id, self.tokens[1], self.device_id, valid_until_ms=None
	63	)
62	64	)
63	65
64	66	result = yield self.store.get_user_by_access_token(self.tokens[1])

73	75	def test_user_delete_access_tokens(self):
74	76	# add some tokens
75	77	yield self.store.register_user(self.user_id, self.pwhash)
76		yield self.store.add_access_token_to_user(
77		self.user_id, self.tokens[0], device_id=None, valid_until_ms=None
	78	yield defer.ensureDeferred(
	79	self.store.add_access_token_to_user(
	80	self.user_id, self.tokens[0], device_id=None, valid_until_ms=None
	81	)
78	82	)
79		yield self.store.add_access_token_to_user(
80		self.user_id, self.tokens[1], self.device_id, valid_until_ms=None
	83	yield defer.ensureDeferred(
	84	self.store.add_access_token_to_user(
	85	self.user_id, self.tokens[1], self.device_id, valid_until_ms=None
	86	)
81	87	)
82	88
83	89	# now delete some

+18

-12

tests/storage/test_room.py less more

36	36	self.alias = RoomAlias.from_string("#a-room-name:test")
37	37	self.u_creator = UserID.from_string("@creator:test")
38	38
39		yield self.store.store_room(
40		self.room.to_string(),
41		room_creator_user_id=self.u_creator.to_string(),
42		is_public=True,
43		room_version=RoomVersions.V1,
	39	yield defer.ensureDeferred(
	40	self.store.store_room(
	41	self.room.to_string(),
	42	room_creator_user_id=self.u_creator.to_string(),
	43	is_public=True,
	44	room_version=RoomVersions.V1,
	45	)
44	46	)
45	47
46	48	@defer.inlineCallbacks

87	89
88	90	self.room = RoomID.from_string("!abcde:test")
89	91
90		yield self.store.store_room(
91		self.room.to_string(),
92		room_creator_user_id="@creator:text",
93		is_public=True,
94		room_version=RoomVersions.V1,
	92	yield defer.ensureDeferred(
	93	self.store.store_room(
	94	self.room.to_string(),
	95	room_creator_user_id="@creator:text",
	96	is_public=True,
	97	room_version=RoomVersions.V1,
	98	)
95	99	)
96	100
97	101	@defer.inlineCallbacks
98	102	def inject_room_event(self, **kwargs):
99		yield self.storage.persistence.persist_event(
100		self.event_factory.create_event(room_id=self.room.to_string(), **kwargs)
	103	yield defer.ensureDeferred(
	104	self.storage.persistence.persist_event(
	105	self.event_factory.create_event(room_id=self.room.to_string(), **kwargs)
	106	)
101	107	)
102	108
103	109	@defer.inlineCallbacks

+7

-7

tests/storage/test_roommember.py less more

178	178	def test_can_rerun_update(self):
179	179	# First make sure we have completed all updates.
180	180	while not self.get_success(
181		self.store.db.updates.has_completed_background_updates()
	181	self.store.db_pool.updates.has_completed_background_updates()
182	182	):
183	183	self.get_success(
184		self.store.db.updates.do_next_background_update(100), by=0.1
	184	self.store.db_pool.updates.do_next_background_update(100), by=0.1
185	185	)
186	186
187	187	# Now let's create a room, which will insert a membership

191	191
192	192	# Register the background update to run again.
193	193	self.get_success(
194		self.store.db.simple_insert(
	194	self.store.db_pool.simple_insert(
195	195	table="background_updates",
196	196	values={
197	197	"update_name": "current_state_events_membership",

202	202	)
203	203
204	204	# ... and tell the DataStore that it hasn't finished all updates yet
205		self.store.db.updates._all_done = False
	205	self.store.db_pool.updates._all_done = False
206	206
207	207	# Now let's actually drive the updates to completion
208	208	while not self.get_success(
209		self.store.db.updates.has_completed_background_updates()
	209	self.store.db_pool.updates.has_completed_background_updates()
210	210	):
211	211	self.get_success(
212		self.store.db.updates.do_next_background_update(100), by=0.1
213		)
	212	self.store.db_pool.updates.do_next_background_update(100), by=0.1
	213	)

+46

-30

tests/storage/test_state.py less more

43	43
44	44	self.room = RoomID.from_string("!abc123:test")
45	45
46		yield self.store.store_room(
47		self.room.to_string(),
48		room_creator_user_id="@creator:text",
49		is_public=True,
50		room_version=RoomVersions.V1,
	46	yield defer.ensureDeferred(
	47	self.store.store_room(
	48	self.room.to_string(),
	49	room_creator_user_id="@creator:text",
	50	is_public=True,
	51	room_version=RoomVersions.V1,
	52	)
51	53	)
52	54
53	55	@defer.inlineCallbacks

67	69	self.event_creation_handler.create_new_client_event(builder)
68	70	)
69	71
70		yield self.storage.persistence.persist_event(event, context)
	72	yield defer.ensureDeferred(
	73	self.storage.persistence.persist_event(event, context)
	74	)
71	75
72	76	return event
73	77

86	90	self.room, self.u_alice, EventTypes.Name, "", {"name": "test room"}
87	91	)
88	92
89		state_group_map = yield self.storage.state.get_state_groups_ids(
90		self.room, [e2.event_id]
	93	state_group_map = yield defer.ensureDeferred(
	94	self.storage.state.get_state_groups_ids(self.room, [e2.event_id])
91	95	)
92	96	self.assertEqual(len(state_group_map), 1)
93	97	state_map = list(state_group_map.values())[0]

105	109	self.room, self.u_alice, EventTypes.Name, "", {"name": "test room"}
106	110	)
107	111
108		state_group_map = yield self.storage.state.get_state_groups(
109		self.room, [e2.event_id]
	112	state_group_map = yield defer.ensureDeferred(
	113	self.storage.state.get_state_groups(self.room, [e2.event_id])
110	114	)
111	115	self.assertEqual(len(state_group_map), 1)
112	116	state_list = list(state_group_map.values())[0]

147	151	)
148	152
149	153	# check we get the full state as of the final event
150		state = yield self.storage.state.get_state_for_event(e5.event_id)
	154	state = yield defer.ensureDeferred(
	155	self.storage.state.get_state_for_event(e5.event_id)
	156	)
151	157
152	158	self.assertIsNotNone(e4)
153	159

163	169	)
164	170
165	171	# check we can filter to the m.room.name event (with a '' state key)
166		state = yield self.storage.state.get_state_for_event(
167		e5.event_id, StateFilter.from_types([(EventTypes.Name, "")])
	172	state = yield defer.ensureDeferred(
	173	self.storage.state.get_state_for_event(
	174	e5.event_id, StateFilter.from_types([(EventTypes.Name, "")])
	175	)
168	176	)
169	177
170	178	self.assertStateMapEqual({(e2.type, e2.state_key): e2}, state)
171	179
172	180	# check we can filter to the m.room.name event (with a wildcard None state key)
173		state = yield self.storage.state.get_state_for_event(
174		e5.event_id, StateFilter.from_types([(EventTypes.Name, None)])
	181	state = yield defer.ensureDeferred(
	182	self.storage.state.get_state_for_event(
	183	e5.event_id, StateFilter.from_types([(EventTypes.Name, None)])
	184	)
175	185	)
176	186
177	187	self.assertStateMapEqual({(e2.type, e2.state_key): e2}, state)
178	188
179	189	# check we can grab the m.room.member events (with a wildcard None state key)
180		state = yield self.storage.state.get_state_for_event(
181		e5.event_id, StateFilter.from_types([(EventTypes.Member, None)])
	190	state = yield defer.ensureDeferred(
	191	self.storage.state.get_state_for_event(
	192	e5.event_id, StateFilter.from_types([(EventTypes.Member, None)])
	193	)
182	194	)
183	195
184	196	self.assertStateMapEqual(

187	199
188	200	# check we can grab a specific room member without filtering out the
189	201	# other event types
190		state = yield self.storage.state.get_state_for_event(
191		e5.event_id,
192		state_filter=StateFilter(
193		types={EventTypes.Member: {self.u_alice.to_string()}},
194		include_others=True,
195		),
	202	state = yield defer.ensureDeferred(
	203	self.storage.state.get_state_for_event(
	204	e5.event_id,
	205	state_filter=StateFilter(
	206	types={EventTypes.Member: {self.u_alice.to_string()}},
	207	include_others=True,
	208	),
	209	)
196	210	)
197	211
198	212	self.assertStateMapEqual(

205	219	)
206	220
207	221	# check that we can grab everything except members
208		state = yield self.storage.state.get_state_for_event(
209		e5.event_id,
210		state_filter=StateFilter(
211		types={EventTypes.Member: set()}, include_others=True
212		),
	222	state = yield defer.ensureDeferred(
	223	self.storage.state.get_state_for_event(
	224	e5.event_id,
	225	state_filter=StateFilter(
	226	types={EventTypes.Member: set()}, include_others=True
	227	),
	228	)
213	229	)
214	230
215	231	self.assertStateMapEqual(

221	237	#######################################################
222	238
223	239	room_id = self.room.to_string()
224		group_ids = yield self.storage.state.get_state_groups_ids(
225		room_id, [e5.event_id]
	240	group_ids = yield defer.ensureDeferred(
	241	self.storage.state.get_state_groups_ids(room_id, [e5.event_id])
226	242	)
227	243	group = list(group_ids.keys())[0]
228	244

+2

-2

tests/storage/test_user_directory.py less more

39	39	def test_search_user_dir(self):
40	40	# normally when alice searches the directory she should just find
41	41	# bob because bobby doesn't share a room with her.
42		r = yield self.store.search_user_dir(ALICE, "bob", 10)
	42	r = yield defer.ensureDeferred(self.store.search_user_dir(ALICE, "bob", 10))
43	43	self.assertFalse(r["limited"])
44	44	self.assertEqual(1, len(r["results"]))
45	45	self.assertDictEqual(

50	50	def test_search_user_dir_all_users(self):
51	51	self.hs.config.user_directory_search_all_users = True
52	52	try:
53		r = yield self.store.search_user_dir(ALICE, "bob", 10)
	53	r = yield defer.ensureDeferred(self.store.search_user_dir(ALICE, "bob", 10))
54	54	self.assertFalse(r["limited"])
55	55	self.assertEqual(2, len(r["results"]))
56	56	self.assertDictEqual(

+18

-2

tests/test_federation.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
0	15	from mock import Mock
1	16
2	17	from twisted.internet.defer import ensureDeferred, maybeDeferred, succeed

9	24
10	25	from tests import unittest
11	26	from tests.server import ThreadedMemoryReactorClock, setup_test_homeserver
	27	from tests.test_utils import make_awaitable
12	28
13	29
14	30	class MessageAcceptTests(unittest.HomeserverTestCase):

94	110	prev_events that said event references.
95	111	"""
96	112
97		def post_json(destination, path, data, headers=None, timeout=0):
	113	async def post_json(destination, path, data, headers=None, timeout=0):
98	114	# If it asks us for new missing events, give them NOTHING
99	115	if path.startswith("/_matrix/federation/v1/get_missing_events/"):
100	116	return {"events": []}

172	188	# Register a mock on the store so that the incoming update doesn't fail because
173	189	# we don't share a room with the user.
174	190	store = self.homeserver.get_datastore()
175		store.get_rooms_for_user = Mock(return_value=succeed(["!someroom:test"]))
	191	store.get_rooms_for_user = Mock(return_value=make_awaitable(["!someroom:test"]))
176	192
177	193	# Manually inject a fake device list update. We need this update to include at
178	194	# least one prev_id so that the user's device list will need to be retried.

+42

-3

tests/test_server.py less more

156	156	self.assertEqual(channel.json_body["error"], "Unrecognized request")
157	157	self.assertEqual(channel.json_body["errcode"], "M_UNRECOGNIZED")
158	158
	159	def test_head_request(self):
	160	"""
	161	JsonResource.handler_for_request gives correctly decoded URL args to
	162	the callback, while Twisted will give the raw bytes of URL query
	163	arguments.
	164	"""
	165
	166	def _callback(request, **kwargs):
	167	return 200, {"result": True}
	168
	169	res = JsonResource(self.homeserver)
	170	res.register_paths(
	171	"GET", [re.compile("^/_matrix/foo$")], _callback, "test_servlet",
	172	)
	173
	174	# The path was registered as GET, but this is a HEAD request.
	175	request, channel = make_request(self.reactor, b"HEAD", b"/_matrix/foo")
	176	render(request, res, self.reactor)
	177
	178	self.assertEqual(channel.result["code"], b"200")
	179	self.assertNotIn("body", channel.result)
	180	self.assertEqual(channel.headers.getRawHeaders(b"Content-Length"), [b"15"])
	181
159	182
160	183	class OptionsResourceTests(unittest.TestCase):
161	184	def setUp(self):

254	277	self.reactor = ThreadedMemoryReactorClock()
255	278
256	279	def test_good_response(self):
257		def callback(request):
	280	async def callback(request):
258	281	request.write(b"response")
259	282	request.finish()
260	283

274	297	with the right location.
275	298	"""
276	299
277		def callback(request, **kwargs):
	300	async def callback(request, **kwargs):
278	301	raise RedirectException(b"/look/an/eagle", 301)
279	302
280	303	res = WrapHtmlRequestHandlerTests.TestResource()

294	317	returned too
295	318	"""
296	319
297		def callback(request, **kwargs):
	320	async def callback(request, **kwargs):
298	321	e = RedirectException(b"/no/over/there", 304)
299	322	e.cookies.append(b"session=yespls")
300	323	raise e

311	334	self.assertEqual(location_headers, [b"/no/over/there"])
312	335	cookies_headers = [v for k, v in headers if k == b"Set-Cookie"]
313	336	self.assertEqual(cookies_headers, [b"session=yespls"])
	337
	338	def test_head_request(self):
	339	"""A head request should work by being turned into a GET request."""
	340
	341	async def callback(request):
	342	request.write(b"response")
	343	request.finish()
	344
	345	res = WrapHtmlRequestHandlerTests.TestResource()
	346	res.callback = callback
	347
	348	request, channel = make_request(self.reactor, b"HEAD", b"/path")
	349	render(request, res, self.reactor)
	350
	351	self.assertEqual(channel.result["code"], b"200")
	352	self.assertNotIn("body", channel.result)

+7

-7

tests/test_state.py less more

212	212	ctx_c = context_store["C"]
213	213	ctx_d = context_store["D"]
214	214
215		prev_state_ids = yield ctx_d.get_prev_state_ids()
	215	prev_state_ids = yield defer.ensureDeferred(ctx_d.get_prev_state_ids())
216	216	self.assertEqual(2, len(prev_state_ids))
217	217
218	218	self.assertEqual(ctx_c.state_group, ctx_d.state_group_before_event)

258	258	ctx_c = context_store["C"]
259	259	ctx_d = context_store["D"]
260	260
261		prev_state_ids = yield ctx_d.get_prev_state_ids()
	261	prev_state_ids = yield defer.ensureDeferred(ctx_d.get_prev_state_ids())
262	262	self.assertSetEqual({"START", "A", "C"}, set(prev_state_ids.values()))
263	263
264	264	self.assertEqual(ctx_c.state_group, ctx_d.state_group_before_event)

317	317	ctx_c = context_store["C"]
318	318	ctx_e = context_store["E"]
319	319
320		prev_state_ids = yield ctx_e.get_prev_state_ids()
	320	prev_state_ids = yield defer.ensureDeferred(ctx_e.get_prev_state_ids())
321	321	self.assertSetEqual({"START", "A", "B", "C"}, set(prev_state_ids.values()))
322	322	self.assertEqual(ctx_c.state_group, ctx_e.state_group_before_event)
323	323	self.assertEqual(ctx_e.state_group_before_event, ctx_e.state_group)

392	392	ctx_b = context_store["B"]
393	393	ctx_d = context_store["D"]
394	394
395		prev_state_ids = yield ctx_d.get_prev_state_ids()
	395	prev_state_ids = yield defer.ensureDeferred(ctx_d.get_prev_state_ids())
396	396	self.assertSetEqual({"A1", "A2", "A3", "A5", "B"}, set(prev_state_ids.values()))
397	397
398	398	self.assertEqual(ctx_b.state_group, ctx_d.state_group_before_event)

424	424	self.state.compute_event_context(event, old_state=old_state)
425	425	)
426	426
427		prev_state_ids = yield context.get_prev_state_ids()
	427	prev_state_ids = yield defer.ensureDeferred(context.get_prev_state_ids())
428	428	self.assertCountEqual((e.event_id for e in old_state), prev_state_ids.values())
429	429
430	430	current_state_ids = yield defer.ensureDeferred(context.get_current_state_ids())

449	449	self.state.compute_event_context(event, old_state=old_state)
450	450	)
451	451
452		prev_state_ids = yield context.get_prev_state_ids()
	452	prev_state_ids = yield defer.ensureDeferred(context.get_prev_state_ids())
453	453	self.assertCountEqual((e.event_id for e in old_state), prev_state_ids.values())
454	454
455	455	current_state_ids = yield defer.ensureDeferred(context.get_current_state_ids())

518	518
519	519	context = yield defer.ensureDeferred(self.state.compute_event_context(event))
520	520
521		prev_state_ids = yield context.get_prev_state_ids()
	521	prev_state_ids = yield defer.ensureDeferred(context.get_prev_state_ids())
522	522
523	523	self.assertEqual({e.event_id for e in old_state}, set(prev_state_ids.values()))
524	524

+16

-10

tests/test_visibility.py less more

39	39	self.store = self.hs.get_datastore()
40	40	self.storage = self.hs.get_storage()
41	41
42		yield create_room(self.hs, TEST_ROOM_ID, "@someone:ROOM")
	42	yield defer.ensureDeferred(create_room(self.hs, TEST_ROOM_ID, "@someone:ROOM"))
43	43
44	44	@defer.inlineCallbacks
45	45	def test_filtering(self):

63	63	evt = yield self.inject_room_member(user, extra_content={"a": "b"})
64	64	events_to_filter.append(evt)
65	65
66		filtered = yield filter_events_for_server(
67		self.storage, "test_server", events_to_filter
	66	filtered = yield defer.ensureDeferred(
	67	filter_events_for_server(self.storage, "test_server", events_to_filter)
68	68	)
69	69
70	70	# the result should be 5 redacted events, and 5 unredacted events.

101	101	yield self.hs.get_datastore().mark_user_erased("@erased:local_hs")
102	102
103	103	# ... and the filtering happens.
104		filtered = yield filter_events_for_server(
105		self.storage, "test_server", events_to_filter
	104	filtered = yield defer.ensureDeferred(
	105	filter_events_for_server(self.storage, "test_server", events_to_filter)
106	106	)
107	107
108	108	for i in range(0, len(events_to_filter)):

139	139	event, context = yield defer.ensureDeferred(
140	140	self.event_creation_handler.create_new_client_event(builder)
141	141	)
142		yield self.storage.persistence.persist_event(event, context)
	142	yield defer.ensureDeferred(
	143	self.storage.persistence.persist_event(event, context)
	144	)
143	145	return event
144	146
145	147	@defer.inlineCallbacks

161	163	self.event_creation_handler.create_new_client_event(builder)
162	164	)
163	165
164		yield self.storage.persistence.persist_event(event, context)
	166	yield defer.ensureDeferred(
	167	self.storage.persistence.persist_event(event, context)
	168	)
165	169	return event
166	170
167	171	@defer.inlineCallbacks

182	186	self.event_creation_handler.create_new_client_event(builder)
183	187	)
184	188
185		yield self.storage.persistence.persist_event(event, context)
	189	yield defer.ensureDeferred(
	190	self.storage.persistence.persist_event(event, context)
	191	)
186	192	return event
187	193
188	194	@defer.inlineCallbacks

264	270	storage.main = test_store
265	271	storage.state = test_store
266	272
267		filtered = yield filter_events_for_server(
268		test_store, "test_server", events_to_filter
	273	filtered = yield defer.ensureDeferred(
	274	filter_events_for_server(test_store, "test_server", events_to_filter)
269	275	)
270	276	logger.info("Filtering took %f seconds", time.time() - start)
271	277

+13

-17

tests/unittest.py less more

240	240	if hasattr(self, "user_id"):
241	241	if self.hijack_auth:
242	242
243		def get_user_by_access_token(token=None, allow_guest=False):
244		return succeed(
245		{
246		"user": UserID.from_string(self.helper.auth_user_id),
247		"token_id": 1,
248		"is_guest": False,
249		}
250		)
251
252		def get_user_by_req(request, allow_guest=False, rights="access"):
253		return succeed(
254		create_requester(
255		UserID.from_string(self.helper.auth_user_id), 1, False, None
256		)
	243	async def get_user_by_access_token(token=None, allow_guest=False):
	244	return {
	245	"user": UserID.from_string(self.helper.auth_user_id),
	246	"token_id": 1,
	247	"is_guest": False,
	248	}
	249
	250	async def get_user_by_req(request, allow_guest=False, rights="access"):
	251	return create_requester(
	252	UserID.from_string(self.helper.auth_user_id), 1, False, None
257	253	)
258	254
259	255	self.hs.get_auth().get_user_by_req = get_user_by_req

421	417
422	418	async def run_bg_updates():
423	419	with LoggingContext("run_bg_updates", request="run_bg_updates-1"):
424		while not await stor.db.updates.has_completed_background_updates():
425		await stor.db.updates.do_next_background_update(1)
	420	while not await stor.db_pool.updates.has_completed_background_updates():
	421	await stor.db_pool.updates.do_next_background_update(1)
426	422
427	423	hs = setup_test_homeserver(self.addCleanup, args, *kwargs)
428	424	stor = hs.get_datastore()

570	566	Add the given event as an extremity to the room.
571	567	"""
572	568	self.get_success(
573		self.hs.get_datastore().db.simple_insert(
	569	self.hs.get_datastore().db_pool.simple_insert(
574	570	table="event_forward_extremities",
575	571	values={"room_id": room_id, "event_id": event_id},
576	572	desc="test_add_extremity",

+11

-33

tests/util/test_retryutils.py less more

25	25	def test_new_destination(self):
26	26	"""A happy-path case with a new destination and a successful operation"""
27	27	store = self.hs.get_datastore()
28		d = get_retry_limiter("test_dest", self.clock, store)
29		self.pump()
30		limiter = self.successResultOf(d)
	28	limiter = self.get_success(get_retry_limiter("test_dest", self.clock, store))
31	29
32	30	# advance the clock a bit before making the request
33	31	self.pump(1)

35	33	with limiter:
36	34	pass
37	35
38		d = store.get_destination_retry_timings("test_dest")
39		self.pump()
40		new_timings = self.successResultOf(d)
	36	new_timings = self.get_success(store.get_destination_retry_timings("test_dest"))
41	37	self.assertIsNone(new_timings)
42	38
43	39	def test_limiter(self):
44	40	"""General test case which walks through the process of a failing request"""
45	41	store = self.hs.get_datastore()
46	42
47		d = get_retry_limiter("test_dest", self.clock, store)
48		self.pump()
49		limiter = self.successResultOf(d)
	43	limiter = self.get_success(get_retry_limiter("test_dest", self.clock, store))
50	44
51	45	self.pump(1)
52	46	try:

57	51	except AssertionError:
58	52	pass
59	53
60		# wait for the update to land
61		self.pump()
62
63		d = store.get_destination_retry_timings("test_dest")
64		self.pump()
65		new_timings = self.successResultOf(d)
	54	new_timings = self.get_success(store.get_destination_retry_timings("test_dest"))
66	55	self.assertEqual(new_timings["failure_ts"], failure_ts)
67	56	self.assertEqual(new_timings["retry_last_ts"], failure_ts)
68	57	self.assertEqual(new_timings["retry_interval"], MIN_RETRY_INTERVAL)
69	58
70	59	# now if we try again we should get a failure
71		d = get_retry_limiter("test_dest", self.clock, store)
72		self.pump()
73		self.failureResultOf(d, NotRetryingDestination)
	60	self.get_failure(
	61	get_retry_limiter("test_dest", self.clock, store), NotRetryingDestination
	62	)
74	63
75	64	#
76	65	# advance the clock and try again
77	66	#
78	67
79	68	self.pump(MIN_RETRY_INTERVAL)
80		d = get_retry_limiter("test_dest", self.clock, store)
81		self.pump()
82		limiter = self.successResultOf(d)
	69	limiter = self.get_success(get_retry_limiter("test_dest", self.clock, store))
83	70
84	71	self.pump(1)
85	72	try:

90	77	except AssertionError:
91	78	pass
92	79
93		# wait for the update to land
94		self.pump()
95
96		d = store.get_destination_retry_timings("test_dest")
97		self.pump()
98		new_timings = self.successResultOf(d)
	80	new_timings = self.get_success(store.get_destination_retry_timings("test_dest"))
99	81	self.assertEqual(new_timings["failure_ts"], failure_ts)
100	82	self.assertEqual(new_timings["retry_last_ts"], retry_ts)
101	83	self.assertGreaterEqual(

109	91	# one more go, with success
110	92	#
111	93	self.pump(MIN_RETRY_INTERVAL * RETRY_MULTIPLIER * 2.0)
112		d = get_retry_limiter("test_dest", self.clock, store)
113		self.pump()
114		limiter = self.successResultOf(d)
	94	limiter = self.get_success(get_retry_limiter("test_dest", self.clock, store))
115	95
116	96	self.pump(1)
117	97	with limiter:

120	100	# wait for the update to land
121	101	self.pump()
122	102
123		d = store.get_destination_retry_timings("test_dest")
124		self.pump()
125		new_timings = self.successResultOf(d)
	103	new_timings = self.get_success(store.get_destination_retry_timings("test_dest"))
126	104	self.assertIsNone(new_timings)

+9

-13

tests/utils.py less more

153	153	"account": {"per_second": 10000, "burst_count": 10000},
154	154	"failed_attempts": {"per_second": 10000, "burst_count": 10000},
155	155	},
	156	"rc_joins": {
	157	"local": {"per_second": 10000, "burst_count": 10000},
	158	"remote": {"per_second": 10000, "burst_count": 10000},
	159	},
156	160	"saml2_enabled": False,
157	161	"public_baseurl": None,
158	162	"default_identity_server": None,

637	641	)
638	642
639	643
640		@defer.inlineCallbacks
641		def create_room(hs, room_id, creator_id):
	644	async def create_room(hs, room_id: str, creator_id: str):
642	645	"""Creates and persist a creation event for the given room
643
644		Args:
645		hs
646		room_id (str)
647		creator_id (str)
648	646	"""
649	647
650	648	persistence_store = hs.get_storage().persistence

652	650	event_builder_factory = hs.get_event_builder_factory()
653	651	event_creation_handler = hs.get_event_creation_handler()
654	652
655		yield store.store_room(
	653	await store.store_room(
656	654	room_id=room_id,
657	655	room_creator_user_id=creator_id,
658	656	is_public=False,

670	668	},
671	669	)
672	670
673		event, context = yield defer.ensureDeferred(
674		event_creation_handler.create_new_client_event(builder)
675		)
676
677		yield persistence_store.persist_event(event, context)
	671	event, context = await event_creation_handler.create_new_client_event(builder)
	672
	673	await persistence_store.persist_event(event, context)

+9

-1

tox.ini less more

178	178	synapse/appservice \
179	179	synapse/config \
180	180	synapse/event_auth.py \
	181	synapse/events/builder.py \
181	182	synapse/events/spamcheck.py \
182	183	synapse/federation \
183	184	synapse/handlers/auth.py \

185	186	synapse/handlers/directory.py \
186	187	synapse/handlers/federation.py \
187	188	synapse/handlers/identity.py \
	189	synapse/handlers/message.py \
188	190	synapse/handlers/oidc_handler.py \
189	191	synapse/handlers/presence.py \
190	192	synapse/handlers/room_member.py \

197	199	synapse/logging/ \
198	200	synapse/metrics \
199	201	synapse/module_api \
	202	synapse/notifier.py \
200	203	synapse/push/pusherpool.py \
201	204	synapse/push/push_rule_evaluator.py \
202	205	synapse/replication \
203	206	synapse/rest \
	207	synapse/server.py \
	208	synapse/server_notices \
204	209	synapse/spam_checker_api \
205		synapse/storage/data_stores/main/ui_auth.py \
	210	synapse/storage/databases/main/ui_auth.py \
206	211	synapse/storage/database.py \
207	212	synapse/storage/engines \
	213	synapse/storage/state.py \
208	214	synapse/storage/util \
209	215	synapse/streams \
	216	synapse/types.py \
210	217	synapse/util/caches/stream_change_cache.py \
	218	synapse/util/metrics.py \
211	219	tests/replication \
212	220	tests/test_utils \
213	221	tests/rest/client/v2_alpha/test_auth.py \