Commit df31d372bad705a18c7c05fc174c760289731a8b - matrix-synapse

+23

-15

.buildkite/pipeline.yml less more

48	48
49	49
50	50	- command:
51		- "python -m pip install tox"
	51	- "apt-get update && apt-get install -y python3.5 python3.5-dev python3-pip libxml2-dev libxslt-dev zlib1g-dev"
	52	- "python3.5 -m pip install tox"
52	53	- "tox -e py35-old,codecov"
53	54	label: ":python: 3.5 / SQLite / Old Deps"
54	55	env:
55	56	TRIAL_FLAGS: "-j 2"
56	57	plugins:
57	58	- docker#v3.0.1:
58		image: "python:3.5"
	59	image: "ubuntu:xenial" # We use xenail to get an old sqlite and python
59	60	propagate-environment: true
60	61	retry:
61	62	automatic:

116	117	limit: 2
117	118
118	119	- label: ":python: 3.5 / :postgres: 9.5"
119		env:
120		TRIAL_FLAGS: "-j 4"
	120	agents:
	121	queue: "medium"
	122	env:
	123	TRIAL_FLAGS: "-j 8"
121	124	command:
122	125	- "bash -c 'python -m pip install tox && python -m tox -e py35-postgres,codecov'"
123	126	plugins:

133	136	limit: 2
134	137
135	138	- label: ":python: 3.7 / :postgres: 9.5"
136		env:
137		TRIAL_FLAGS: "-j 4"
	139	agents:
	140	queue: "medium"
	141	env:
	142	TRIAL_FLAGS: "-j 8"
138	143	command:
139	144	- "bash -c 'python -m pip install tox && python -m tox -e py37-postgres,codecov'"
140	145	plugins:

150	155	limit: 2
151	156
152	157	- label: ":python: 3.7 / :postgres: 11"
153		env:
154		TRIAL_FLAGS: "-j 4"
	158	agents:
	159	queue: "medium"
	160	env:
	161	TRIAL_FLAGS: "-j 8"
155	162	command:
156	163	- "bash -c 'python -m pip install tox && python -m tox -e py37-postgres,codecov'"
157	164	plugins:

213	220	env:
214	221	POSTGRES: "1"
215	222	WORKERS: "1"
	223	BLACKLIST: "synapse-blacklist-with-workers"
216	224	command:
217	225	- "bash .buildkite/merge_base_branch.sh"
	226	- "bash -c 'cat /src/sytest-blacklist /src/.buildkite/worker-blacklist > /src/synapse-blacklist-with-workers'"
218	227	- "bash /synapse_sytest.sh"
219	228	plugins:
220	229	- docker#v3.0.1:

222	231	propagate-environment: true
223	232	always-pull: true
224	233	workdir: "/src"
225		soft_fail: true
226		retry:
227		automatic:
228		- exit_status: -1
229		limit: 2
230		- exit_status: 2
231		limit: 2
	234	retry:
	235	automatic:
	236	- exit_status: -1
	237	limit: 2
	238	- exit_status: 2
	239	limit: 2

+30

-0

.buildkite/worker-blacklist less more

	0	# This file serves as a blacklist for SyTest tests that we expect will fail in
	1	# Synapse when run under worker mode. For more details, see sytest-blacklist.
	2
	3	Message history can be paginated
	4
	5	Can re-join room if re-invited
	6
	7	/upgrade creates a new room
	8
	9	The only membership state included in an initial sync is for all the senders in the timeline
	10
	11	Local device key changes get to remote servers
	12
	13	If remote user leaves room we no longer receive device updates
	14
	15	Forgotten room messages cannot be paginated
	16
	17	Inbound federation can get public room list
	18
	19	Members from the gap are included in gappy incr LL sync
	20
	21	Leaves are present in non-gapped incremental syncs
	22
	23	Old leaves are present in gapped incremental syncs
	24
	25	User sees updates to presence from other users in the incremental sync.
	26
	27	Gapped incremental syncs include all state changes
	28
	29	Old members are included in gappy incr LL sync if they start speaking

+1

-2

.codecov.yml less more

0		comment:
1		layout: "diff"
	0	comment: off
2	1
3	2	coverage:
4	3	status:

+1

-0

.gitignore less more

15	15	/*.log
16	16	/*.log.config
17	17	/*.pid
	18	/.python-version
18	19	/*.signing.key
19	20	/env/
20	21	/homeserver*.yaml

+84

-0

CHANGES.md less more

	0	Synapse 1.3.0 (2019-08-15)
	1	==========================
	2
	3	Bugfixes
	4	--------
	5
	6	- Fix 500 Internal Server Error on `publicRooms` when the public room list was
	7	cached. ([\#5851](https://github.com/matrix-org/synapse/issues/5851))
	8
	9
	10	Synapse 1.3.0rc1 (2019-08-13)
	11	==========================
	12
	13	Features
	14	--------
	15
	16	- Use `M_USER_DEACTIVATED` instead of `M_UNKNOWN` for errcode when a deactivated user attempts to login. ([\#5686](https://github.com/matrix-org/synapse/issues/5686))
	17	- Add sd_notify hooks to ease systemd integration and allows usage of Type=Notify. ([\#5732](https://github.com/matrix-org/synapse/issues/5732))
	18	- Synapse will no longer serve any media repo admin endpoints when `enable_media_repo` is set to False in the configuration. If a media repo worker is used, the admin APIs relating to the media repo will be served from it instead. ([\#5754](https://github.com/matrix-org/synapse/issues/5754), [\#5848](https://github.com/matrix-org/synapse/issues/5848))
	19	- Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events) over federation. This option can be used to prevent adverse performance on resource-constrained homeservers. ([\#5783](https://github.com/matrix-org/synapse/issues/5783))
	20	- Allow defining HTML templates to serve the user on account renewal attempt when using the account validity feature. ([\#5807](https://github.com/matrix-org/synapse/issues/5807))
	21
	22
	23	Bugfixes
	24	--------
	25
	26	- Fix UISIs during homeserver outage. ([\#5693](https://github.com/matrix-org/synapse/issues/5693), [\#5789](https://github.com/matrix-org/synapse/issues/5789))
	27	- Fix stack overflow in server key lookup code. ([\#5724](https://github.com/matrix-org/synapse/issues/5724))
	28	- start.sh no longer uses deprecated cli option. ([\#5725](https://github.com/matrix-org/synapse/issues/5725))
	29	- Log when we receive an event receipt from an unexpected origin. ([\#5743](https://github.com/matrix-org/synapse/issues/5743))
	30	- Fix debian packaging scripts to correctly build sid packages. ([\#5775](https://github.com/matrix-org/synapse/issues/5775))
	31	- Correctly handle redactions of redactions. ([\#5788](https://github.com/matrix-org/synapse/issues/5788))
	32	- Return 404 instead of 403 when accessing /rooms/{roomId}/event/{eventId} for an event without the appropriate permissions. ([\#5798](https://github.com/matrix-org/synapse/issues/5798))
	33	- Fix check that tombstone is a state event in push rules. ([\#5804](https://github.com/matrix-org/synapse/issues/5804))
	34	- Fix error when trying to login as a deactivated user when using a worker to handle login. ([\#5806](https://github.com/matrix-org/synapse/issues/5806))
	35	- Fix bug where user `/sync` stream could get wedged in rare circumstances. ([\#5825](https://github.com/matrix-org/synapse/issues/5825))
	36	- The purge_remote_media.sh script was fixed. ([\#5839](https://github.com/matrix-org/synapse/issues/5839))
	37
	38
	39	Deprecations and Removals
	40	-------------------------
	41
	42	- Synapse now no longer accepts the `-v`/`--verbose`, `-f`/`--log-file`, or `--log-config` command line flags, and removes the deprecated `verbose` and `log_file` configuration file options. Users of these options should migrate their options into the dedicated log configuration. ([\#5678](https://github.com/matrix-org/synapse/issues/5678), [\#5729](https://github.com/matrix-org/synapse/issues/5729))
	43	- Remove non-functional 'expire_access_token' setting. ([\#5782](https://github.com/matrix-org/synapse/issues/5782))
	44
	45
	46	Internal Changes
	47	----------------
	48
	49	- Make Jaeger fully configurable. ([\#5694](https://github.com/matrix-org/synapse/issues/5694))
	50	- Add precautionary measures to prevent future abuse of `window.opener` in default welcome page. ([\#5695](https://github.com/matrix-org/synapse/issues/5695))
	51	- Reduce database IO usage by optimising queries for current membership. ([\#5706](https://github.com/matrix-org/synapse/issues/5706), [\#5738](https://github.com/matrix-org/synapse/issues/5738), [\#5746](https://github.com/matrix-org/synapse/issues/5746), [\#5752](https://github.com/matrix-org/synapse/issues/5752), [\#5770](https://github.com/matrix-org/synapse/issues/5770), [\#5774](https://github.com/matrix-org/synapse/issues/5774), [\#5792](https://github.com/matrix-org/synapse/issues/5792), [\#5793](https://github.com/matrix-org/synapse/issues/5793))
	52	- Improve caching when fetching `get_filtered_current_state_ids`. ([\#5713](https://github.com/matrix-org/synapse/issues/5713))
	53	- Don't accept opentracing data from clients. ([\#5715](https://github.com/matrix-org/synapse/issues/5715))
	54	- Speed up PostgreSQL unit tests in CI. ([\#5717](https://github.com/matrix-org/synapse/issues/5717))
	55	- Update the coding style document. ([\#5719](https://github.com/matrix-org/synapse/issues/5719))
	56	- Improve database query performance when recording retry intervals for remote hosts. ([\#5720](https://github.com/matrix-org/synapse/issues/5720))
	57	- Add a set of opentracing utils. ([\#5722](https://github.com/matrix-org/synapse/issues/5722))
	58	- Cache result of get_version_string to reduce overhead of `/version` federation requests. ([\#5730](https://github.com/matrix-org/synapse/issues/5730))
	59	- Return 'user_type' in admin API user endpoints results. ([\#5731](https://github.com/matrix-org/synapse/issues/5731))
	60	- Don't package the sytest test blacklist file. ([\#5733](https://github.com/matrix-org/synapse/issues/5733))
	61	- Replace uses of returnValue with plain return, as returnValue is not needed on Python 3. ([\#5736](https://github.com/matrix-org/synapse/issues/5736))
	62	- Blacklist some flakey tests in worker mode. ([\#5740](https://github.com/matrix-org/synapse/issues/5740))
	63	- Fix some error cases in the caching layer. ([\#5749](https://github.com/matrix-org/synapse/issues/5749))
	64	- Add a prometheus metric for pending cache lookups. ([\#5750](https://github.com/matrix-org/synapse/issues/5750))
	65	- Stop trying to fetch events with event_id=None. ([\#5753](https://github.com/matrix-org/synapse/issues/5753))
	66	- Convert RedactionTestCase to modern test style. ([\#5768](https://github.com/matrix-org/synapse/issues/5768))
	67	- Allow looping calls to be given arguments. ([\#5780](https://github.com/matrix-org/synapse/issues/5780))
	68	- Set the logs emitted when checking typing and presence timeouts to DEBUG level, not INFO. ([\#5785](https://github.com/matrix-org/synapse/issues/5785))
	69	- Remove DelayedCall debugging from the test suite, as it is no longer required in the vast majority of Synapse's tests. ([\#5787](https://github.com/matrix-org/synapse/issues/5787))
	70	- Remove some spurious exceptions from the logs where we failed to talk to a remote server. ([\#5790](https://github.com/matrix-org/synapse/issues/5790))
	71	- Improve performance when making `.well-known` requests by sharing the SSL options between requests. ([\#5794](https://github.com/matrix-org/synapse/issues/5794))
	72	- Disable codecov GitHub comments on PRs. ([\#5796](https://github.com/matrix-org/synapse/issues/5796))
	73	- Don't allow clients to send tombstone events that reference the room it's sent in. ([\#5801](https://github.com/matrix-org/synapse/issues/5801))
	74	- Deny redactions of events sent in a different room. ([\#5802](https://github.com/matrix-org/synapse/issues/5802))
	75	- Deny sending well known state types as non-state events. ([\#5805](https://github.com/matrix-org/synapse/issues/5805))
	76	- Handle incorrectly encoded query params correctly by returning a 400. ([\#5808](https://github.com/matrix-org/synapse/issues/5808))
	77	- Handle pusher being deleted during processing rather than logging an exception. ([\#5809](https://github.com/matrix-org/synapse/issues/5809))
	78	- Return 502 not 500 when failing to reach any remote server. ([\#5810](https://github.com/matrix-org/synapse/issues/5810))
	79	- Reduce global pauses in the events stream caused by expensive state resolution during persistence. ([\#5826](https://github.com/matrix-org/synapse/issues/5826))
	80	- Add a lower bound to well-known lookup cache time to avoid repeated lookups. ([\#5836](https://github.com/matrix-org/synapse/issues/5836))
	81	- Whitelist history visbility sytests in worker mode tests. ([\#5843](https://github.com/matrix-org/synapse/issues/5843))
	82
	83
0	84	Synapse 1.2.1 (2019-07-26)
1	85	==========================
2	86

+1

-1

MANIFEST.in less more

6	6	include demo/demo.tls.dh
7	7	include demo/*.py
8	8	include demo/*.sh
9		include sytest-blacklist
10	9
11	10	recursive-include synapse/storage/schema *.sql
12	11	recursive-include synapse/storage/schema *.sql.postgres

33	32	exclude .dockerignore
34	33	exclude test_postgresql.sh
35	34	exclude .editorconfig
	35	exclude sytest-blacklist
36	36
37	37	include pyproject.toml
38	38	recursive-include changelog.d *

+1

-1

contrib/purge_api/purge_remote_media.sh less more

50	50	# finally start pruning media:
51	51	###############################################################################
52	52	set -x # for debugging the generated string
53		curl --header "Authorization: Bearer $TOKEN" -v POST "$API_URL/admin/purge_media_cache/?before_ts=$UNIX_TIMESTAMP"
	53	curl --header "Authorization: Bearer $TOKEN" -X POST "$API_URL/admin/purge_media_cache/?before_ts=$UNIX_TIMESTAMP"

+3

-1

contrib/systemd/matrix-synapse.service less more

13	13	Description=Synapse Matrix homeserver
14	14
15	15	[Service]
16		Type=simple
	16	Type=notify
	17	NotifyAccess=main
	18	ExecReload=/bin/kill -HUP $MAINPID
17	19	Restart=on-abort
18	20
19	21	User=synapse

+2

-1

contrib/systemd-with-workers/system/matrix-synapse-worker@.service less more

3	3	BindsTo=matrix-synapse.service
4	4
5	5	[Service]
6		Type=simple
	6	Type=notify
	7	NotifyAccess=main
7	8	User=matrix-synapse
8	9	WorkingDirectory=/var/lib/matrix-synapse
9	10	EnvironmentFile=/etc/default/matrix-synapse

+2

-1

contrib/systemd-with-workers/system/matrix-synapse.service less more

1	1	Description=Synapse Matrix Homeserver
2	2
3	3	[Service]
4		Type=simple
	4	Type=notify
	5	NotifyAccess=main
5	6	User=matrix-synapse
6	7	WorkingDirectory=/var/lib/matrix-synapse
7	8	EnvironmentFile=/etc/default/matrix-synapse

+7

-7

debian/changelog less more

0		matrix-synapse-py3 (1.2.1) stable; urgency=medium
1
2		* New synapse release 1.2.1.
3
4		-- Synapse Packaging team <packages@matrix.org> Fri, 26 Jul 2019 11:32:47 +0100
	0	matrix-synapse-py3 (1.3.0) stable; urgency=medium
	1
	2	[ Andrew Morgan ]
	3	* Remove libsqlite3-dev from required build dependencies.
5	4
6	5	matrix-synapse-py3 (1.2.0) stable; urgency=medium
7	6

13	12
14	13	[ Synapse Packaging team ]
15	14	* New synapse release 1.2.0.
16
17		-- Synapse Packaging team <packages@matrix.org> Thu, 25 Jul 2019 14:10:07 +0100
	15	* New synapse release 1.3.0.
	16
	17	-- Synapse Packaging team <packages@matrix.org> Thu, 15 Aug 2019 12:04:23 +0100
18	18
19	19	matrix-synapse-py3 (1.1.0) stable; urgency=medium
20	20

+0

-1

debian/control less more

14	14	python3-setuptools,
15	15	python3-pip,
16	16	python3-venv,
17		libsqlite3-dev,
18	17	tar,
19	18	Standards-Version: 3.9.8
20	19	Homepage: https://github.com/matrix-org/synapse

+3

-4

demo/start.sh less more

28	28
29	29	if ! grep -F "Customisation made by demo/start.sh" -q $DIR/etc/$port.config; then
30	30	printf '\n\n# Customisation made by demo/start.sh\n' >> $DIR/etc/$port.config
31
	31
32	32	echo 'enable_registration: true' >> $DIR/etc/$port.config
33	33
34	34	# Warning, this heredoc depends on the interaction of tabs and spaces. Please don't

42	42	tls: true
43	43	resources:
44	44	- names: [client, federation]
45
	45
46	46	- port: $port
47	47	tls: false
48	48	bind_addresses: ['::1', '127.0.0.1']

67	67
68	68	# Generate tls keys
69	69	openssl req -x509 -newkey rsa:4096 -keyout $DIR/etc/localhost\:$https_port.tls.key -out $DIR/etc/localhost\:$https_port.tls.crt -days 365 -nodes -subj "/O=matrix"
70
	70
71	71	# Ignore keys from the trusted keys server
72	72	echo '# Ignore keys from the trusted keys server' >> $DIR/etc/$port.config
73	73	echo 'trusted_key_servers:' >> $DIR/etc/$port.config

119	119	python3 -m synapse.app.homeserver \
120	120	--config-path "$DIR/etc/$port.config" \
121	121	-D \
122		-vv \
123	122
124	123	popd
125	124	done

+5

-0

docker/Dockerfile-dhvirtualenv less more

41	41	###
42	42	FROM ${distro}
43	43
	44	# Get the distro we want to pull from as a dynamic build variable
	45	# (We need to define it in each build stage)
	46	ARG distro=""
	47	ENV distro ${distro}
	48
44	49	# Install the build dependencies
45	50	#
46	51	# NB: keep this list in sync with the list of build-deps in debian/control

+2

-1

docker/build_debian.sh less more

3	3
4	4	set -ex
5	5
6		DIST=`lsb_release -c -s`
	6	# Get the codename from distro env
	7	DIST=`cut -d ':' -f2 <<< $distro`
7	8
8	9	# we get a read-only copy of the source: make a writeable copy
9	10	cp -aT /synapse/source /synapse/build

+99

-35

docs/code_style.rst less more

0		# Code Style
	0	Code Style
	1	==========
	2
	3	Formatting tools
	4	----------------
1	5
2	6	The Synapse codebase uses a number of code formatting tools in order to
3	7	quickly and automatically check for formatting (and sometimes logical) errors

5	9
6	10	The necessary tools are detailed below.
7	11
8		## Formatting tools
	12	- black
9	13
10		The Synapse codebase uses [black](https://pypi.org/project/black/) as an
11		opinionated code formatter, ensuring all comitted code is properly
12		formatted.
	14	The Synapse codebase uses `black <https://pypi.org/project/black/>`_ as an
	15	opinionated code formatter, ensuring all comitted code is properly
	16	formatted.
13	17
14		First install ``black`` with::
	18	First install ``black`` with::
15	19
16		pip install --upgrade black
	20	pip install --upgrade black
17	21
18		Have ``black`` auto-format your code (it shouldn't change any
19		functionality) with::
	22	Have ``black`` auto-format your code (it shouldn't change any functionality)
	23	with::
20	24
21		black . --exclude="\.tox\|build\|env"
	25	black . --exclude="\.tox\|build\|env"
22	26
23	27	- flake8
24	28

53	57	workflow. It is not, however, recommended to run ``flake8`` on save as it
54	58	takes a while and is very resource intensive.
55	59
56		## General rules
	60	General rules
	61	-------------
57	62
58	63	- Naming:
59	64
60	65	- Use camel case for class and type names
61	66	- Use underscores for functions and variables.
62	67
63		- Use double quotes ``"foo"`` rather than single quotes ``'foo'``.
64
65		- Comments: should follow the `google code style
66		<http://google.github.io/styleguide/pyguide.html?showone=Comments#Comments>`_.
	68	- Docstrings: should follow the `google code style
	69	<https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings>`_.
67	70	This is so that we can generate documentation with `sphinx
68	71	<http://sphinxcontrib-napoleon.readthedocs.org/en/latest/>`_. See the
69	72	`examples

71	74	in the sphinx documentation.
72	75
73	76	- Imports:
	77
	78	- Imports should be sorted by ``isort`` as described above.
74	79
75	80	- Prefer to import classes and functions rather than packages or modules.
76	81

91	96	This goes against the advice in the Google style guide, but it means that
92	97	errors in the name are caught early (at import time).
93	98
94		- Multiple imports from the same package can be combined onto one line::
95
96		from synapse.types import GroupID, RoomID, UserID
97
98		An effort should be made to keep the individual imports in alphabetical
99		order.
100
101		If the list becomes long, wrap it with parentheses and split it over
102		multiple lines.
103
104		- As per `PEP-8 <https://www.python.org/dev/peps/pep-0008/#imports>`_,
105		imports should be grouped in the following order, with a blank line between
106		each group:
107
108		1. standard library imports
109		2. related third party imports
110		3. local application/library specific imports
111
112		- Imports within each group should be sorted alphabetically by module name.
113
114	99	- Avoid wildcard imports (``from synapse.types import *``) and relative
115	100	imports (``from .types import UserID``).
	101
	102	Configuration file format
	103	-------------------------
	104
	105	The `sample configuration file <./sample_config.yaml>`_ acts as a reference to
	106	Synapse's configuration options for server administrators. Remember that many
	107	readers will be unfamiliar with YAML and server administration in general, so
	108	that it is important that the file be as easy to understand as possible, which
	109	includes following a consistent format.
	110
	111	Some guidelines follow:
	112
	113	* Sections should be separated with a heading consisting of a single line
	114	prefixed and suffixed with ``##``. There should be two blank lines
	115	before the section header, and one after.
	116
	117	* Each option should be listed in the file with the following format:
	118
	119	* A comment describing the setting. Each line of this comment should be
	120	prefixed with a hash (``#``) and a space.
	121
	122	The comment should describe the default behaviour (ie, what happens if
	123	the setting is omitted), as well as what the effect will be if the
	124	setting is changed.
	125
	126	Often, the comment end with something like "uncomment the
	127	following to \<do action>".
	128
	129	* A line consisting of only ``#``.
	130
	131	* A commented-out example setting, prefixed with only ``#``.
	132
	133	For boolean (on/off) options, convention is that this example should be
	134	the opposite to the default (so the comment will end with "Uncomment
	135	the following to enable [or disable] \<feature\>." For other options,
	136	the example should give some non-default value which is likely to be
	137	useful to the reader.
	138
	139	* There should be a blank line between each option.
	140
	141	* Where several settings are grouped into a single dict, avoid the
	142	convention where the whole block is commented out, resulting in comment
	143	lines starting ``# #``, as this is hard to read and confusing to
	144	edit. Instead, leave the top-level config option uncommented, and follow
	145	the conventions above for sub-options. Ensure that your code correctly
	146	handles the top-level option being set to ``None`` (as it will be if no
	147	sub-options are enabled).
	148
	149	* Lines should be wrapped at 80 characters.
	150
	151	Example::
	152
	153	## Frobnication ##
	154
	155	# The frobnicator will ensure that all requests are fully frobnicated.
	156	# To enable it, uncomment the following.
	157	#
	158	#frobnicator_enabled: true
	159
	160	# By default, the frobnicator will frobnicate with the default frobber.
	161	# The following will make it use an alternative frobber.
	162	#
	163	#frobincator_frobber: special_frobber
	164
	165	# Settings for the frobber
	166	#
	167	frobber:
	168	# frobbing speed. Defaults to 1.
	169	#
	170	#speed: 10
	171
	172	# frobbing distance. Defaults to 1000.
	173	#
	174	#distance: 100
	175
	176	Note that the sample configuration is generated from the synapse code and is
	177	maintained by a script, ``scripts-dev/generate_sample_config``. Making sure
	178	that the output from this script matches the desired format is left as an
	179	exercise for the reader!

+1

-1

docs/log_contexts.rst less more

147	147	d = more_stuff()
148	148	result = yield d # also fine, of course
149	149
150		defer.returnValue(result)
	150	return result
151	151
152	152	def nonInlineCallbacksFun():
153	153	logger.debug("just a wrapper really")

+50

-4

docs/sample_config.yaml less more

277	277	# Used by phonehome stats to group together related servers.
278	278	#server_context: context
279	279
	280	# Resource-constrained Homeserver Settings
	281	#
	282	# If limit_remote_rooms.enabled is True, the room complexity will be
	283	# checked before a user joins a new remote room. If it is above
	284	# limit_remote_rooms.complexity, it will disallow joining or
	285	# instantly leave.
	286	#
	287	# limit_remote_rooms.complexity_error can be set to customise the text
	288	# displayed to the user when a room above the complexity threshold has
	289	# its join cancelled.
	290	#
	291	# Uncomment the below lines to enable:
	292	#limit_remote_rooms:
	293	# enabled: True
	294	# complexity: 1.0
	295	# complexity_error: "This room is too complex."
	296
280	297	# Whether to require a user to be in the room to add an alias to it.
281	298	# Defaults to 'true'.
282	299	#

546	563	#federation_rr_transactions_per_room_per_second: 50
547	564
548	565
	566
	567	## Media Store ##
	568
	569	# Enable the media store service in the Synapse master. Uncomment the
	570	# following if you are using a separate media store worker.
	571	#
	572	#enable_media_repo: false
549	573
550	574	# Directory where uploaded images and attachments are stored.
551	575	#

784	808	# period: 6w
785	809	# renew_at: 1w
786	810	# renew_email_subject: "Renew your %(app)s account"
	811	# # Directory in which Synapse will try to find the HTML files to serve to the
	812	# # user when trying to renew an account. Optional, defaults to
	813	# # synapse/res/templates.
	814	# template_dir: "res/templates"
	815	# # HTML to be displayed to the user after they successfully renewed their
	816	# # account. Optional.
	817	# account_renewed_html_path: "account_renewed.html"
	818	# # HTML to be displayed when the user tries to renew an account with an invalid
	819	# # renewal token. Optional.
	820	# invalid_token_html_path: "invalid_token.html"
787	821
788	822	# Time that a user's session remains valid for, after they log in.
789	823	#

923	957	# a secret key is derived from the signing key.
924	958	#
925	959	# macaroon_secret_key: <PRIVATE STRING>
926
927		# Used to enable access token expiration.
928		#
929		#expire_access_token: False
930	960
931	961	# a secret which is used to calculate HMACs for form values, to stop
932	962	# falsification of values. Must be specified for the User Consent

1429	1459	#
1430	1460	#homeserver_whitelist:
1431	1461	# - ".*"
	1462
	1463	# Jaeger can be configured to sample traces at different rates.
	1464	# All configuration options provided by Jaeger can be set here.
	1465	# Jaeger's configuration mostly related to trace sampling which
	1466	# is documented here:
	1467	# https://www.jaegertracing.io/docs/1.13/sampling/.
	1468	#
	1469	#jaeger_config:
	1470	# sampler:
	1471	# type: const
	1472	# param: 1
	1473
	1474	# Logging whether spans were started and reported
	1475	#
	1476	# logging:
	1477	# false

+7

-0

docs/workers.rst less more

205	205
206	206	/_matrix/media/
207	207
	208	And the following regular expressions matching media-specific administration
	209	APIs::
	210
	211	^/_synapse/admin/v1/purge_media_cache$
	212	^/_synapse/admin/v1/room/.*/media$
	213	^/_synapse/admin/v1/quarantine_media/.*$
	214
208	215	You should also set ``enable_media_repo: False`` in the shared configuration
209	216	file to stop the main synapse running background jobs related to managing the
210	217	media repository.

+1

-1

synapse/__init__.py less more

34	34	except ImportError:
35	35	pass
36	36
37		__version__ = "1.2.1"
	37	__version__ = "1.3.0"

+25

-47

synapse/api/auth.py less more

127	127	)
128	128
129	129	self._check_joined_room(member, user_id, room_id)
130		defer.returnValue(member)
	130	return member
131	131
132	132	@defer.inlineCallbacks
133	133	def check_user_was_in_room(self, room_id, user_id):

155	155	if forgot:
156	156	raise AuthError(403, "User %s not in room %s" % (user_id, room_id))
157	157
158		defer.returnValue(member)
	158	return member
159	159
160	160	@defer.inlineCallbacks
161	161	def check_host_in_room(self, room_id, host):
162	162	with Measure(self.clock, "check_host_in_room"):
163	163	latest_event_ids = yield self.store.is_host_joined(room_id, host)
164		defer.returnValue(latest_event_ids)
	164	return latest_event_ids
165	165
166	166	def _check_joined_room(self, member, user_id, room_id):
167	167	if not member or member.membership != Membership.JOIN:

218	218	device_id="dummy-device", # stubbed
219	219	)
220	220
221		defer.returnValue(
222		synapse.types.create_requester(user_id, app_service=app_service)
223		)
	221	return synapse.types.create_requester(user_id, app_service=app_service)
224	222
225	223	user_info = yield self.get_user_by_access_token(access_token, rights)
226	224	user = user_info["user"]

261	259
262	260	request.authenticated_entity = user.to_string()
263	261
264		defer.returnValue(
265		synapse.types.create_requester(
266		user, token_id, is_guest, device_id, app_service=app_service
267		)
	262	return synapse.types.create_requester(
	263	user, token_id, is_guest, device_id, app_service=app_service
268	264	)
269	265	except KeyError:
270	266	raise MissingClientTokenError()

275	271	self.get_access_token_from_request(request)
276	272	)
277	273	if app_service is None:
278		defer.returnValue((None, None))
	274	return (None, None)
279	275
280	276	if app_service.ip_range_whitelist:
281	277	ip_address = IPAddress(self.hs.get_ip_from_request(request))
282	278	if ip_address not in app_service.ip_range_whitelist:
283		defer.returnValue((None, None))
	279	return (None, None)
284	280
285	281	if b"user_id" not in request.args:
286		defer.returnValue((app_service.sender, app_service))
	282	return (app_service.sender, app_service)
287	283
288	284	user_id = request.args[b"user_id"][0].decode("utf8")
289	285	if app_service.sender == user_id:
290		defer.returnValue((app_service.sender, app_service))
	286	return (app_service.sender, app_service)
291	287
292	288	if not app_service.is_interested_in_user(user_id):
293	289	raise AuthError(403, "Application service cannot masquerade as this user.")
294	290	if not (yield self.store.get_user_by_id(user_id)):
295	291	raise AuthError(403, "Application service has not registered this user")
296		defer.returnValue((user_id, app_service))
	292	return (user_id, app_service)
297	293
298	294	@defer.inlineCallbacks
299	295	def get_user_by_access_token(self, token, rights="access"):

329	325	msg="Access token has expired", soft_logout=True
330	326	)
331	327
332		defer.returnValue(r)
	328	return r
333	329
334	330	# otherwise it needs to be a valid macaroon
335	331	try:

377	373	}
378	374	else:
379	375	raise RuntimeError("Unknown rights setting %s", rights)
380		defer.returnValue(ret)
	376	return ret
381	377	except (
382	378	_InvalidMacaroonException,
383	379	pymacaroons.exceptions.MacaroonException,

413	409	try:
414	410	user_id = self.get_user_id_from_macaroon(macaroon)
415	411
416		has_expiry = False
417	412	guest = False
418	413	for caveat in macaroon.caveats:
419		if caveat.caveat_id.startswith("time "):
420		has_expiry = True
421		elif caveat.caveat_id == "guest = true":
	414	if caveat.caveat_id == "guest = true":
422	415	guest = True
423	416
424		self.validate_macaroon(
425		macaroon, rights, self.hs.config.expire_access_token, user_id=user_id
426		)
	417	self.validate_macaroon(macaroon, rights, user_id=user_id)
427	418	except (pymacaroons.exceptions.MacaroonException, TypeError, ValueError):
428	419	raise InvalidClientTokenError("Invalid macaroon passed.")
429	420
430		if not has_expiry and rights == "access":
	421	if rights == "access":
431	422	self.token_cache[token] = (user_id, guest)
432	423
433	424	return user_id, guest

453	444	return caveat.caveat_id[len(user_prefix) :]
454	445	raise InvalidClientTokenError("No user caveat in macaroon")
455	446
456		def validate_macaroon(self, macaroon, type_string, verify_expiry, user_id):
	447	def validate_macaroon(self, macaroon, type_string, user_id):
457	448	"""
458	449	validate that a Macaroon is understood by and was signed by this server.
459	450

461	452	macaroon(pymacaroons.Macaroon): The macaroon to validate
462	453	type_string(str): The kind of token required (e.g. "access",
463	454	"delete_pusher")
464		verify_expiry(bool): Whether to verify whether the macaroon has expired.
465	455	user_id (str): The user_id required
466	456	"""
467	457	v = pymacaroons.Verifier()

474	464	v.satisfy_exact("type = " + type_string)
475	465	v.satisfy_exact("user_id = %s" % user_id)
476	466	v.satisfy_exact("guest = true")
477
478		# verify_expiry should really always be True, but there exist access
479		# tokens in the wild which expire when they should not, so we can't
480		# enforce expiry yet (so we have to allow any caveat starting with
481		# 'time < ' in access tokens).
482		#
483		# On the other hand, short-term login tokens (as used by CAS login, for
484		# example) have an expiry time which we do want to enforce.
485
486		if verify_expiry:
487		v.satisfy_general(self._verify_expiry)
488		else:
489		v.satisfy_general(lambda c: c.startswith("time < "))
	467	v.satisfy_general(self._verify_expiry)
490	468
491	469	# access_tokens include a nonce for uniqueness: any value is acceptable
492	470	v.satisfy_general(lambda c: c.startswith("nonce = "))

505	483	def _look_up_user_by_access_token(self, token):
506	484	ret = yield self.store.get_user_by_access_token(token)
507	485	if not ret:
508		defer.returnValue(None)
	486	return None
509	487
510	488	# we use ret.get() below because lots of unit tests stub out
511	489	# get_user_by_access_token in a way where it only returns a couple of

517	495	"device_id": ret.get("device_id"),
518	496	"valid_until_ms": ret.get("valid_until_ms"),
519	497	}
520		defer.returnValue(user_info)
	498	return user_info
521	499
522	500	def get_appservice_by_req(self, request):
523	501	token = self.get_access_token_from_request(request)

542	520	@defer.inlineCallbacks
543	521	def compute_auth_events(self, event, current_state_ids, for_verification=False):
544	522	if event.type == EventTypes.Create:
545		defer.returnValue([])
	523	return []
546	524
547	525	auth_ids = []
548	526

603	581	if member_event.content["membership"] == Membership.JOIN:
604	582	auth_ids.append(member_event.event_id)
605	583
606		defer.returnValue(auth_ids)
	584	return auth_ids
607	585
608	586	@defer.inlineCallbacks
609	587	def check_can_change_room_list(self, room_id, user):

617	595
618	596	is_admin = yield self.is_server_admin(user)
619	597	if is_admin:
620		defer.returnValue(True)
	598	return True
621	599
622	600	user_id = user.to_string()
623	601	yield self.check_joined_room(room_id, user_id)

711	689	# * The user is a guest user, and has joined the room
712	690	# else it will throw.
713	691	member_event = yield self.check_user_was_in_room(room_id, user_id)
714		defer.returnValue((member_event.membership, member_event.event_id))
	692	return (member_event.membership, member_event.event_id)
715	693	except AuthError:
716	694	visibility = yield self.state.get_current_state(
717	695	room_id, EventTypes.RoomHistoryVisibility, ""

720	698	visibility
721	699	and visibility.content["history_visibility"] == "world_readable"
722	700	):
723		defer.returnValue((Membership.JOIN, None))
	701	return (Membership.JOIN, None)
724	702	return
725	703	raise AuthError(
726	704	403, "Guest access not allowed", errcode=Codes.GUEST_ACCESS_FORBIDDEN

+2

-1

synapse/api/errors.py less more

60	60	INCOMPATIBLE_ROOM_VERSION = "M_INCOMPATIBLE_ROOM_VERSION"
61	61	WRONG_ROOM_KEYS_VERSION = "M_WRONG_ROOM_KEYS_VERSION"
62	62	EXPIRED_ACCOUNT = "ORG_MATRIX_EXPIRED_ACCOUNT"
	63	USER_DEACTIVATED = "M_USER_DEACTIVATED"
63	64
64	65
65	66	class CodeMessageException(RuntimeError):

150	151	msg (str): The human-readable error message
151	152	"""
152	153	super(UserDeactivatedError, self).__init__(
153		code=http_client.FORBIDDEN, msg=msg, errcode=Codes.UNKNOWN
	154	code=http_client.FORBIDDEN, msg=msg, errcode=Codes.USER_DEACTIVATED
154	155	)
155	156
156	157

+1

-1

synapse/api/filtering.py less more

131	131	@defer.inlineCallbacks
132	132	def get_user_filter(self, user_localpart, filter_id):
133	133	result = yield self.store.get_user_filter(user_localpart, filter_id)
134		defer.returnValue(FilterCollection(result))
	134	return FilterCollection(result)
135	135
136	136	def add_user_filter(self, user_localpart, user_filter):
137	137	self.check_valid_filter(user_filter)

+29

-0

synapse/app/_base.py less more

14	14
15	15	import gc
16	16	import logging
	17	import os
17	18	import signal
18	19	import sys
19	20	import traceback
20	21
	22	import sdnotify
21	23	from daemonize import Daemonize
22	24
23	25	from twisted.internet import defer, error, reactor

241	243	if hasattr(signal, "SIGHUP"):
242	244
243	245	def handle_sighup(args, *kwargs):
	246	# Tell systemd our state, if we're using it. This will silently fail if
	247	# we're not using systemd.
	248	sd_channel = sdnotify.SystemdNotifier()
	249	sd_channel.notify("RELOADING=1")
	250
244	251	for i in _sighup_callbacks:
245	252	i(hs)
	253
	254	sd_channel.notify("READY=1")
246	255
247	256	signal.signal(signal.SIGHUP, handle_sighup)
248	257

259	268	hs.get_datastore().start_profiling()
260	269
261	270	setup_sentry(hs)
	271	setup_sdnotify(hs)
262	272	except Exception:
263	273	traceback.print_exc(file=sys.stderr)
264	274	reactor = hs.get_reactor()

289	299	name = hs.config.worker_name if hs.config.worker_name else "master"
290	300	scope.set_tag("worker_app", app)
291	301	scope.set_tag("worker_name", name)
	302
	303
	304	def setup_sdnotify(hs):
	305	"""Adds process state hooks to tell systemd what we are up to.
	306	"""
	307
	308	# Tell systemd our state, if we're using it. This will silently fail if
	309	# we're not using systemd.
	310	sd_channel = sdnotify.SystemdNotifier()
	311
	312	hs.get_reactor().addSystemEventTrigger(
	313	"after",
	314	"startup",
	315	lambda: sd_channel.notify("READY=1\nMAINPID=%s" % (os.getpid())),
	316	)
	317
	318	hs.get_reactor().addSystemEventTrigger(
	319	"before", "shutdown", lambda: sd_channel.notify("STOPPING=1")
	320	)
292	321
293	322
294	323	def install_dns_limiter(reactor, max_dns_requests_in_flight=100):

+3

-1

synapse/app/appservice.py less more

167	167	)
168	168
169	169	ps.setup()
170		reactor.callWhenRunning(_base.start, ps, config.worker_listeners)
	170	reactor.addSystemEventTrigger(
	171	"before", "startup", _base.start, ps, config.worker_listeners
	172	)
171	173
172	174	_base.start_worker_reactor("synapse-appservice", config)
173	175

+3

-1

synapse/app/client_reader.py less more

193	193	)
194	194
195	195	ss.setup()
196		reactor.callWhenRunning(_base.start, ss, config.worker_listeners)
	196	reactor.addSystemEventTrigger(
	197	"before", "startup", _base.start, ss, config.worker_listeners
	198	)
197	199
198	200	_base.start_worker_reactor("synapse-client-reader", config)
199	201

+3

-1

synapse/app/event_creator.py less more

192	192	)
193	193
194	194	ss.setup()
195		reactor.callWhenRunning(_base.start, ss, config.worker_listeners)
	195	reactor.addSystemEventTrigger(
	196	"before", "startup", _base.start, ss, config.worker_listeners
	197	)
196	198
197	199	_base.start_worker_reactor("synapse-event-creator", config)
198	200

+3

-1

synapse/app/federation_reader.py less more

174	174	)
175	175
176	176	ss.setup()
177		reactor.callWhenRunning(_base.start, ss, config.worker_listeners)
	177	reactor.addSystemEventTrigger(
	178	"before", "startup", _base.start, ss, config.worker_listeners
	179	)
178	180
179	181	_base.start_worker_reactor("synapse-federation-reader", config)
180	182

+3

-1

synapse/app/federation_sender.py less more

197	197	)
198	198
199	199	ss.setup()
200		reactor.callWhenRunning(_base.start, ss, config.worker_listeners)
	200	reactor.addSystemEventTrigger(
	201	"before", "startup", _base.start, ss, config.worker_listeners
	202	)
201	203
202	204	_base.start_worker_reactor("synapse-federation-sender", config)
203	205

+7

-5

synapse/app/frontend_proxy.py less more

69	69	except HttpResponseException as e:
70	70	raise e.to_synapse_error()
71	71
72		defer.returnValue((200, result))
	72	return (200, result)
73	73
74	74	@defer.inlineCallbacks
75	75	def on_PUT(self, request, user_id):
76	76	yield self.auth.get_user_by_req(request)
77		defer.returnValue((200, {}))
	77	return (200, {})
78	78
79	79
80	80	class KeyUploadServlet(RestServlet):

125	125	self.main_uri + request.uri.decode("ascii"), body, headers=headers
126	126	)
127	127
128		defer.returnValue((200, result))
	128	return (200, result)
129	129	else:
130	130	# Just interested in counts.
131	131	result = yield self.store.count_e2e_one_time_keys(user_id, device_id)
132		defer.returnValue((200, {"one_time_key_counts": result}))
	132	return (200, {"one_time_key_counts": result})
133	133
134	134
135	135	class FrontendProxySlavedStore(

246	246	)
247	247
248	248	ss.setup()
249		reactor.callWhenRunning(_base.start, ss, config.worker_listeners)
	249	reactor.addSystemEventTrigger(
	250	"before", "startup", _base.start, ss, config.worker_listeners
	251	)
250	252
251	253	_base.start_worker_reactor("synapse-frontend-proxy", config)
252	254

+2

-2

synapse/app/homeserver.py less more

405	405	if provision:
406	406	yield acme.provision_certificate()
407	407
408		defer.returnValue(provision)
	408	return provision
409	409
410	410	@defer.inlineCallbacks
411	411	def reprovision_acme():

446	446	reactor.stop()
447	447	sys.exit(1)
448	448
449		reactor.callWhenRunning(start)
	449	reactor.addSystemEventTrigger("before", "startup", start)
450	450
451	451	return hs
452	452

+12

-1

synapse/app/media_repository.py less more

25	25	from synapse.config._base import ConfigError
26	26	from synapse.config.homeserver import HomeServerConfig
27	27	from synapse.config.logger import setup_logging
	28	from synapse.http.server import JsonResource
28	29	from synapse.http.site import SynapseSite
29	30	from synapse.logging.context import LoggingContext
30	31	from synapse.metrics import METRICS_PREFIX, MetricsResource, RegistryProxy

34	35	from synapse.replication.slave.storage.registration import SlavedRegistrationStore
35	36	from synapse.replication.slave.storage.transactions import SlavedTransactionStore
36	37	from synapse.replication.tcp.client import ReplicationClientHandler
	38	from synapse.rest.admin import register_servlets_for_media_repo
37	39	from synapse.rest.media.v0.content_repository import ContentRepoResource
38	40	from synapse.server import HomeServer
39	41	from synapse.storage.engines import create_engine

70	72	resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
71	73	elif name == "media":
72	74	media_repo = self.get_media_repository_resource()
	75
	76	# We need to serve the admin servlets for media on the
	77	# worker.
	78	admin_resource = JsonResource(self, canonical_json=False)
	79	register_servlets_for_media_repo(self, admin_resource)
	80
73	81	resources.update(
74	82	{
75	83	MEDIA_PREFIX: media_repo,

77	85	CONTENT_REPO_PREFIX: ContentRepoResource(
78	86	self, self.config.uploads_path
79	87	),
	88	"/_synapse/admin": admin_resource,
80	89	}
81	90	)
82	91

160	169	)
161	170
162	171	ss.setup()
163		reactor.callWhenRunning(_base.start, ss, config.worker_listeners)
	172	reactor.addSystemEventTrigger(
	173	"before", "startup", _base.start, ss, config.worker_listeners
	174	)
164	175
165	176	_base.start_worker_reactor("synapse-media-repository", config)
166	177

+1

-1

synapse/app/pusher.py less more

215	215	_base.start(ps, config.worker_listeners)
216	216	ps.get_pusherpool().start()
217	217
218		reactor.callWhenRunning(start)
	218	reactor.addSystemEventTrigger("before", "startup", start)
219	219
220	220	_base.start_worker_reactor("synapse-pusher", config)
221	221

+3

-1

synapse/app/synchrotron.py less more

450	450	)
451	451
452	452	ss.setup()
453		reactor.callWhenRunning(_base.start, ss, config.worker_listeners)
	453	reactor.addSystemEventTrigger(
	454	"before", "startup", _base.start, ss, config.worker_listeners
	455	)
454	456
455	457	_base.start_worker_reactor("synapse-synchrotron", config)
456	458

+3

-1

synapse/app/user_dir.py less more

223	223	)
224	224
225	225	ss.setup()
226		reactor.callWhenRunning(_base.start, ss, config.worker_listeners)
	226	reactor.addSystemEventTrigger(
	227	"before", "startup", _base.start, ss, config.worker_listeners
	228	)
227	229
228	230	_base.start_worker_reactor("synapse-user-dir", config)
229	231

+15

-15

synapse/appservice/__init__.py less more

174	174	@defer.inlineCallbacks
175	175	def _matches_user(self, event, store):
176	176	if not event:
177		defer.returnValue(False)
	177	return False
178	178
179	179	if self.is_interested_in_user(event.sender):
180		defer.returnValue(True)
	180	return True
181	181	# also check m.room.member state key
182	182	if event.type == EventTypes.Member and self.is_interested_in_user(
183	183	event.state_key
184	184	):
185		defer.returnValue(True)
	185	return True
186	186
187	187	if not store:
188		defer.returnValue(False)
	188	return False
189	189
190	190	does_match = yield self._matches_user_in_member_list(event.room_id, store)
191		defer.returnValue(does_match)
	191	return does_match
192	192
193	193	@cachedInlineCallbacks(num_args=1, cache_context=True)
194	194	def _matches_user_in_member_list(self, room_id, store, cache_context):

199	199	# check joined member events
200	200	for user_id in member_list:
201	201	if self.is_interested_in_user(user_id):
202		defer.returnValue(True)
203		defer.returnValue(False)
	202	return True
	203	return False
204	204
205	205	def _matches_room_id(self, event):
206	206	if hasattr(event, "room_id"):

210	210	@defer.inlineCallbacks
211	211	def _matches_aliases(self, event, store):
212	212	if not store or not event:
213		defer.returnValue(False)
	213	return False
214	214
215	215	alias_list = yield store.get_aliases_for_room(event.room_id)
216	216	for alias in alias_list:
217	217	if self.is_interested_in_alias(alias):
218		defer.returnValue(True)
219		defer.returnValue(False)
	218	return True
	219	return False
220	220
221	221	@defer.inlineCallbacks
222	222	def is_interested(self, event, store=None):

230	230	"""
231	231	# Do cheap checks first
232	232	if self._matches_room_id(event):
233		defer.returnValue(True)
	233	return True
234	234
235	235	if (yield self._matches_aliases(event, store)):
236		defer.returnValue(True)
	236	return True
237	237
238	238	if (yield self._matches_user(event, store)):
239		defer.returnValue(True)
240
241		defer.returnValue(False)
	239	return True
	240
	241	return False
242	242
243	243	def is_interested_in_user(self, user_id):
244	244	return (

+19

-19

synapse/appservice/api.py less more

96	96	@defer.inlineCallbacks
97	97	def query_user(self, service, user_id):
98	98	if service.url is None:
99		defer.returnValue(False)
	99	return False
100	100	uri = service.url + ("/users/%s" % urllib.parse.quote(user_id))
101	101	response = None
102	102	try:
103	103	response = yield self.get_json(uri, {"access_token": service.hs_token})
104	104	if response is not None: # just an empty json object
105		defer.returnValue(True)
	105	return True
106	106	except CodeMessageException as e:
107	107	if e.code == 404:
108		defer.returnValue(False)
	108	return False
109	109	return
110	110	logger.warning("query_user to %s received %s", uri, e.code)
111	111	except Exception as ex:
112	112	logger.warning("query_user to %s threw exception %s", uri, ex)
113		defer.returnValue(False)
	113	return False
114	114
115	115	@defer.inlineCallbacks
116	116	def query_alias(self, service, alias):
117	117	if service.url is None:
118		defer.returnValue(False)
	118	return False
119	119	uri = service.url + ("/rooms/%s" % urllib.parse.quote(alias))
120	120	response = None
121	121	try:
122	122	response = yield self.get_json(uri, {"access_token": service.hs_token})
123	123	if response is not None: # just an empty json object
124		defer.returnValue(True)
	124	return True
125	125	except CodeMessageException as e:
126	126	logger.warning("query_alias to %s received %s", uri, e.code)
127	127	if e.code == 404:
128		defer.returnValue(False)
	128	return False
129	129	return
130	130	except Exception as ex:
131	131	logger.warning("query_alias to %s threw exception %s", uri, ex)
132		defer.returnValue(False)
	132	return False
133	133
134	134	@defer.inlineCallbacks
135	135	def query_3pe(self, service, kind, protocol, fields):

140	140	else:
141	141	raise ValueError("Unrecognised 'kind' argument %r to query_3pe()", kind)
142	142	if service.url is None:
143		defer.returnValue([])
	143	return []
144	144
145	145	uri = "%s%s/thirdparty/%s/%s" % (
146	146	service.url,

154	154	logger.warning(
155	155	"query_3pe to %s returned an invalid response %r", uri, response
156	156	)
157		defer.returnValue([])
	157	return []
158	158
159	159	ret = []
160	160	for r in response:

165	165	"query_3pe to %s returned an invalid result %r", uri, r
166	166	)
167	167
168		defer.returnValue(ret)
	168	return ret
169	169	except Exception as ex:
170	170	logger.warning("query_3pe to %s threw exception %s", uri, ex)
171		defer.returnValue([])
	171	return []
172	172
173	173	def get_3pe_protocol(self, service, protocol):
174	174	if service.url is None:
175		defer.returnValue({})
	175	return {}
176	176
177	177	@defer.inlineCallbacks
178	178	def _get():

188	188	logger.warning(
189	189	"query_3pe_protocol to %s did not return a" " valid result", uri
190	190	)
191		defer.returnValue(None)
	191	return None
192	192
193	193	for instance in info.get("instances", []):
194	194	network_id = instance.get("network_id", None)

197	197	service.id, network_id
198	198	).to_string()
199	199
200		defer.returnValue(info)
	200	return info
201	201	except Exception as ex:
202	202	logger.warning("query_3pe_protocol to %s threw exception %s", uri, ex)
203		defer.returnValue(None)
	203	return None
204	204
205	205	key = (service.id, protocol)
206	206	return self.protocol_meta_cache.wrap(key, _get)

208	208	@defer.inlineCallbacks
209	209	def push_bulk(self, service, events, txn_id=None):
210	210	if service.url is None:
211		defer.returnValue(True)
	211	return True
212	212
213	213	events = self._serialize(events)
214	214

228	228	)
229	229	sent_transactions_counter.labels(service.id).inc()
230	230	sent_events_counter.labels(service.id).inc(len(events))
231		defer.returnValue(True)
	231	return True
232	232	return
233	233	except CodeMessageException as e:
234	234	logger.warning("push_bulk to %s received %s", uri, e.code)
235	235	except Exception as ex:
236	236	logger.warning("push_bulk to %s threw exception %s", uri, ex)
237	237	failed_transactions_counter.labels(service.id).inc()
238		defer.returnValue(False)
	238	return False
239	239
240	240	def _serialize(self, events):
241	241	time_now = self.clock.time_msec()

+2

-2

synapse/appservice/scheduler.py less more

192	192	@defer.inlineCallbacks
193	193	def _is_service_up(self, service):
194	194	state = yield self.store.get_appservice_state(service)
195		defer.returnValue(state == ApplicationServiceState.UP or state is None)
	195	return state == ApplicationServiceState.UP or state is None
196	196
197	197
198	198	class _Recoverer(object):

207	207	r.service.id,
208	208	)
209	209	r.recover()
210		defer.returnValue(recoverers)
	210	return recoverers
211	211
212	212	def __init__(self, clock, store, as_api, service, callback):
213	213	self.clock = clock

+0

-6

synapse/config/key.py less more

115	115	seed = bytes(self.signing_key[0])
116	116	self.macaroon_secret_key = hashlib.sha256(seed).digest()
117	117
118		self.expire_access_token = config.get("expire_access_token", False)
119
120	118	# a secret which is used to calculate HMACs for form values, to stop
121	119	# falsification of values
122	120	self.form_secret = config.get("form_secret", None)

142	140	# a secret key is derived from the signing key.
143	141	#
144	142	%(macaroon_secret_key)s
145
146		# Used to enable access token expiration.
147		#
148		#expire_access_token: False
149	143
150	144	# a secret which is used to calculate HMACs for form values, to stop
151	145	# falsification of values. Must be specified for the User Consent

+13

-70

synapse/config/logger.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14
14	15	import logging
15	16	import logging.config
16	17	import os

74	75
75	76	class LoggingConfig(Config):
76	77	def read_config(self, config, **kwargs):
77		self.verbosity = config.get("verbose", 0)
	78	self.log_config = self.abspath(config.get("log_config"))
78	79	self.no_redirect_stdio = config.get("no_redirect_stdio", False)
79		self.log_config = self.abspath(config.get("log_config"))
80		self.log_file = self.abspath(config.get("log_file"))
81	80
82	81	def generate_config_section(self, config_dir_path, server_name, **kwargs):
83	82	log_config = os.path.join(config_dir_path, server_name + ".log.config")

93	92	)
94	93
95	94	def read_arguments(self, args):
96		if args.verbose is not None:
97		self.verbosity = args.verbose
98	95	if args.no_redirect_stdio is not None:
99	96	self.no_redirect_stdio = args.no_redirect_stdio
100		if args.log_config is not None:
101		self.log_config = args.log_config
102		if args.log_file is not None:
103		self.log_file = args.log_file
104	97
105	98	@staticmethod
106	99	def add_arguments(parser):
107	100	logging_group = parser.add_argument_group("logging")
108		logging_group.add_argument(
109		"-v",
110		"--verbose",
111		dest="verbose",
112		action="count",
113		help="The verbosity level. Specify multiple times to increase "
114		"verbosity. (Ignored if --log-config is specified.)",
115		)
116		logging_group.add_argument(
117		"-f",
118		"--log-file",
119		dest="log_file",
120		help="File to log to. (Ignored if --log-config is specified.)",
121		)
122		logging_group.add_argument(
123		"--log-config",
124		dest="log_config",
125		default=None,
126		help="Python logging config file",
127		)
128	101	logging_group.add_argument(
129	102	"-n",
130	103	"--no-redirect-stdio",

152	125	config (LoggingConfig \| synapse.config.workers.WorkerConfig):
153	126	configuration data
154	127
155		use_worker_options (bool): True to use 'worker_log_config' and
156		'worker_log_file' options instead of 'log_config' and 'log_file'.
	128	use_worker_options (bool): True to use the 'worker_log_config' option
	129	instead of 'log_config'.
157	130
158	131	register_sighup (func \| None): Function to call to register a
159	132	sighup handler.
160	133	"""
161	134	log_config = config.worker_log_config if use_worker_options else config.log_config
162		log_file = config.worker_log_file if use_worker_options else config.log_file
163
164		log_format = (
165		"%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s"
166		" - %(message)s"
167		)
168	135
169	136	if log_config is None:
170		# We don't have a logfile, so fall back to the 'verbosity' param from
171		# the config or cmdline. (Note that we generate a log config for new
172		# installs, so this will be an unusual case)
173		level = logging.INFO
174		level_for_storage = logging.INFO
175		if config.verbosity:
176		level = logging.DEBUG
177		if config.verbosity > 1:
178		level_for_storage = logging.DEBUG
	137	log_format = (
	138	"%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s"
	139	" - %(message)s"
	140	)
179	141
180	142	logger = logging.getLogger("")
181		logger.setLevel(level)
182
183		logging.getLogger("synapse.storage.SQL").setLevel(level_for_storage)
	143	logger.setLevel(logging.INFO)
	144	logging.getLogger("synapse.storage.SQL").setLevel(logging.INFO)
184	145
185	146	formatter = logging.Formatter(log_format)
186		if log_file:
187		# TODO: Customisable file size / backup count
188		handler = logging.handlers.RotatingFileHandler(
189		log_file, maxBytes=(1000 * 1000 * 100), backupCount=3, encoding="utf8"
190		)
191
192		def sighup(signum, stack):
193		logger.info("Closing log file due to SIGHUP")
194		handler.doRollover()
195		logger.info("Opened new log file due to SIGHUP")
196
197		else:
198		handler = logging.StreamHandler()
199
200		def sighup(*args):
201		pass
202
	147
	148	handler = logging.StreamHandler()
203	149	handler.setFormatter(formatter)
204
205	150	handler.addFilter(LoggingContextFilter(request=""))
206
207	151	logger.addHandler(handler)
208	152	else:
209	153

217	161	logging.info("Reloaded log config from %s due to SIGHUP", log_config)
218	162
219	163	load_log_config()
220
221		appbase.register_sighup(sighup)
	164	appbase.register_sighup(sighup)
222	165
223	166	# make sure that the first thing we log is a thing we can grep backwards
224	167	# for

+43

-2

synapse/config/registration.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
	15	import os
15	16	from distutils.util import strtobool
	17
	18	import pkg_resources
16	19
17	20	from synapse.config._base import Config, ConfigError
18	21	from synapse.types import RoomAlias

40	43
41	44	self.startup_job_max_delta = self.period * 10.0 / 100.0
42	45
43		if self.renew_by_email_enabled and "public_baseurl" not in synapse_config:
44		raise ConfigError("Can't send renewal emails without 'public_baseurl'")
	46	if self.renew_by_email_enabled:
	47	if "public_baseurl" not in synapse_config:
	48	raise ConfigError("Can't send renewal emails without 'public_baseurl'")
	49
	50	template_dir = config.get("template_dir")
	51
	52	if not template_dir:
	53	template_dir = pkg_resources.resource_filename("synapse", "res/templates")
	54
	55	if "account_renewed_html_path" in config:
	56	file_path = os.path.join(template_dir, config["account_renewed_html_path"])
	57
	58	self.account_renewed_html_content = self.read_file(
	59	file_path, "account_validity.account_renewed_html_path"
	60	)
	61	else:
	62	self.account_renewed_html_content = (
	63	"<html><body>Your account has been successfully renewed.</body><html>"
	64	)
	65
	66	if "invalid_token_html_path" in config:
	67	file_path = os.path.join(template_dir, config["invalid_token_html_path"])
	68
	69	self.invalid_token_html_content = self.read_file(
	70	file_path, "account_validity.invalid_token_html_path"
	71	)
	72	else:
	73	self.invalid_token_html_content = (
	74	"<html><body>Invalid renewal token.</body><html>"
	75	)
45	76
46	77
47	78	class RegistrationConfig(Config):

144	175	# period: 6w
145	176	# renew_at: 1w
146	177	# renew_email_subject: "Renew your %%(app)s account"
	178	# # Directory in which Synapse will try to find the HTML files to serve to the
	179	# # user when trying to renew an account. Optional, defaults to
	180	# # synapse/res/templates.
	181	# template_dir: "res/templates"
	182	# # HTML to be displayed to the user after they successfully renewed their
	183	# # account. Optional.
	184	# account_renewed_html_path: "account_renewed.html"
	185	# # HTML to be displayed when the user tries to renew an account with an invalid
	186	# # renewal token. Optional.
	187	# invalid_token_html_path: "invalid_token.html"
147	188
148	189	# Time that a user's session remains valid for, after they log in.
149	190	#

+20

-0

synapse/config/repository.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14
14	15	import os
15	16	from collections import namedtuple
16	17

86	87
87	88	class ContentRepositoryConfig(Config):
88	89	def read_config(self, config, **kwargs):
	90
	91	# Only enable the media repo if either the media repo is enabled or the
	92	# current worker app is the media repo.
	93	if (
	94	self.enable_media_repo is False
	95	and config.get("worker_app") != "synapse.app.media_repository"
	96	):
	97	self.can_load_media_repo = False
	98	return
	99	else:
	100	self.can_load_media_repo = True
	101
89	102	self.max_upload_size = self.parse_size(config.get("max_upload_size", "10M"))
90	103	self.max_image_pixels = self.parse_size(config.get("max_image_pixels", "32M"))
91	104	self.max_spider_size = self.parse_size(config.get("max_spider_size", "10M"))

201	214
202	215	return (
203	216	r"""
	217	## Media Store ##
	218
	219	# Enable the media store service in the Synapse master. Uncomment the
	220	# following if you are using a separate media store worker.
	221	#
	222	#enable_media_repo: false
	223
204	224	# Directory where uploaded images and attachments are stored.
205	225	#
206	226	media_store_path: "%(media_store)s"

+41

-0

synapse/config/server.py less more

17	17	import logging
18	18	import os.path
19	19
	20	import attr
20	21	from netaddr import IPSet
21	22
22	23	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS

36	37	DEFAULT_BIND_ADDRESSES = ["::", "0.0.0.0"]
37	38
38	39	DEFAULT_ROOM_VERSION = "4"
	40
	41	ROOM_COMPLEXITY_TOO_GREAT = (
	42	"Your homeserver is unable to join rooms this large or complex. "
	43	"Please speak to your server administrator, or upgrade your instance "
	44	"to join this room."
	45	)
39	46
40	47
41	48	class ServerConfig(Config):

245	252	_warn_if_webclient_configured(self.listeners)
246	253
247	254	self.gc_thresholds = read_gc_thresholds(config.get("gc_thresholds", None))
	255
	256	@attr.s
	257	class LimitRemoteRoomsConfig(object):
	258	enabled = attr.ib(
	259	validator=attr.validators.instance_of(bool), default=False
	260	)
	261	complexity = attr.ib(
	262	validator=attr.validators.instance_of((int, float)), default=1.0
	263	)
	264	complexity_error = attr.ib(
	265	validator=attr.validators.instance_of(str),
	266	default=ROOM_COMPLEXITY_TOO_GREAT,
	267	)
	268
	269	self.limit_remote_rooms = LimitRemoteRoomsConfig(
	270	**config.get("limit_remote_rooms", {})
	271	)
248	272
249	273	bind_port = config.get("bind_port")
250	274	if bind_port:

616	640	# Used by phonehome stats to group together related servers.
617	641	#server_context: context
618	642
	643	# Resource-constrained Homeserver Settings
	644	#
	645	# If limit_remote_rooms.enabled is True, the room complexity will be
	646	# checked before a user joins a new remote room. If it is above
	647	# limit_remote_rooms.complexity, it will disallow joining or
	648	# instantly leave.
	649	#
	650	# limit_remote_rooms.complexity_error can be set to customise the text
	651	# displayed to the user when a room above the complexity threshold has
	652	# its join cancelled.
	653	#
	654	# Uncomment the below lines to enable:
	655	#limit_remote_rooms:
	656	# enabled: True
	657	# complexity: 1.0
	658	# complexity_error: "This room is too complex."
	659
619	660	# Whether to require a user to be in the room to add an alias to it.
620	661	# Defaults to 'true'.
621	662	#

+22

-0

synapse/config/tracer.py less more

22	22	opentracing_config = {}
23	23
24	24	self.opentracer_enabled = opentracing_config.get("enabled", False)
	25
	26	self.jaeger_config = opentracing_config.get(
	27	"jaeger_config",
	28	{"sampler": {"type": "const", "param": 1}, "logging": False},
	29	)
	30
25	31	if not self.opentracer_enabled:
26	32	return
27	33

55	61	#
56	62	#homeserver_whitelist:
57	63	# - ".*"
	64
	65	# Jaeger can be configured to sample traces at different rates.
	66	# All configuration options provided by Jaeger can be set here.
	67	# Jaeger's configuration mostly related to trace sampling which
	68	# is documented here:
	69	# https://www.jaegertracing.io/docs/1.13/sampling/.
	70	#
	71	#jaeger_config:
	72	# sampler:
	73	# type: const
	74	# param: 1
	75
	76	# Logging whether spans were started and reported
	77	#
	78	# logging:
	79	# false
58	80	"""

+0

-5

synapse/config/workers.py less more

30	30	self.worker_listeners = config.get("worker_listeners", [])
31	31	self.worker_daemonize = config.get("worker_daemonize")
32	32	self.worker_pid_file = config.get("worker_pid_file")
33		self.worker_log_file = config.get("worker_log_file")
34	33	self.worker_log_config = config.get("worker_log_config")
35	34
36	35	# The host used to connect to the main synapse

77	76
78	77	if args.daemonize is not None:
79	78	self.worker_daemonize = args.daemonize
80		if args.log_config is not None:
81		self.worker_log_config = args.log_config
82		if args.log_file is not None:
83		self.worker_log_file = args.log_file
84	79	if args.manhole is not None:
85	80	self.worker_manhole = args.worker_manhole

+8

-0

synapse/crypto/context_factory.py less more

30	30	platformTrust,
31	31	)
32	32	from twisted.python.failure import Failure
	33	from twisted.web.iweb import IPolicyForHTTPS
33	34
34	35	logger = logging.getLogger(__name__)
35	36

73	74	return self._context
74	75
75	76
	77	@implementer(IPolicyForHTTPS)
76	78	class ClientTLSOptionsFactory(object):
77	79	"""Factory for Twisted SSLClientConnectionCreators that are used to make connections
78	80	to remote servers for federation.

145	147	f = Failure()
146	148	tls_protocol.failVerification(f)
147	149
	150	def creatorForNetloc(self, hostname, port):
	151	"""Implements the IPolicyForHTTPS interace so that this can be passed
	152	directly to agents.
	153	"""
	154	return self.get_options(hostname)
	155
148	156
149	157	@implementer(IOpenSSLClientConnectionCreator)
150	158	class SSLClientConnectionCreator(object):

+49

-59

synapse/crypto/keyring.py less more

237	237	"""
238	238
239	239	try:
240		# create a deferred for each server we're going to look up the keys
241		# for; we'll resolve them once we have completed our lookups.
242		# These will be passed into wait_for_previous_lookups to block
243		# any other lookups until we have finished.
244		# The deferreds are called with no logcontext.
245		server_to_deferred = {
246		rq.server_name: defer.Deferred() for rq in verify_requests
247		}
248
249		# We want to wait for any previous lookups to complete before
250		# proceeding.
251		yield self.wait_for_previous_lookups(server_to_deferred)
252
253		# Actually start fetching keys.
254		self._get_server_verify_keys(verify_requests)
255
256		# When we've finished fetching all the keys for a given server_name,
257		# resolve the deferred passed to `wait_for_previous_lookups` so that
258		# any lookups waiting will proceed.
259		#
260		# map from server name to a set of request ids
	240	ctx = LoggingContext.current_context()
	241
	242	# map from server name to a set of outstanding request ids
261	243	server_to_request_ids = {}
262	244
263	245	for verify_request in verify_requests:

265	247	request_id = id(verify_request)
266	248	server_to_request_ids.setdefault(server_name, set()).add(request_id)
267	249
268		def remove_deferreds(res, verify_request):
	250	# Wait for any previous lookups to complete before proceeding.
	251	yield self.wait_for_previous_lookups(server_to_request_ids.keys())
	252
	253	# take out a lock on each of the servers by sticking a Deferred in
	254	# key_downloads
	255	for server_name in server_to_request_ids.keys():
	256	self.key_downloads[server_name] = defer.Deferred()
	257	logger.debug("Got key lookup lock on %s", server_name)
	258
	259	# When we've finished fetching all the keys for a given server_name,
	260	# drop the lock by resolving the deferred in key_downloads.
	261	def drop_server_lock(server_name):
	262	d = self.key_downloads.pop(server_name)
	263	d.callback(None)
	264
	265	def lookup_done(res, verify_request):
269	266	server_name = verify_request.server_name
270		request_id = id(verify_request)
271		server_to_request_ids[server_name].discard(request_id)
272		if not server_to_request_ids[server_name]:
273		d = server_to_deferred.pop(server_name, None)
274		if d:
275		d.callback(None)
	267	server_requests = server_to_request_ids[server_name]
	268	server_requests.remove(id(verify_request))
	269
	270	# if there are no more requests for this server, we can drop the lock.
	271	if not server_requests:
	272	with PreserveLoggingContext(ctx):
	273	logger.debug("Releasing key lookup lock on %s", server_name)
	274
	275	# ... but not immediately, as that can cause stack explosions if
	276	# we get a long queue of lookups.
	277	self.clock.call_later(0, drop_server_lock, server_name)
	278
276	279	return res
277	280
278	281	for verify_request in verify_requests:
279		verify_request.key_ready.addBoth(remove_deferreds, verify_request)
	282	verify_request.key_ready.addBoth(lookup_done, verify_request)
	283
	284	# Actually start fetching keys.
	285	self._get_server_verify_keys(verify_requests)
280	286	except Exception:
281	287	logger.exception("Error starting key lookups")
282	288
283	289	@defer.inlineCallbacks
284		def wait_for_previous_lookups(self, server_to_deferred):
	290	def wait_for_previous_lookups(self, server_names):
285	291	"""Waits for any previous key lookups for the given servers to finish.
286	292
287	293	Args:
288		server_to_deferred (dict[str, Deferred]): server_name to deferred which gets
289		resolved once we've finished looking up keys for that server.
290		The Deferreds should be regular twisted ones which call their
291		callbacks with no logcontext.
292
293		Returns: a Deferred which resolves once all key lookups for the given
294		servers have completed. Follows the synapse rules of logcontext
295		preservation.
	294	server_names (Iterable[str]): list of servers which we want to look up
	295
	296	Returns:
	297	Deferred[None]: resolves once all key lookups for the given servers have
	298	completed. Follows the synapse rules of logcontext preservation.
296	299	"""
297	300	loop_count = 1
298	301	while True:
299	302	wait_on = [
300	303	(server_name, self.key_downloads[server_name])
301		for server_name in server_to_deferred.keys()
	304	for server_name in server_names
302	305	if server_name in self.key_downloads
303	306	]
304	307	if not wait_on:

312	315	yield defer.DeferredList((w[1] for w in wait_on))
313	316
314	317	loop_count += 1
315
316		ctx = LoggingContext.current_context()
317
318		def rm(r, server_name_):
319		with PreserveLoggingContext(ctx):
320		logger.debug("Releasing key lookup lock on %s", server_name_)
321		self.key_downloads.pop(server_name_, None)
322		return r
323
324		for server_name, deferred in server_to_deferred.items():
325		logger.debug("Got key lookup lock on %s", server_name)
326		self.key_downloads[server_name] = deferred
327		deferred.addBoth(rm, server_name)
328	318
329	319	def _get_server_verify_keys(self, verify_requests):
330	320	"""Tries to find at least one key for each verify request

471	461	keys = {}
472	462	for (server_name, key_id), key in res.items():
473	463	keys.setdefault(server_name, {})[key_id] = key
474		defer.returnValue(keys)
	464	return keys
475	465
476	466
477	467	class BaseV2KeyFetcher(object):

575	565	).addErrback(unwrapFirstError)
576	566	)
577	567
578		defer.returnValue(verify_keys)
	568	return verify_keys
579	569
580	570
581	571	class PerspectivesKeyFetcher(BaseV2KeyFetcher):

597	587	result = yield self.get_server_verify_key_v2_indirect(
598	588	keys_to_fetch, key_server
599	589	)
600		defer.returnValue(result)
	590	return result
601	591	except KeyLookupError as e:
602	592	logger.warning(
603	593	"Key lookup failed from %r: %s", key_server.server_name, e

610	600	str(e),
611	601	)
612	602
613		defer.returnValue({})
	603	return {}
614	604
615	605	results = yield make_deferred_yieldable(
616	606	defer.gatherResults(

624	614	for server_name, keys in result.items():
625	615	union_of_keys.setdefault(server_name, {}).update(keys)
626	616
627		defer.returnValue(union_of_keys)
	617	return union_of_keys
628	618
629	619	@defer.inlineCallbacks
630	620	def get_server_verify_key_v2_indirect(self, keys_to_fetch, key_server):

710	700	perspective_name, time_now_ms, added_keys
711	701	)
712	702
713		defer.returnValue(keys)
	703	return keys
714	704
715	705	def _validate_perspectives_response(self, key_server, response):
716	706	"""Optionally check the signature on the result of a /key/query request

852	842	)
853	843	keys.update(response_keys)
854	844
855		defer.returnValue(keys)
	845	return keys
856	846
857	847
858	848	@defer.inlineCallbacks

+7

-9

synapse/events/builder.py less more

143	143	if self._origin_server_ts is not None:
144	144	event_dict["origin_server_ts"] = self._origin_server_ts
145	145
146		defer.returnValue(
147		create_local_event_from_event_dict(
148		clock=self._clock,
149		hostname=self._hostname,
150		signing_key=self._signing_key,
151		format_version=self.format_version,
152		event_dict=event_dict,
153		internal_metadata_dict=self.internal_metadata.get_dict(),
154		)
	146	return create_local_event_from_event_dict(
	147	clock=self._clock,
	148	hostname=self._hostname,
	149	signing_key=self._signing_key,
	150	format_version=self.format_version,
	151	event_dict=event_dict,
	152	internal_metadata_dict=self.internal_metadata.get_dict(),
155	153	)
156	154
157	155

+13

-15

synapse/events/snapshot.py less more

132	132	else:
133	133	prev_state_id = None
134	134
135		defer.returnValue(
136		{
137		"prev_state_id": prev_state_id,
138		"event_type": event.type,
139		"event_state_key": event.state_key if event.is_state() else None,
140		"state_group": self.state_group,
141		"rejected": self.rejected,
142		"prev_group": self.prev_group,
143		"delta_ids": _encode_state_dict(self.delta_ids),
144		"prev_state_events": self.prev_state_events,
145		"app_service_id": self.app_service.id if self.app_service else None,
146		}
147		)
	135	return {
	136	"prev_state_id": prev_state_id,
	137	"event_type": event.type,
	138	"event_state_key": event.state_key if event.is_state() else None,
	139	"state_group": self.state_group,
	140	"rejected": self.rejected,
	141	"prev_group": self.prev_group,
	142	"delta_ids": _encode_state_dict(self.delta_ids),
	143	"prev_state_events": self.prev_state_events,
	144	"app_service_id": self.app_service.id if self.app_service else None,
	145	}
148	146
149	147	@staticmethod
150	148	def deserialize(store, input):

201	199
202	200	yield make_deferred_yieldable(self._fetching_state_deferred)
203	201
204		defer.returnValue(self._current_state_ids)
	202	return self._current_state_ids
205	203
206	204	@defer.inlineCallbacks
207	205	def get_prev_state_ids(self, store):

221	219
222	220	yield make_deferred_yieldable(self._fetching_state_deferred)
223	221
224		defer.returnValue(self._prev_state_ids)
	222	return self._prev_state_ids
225	223
226	224	def get_cached_current_state_ids(self):
227	225	"""Gets the current state IDs if we have them already cached.

+4

-4

synapse/events/third_party_rules.py less more

50	50	defer.Deferred[bool]: True if the event should be allowed, False if not.
51	51	"""
52	52	if self.third_party_rules is None:
53		defer.returnValue(True)
	53	return True
54	54
55	55	prev_state_ids = yield context.get_prev_state_ids(self.store)
56	56

60	60	state_events[key] = yield self.store.get_event(event_id, allow_none=True)
61	61
62	62	ret = yield self.third_party_rules.check_event_allowed(event, state_events)
63		defer.returnValue(ret)
	63	return ret
64	64
65	65	@defer.inlineCallbacks
66	66	def on_create_room(self, requester, config, is_requester_admin):

97	97	"""
98	98
99	99	if self.third_party_rules is None:
100		defer.returnValue(True)
	100	return True
101	101
102	102	state_ids = yield self.store.get_filtered_current_state_ids(room_id)
103	103	room_state_events = yield self.store.get_events(state_ids.values())

109	109	ret = yield self.third_party_rules.check_threepid_can_be_invited(
110	110	medium, address, state_events
111	111	)
112		defer.returnValue(ret)
	112	return ret

+2

-2

synapse/events/utils.py less more

359	359	"""
360	360	# To handle the case of presence events and the like
361	361	if not isinstance(event, EventBase):
362		defer.returnValue(event)
	362	return event
363	363
364	364	event_id = event.event_id
365	365	serialized_event = serialize_event(event, time_now, **kwargs)

405	405	"sender": edit.sender,
406	406	}
407	407
408		defer.returnValue(serialized_event)
	408	return serialized_event
409	409
410	410	def serialize_events(self, events, time_now, **kwargs):
411	411	"""Serializes multiple events.

+18

-2

synapse/events/validator.py less more

94	94
95	95	elif event.type == EventTypes.Topic:
96	96	self._ensure_strings(event.content, ["topic"])
97
	97	self._ensure_state_event(event)
98	98	elif event.type == EventTypes.Name:
99	99	self._ensure_strings(event.content, ["name"])
100
	100	self._ensure_state_event(event)
101	101	elif event.type == EventTypes.Member:
102	102	if "membership" not in event.content:
103	103	raise SynapseError(400, "Content has not membership key")

105	105	if event.content["membership"] not in Membership.LIST:
106	106	raise SynapseError(400, "Invalid membership key")
107	107
	108	self._ensure_state_event(event)
	109	elif event.type == EventTypes.Tombstone:
	110	if "replacement_room" not in event.content:
	111	raise SynapseError(400, "Content has no replacement_room key")
	112
	113	if event.content["replacement_room"] == event.room_id:
	114	raise SynapseError(
	115	400, "Tombstone cannot reference the room it was sent in"
	116	)
	117
	118	self._ensure_state_event(event)
	119
108	120	def _ensure_strings(self, d, keys):
109	121	for s in keys:
110	122	if s not in d:
111	123	raise SynapseError(400, "'%s' not in content" % (s,))
112	124	if not isinstance(d[s], string_types):
113	125	raise SynapseError(400, "'%s' not a string type" % (s,))
	126
	127	def _ensure_state_event(self, event):
	128	if not event.is_state():
	129	raise SynapseError(400, "'%s' must be state events" % (event.type,))

+3

-3

synapse/federation/federation_base.py less more

105	105	"Failed to find copy of %s with valid signature", pdu.event_id
106	106	)
107	107
108		defer.returnValue(res)
	108	return res
109	109
110	110	handle = preserve_fn(handle_check_result)
111	111	deferreds2 = [handle(pdu, deferred) for pdu, deferred in zip(pdus, deferreds)]

115	115	).addErrback(unwrapFirstError)
116	116
117	117	if include_none:
118		defer.returnValue(valid_pdus)
	118	return valid_pdus
119	119	else:
120		defer.returnValue([p for p in valid_pdus if p])
	120	return [p for p in valid_pdus if p]
121	121
122	122	def _check_sigs_and_hash(self, room_version, pdu):
123	123	return make_deferred_yieldable(

+61

-28

synapse/federation/federation_client.py less more

212	212	).addErrback(unwrapFirstError)
213	213	)
214	214
215		defer.returnValue(pdus)
	215	return pdus
216	216
217	217	@defer.inlineCallbacks
218	218	@log_function

244	244
245	245	ev = self._get_pdu_cache.get(event_id)
246	246	if ev:
247		defer.returnValue(ev)
	247	return ev
248	248
249	249	pdu_attempts = self.pdu_destination_tried.setdefault(event_id, {})
250	250

306	306	if signed_pdu:
307	307	self._get_pdu_cache[event_id] = signed_pdu
308	308
309		defer.returnValue(signed_pdu)
	309	return signed_pdu
310	310
311	311	@defer.inlineCallbacks
312	312	@log_function

354	354
355	355	auth_chain.sort(key=lambda e: e.depth)
356	356
357		defer.returnValue((pdus, auth_chain))
	357	return (pdus, auth_chain)
358	358	except HttpResponseException as e:
359	359	if e.code == 400 or e.code == 404:
360	360	logger.info("Failed to use get_room_state_ids API, falling back")

403	403
404	404	signed_auth.sort(key=lambda e: e.depth)
405	405
406		defer.returnValue((signed_pdus, signed_auth))
	406	return (signed_pdus, signed_auth)
407	407
408	408	@defer.inlineCallbacks
409	409	def get_events_from_store_or_dest(self, destination, room_id, event_ids):

428	428	missing_events.discard(k)
429	429
430	430	if not missing_events:
431		defer.returnValue((signed_events, failed_to_fetch))
	431	return (signed_events, failed_to_fetch)
432	432
433	433	logger.debug(
434	434	"Fetching unknown state/auth events %s for room %s",

464	464	# We removed all events we successfully fetched from `batch`
465	465	failed_to_fetch.update(batch)
466	466
467		defer.returnValue((signed_events, failed_to_fetch))
	467	return (signed_events, failed_to_fetch)
468	468
469	469	@defer.inlineCallbacks
470	470	@log_function

484	484
485	485	signed_auth.sort(key=lambda e: e.depth)
486	486
487		defer.returnValue(signed_auth)
	487	return signed_auth
488	488
489	489	@defer.inlineCallbacks
490	490	def _try_destination_list(self, description, destinations, callback):

510	510	The [Deferred] result of callback, if it succeeds
511	511
512	512	Raises:
513		SynapseError if the chosen remote server returns a 300/400 code.
514
515		RuntimeError if no servers were reachable.
	513	SynapseError if the chosen remote server returns a 300/400 code, or
	514	no servers were reachable.
516	515	"""
517	516	for destination in destinations:
518	517	if destination == self.server_name:

520	519
521	520	try:
522	521	res = yield callback(destination)
523		defer.returnValue(res)
	522	return res
524	523	except InvalidResponseError as e:
525	524	logger.warn("Failed to %s via %s: %s", description, destination, e)
526	525	except HttpResponseException as e:

537	536	except Exception:
538	537	logger.warn("Failed to %s via %s", description, destination, exc_info=1)
539	538
540		raise RuntimeError("Failed to %s via any server" % (description,))
	539	raise SynapseError(502, "Failed to %s via any server" % (description,))
541	540
542	541	def make_membership_event(
543	542	self, destinations, room_id, user_id, membership, content, params

614	613	event_dict=pdu_dict,
615	614	)
616	615
617		defer.returnValue((destination, ev, event_format))
	616	return (destination, ev, event_format)
618	617
619	618	return self._try_destination_list(
620	619	"make_" + membership, destinations, send_request

727	726
728	727	check_authchain_validity(signed_auth)
729	728
730		defer.returnValue(
731		{
732		"state": signed_state,
733		"auth_chain": signed_auth,
734		"origin": destination,
735		}
736		)
	729	return {
	730	"state": signed_state,
	731	"auth_chain": signed_auth,
	732	"origin": destination,
	733	}
737	734
738	735	return self._try_destination_list("send_join", destinations, send_request)
739	736

757	754
758	755	# FIXME: We should handle signature failures more gracefully.
759	756
760		defer.returnValue(pdu)
	757	return pdu
761	758
762	759	@defer.inlineCallbacks
763	760	def _do_send_invite(self, destination, pdu, room_version):

785	782	"invite_room_state": pdu.unsigned.get("invite_room_state", []),
786	783	},
787	784	)
788		defer.returnValue(content)
	785	return content
789	786	except HttpResponseException as e:
790	787	if e.code in [400, 404]:
791	788	err = e.to_synapse_error()

820	817	event_id=pdu.event_id,
821	818	content=pdu.get_pdu_json(time_now),
822	819	)
823		defer.returnValue(content)
	820	return content
824	821
825	822	def send_leave(self, destinations, pdu):
826	823	"""Sends a leave event to one of a list of homeservers.

855	852	)
856	853
857	854	logger.debug("Got content: %s", content)
858		defer.returnValue(None)
	855	return None
859	856
860	857	return self._try_destination_list("send_leave", destinations, send_request)
861	858

916	913	"missing": content.get("missing", []),
917	914	}
918	915
919		defer.returnValue(ret)
	916	return ret
920	917
921	918	@defer.inlineCallbacks
922	919	def get_missing_events(

973	970	# get_missing_events
974	971	signed_events = []
975	972
976		defer.returnValue(signed_events)
	973	return signed_events
977	974
978	975	@defer.inlineCallbacks
979	976	def forward_third_party_invite(self, destinations, room_id, event_dict):

985	982	yield self.transport_layer.exchange_third_party_invite(
986	983	destination=destination, room_id=room_id, event_dict=event_dict
987	984	)
988		defer.returnValue(None)
	985	return None
989	986	except CodeMessageException:
990	987	raise
991	988	except Exception as e:

994	991	)
995	992
996	993	raise RuntimeError("Failed to send to any server.")
	994
	995	@defer.inlineCallbacks
	996	def get_room_complexity(self, destination, room_id):
	997	"""
	998	Fetch the complexity of a remote room from another server.
	999
	1000	Args:
	1001	destination (str): The remote server
	1002	room_id (str): The room ID to ask about.
	1003
	1004	Returns:
	1005	Deferred[dict] or Deferred[None]: Dict contains the complexity
	1006	metric versions, while None means we could not fetch the complexity.
	1007	"""
	1008	try:
	1009	complexity = yield self.transport_layer.get_room_complexity(
	1010	destination=destination, room_id=room_id
	1011	)
	1012	defer.returnValue(complexity)
	1013	except CodeMessageException as e:
	1014	# We didn't manage to get it -- probably a 404. We are okay if other
	1015	# servers don't give it to us.
	1016	logger.debug(
	1017	"Failed to fetch room complexity via %s for %s, got a %d",
	1018	destination,
	1019	room_id,
	1020	e.code,
	1021	)
	1022	except Exception:
	1023	logger.exception(
	1024	"Failed to fetch room complexity via %s for %s", destination, room_id
	1025	)
	1026
	1027	# If we don't manage to find it, return None. It's not an error if a
	1028	# server doesn't give it to us.
	1029	defer.returnValue(None)

+32

-45

synapse/federation/federation_server.py less more

98	98
99	99	res = self._transaction_from_pdus(pdus).get_dict()
100	100
101		defer.returnValue((200, res))
	101	return (200, res)
102	102
103	103	@defer.inlineCallbacks
104	104	@log_function

125	125	origin, transaction, request_time
126	126	)
127	127
128		defer.returnValue(result)
	128	return result
129	129
130	130	@defer.inlineCallbacks
131	131	def _handle_incoming_transaction(self, origin, transaction, request_time):

146	146	"[%s] We've already responded to this request",
147	147	transaction.transaction_id,
148	148	)
149		defer.returnValue(response)
150		return
	149	return response
151	150
152	151	logger.debug("[%s] Transaction is new", transaction.transaction_id)
153	152

162	161	yield self.transaction_actions.set_response(
163	162	origin, transaction, 400, response
164	163	)
165		defer.returnValue((400, response))
	164	return (400, response)
166	165
167	166	received_pdus_counter.inc(len(transaction.pdus))
168	167

264	263	logger.debug("Returning: %s", str(response))
265	264
266	265	yield self.transaction_actions.set_response(origin, transaction, 200, response)
267		defer.returnValue((200, response))
	266	return (200, response)
268	267
269	268	@defer.inlineCallbacks
270	269	def received_edu(self, origin, edu_type, content):

297	296	event_id,
298	297	)
299	298
300		defer.returnValue((200, resp))
	299	return (200, resp)
301	300
302	301	@defer.inlineCallbacks
303	302	def on_state_ids_request(self, origin, room_id, event_id):

314	313	state_ids = yield self.handler.get_state_ids_for_pdu(room_id, event_id)
315	314	auth_chain_ids = yield self.store.get_auth_chain_ids(state_ids)
316	315
317		defer.returnValue(
318		(200, {"pdu_ids": state_ids, "auth_chain_ids": auth_chain_ids})
319		)
	316	return (200, {"pdu_ids": state_ids, "auth_chain_ids": auth_chain_ids})
320	317
321	318	@defer.inlineCallbacks
322	319	def _on_context_state_request_compute(self, room_id, event_id):

335	332	)
336	333	)
337	334
338		defer.returnValue(
339		{
340		"pdus": [pdu.get_pdu_json() for pdu in pdus],
341		"auth_chain": [pdu.get_pdu_json() for pdu in auth_chain],
342		}
343		)
	335	return {
	336	"pdus": [pdu.get_pdu_json() for pdu in pdus],
	337	"auth_chain": [pdu.get_pdu_json() for pdu in auth_chain],
	338	}
344	339
345	340	@defer.inlineCallbacks
346	341	@log_function

348	343	pdu = yield self.handler.get_persisted_pdu(origin, event_id)
349	344
350	345	if pdu:
351		defer.returnValue((200, self._transaction_from_pdus([pdu]).get_dict()))
	346	return (200, self._transaction_from_pdus([pdu]).get_dict())
352	347	else:
353		defer.returnValue((404, ""))
	348	return (404, "")
354	349
355	350	@defer.inlineCallbacks
356	351	def on_query_request(self, query_type, args):
357	352	received_queries_counter.labels(query_type).inc()
358	353	resp = yield self.registry.on_query(query_type, args)
359		defer.returnValue((200, resp))
	354	return (200, resp)
360	355
361	356	@defer.inlineCallbacks
362	357	def on_make_join_request(self, origin, room_id, user_id, supported_versions):

370	365
371	366	pdu = yield self.handler.on_make_join_request(origin, room_id, user_id)
372	367	time_now = self._clock.time_msec()
373		defer.returnValue(
374		{"event": pdu.get_pdu_json(time_now), "room_version": room_version}
375		)
	368	return {"event": pdu.get_pdu_json(time_now), "room_version": room_version}
376	369
377	370	@defer.inlineCallbacks
378	371	def on_invite_request(self, origin, content, room_version):

390	383	yield self.check_server_matches_acl(origin_host, pdu.room_id)
391	384	ret_pdu = yield self.handler.on_invite_request(origin, pdu)
392	385	time_now = self._clock.time_msec()
393		defer.returnValue({"event": ret_pdu.get_pdu_json(time_now)})
	386	return {"event": ret_pdu.get_pdu_json(time_now)}
394	387
395	388	@defer.inlineCallbacks
396	389	def on_send_join_request(self, origin, content, room_id):

406	399	logger.debug("on_send_join_request: pdu sigs: %s", pdu.signatures)
407	400	res_pdus = yield self.handler.on_send_join_request(origin, pdu)
408	401	time_now = self._clock.time_msec()
409		defer.returnValue(
410		(
411		200,
412		{
413		"state": [p.get_pdu_json(time_now) for p in res_pdus["state"]],
414		"auth_chain": [
415		p.get_pdu_json(time_now) for p in res_pdus["auth_chain"]
416		],
417		},
418		)
	402	return (
	403	200,
	404	{
	405	"state": [p.get_pdu_json(time_now) for p in res_pdus["state"]],
	406	"auth_chain": [
	407	p.get_pdu_json(time_now) for p in res_pdus["auth_chain"]
	408	],
	409	},
419	410	)
420	411
421	412	@defer.inlineCallbacks

427	418	room_version = yield self.store.get_room_version(room_id)
428	419
429	420	time_now = self._clock.time_msec()
430		defer.returnValue(
431		{"event": pdu.get_pdu_json(time_now), "room_version": room_version}
432		)
	421	return {"event": pdu.get_pdu_json(time_now), "room_version": room_version}
433	422
434	423	@defer.inlineCallbacks
435	424	def on_send_leave_request(self, origin, content, room_id):

444	433
445	434	logger.debug("on_send_leave_request: pdu sigs: %s", pdu.signatures)
446	435	yield self.handler.on_send_leave_request(origin, pdu)
447		defer.returnValue((200, {}))
	436	return (200, {})
448	437
449	438	@defer.inlineCallbacks
450	439	def on_event_auth(self, origin, room_id, event_id):

455	444	time_now = self._clock.time_msec()
456	445	auth_pdus = yield self.handler.on_event_auth(event_id)
457	446	res = {"auth_chain": [a.get_pdu_json(time_now) for a in auth_pdus]}
458		defer.returnValue((200, res))
	447	return (200, res)
459	448
460	449	@defer.inlineCallbacks
461	450	def on_query_auth_request(self, origin, content, room_id, event_id):

508	497	"missing": ret.get("missing", []),
509	498	}
510	499
511		defer.returnValue((200, send_content))
	500	return (200, send_content)
512	501
513	502	@log_function
514	503	def on_query_client_keys(self, origin, content):

547	536	),
548	537	)
549	538
550		defer.returnValue({"one_time_keys": json_result})
	539	return {"one_time_keys": json_result}
551	540
552	541	@defer.inlineCallbacks
553	542	@log_function

579	568
580	569	time_now = self._clock.time_msec()
581	570
582		defer.returnValue(
583		{"events": [ev.get_pdu_json(time_now) for ev in missing_events]}
584		)
	571	return {"events": [ev.get_pdu_json(time_now) for ev in missing_events]}
585	572
586	573	@log_function
587	574	def on_openid_userinfo(self, token):

675	662	ret = yield self.handler.exchange_third_party_invite(
676	663	sender_user_id, target_user_id, room_id, signed
677	664	)
678		defer.returnValue(ret)
	665	return ret
679	666
680	667	@defer.inlineCallbacks
681	668	def on_exchange_third_party_invite_request(self, origin, room_id, event_dict):
682	669	ret = yield self.handler.on_exchange_third_party_invite_request(
683	670	origin, room_id, event_dict
684	671	)
685		defer.returnValue(ret)
	672	return ret
686	673
687	674	@defer.inlineCallbacks
688	675	def check_server_matches_acl(self, server_name, room_id):

+2

-2

synapse/federation/sender/per_destination_queue.py less more

373	373
374	374	assert len(edus) <= limit, "get_devices_by_remote returned too many EDUs"
375	375
376		defer.returnValue((edus, now_stream_id))
	376	return (edus, now_stream_id)
377	377
378	378	@defer.inlineCallbacks
379	379	def _get_to_device_message_edus(self, limit):

392	392	for content in contents
393	393	]
394	394
395		defer.returnValue((edus, stream_id))
	395	return (edus, stream_id)

+1

-1

synapse/federation/sender/transaction_manager.py less more

132	132	)
133	133	success = False
134	134
135		defer.returnValue(success)
	135	return success

+39

-22

synapse/federation/transport/client.py less more

20	20	from twisted.internet import defer
21	21
22	22	from synapse.api.constants import Membership
23		from synapse.api.urls import FEDERATION_V1_PREFIX, FEDERATION_V2_PREFIX
	23	from synapse.api.urls import (
	24	FEDERATION_UNSTABLE_PREFIX,
	25	FEDERATION_V1_PREFIX,
	26	FEDERATION_V2_PREFIX,
	27	)
24	28	from synapse.logging.utils import log_function
25	29
26	30	logger = logging.getLogger(__name__)

182	186	try_trailing_slash_on_400=True,
183	187	)
184	188
185		defer.returnValue(response)
	189	return response
186	190
187	191	@defer.inlineCallbacks
188	192	@log_function

200	204	ignore_backoff=ignore_backoff,
201	205	)
202	206
203		defer.returnValue(content)
	207	return content
204	208
205	209	@defer.inlineCallbacks
206	210	@log_function

258	262	ignore_backoff=ignore_backoff,
259	263	)
260	264
261		defer.returnValue(content)
	265	return content
262	266
263	267	@defer.inlineCallbacks
264	268	@log_function

269	273	destination=destination, path=path, data=content
270	274	)
271	275
272		defer.returnValue(response)
	276	return response
273	277
274	278	@defer.inlineCallbacks
275	279	@log_function

287	291	ignore_backoff=True,
288	292	)
289	293
290		defer.returnValue(response)
	294	return response
291	295
292	296	@defer.inlineCallbacks
293	297	@log_function

298	302	destination=destination, path=path, data=content, ignore_backoff=True
299	303	)
300	304
301		defer.returnValue(response)
	305	return response
302	306
303	307	@defer.inlineCallbacks
304	308	@log_function

309	313	destination=destination, path=path, data=content, ignore_backoff=True
310	314	)
311	315
312		defer.returnValue(response)
	316	return response
313	317
314	318	@defer.inlineCallbacks
315	319	@log_function

338	342	destination=remote_server, path=path, args=args, ignore_backoff=True
339	343	)
340	344
341		defer.returnValue(response)
	345	return response
342	346
343	347	@defer.inlineCallbacks
344	348	@log_function

349	353	destination=destination, path=path, data=event_dict
350	354	)
351	355
352		defer.returnValue(response)
	356	return response
353	357
354	358	@defer.inlineCallbacks
355	359	@log_function

358	362
359	363	content = yield self.client.get_json(destination=destination, path=path)
360	364
361		defer.returnValue(content)
	365	return content
362	366
363	367	@defer.inlineCallbacks
364	368	@log_function

369	373	destination=destination, path=path, data=content
370	374	)
371	375
372		defer.returnValue(content)
	376	return content
373	377
374	378	@defer.inlineCallbacks
375	379	@log_function

401	405	content = yield self.client.post_json(
402	406	destination=destination, path=path, data=query_content, timeout=timeout
403	407	)
404		defer.returnValue(content)
	408	return content
405	409
406	410	@defer.inlineCallbacks
407	411	@log_function

425	429	content = yield self.client.get_json(
426	430	destination=destination, path=path, timeout=timeout
427	431	)
428		defer.returnValue(content)
	432	return content
429	433
430	434	@defer.inlineCallbacks
431	435	@log_function

459	463	content = yield self.client.post_json(
460	464	destination=destination, path=path, data=query_content, timeout=timeout
461	465	)
462		defer.returnValue(content)
	466	return content
463	467
464	468	@defer.inlineCallbacks
465	469	@log_function

487	491	timeout=timeout,
488	492	)
489	493
490		defer.returnValue(content)
	494	return content
491	495
492	496	@log_function
493	497	def get_group_profile(self, destination, group_id, requester_user_id):

934	938	destination=destination, path=path, data=content, ignore_backoff=True
935	939	)
936	940
	941	def get_room_complexity(self, destination, room_id):
	942	"""
	943	Args:
	944	destination (str): The remote server
	945	room_id (str): The room ID to ask about.
	946	"""
	947	path = _create_path(FEDERATION_UNSTABLE_PREFIX, "/rooms/%s/complexity", room_id)
	948
	949	return self.client.get_json(destination=destination, path=path)
	950
	951
	952	def _create_path(federation_prefix, path, *args):
	953	"""
	954	Ensures that all args are url encoded.
	955	"""
	956	return federation_prefix + path % tuple(urllib.parse.quote(arg, "") for arg in args)
	957
937	958
938	959	def _create_v1_path(path, *args):
939	960	"""Creates a path against V1 federation API from the path template and

950	971	Returns:
951	972	str
952	973	"""
953		return FEDERATION_V1_PREFIX + path % tuple(
954		urllib.parse.quote(arg, "") for arg in args
955		)
	974	return _create_path(FEDERATION_V1_PREFIX, path, *args)
956	975
957	976
958	977	def _create_v2_path(path, *args):

970	989	Returns:
971	990	str
972	991	"""
973		return FEDERATION_V2_PREFIX + path % tuple(
974		urllib.parse.quote(arg, "") for arg in args
975		)
	992	return _create_path(FEDERATION_V2_PREFIX, path, *args)

+8

-2

synapse/federation/transport/server.py less more

17	17	import functools
18	18	import logging
19	19	import re
	20
	21	from twisted.internet.defer import maybeDeferred
20	22
21	23	import synapse
22	24	import synapse.logging.opentracing as opentracing

744	746	else:
745	747	network_tuple = ThirdPartyInstanceID(None, None)
746	748
747		data = await self.handler.get_local_public_room_list(
748		limit, since_token, network_tuple=network_tuple, from_federation=True
	749	data = await maybeDeferred(
	750	self.handler.get_local_public_room_list,
	751	limit,
	752	since_token,
	753	network_tuple=network_tuple,
	754	from_federation=True,
749	755	)
750	756	return 200, data
751	757

+1

-1

synapse/groups/attestations.py less more

156	156
157	157	yield self.store.update_remote_attestion(group_id, user_id, attestation)
158	158
159		defer.returnValue({})
	159	return {}
160	160
161	161	def _start_renew_attestations(self):
162	162	return run_as_background_process("renew_attestations", self._renew_attestations)

+42

-50

synapse/groups/groups_server.py less more

84	84	if not is_admin:
85	85	raise SynapseError(403, "User is not admin in group")
86	86
87		defer.returnValue(group)
	87	return group
88	88
89	89	@defer.inlineCallbacks
90	90	def get_group_summary(self, group_id, requester_user_id):

150	150	group_id, requester_user_id
151	151	)
152	152
153		defer.returnValue(
154		{
155		"profile": profile,
156		"users_section": {
157		"users": users,
158		"roles": roles,
159		"total_user_count_estimate": 0, # TODO
160		},
161		"rooms_section": {
162		"rooms": rooms,
163		"categories": categories,
164		"total_room_count_estimate": 0, # TODO
165		},
166		"user": membership_info,
167		}
168		)
	153	return {
	154	"profile": profile,
	155	"users_section": {
	156	"users": users,
	157	"roles": roles,
	158	"total_user_count_estimate": 0, # TODO
	159	},
	160	"rooms_section": {
	161	"rooms": rooms,
	162	"categories": categories,
	163	"total_room_count_estimate": 0, # TODO
	164	},
	165	"user": membership_info,
	166	}
169	167
170	168	@defer.inlineCallbacks
171	169	def update_group_summary_room(

191	189	is_public=is_public,
192	190	)
193	191
194		defer.returnValue({})
	192	return {}
195	193
196	194	@defer.inlineCallbacks
197	195	def delete_group_summary_room(

207	205	group_id=group_id, room_id=room_id, category_id=category_id
208	206	)
209	207
210		defer.returnValue({})
	208	return {}
211	209
212	210	@defer.inlineCallbacks
213	211	def set_group_join_policy(self, group_id, requester_user_id, content):

227	225
228	226	yield self.store.set_group_join_policy(group_id, join_policy=join_policy)
229	227
230		defer.returnValue({})
	228	return {}
231	229
232	230	@defer.inlineCallbacks
233	231	def get_group_categories(self, group_id, requester_user_id):

236	234	yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
237	235
238	236	categories = yield self.store.get_group_categories(group_id=group_id)
239		defer.returnValue({"categories": categories})
	237	return {"categories": categories}
240	238
241	239	@defer.inlineCallbacks
242	240	def get_group_category(self, group_id, requester_user_id, category_id):

248	246	group_id=group_id, category_id=category_id
249	247	)
250	248
251		defer.returnValue(res)
	249	return res
252	250
253	251	@defer.inlineCallbacks
254	252	def update_group_category(self, group_id, requester_user_id, category_id, content):

268	266	profile=profile,
269	267	)
270	268
271		defer.returnValue({})
	269	return {}
272	270
273	271	@defer.inlineCallbacks
274	272	def delete_group_category(self, group_id, requester_user_id, category_id):

282	280	group_id=group_id, category_id=category_id
283	281	)
284	282
285		defer.returnValue({})
	283	return {}
286	284
287	285	@defer.inlineCallbacks
288	286	def get_group_roles(self, group_id, requester_user_id):

291	289	yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
292	290
293	291	roles = yield self.store.get_group_roles(group_id=group_id)
294		defer.returnValue({"roles": roles})
	292	return {"roles": roles}
295	293
296	294	@defer.inlineCallbacks
297	295	def get_group_role(self, group_id, requester_user_id, role_id):

300	298	yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
301	299
302	300	res = yield self.store.get_group_role(group_id=group_id, role_id=role_id)
303		defer.returnValue(res)
	301	return res
304	302
305	303	@defer.inlineCallbacks
306	304	def update_group_role(self, group_id, requester_user_id, role_id, content):

318	316	group_id=group_id, role_id=role_id, is_public=is_public, profile=profile
319	317	)
320	318
321		defer.returnValue({})
	319	return {}
322	320
323	321	@defer.inlineCallbacks
324	322	def delete_group_role(self, group_id, requester_user_id, role_id):

330	328
331	329	yield self.store.remove_group_role(group_id=group_id, role_id=role_id)
332	330
333		defer.returnValue({})
	331	return {}
334	332
335	333	@defer.inlineCallbacks
336	334	def update_group_summary_user(

354	352	is_public=is_public,
355	353	)
356	354
357		defer.returnValue({})
	355	return {}
358	356
359	357	@defer.inlineCallbacks
360	358	def delete_group_summary_user(self, group_id, requester_user_id, user_id, role_id):

368	366	group_id=group_id, user_id=user_id, role_id=role_id
369	367	)
370	368
371		defer.returnValue({})
	369	return {}
372	370
373	371	@defer.inlineCallbacks
374	372	def get_group_profile(self, group_id, requester_user_id):

390	388	group_description = {key: group[key] for key in cols}
391	389	group_description["is_openly_joinable"] = group["join_policy"] == "open"
392	390
393		defer.returnValue(group_description)
	391	return group_description
394	392	else:
395	393	raise SynapseError(404, "Unknown group")
396	394

460	458
461	459	# TODO: If admin add lists of users whose attestations have timed out
462	460
463		defer.returnValue(
464		{"chunk": chunk, "total_user_count_estimate": len(user_results)}
465		)
	461	return {"chunk": chunk, "total_user_count_estimate": len(user_results)}
466	462
467	463	@defer.inlineCallbacks
468	464	def get_invited_users_in_group(self, group_id, requester_user_id):

493	489	logger.warn("Error getting profile for %s: %s", user_id, e)
494	490	user_profiles.append(user_profile)
495	491
496		defer.returnValue(
497		{"chunk": user_profiles, "total_user_count_estimate": len(invited_users)}
498		)
	492	return {"chunk": user_profiles, "total_user_count_estimate": len(invited_users)}
499	493
500	494	@defer.inlineCallbacks
501	495	def get_rooms_in_group(self, group_id, requester_user_id):

532	526
533	527	chunk.sort(key=lambda e: -e["num_joined_members"])
534	528
535		defer.returnValue(
536		{"chunk": chunk, "total_room_count_estimate": len(room_results)}
537		)
	529	return {"chunk": chunk, "total_room_count_estimate": len(room_results)}
538	530
539	531	@defer.inlineCallbacks
540	532	def add_room_to_group(self, group_id, requester_user_id, room_id, content):

550	542
551	543	yield self.store.add_room_to_group(group_id, room_id, is_public=is_public)
552	544
553		defer.returnValue({})
	545	return {}
554	546
555	547	@defer.inlineCallbacks
556	548	def update_room_in_group(

573	565	else:
574	566	raise SynapseError(400, "Uknown config option")
575	567
576		defer.returnValue({})
	568	return {}
577	569
578	570	@defer.inlineCallbacks
579	571	def remove_room_from_group(self, group_id, requester_user_id, room_id):

585	577
586	578	yield self.store.remove_room_from_group(group_id, room_id)
587	579
588		defer.returnValue({})
	580	return {}
589	581
590	582	@defer.inlineCallbacks
591	583	def invite_to_group(self, group_id, user_id, requester_user_id, content):

643	635	)
644	636	elif res["state"] == "invite":
645	637	yield self.store.add_group_invite(group_id, user_id)
646		defer.returnValue({"state": "invite"})
	638	return {"state": "invite"}
647	639	elif res["state"] == "reject":
648		defer.returnValue({"state": "reject"})
	640	return {"state": "reject"}
649	641	else:
650	642	raise SynapseError(502, "Unknown state returned by HS")
651	643

678	670	remote_attestation=remote_attestation,
679	671	)
680	672
681		defer.returnValue(local_attestation)
	673	return local_attestation
682	674
683	675	@defer.inlineCallbacks
684	676	def accept_invite(self, group_id, requester_user_id, content):

698	690
699	691	local_attestation = yield self._add_user(group_id, requester_user_id, content)
700	692
701		defer.returnValue({"state": "join", "attestation": local_attestation})
	693	return {"state": "join", "attestation": local_attestation}
702	694
703	695	@defer.inlineCallbacks
704	696	def join_group(self, group_id, requester_user_id, content):

715	707
716	708	local_attestation = yield self._add_user(group_id, requester_user_id, content)
717	709
718		defer.returnValue({"state": "join", "attestation": local_attestation})
	710	return {"state": "join", "attestation": local_attestation}
719	711
720	712	@defer.inlineCallbacks
721	713	def knock(self, group_id, requester_user_id, content):

768	760	if not self.hs.is_mine_id(user_id):
769	761	yield self.store.maybe_delete_remote_profile_cache(user_id)
770	762
771		defer.returnValue({})
	763	return {}
772	764
773	765	@defer.inlineCallbacks
774	766	def create_group(self, group_id, requester_user_id, content):

844	836	avatar_url=user_profile.get("avatar_url"),
845	837	)
846	838
847		defer.returnValue({"group_id": group_id})
	839	return {"group_id": group_id}
848	840
849	841	@defer.inlineCallbacks
850	842	def delete_group(self, group_id, requester_user_id):

+2

-2

synapse/handlers/account_data.py less more

50	50	{"type": account_data_type, "content": content, "room_id": room_id}
51	51	)
52	52
53		defer.returnValue((results, current_stream_id))
	53	return (results, current_stream_id)
54	54
55	55	@defer.inlineCallbacks
56	56	def get_pagination_rows(self, user, config, key):
57		defer.returnValue(([], config.to_id))
	57	return ([], config.to_id)

+13

-5

synapse/handlers/account_validity.py less more

192	192	if threepid["medium"] == "email":
193	193	addresses.append(threepid["address"])
194	194
195		defer.returnValue(addresses)
	195	return addresses
196	196
197	197	@defer.inlineCallbacks
198	198	def _get_renewal_token(self, user_id):

213	213	try:
214	214	renewal_token = stringutils.random_string(32)
215	215	yield self.store.set_renewal_token_for_user(user_id, renewal_token)
216		defer.returnValue(renewal_token)
	216	return renewal_token
217	217	except StoreError:
218	218	attempts += 1
219	219	raise StoreError(500, "Couldn't generate a unique string as refresh string.")

225	225
226	226	Args:
227	227	renewal_token (str): Token sent with the renewal request.
228		"""
229		user_id = yield self.store.get_user_from_renewal_token(renewal_token)
	228	Returns:
	229	bool: Whether the provided token is valid.
	230	"""
	231	try:
	232	user_id = yield self.store.get_user_from_renewal_token(renewal_token)
	233	except StoreError:
	234	defer.returnValue(False)
	235
230	236	logger.debug("Renewing an account for user %s", user_id)
231	237	yield self.renew_account_for_user(user_id)
	238
	239	defer.returnValue(True)
232	240
233	241	@defer.inlineCallbacks
234	242	def renew_account_for_user(self, user_id, expiration_ts=None, email_sent=False):

253	261	user_id=user_id, expiration_ts=expiration_ts, email_sent=email_sent
254	262	)
255	263
256		defer.returnValue(expiration_ts)
	264	return expiration_ts

+1

-1

synapse/handlers/acme.py less more

99	99	logger.exception("Failed saving!")
100	100	raise
101	101
102		defer.returnValue(True)
	102	return True

+5

-5

synapse/handlers/admin.py less more

48	48	"devices": {"": {"sessions": [{"connections": connections}]}},
49	49	}
50	50
51		defer.returnValue(ret)
	51	return ret
52	52
53	53	@defer.inlineCallbacks
54	54	def get_users(self):

60	60	"""
61	61	ret = yield self.store.get_users()
62	62
63		defer.returnValue(ret)
	63	return ret
64	64
65	65	@defer.inlineCallbacks
66	66	def get_users_paginate(self, order, start, limit):

77	77	"""
78	78	ret = yield self.store.get_users_paginate(order, start, limit)
79	79
80		defer.returnValue(ret)
	80	return ret
81	81
82	82	@defer.inlineCallbacks
83	83	def search_users(self, term):

91	91	"""
92	92	ret = yield self.store.search_users(term)
93	93
94		defer.returnValue(ret)
	94	return ret
95	95
96	96	@defer.inlineCallbacks
97	97	def export_user_data(self, user_id, writer):

224	224	state = yield self.store.get_state_for_event(event_id)
225	225	writer.write_state(room_id, event_id, state)
226	226
227		defer.returnValue(writer.finished())
	227	return writer.finished()
228	228
229	229
230	230	class ExfiltrationWriter(object):

+11

-11

synapse/handlers/appservice.py less more

166	166	for user_service in user_query_services:
167	167	is_known_user = yield self.appservice_api.query_user(user_service, user_id)
168	168	if is_known_user:
169		defer.returnValue(True)
170		defer.returnValue(False)
	169	return True
	170	return False
171	171
172	172	@defer.inlineCallbacks
173	173	def query_room_alias_exists(self, room_alias):

191	191	if is_known_alias:
192	192	# the alias exists now so don't query more ASes.
193	193	result = yield self.store.get_association_from_room_alias(room_alias)
194		defer.returnValue(result)
	194	return result
195	195
196	196	@defer.inlineCallbacks
197	197	def query_3pe(self, kind, protocol, fields):

214	214	if success:
215	215	ret.extend(result)
216	216
217		defer.returnValue(ret)
	217	return ret
218	218
219	219	@defer.inlineCallbacks
220	220	def get_3pe_protocols(self, only_protocol=None):

253	253	for p in protocols.keys():
254	254	protocols[p] = _merge_instances(protocols[p])
255	255
256		defer.returnValue(protocols)
	256	return protocols
257	257
258	258	@defer.inlineCallbacks
259	259	def _get_services_for_event(self, event):

275	275	if (yield s.is_interested(event, self.store)):
276	276	interested_list.append(s)
277	277
278		defer.returnValue(interested_list)
	278	return interested_list
279	279
280	280	def _get_services_for_user(self, user_id):
281	281	services = self.store.get_app_services()

292	292	if not self.is_mine_id(user_id):
293	293	# we don't know if they are unknown or not since it isn't one of our
294	294	# users. We can't poke ASes.
295		defer.returnValue(False)
	295	return False
296	296	return
297	297
298	298	user_info = yield self.store.get_user_by_id(user_id)
299	299	if user_info:
300		defer.returnValue(False)
	300	return False
301	301	return
302	302
303	303	# user not found; could be the AS though, so check.
304	304	services = self.store.get_app_services()
305	305	service_list = [s for s in services if s.sender == user_id]
306		defer.returnValue(len(service_list) == 0)
	306	return len(service_list) == 0
307	307
308	308	@defer.inlineCallbacks
309	309	def _check_user_exists(self, user_id):
310	310	unknown_user = yield self._is_unknown_user(user_id)
311	311	if unknown_user:
312	312	exists = yield self.query_user_exists(user_id)
313		defer.returnValue(exists)
314		defer.returnValue(True)
	313	return exists
	314	return True

+24

-24

synapse/handlers/auth.py less more

154	154	if user_id != requester.user.to_string():
155	155	raise AuthError(403, "Invalid auth")
156	156
157		defer.returnValue(params)
	157	return params
158	158
159	159	@defer.inlineCallbacks
160	160	def check_auth(self, flows, clientdict, clientip, password_servlet=False):

279	279	creds,
280	280	list(clientdict),
281	281	)
282		defer.returnValue((creds, clientdict, session["id"]))
	282	return (creds, clientdict, session["id"])
283	283
284	284	ret = self._auth_dict_for_flows(flows, session)
285	285	ret["completed"] = list(creds)

306	306	if result:
307	307	creds[stagetype] = result
308	308	self._save_session(sess)
309		defer.returnValue(True)
310		defer.returnValue(False)
	309	return True
	310	return False
311	311
312	312	def get_session_id(self, clientdict):
313	313	"""

378	378	res = yield checker(
379	379	authdict, clientip=clientip, password_servlet=password_servlet
380	380	)
381		defer.returnValue(res)
	381	return res
382	382
383	383	# build a v1-login-style dict out of the authdict and fall back to the
384	384	# v1 code

388	388	raise SynapseError(400, "", Codes.MISSING_PARAM)
389	389
390	390	(canonical_id, callback) = yield self.validate_login(user_id, authdict)
391		defer.returnValue(canonical_id)
	391	return canonical_id
392	392
393	393	@defer.inlineCallbacks
394	394	def _check_recaptcha(self, authdict, clientip, **kwargs):

432	432	resp_body.get("hostname"),
433	433	)
434	434	if resp_body["success"]:
435		defer.returnValue(True)
	435	return True
436	436	raise LoginError(401, "", errcode=Codes.UNAUTHORIZED)
437	437
438	438	def _check_email_identity(self, authdict, **kwargs):

501	501
502	502	threepid["threepid_creds"] = authdict["threepid_creds"]
503	503
504		defer.returnValue(threepid)
	504	return threepid
505	505
506	506	def _get_params_recaptcha(self):
507	507	return {"public_key": self.hs.config.recaptcha_public_key}

605	605	yield self.store.delete_access_token(access_token)
606	606	raise StoreError(400, "Login raced against device deletion")
607	607
608		defer.returnValue(access_token)
	608	return access_token
609	609
610	610	@defer.inlineCallbacks
611	611	def check_user_exists(self, user_id):

628	628	self.ratelimit_login_per_account(user_id)
629	629	res = yield self._find_user_id_and_pwd_hash(user_id)
630	630	if res is not None:
631		defer.returnValue(res[0])
632		defer.returnValue(None)
	631	return res[0]
	632	return None
633	633
634	634	@defer.inlineCallbacks
635	635	def _find_user_id_and_pwd_hash(self, user_id):

660	660	user_id,
661	661	user_infos.keys(),
662	662	)
663		defer.returnValue(result)
	663	return result
664	664
665	665	def get_supported_login_types(self):
666	666	"""Get a the login types supported for the /login API

721	721	known_login_type = True
722	722	is_valid = yield provider.check_password(qualified_user_id, password)
723	723	if is_valid:
724		defer.returnValue((qualified_user_id, None))
	724	return (qualified_user_id, None)
725	725
726	726	if not hasattr(provider, "get_supported_login_types") or not hasattr(
727	727	provider, "check_auth"

755	755	if result:
756	756	if isinstance(result, str):
757	757	result = (result, None)
758		defer.returnValue(result)
	758	return result
759	759
760	760	if login_type == LoginType.PASSWORD and self.hs.config.password_localdb_enabled:
761	761	known_login_type = True

765	765	)
766	766
767	767	if canonical_user_id:
768		defer.returnValue((canonical_user_id, None))
	768	return (canonical_user_id, None)
769	769
770	770	if not known_login_type:
771	771	raise SynapseError(400, "Unknown login type %s" % login_type)

813	813	if isinstance(result, str):
814	814	# If it's a str, set callback function to None
815	815	result = (result, None)
816		defer.returnValue(result)
817
818		defer.returnValue((None, None))
	816	return result
	817
	818	return (None, None)
819	819
820	820	@defer.inlineCallbacks
821	821	def _check_local_password(self, user_id, password):

837	837	"""
838	838	lookupres = yield self._find_user_id_and_pwd_hash(user_id)
839	839	if not lookupres:
840		defer.returnValue(None)
	840	return None
841	841	(user_id, password_hash) = lookupres
842	842
843	843	# If the password hash is None, the account has likely been deactivated

849	849	result = yield self.validate_hash(password, password_hash)
850	850	if not result:
851	851	logger.warn("Failed password login for user %s", user_id)
852		defer.returnValue(None)
853		defer.returnValue(user_id)
	852	return None
	853	return user_id
854	854
855	855	@defer.inlineCallbacks
856	856	def validate_short_term_login_token_and_get_user_id(self, login_token):

859	859	try:
860	860	macaroon = pymacaroons.Macaroon.deserialize(login_token)
861	861	user_id = auth_api.get_user_id_from_macaroon(macaroon)
862		auth_api.validate_macaroon(macaroon, "login", True, user_id)
	862	auth_api.validate_macaroon(macaroon, "login", user_id)
863	863	except Exception:
864	864	raise AuthError(403, "Invalid token", errcode=Codes.FORBIDDEN)
865	865	self.ratelimit_login_per_account(user_id)
866	866	yield self.auth.check_auth_blocking(user_id)
867		defer.returnValue(user_id)
	867	return user_id
868	868
869	869	@defer.inlineCallbacks
870	870	def delete_access_token(self, access_token):

975	975	)
976	976
977	977	yield self.store.user_delete_threepid(user_id, medium, address)
978		defer.returnValue(result)
	978	return result
979	979
980	980	def _save_session(self, session):
981	981	# TODO: Persistent storage

+1

-1

synapse/handlers/deactivate_account.py less more

124	124	# Mark the user as deactivated.
125	125	yield self.store.set_user_deactivated_status(user_id, True)
126	126
127		defer.returnValue(identity_server_supports_unbinding)
	127	return identity_server_supports_unbinding
128	128
129	129	def _start_user_parting(self):
130	130	"""

+87

-85

synapse/handlers/device.py less more

63	63	for device in devices:
64	64	_update_device_from_client_ips(device, ips)
65	65
66		defer.returnValue(devices)
	66	return devices
67	67
68	68	@defer.inlineCallbacks
69	69	def get_device(self, user_id, device_id):

84	84	raise errors.NotFoundError
85	85	ips = yield self.store.get_last_client_ip_by_device(user_id, device_id)
86	86	_update_device_from_client_ips(device, ips)
87		defer.returnValue(device)
	87	return device
88	88
89	89	@measure_func("device.get_user_ids_changed")
90	90	@defer.inlineCallbacks

199	199	possibly_joined = []
200	200	possibly_left = []
201	201
202		defer.returnValue(
203		{"changed": list(possibly_joined), "left": list(possibly_left)}
204		)
	202	return {"changed": list(possibly_joined), "left": list(possibly_left)}
205	203
206	204
207	205	class DeviceHandler(DeviceWorkerHandler):

210	208
211	209	self.federation_sender = hs.get_federation_sender()
212	210
213		self._edu_updater = DeviceListEduUpdater(hs, self)
	211	self.device_list_updater = DeviceListUpdater(hs, self)
214	212
215	213	federation_registry = hs.get_federation_registry()
216	214
217	215	federation_registry.register_edu_handler(
218		"m.device_list_update", self._edu_updater.incoming_device_list_update
	216	"m.device_list_update", self.device_list_updater.incoming_device_list_update
219	217	)
220	218	federation_registry.register_query_handler(
221	219	"user_devices", self.on_federation_query_user_devices

249	247	)
250	248	if new_device:
251	249	yield self.notify_device_update(user_id, [device_id])
252		defer.returnValue(device_id)
	250	return device_id
253	251
254	252	# if the device id is not specified, we'll autogen one, but loop a few
255	253	# times in case of a clash.

263	261	)
264	262	if new_device:
265	263	yield self.notify_device_update(user_id, [device_id])
266		defer.returnValue(device_id)
	264	return device_id
267	265	attempts += 1
268	266
269	267	raise errors.StoreError(500, "Couldn't generate a device ID.")

410	408	@defer.inlineCallbacks
411	409	def on_federation_query_user_devices(self, user_id):
412	410	stream_id, devices = yield self.store.get_devices_with_keys_by_user(user_id)
413		defer.returnValue(
414		{"user_id": user_id, "stream_id": stream_id, "devices": devices}
415		)
	411	return {"user_id": user_id, "stream_id": stream_id, "devices": devices}
416	412
417	413	@defer.inlineCallbacks
418	414	def user_left_room(self, user, room_id):

429	425	device.update({"last_seen_ts": ip.get("last_seen"), "last_seen_ip": ip.get("ip")})
430	426
431	427
432		class DeviceListEduUpdater(object):
	428	class DeviceListUpdater(object):
433	429	"Handles incoming device list updates from federation and updates the DB"
434	430
435	431	def __init__(self, hs, device_handler):

522	518	logger.debug("Need to re-sync devices for %r? %r", user_id, resync)
523	519
524	520	if resync:
525		# Fetch all devices for the user.
526		origin = get_domain_from_id(user_id)
527		try:
528		result = yield self.federation.query_user_devices(origin, user_id)
529		except (
530		NotRetryingDestination,
531		RequestSendFailed,
532		HttpResponseException,
533		):
534		# TODO: Remember that we are now out of sync and try again
535		# later
536		logger.warn("Failed to handle device list update for %s", user_id)
537		# We abort on exceptions rather than accepting the update
538		# as otherwise synapse will 'forget' that its device list
539		# is out of date. If we bail then we will retry the resync
540		# next time we get a device list update for this user_id.
541		# This makes it more likely that the device lists will
542		# eventually become consistent.
543		return
544		except FederationDeniedError as e:
545		logger.info(e)
546		return
547		except Exception:
548		# TODO: Remember that we are now out of sync and try again
549		# later
550		logger.exception(
551		"Failed to handle device list update for %s", user_id
552		)
553		return
554
555		stream_id = result["stream_id"]
556		devices = result["devices"]
557
558		# If the remote server has more than ~1000 devices for this user
559		# we assume that something is going horribly wrong (e.g. a bot
560		# that logs in and creates a new device every time it tries to
561		# send a message). Maintaining lots of devices per user in the
562		# cache can cause serious performance issues as if this request
563		# takes more than 60s to complete, internal replication from the
564		# inbound federation worker to the synapse master may time out
565		# causing the inbound federation to fail and causing the remote
566		# server to retry, causing a DoS. So in this scenario we give
567		# up on storing the total list of devices and only handle the
568		# delta instead.
569		if len(devices) > 1000:
570		logger.warn(
571		"Ignoring device list snapshot for %s as it has >1K devs (%d)",
572		user_id,
573		len(devices),
574		)
575		devices = []
576
577		for device in devices:
578		logger.debug(
579		"Handling resync update %r/%r, ID: %r",
580		user_id,
581		device["device_id"],
582		stream_id,
583		)
584
585		yield self.store.update_remote_device_list_cache(
586		user_id, devices, stream_id
587		)
588		device_ids = [device["device_id"] for device in devices]
589		yield self.device_handler.notify_device_update(user_id, device_ids)
590
591		# We clobber the seen updates since we've re-synced from a given
592		# point.
593		self._seen_updates[user_id] = set([stream_id])
	521	yield self.user_device_resync(user_id)
594	522	else:
595	523	# Simply update the single device, since we know that is the only
596	524	# change (because of the single prev_id matching the current cache)

622	550	for _, stream_id, prev_ids, _ in updates:
623	551	if not prev_ids:
624	552	# We always do a resync if there are no previous IDs
625		defer.returnValue(True)
	553	return True
626	554
627	555	for prev_id in prev_ids:
628	556	if prev_id == extremity:

632	560	elif prev_id in stream_id_in_updates:
633	561	continue
634	562	else:
635		defer.returnValue(True)
	563	return True
636	564
637	565	stream_id_in_updates.add(stream_id)
638	566
639		defer.returnValue(False)
	567	return False
	568
	569	@defer.inlineCallbacks
	570	def user_device_resync(self, user_id):
	571	"""Fetches all devices for a user and updates the device cache with them.
	572
	573	Args:
	574	user_id (str): The user's id whose device_list will be updated.
	575	Returns:
	576	Deferred[dict]: a dict with device info as under the "devices" in the result of this
	577	request:
	578	https://matrix.org/docs/spec/server_server/r0.1.2#get-matrix-federation-v1-user-devices-userid
	579	"""
	580	# Fetch all devices for the user.
	581	origin = get_domain_from_id(user_id)
	582	try:
	583	result = yield self.federation.query_user_devices(origin, user_id)
	584	except (NotRetryingDestination, RequestSendFailed, HttpResponseException):
	585	# TODO: Remember that we are now out of sync and try again
	586	# later
	587	logger.warn("Failed to handle device list update for %s", user_id)
	588	# We abort on exceptions rather than accepting the update
	589	# as otherwise synapse will 'forget' that its device list
	590	# is out of date. If we bail then we will retry the resync
	591	# next time we get a device list update for this user_id.
	592	# This makes it more likely that the device lists will
	593	# eventually become consistent.
	594	return
	595	except FederationDeniedError as e:
	596	logger.info(e)
	597	return
	598	except Exception:
	599	# TODO: Remember that we are now out of sync and try again
	600	# later
	601	logger.exception("Failed to handle device list update for %s", user_id)
	602	return
	603	stream_id = result["stream_id"]
	604	devices = result["devices"]
	605
	606	# If the remote server has more than ~1000 devices for this user
	607	# we assume that something is going horribly wrong (e.g. a bot
	608	# that logs in and creates a new device every time it tries to
	609	# send a message). Maintaining lots of devices per user in the
	610	# cache can cause serious performance issues as if this request
	611	# takes more than 60s to complete, internal replication from the
	612	# inbound federation worker to the synapse master may time out
	613	# causing the inbound federation to fail and causing the remote
	614	# server to retry, causing a DoS. So in this scenario we give
	615	# up on storing the total list of devices and only handle the
	616	# delta instead.
	617	if len(devices) > 1000:
	618	logger.warn(
	619	"Ignoring device list snapshot for %s as it has >1K devs (%d)",
	620	user_id,
	621	len(devices),
	622	)
	623	devices = []
	624
	625	for device in devices:
	626	logger.debug(
	627	"Handling resync update %r/%r, ID: %r",
	628	user_id,
	629	device["device_id"],
	630	stream_id,
	631	)
	632
	633	yield self.store.update_remote_device_list_cache(user_id, devices, stream_id)
	634	device_ids = [device["device_id"] for device in devices]
	635	yield self.device_handler.notify_device_update(user_id, device_ids)
	636
	637	# We clobber the seen updates since we've re-synced from a given
	638	# point.
	639	self._seen_updates[user_id] = set([stream_id])
	640
	641	defer.returnValue(result)

+7

-8

synapse/handlers/directory.py less more

209	209	except AuthError as e:
210	210	logger.info("Failed to update alias events: %s", e)
211	211
212		defer.returnValue(room_id)
	212	return room_id
213	213
214	214	@defer.inlineCallbacks
215	215	def delete_appservice_association(self, service, room_alias):

228	228
229	229	room_id = yield self.store.delete_room_alias(room_alias)
230	230
231		defer.returnValue(room_id)
	231	return room_id
232	232
233	233	@defer.inlineCallbacks
234	234	def get_association(self, room_alias):

276	276	else:
277	277	servers = list(servers)
278	278
279		defer.returnValue({"room_id": room_id, "servers": servers})
280		return
	279	return {"room_id": room_id, "servers": servers}
281	280
282	281	@defer.inlineCallbacks
283	282	def on_directory_query(self, args):

288	287	result = yield self.get_association_from_room_alias(room_alias)
289	288
290	289	if result is not None:
291		defer.returnValue({"room_id": result.room_id, "servers": result.servers})
	290	return {"room_id": result.room_id, "servers": result.servers}
292	291	else:
293	292	raise SynapseError(
294	293	404,

341	340	# Query AS to see if it exists
342	341	as_handler = self.appservice_handler
343	342	result = yield as_handler.query_room_alias_exists(room_alias)
344		defer.returnValue(result)
	343	return result
345	344
346	345	def can_modify_alias(self, alias, user_id=None):
347	346	# Any application service "interested" in an alias they are regexing on

368	367	creator = yield self.store.get_room_alias_creator(alias.to_string())
369	368
370	369	if creator is not None and creator == user_id:
371		defer.returnValue(True)
	370	return True
372	371
373	372	is_admin = yield self.auth.is_server_admin(UserID.from_string(user_id))
374		defer.returnValue(is_admin)
	373	return is_admin
375	374
376	375	@defer.inlineCallbacks
377	376	def edit_published_room_list(self, requester, room_id, visibility):

+65

-10

synapse/handlers/e2e_keys.py less more

24	24	from synapse.api.errors import CodeMessageException, SynapseError
25	25	from synapse.logging.context import make_deferred_yieldable, run_in_background
26	26	from synapse.types import UserID, get_domain_from_id
	27	from synapse.util import unwrapFirstError
27	28	from synapse.util.retryutils import NotRetryingDestination
28	29
29	30	logger = logging.getLogger(__name__)

64	65	}
65	66	}
66	67	"""
	68
67	69	device_keys_query = query_body.get("device_keys", {})
68	70
69	71	# separate users by domain.

120	122	# Now fetch any devices that we don't have in our cache
121	123	@defer.inlineCallbacks
122	124	def do_remote_query(destination):
	125	"""This is called when we are querying the device list of a user on
	126	a remote homeserver and their device list is not in the device list
	127	cache. If we share a room with this user and we're not querying for
	128	specific user we will update the cache
	129	with their device list."""
	130
123	131	destination_query = remote_queries_not_in_cache[destination]
	132
	133	# We first consider whether we wish to update the device list cache with
	134	# the users device list. We want to track a user's devices when the
	135	# authenticated user shares a room with the queried user and the query
	136	# has not specified a particular device.
	137	# If we update the cache for the queried user we remove them from further
	138	# queries. We use the more efficient batched query_client_keys for all
	139	# remaining users
	140	user_ids_updated = []
	141	for (user_id, device_list) in destination_query.items():
	142	if user_id in user_ids_updated:
	143	continue
	144
	145	if device_list:
	146	continue
	147
	148	room_ids = yield self.store.get_rooms_for_user(user_id)
	149	if not room_ids:
	150	continue
	151
	152	# We've decided we're sharing a room with this user and should
	153	# probably be tracking their device lists. However, we haven't
	154	# done an initial sync on the device list so we do it now.
	155	try:
	156	user_devices = yield self.device_handler.device_list_updater.user_device_resync(
	157	user_id
	158	)
	159	user_devices = user_devices["devices"]
	160	for device in user_devices:
	161	results[user_id] = {device["device_id"]: device["keys"]}
	162	user_ids_updated.append(user_id)
	163	except Exception as e:
	164	failures[destination] = _exception_to_failure(e)
	165
	166	if len(destination_query) == len(user_ids_updated):
	167	# We've updated all the users in the query and we do not need to
	168	# make any further remote calls.
	169	return
	170
	171	# Remove all the users from the query which we have updated
	172	for user_id in user_ids_updated:
	173	destination_query.pop(user_id)
	174
124	175	try:
125	176	remote_result = yield self.federation.query_client_keys(
126	177	destination, {"device_keys": destination_query}, timeout=timeout

131	182	results[user_id] = keys
132	183
133	184	except Exception as e:
134		failures[destination] = _exception_to_failure(e)
	185	failure = _exception_to_failure(e)
	186	failures[destination] = failure
135	187
136	188	yield make_deferred_yieldable(
137	189	defer.gatherResults(

140	192	for destination in remote_queries_not_in_cache
141	193	],
142	194	consumeErrors=True,
143		)
144		)
145
146		defer.returnValue({"device_keys": results, "failures": failures})
	195	).addErrback(unwrapFirstError)
	196	)
	197
	198	return {"device_keys": results, "failures": failures}
147	199
148	200	@defer.inlineCallbacks
149	201	def query_local_devices(self, query):

188	240	r["unsigned"]["device_display_name"] = display_name
189	241	result_dict[user_id][device_id] = r
190	242
191		defer.returnValue(result_dict)
	243	return result_dict
192	244
193	245	@defer.inlineCallbacks
194	246	def on_federation_query_client_keys(self, query_body):

196	248	"""
197	249	device_keys_query = query_body.get("device_keys", {})
198	250	res = yield self.query_local_devices(device_keys_query)
199		defer.returnValue({"device_keys": res})
	251	return {"device_keys": res}
200	252
201	253	@defer.inlineCallbacks
202	254	def claim_one_time_keys(self, query, timeout):

233	285	for user_id, keys in remote_result["one_time_keys"].items():
234	286	if user_id in device_keys:
235	287	json_result[user_id] = keys
	288
236	289	except Exception as e:
237		failures[destination] = _exception_to_failure(e)
	290	failure = _exception_to_failure(e)
	291	failures[destination] = failure
238	292
239	293	yield make_deferred_yieldable(
240	294	defer.gatherResults(

258	312	),
259	313	)
260	314
261		defer.returnValue({"one_time_keys": json_result, "failures": failures})
	315	return {"one_time_keys": json_result, "failures": failures}
262	316
263	317	@defer.inlineCallbacks
264	318	def upload_keys_for_user(self, user_id, device_id, keys):
	319
265	320	time_now = self.clock.time_msec()
266	321
267	322	# TODO: Validate the JSON to make sure it has the right keys.

296	351
297	352	result = yield self.store.count_e2e_one_time_keys(user_id, device_id)
298	353
299		defer.returnValue({"one_time_key_counts": result})
	354	return {"one_time_key_counts": result}
300	355
301	356	@defer.inlineCallbacks
302	357	def _upload_one_time_keys_for_user(

+4

-4

synapse/handlers/e2e_room_keys.py less more

83	83	user_id, version, room_id, session_id
84	84	)
85	85
86		defer.returnValue(results)
	86	return results
87	87
88	88	@defer.inlineCallbacks
89	89	def delete_room_keys(self, user_id, version, room_id=None, session_id=None):

261	261	new_version = yield self.store.create_e2e_room_keys_version(
262	262	user_id, version_info
263	263	)
264		defer.returnValue(new_version)
	264	return new_version
265	265
266	266	@defer.inlineCallbacks
267	267	def get_version_info(self, user_id, version=None):

291	291	raise NotFoundError("Unknown backup version")
292	292	else:
293	293	raise
294		defer.returnValue(res)
	294	return res
295	295
296	296	@defer.inlineCallbacks
297	297	def delete_version(self, user_id, version=None):

349	349	user_id, version, version_info
350	350	)
351	351
352		defer.returnValue({})
	352	return {}

+3

-3

synapse/handlers/events.py less more

142	142	"end": tokens[1].to_string(),
143	143	}
144	144
145		defer.returnValue(chunk)
	145	return chunk
146	146
147	147
148	148	class EventHandler(BaseHandler):

165	165	event = yield self.store.get_event(event_id, check_room_id=room_id)
166	166
167	167	if not event:
168		defer.returnValue(None)
	168	return None
169	169	return
170	170
171	171	users = yield self.store.get_users_in_room(event.room_id)

178	178	if not filtered:
179	179	raise AuthError(403, "You don't have permission to access that event.")
180	180
181		defer.returnValue(event)
	181	return event

+68

-42

synapse/handlers/federation.py less more

209	209	event_id,
210	210	origin,
211	211	)
212		defer.returnValue(None)
	212	return None
213	213
214	214	state = None
215	215	auth_chain = []

675	675	events = [e for e in events if e.event_id not in seen_events]
676	676
677	677	if not events:
678		defer.returnValue([])
	678	return []
679	679
680	680	event_map = {e.event_id: e for e in events}
681	681

837	837	# TODO: We can probably do something more clever here.
838	838	yield self._handle_new_event(dest, event, backfilled=True)
839	839
840		defer.returnValue(events)
	840	return events
841	841
842	842	@defer.inlineCallbacks
843	843	def maybe_backfill(self, room_id, current_depth):

893	893	)
894	894
895	895	if not filtered_extremities:
896		defer.returnValue(False)
	896	return False
897	897
898	898	# Check if we reached a point where we should start backfilling.
899	899	sorted_extremeties_tuple = sorted(extremities.items(), key=lambda e: -int(e[1]))

964	964	# If this succeeded then we probably already have the
965	965	# appropriate stuff.
966	966	# TODO: We can probably do something more intelligent here.
967		defer.returnValue(True)
	967	return True
968	968	except SynapseError as e:
969	969	logger.info("Failed to backfill from %s because %s", dom, e)
970	970	continue

976	976	continue
977	977	except NotRetryingDestination as e:
978	978	logger.info(str(e))
	979	continue
	980	except RequestSendFailed as e:
	981	logger.info("Falied to get backfill from %s because %s", dom, e)
979	982	continue
980	983	except FederationDeniedError as e:
981	984	logger.info(e)

984	987	logger.exception("Failed to backfill from %s because %s", dom, e)
985	988	continue
986	989
987		defer.returnValue(False)
	990	return False
988	991
989	992	success = yield try_backfill(likely_domains)
990	993	if success:
991		defer.returnValue(True)
	994	return True
992	995
993	996	# Huh, well those domains didn't work out. Lets try some domains
994	997	# from the time.

1030	1033	[dom for dom, _ in likely_domains if dom not in tried_domains]
1031	1034	)
1032	1035	if success:
1033		defer.returnValue(True)
	1036	return True
1034	1037
1035	1038	tried_domains.update(dom for dom, _ in likely_domains)
1036	1039
1037		defer.returnValue(False)
	1040	return False
1038	1041
1039	1042	def _sanity_check_event(self, ev):
1040	1043	"""

1081	1084	pdu=event,
1082	1085	)
1083	1086
1084		defer.returnValue(pdu)
	1087	return pdu
1085	1088
1086	1089	@defer.inlineCallbacks
1087	1090	def on_event_auth(self, event_id):

1089	1092	auth = yield self.store.get_auth_chain(
1090	1093	[auth_id for auth_id in event.auth_event_ids()], include_given=True
1091	1094	)
1092		defer.returnValue([e for e in auth])
	1095	return [e for e in auth]
1093	1096
1094	1097	@log_function
1095	1098	@defer.inlineCallbacks

1176	1179
1177	1180	run_in_background(self._handle_queued_pdus, room_queue)
1178	1181
1179		defer.returnValue(True)
	1182	return True
1180	1183
1181	1184	@defer.inlineCallbacks
1182	1185	def _handle_queued_pdus(self, room_queue):

1263	1266	room_version, event, context, do_sig_check=False
1264	1267	)
1265	1268
1266		defer.returnValue(event)
	1269	return event
1267	1270
1268	1271	@defer.inlineCallbacks
1269	1272	@log_function

1324	1327
1325	1328	state = yield self.store.get_events(list(prev_state_ids.values()))
1326	1329
1327		defer.returnValue({"state": list(state.values()), "auth_chain": auth_chain})
	1330	return {"state": list(state.values()), "auth_chain": auth_chain}
1328	1331
1329	1332	@defer.inlineCallbacks
1330	1333	def on_invite_request(self, origin, pdu):

1380	1383	context = yield self.state_handler.compute_event_context(event)
1381	1384	yield self.persist_events_and_notify([(event, context)])
1382	1385
1383		defer.returnValue(event)
	1386	return event
1384	1387
1385	1388	@defer.inlineCallbacks
1386	1389	def do_remotely_reject_invite(self, target_hosts, room_id, user_id):

1405	1408	context = yield self.state_handler.compute_event_context(event)
1406	1409	yield self.persist_events_and_notify([(event, context)])
1407	1410
1408		defer.returnValue(event)
	1411	return event
1409	1412
1410	1413	@defer.inlineCallbacks
1411	1414	def _make_and_verify_event(

1423	1426	assert event.user_id == user_id
1424	1427	assert event.state_key == user_id
1425	1428	assert event.room_id == room_id
1426		defer.returnValue((origin, event, format_ver))
	1429	return (origin, event, format_ver)
1427	1430
1428	1431	@defer.inlineCallbacks
1429	1432	@log_function

1483	1486	logger.warn("Failed to create new leave %r because %s", event, e)
1484	1487	raise e
1485	1488
1486		defer.returnValue(event)
	1489	return event
1487	1490
1488	1491	@defer.inlineCallbacks
1489	1492	@log_function

1516	1519	event.signatures,
1517	1520	)
1518	1521
1519		defer.returnValue(None)
	1522	return None
1520	1523
1521	1524	@defer.inlineCallbacks
1522	1525	def get_state_for_pdu(self, room_id, event_id):

1544	1547	del results[(event.type, event.state_key)]
1545	1548
1546	1549	res = list(results.values())
1547		defer.returnValue(res)
	1550	return res
1548	1551	else:
1549		defer.returnValue([])
	1552	return []
1550	1553
1551	1554	@defer.inlineCallbacks
1552	1555	def get_state_ids_for_pdu(self, room_id, event_id):

1571	1574	else:
1572	1575	results.pop((event.type, event.state_key), None)
1573	1576
1574		defer.returnValue(list(results.values()))
	1577	return list(results.values())
1575	1578	else:
1576		defer.returnValue([])
	1579	return []
1577	1580
1578	1581	@defer.inlineCallbacks
1579	1582	@log_function

1586	1589
1587	1590	events = yield filter_events_for_server(self.store, origin, events)
1588	1591
1589		defer.returnValue(events)
	1592	return events
1590	1593
1591	1594	@defer.inlineCallbacks
1592	1595	@log_function

1616	1619
1617	1620	events = yield filter_events_for_server(self.store, origin, [event])
1618	1621	event = events[0]
1619		defer.returnValue(event)
	1622	return event
1620	1623	else:
1621		defer.returnValue(None)
	1624	return None
1622	1625
1623	1626	def get_min_depth_for_context(self, context):
1624	1627	return self.store.get_min_depth(context)

1650	1653	self.store.remove_push_actions_from_staging, event.event_id
1651	1654	)
1652	1655
1653		defer.returnValue(context)
	1656	return context
1654	1657
1655	1658	@defer.inlineCallbacks
1656	1659	def _handle_new_events(self, origin, event_infos, backfilled=False):

1673	1676	auth_events=ev_info.get("auth_events"),
1674	1677	backfilled=backfilled,
1675	1678	)
1676		defer.returnValue(res)
	1679	return res
1677	1680
1678	1681	contexts = yield make_deferred_yieldable(
1679	1682	defer.gatherResults(

1832	1835	if event.type == EventTypes.GuestAccess and not context.rejected:
1833	1836	yield self.maybe_kick_guest_users(event)
1834	1837
1835		defer.returnValue(context)
	1838	return context
1836	1839
1837	1840	@defer.inlineCallbacks
1838	1841	def _check_for_soft_fail(self, event, state, backfilled):

1951	1954
1952	1955	logger.debug("on_query_auth returning: %s", ret)
1953	1956
1954		defer.returnValue(ret)
	1957	return ret
1955	1958
1956	1959	@defer.inlineCallbacks
1957	1960	def on_get_missing_events(

1974	1977	self.store, origin, missing_events
1975	1978	)
1976	1979
1977		defer.returnValue(missing_events)
	1980	return missing_events
1978	1981
1979	1982	@defer.inlineCallbacks
1980	1983	@log_function

2450	2453
2451	2454	logger.debug("construct_auth_difference returning")
2452	2455
2453		defer.returnValue(
2454		{
2455		"auth_chain": local_auth,
2456		"rejects": {
2457		e.event_id: {"reason": reason_map[e.event_id], "proof": None}
2458		for e in base_remote_rejected
2459		},
2460		"missing": [e.event_id for e in missing_locals],
2461		}
2462		)
	2456	return {
	2457	"auth_chain": local_auth,
	2458	"rejects": {
	2459	e.event_id: {"reason": reason_map[e.event_id], "proof": None}
	2460	for e in base_remote_rejected
	2461	},
	2462	"missing": [e.event_id for e in missing_locals],
	2463	}
2463	2464
2464	2465	@defer.inlineCallbacks
2465	2466	@log_function

2607	2608	builder=builder
2608	2609	)
2609	2610	EventValidator().validate_new(event)
2610		defer.returnValue((event, context))
	2611	return (event, context)
2611	2612
2612	2613	@defer.inlineCallbacks
2613	2614	def _check_signature(self, event, context):

2797	2798	)
2798	2799	else:
2799	2800	return user_joined_room(self.distributor, user, room_id)
	2801
	2802	@defer.inlineCallbacks
	2803	def get_room_complexity(self, remote_room_hosts, room_id):
	2804	"""
	2805	Fetch the complexity of a remote room over federation.
	2806
	2807	Args:
	2808	remote_room_hosts (list[str]): The remote servers to ask.
	2809	room_id (str): The room ID to ask about.
	2810
	2811	Returns:
	2812	Deferred[dict] or Deferred[None]: Dict contains the complexity
	2813	metric versions, while None means we could not fetch the complexity.
	2814	"""
	2815
	2816	for host in remote_room_hosts:
	2817	res = yield self.federation_client.get_room_complexity(host, room_id)
	2818
	2819	# We got a result, return it.
	2820	if res:
	2821	defer.returnValue(res)
	2822
	2823	# We fell off the bottom, couldn't get the complexity from anyone. Oh
	2824	# well.
	2825	defer.returnValue(None)

+73

-48

synapse/handlers/groups_local.py less more

125	125	group_id, requester_user_id
126	126	)
127	127	else:
128		res = yield self.transport_client.get_group_summary(
129		get_domain_from_id(group_id), group_id, requester_user_id
130		)
	128	try:
	129	res = yield self.transport_client.get_group_summary(
	130	get_domain_from_id(group_id), group_id, requester_user_id
	131	)
	132	except RequestSendFailed:
	133	raise SynapseError(502, "Failed to contact group server")
131	134
132	135	group_server_name = get_domain_from_id(group_id)
133	136

161	164
162	165	res.setdefault("user", {})["is_publicised"] = is_publicised
163	166
164		defer.returnValue(res)
	167	return res
165	168
166	169	@defer.inlineCallbacks
167	170	def create_group(self, group_id, user_id, content):

182	185
183	186	content["user_profile"] = yield self.profile_handler.get_profile(user_id)
184	187
185		res = yield self.transport_client.create_group(
186		get_domain_from_id(group_id), group_id, user_id, content
187		)
	188	try:
	189	res = yield self.transport_client.create_group(
	190	get_domain_from_id(group_id), group_id, user_id, content
	191	)
	192	except RequestSendFailed:
	193	raise SynapseError(502, "Failed to contact group server")
188	194
189	195	remote_attestation = res["attestation"]
190	196	yield self.attestations.verify_attestation(

206	212	)
207	213	self.notifier.on_new_event("groups_key", token, users=[user_id])
208	214
209		defer.returnValue(res)
	215	return res
210	216
211	217	@defer.inlineCallbacks
212	218	def get_users_in_group(self, group_id, requester_user_id):

216	222	res = yield self.groups_server_handler.get_users_in_group(
217	223	group_id, requester_user_id
218	224	)
219		defer.returnValue(res)
	225	return res
220	226
221	227	group_server_name = get_domain_from_id(group_id)
222	228
223		res = yield self.transport_client.get_users_in_group(
224		get_domain_from_id(group_id), group_id, requester_user_id
225		)
	229	try:
	230	res = yield self.transport_client.get_users_in_group(
	231	get_domain_from_id(group_id), group_id, requester_user_id
	232	)
	233	except RequestSendFailed:
	234	raise SynapseError(502, "Failed to contact group server")
226	235
227	236	chunk = res["chunk"]
228	237	valid_entries = []

243	252
244	253	res["chunk"] = valid_entries
245	254
246		defer.returnValue(res)
	255	return res
247	256
248	257	@defer.inlineCallbacks
249	258	def join_group(self, group_id, user_id, content):

257	266	local_attestation = self.attestations.create_attestation(group_id, user_id)
258	267	content["attestation"] = local_attestation
259	268
260		res = yield self.transport_client.join_group(
261		get_domain_from_id(group_id), group_id, user_id, content
262		)
	269	try:
	270	res = yield self.transport_client.join_group(
	271	get_domain_from_id(group_id), group_id, user_id, content
	272	)
	273	except RequestSendFailed:
	274	raise SynapseError(502, "Failed to contact group server")
263	275
264	276	remote_attestation = res["attestation"]
265	277

284	296	)
285	297	self.notifier.on_new_event("groups_key", token, users=[user_id])
286	298
287		defer.returnValue({})
	299	return {}
288	300
289	301	@defer.inlineCallbacks
290	302	def accept_invite(self, group_id, user_id, content):

298	310	local_attestation = self.attestations.create_attestation(group_id, user_id)
299	311	content["attestation"] = local_attestation
300	312
301		res = yield self.transport_client.accept_group_invite(
302		get_domain_from_id(group_id), group_id, user_id, content
303		)
	313	try:
	314	res = yield self.transport_client.accept_group_invite(
	315	get_domain_from_id(group_id), group_id, user_id, content
	316	)
	317	except RequestSendFailed:
	318	raise SynapseError(502, "Failed to contact group server")
304	319
305	320	remote_attestation = res["attestation"]
306	321

325	340	)
326	341	self.notifier.on_new_event("groups_key", token, users=[user_id])
327	342
328		defer.returnValue({})
	343	return {}
329	344
330	345	@defer.inlineCallbacks
331	346	def invite(self, group_id, user_id, requester_user_id, config):

337	352	group_id, user_id, requester_user_id, content
338	353	)
339	354	else:
340		res = yield self.transport_client.invite_to_group(
341		get_domain_from_id(group_id),
342		group_id,
343		user_id,
344		requester_user_id,
345		content,
346		)
347
348		defer.returnValue(res)
	355	try:
	356	res = yield self.transport_client.invite_to_group(
	357	get_domain_from_id(group_id),
	358	group_id,
	359	user_id,
	360	requester_user_id,
	361	content,
	362	)
	363	except RequestSendFailed:
	364	raise SynapseError(502, "Failed to contact group server")
	365
	366	return res
349	367
350	368	@defer.inlineCallbacks
351	369	def on_invite(self, group_id, user_id, content):

376	394	logger.warn("No profile for user %s: %s", user_id, e)
377	395	user_profile = {}
378	396
379		defer.returnValue({"state": "invite", "user_profile": user_profile})
	397	return {"state": "invite", "user_profile": user_profile}
380	398
381	399	@defer.inlineCallbacks
382	400	def remove_user_from_group(self, group_id, user_id, requester_user_id, content):

397	415	)
398	416	else:
399	417	content["requester_user_id"] = requester_user_id
400		res = yield self.transport_client.remove_user_from_group(
401		get_domain_from_id(group_id),
402		group_id,
403		requester_user_id,
404		user_id,
405		content,
406		)
407
408		defer.returnValue(res)
	418	try:
	419	res = yield self.transport_client.remove_user_from_group(
	420	get_domain_from_id(group_id),
	421	group_id,
	422	requester_user_id,
	423	user_id,
	424	content,
	425	)
	426	except RequestSendFailed:
	427	raise SynapseError(502, "Failed to contact group server")
	428
	429	return res
409	430
410	431	@defer.inlineCallbacks
411	432	def user_removed_from_group(self, group_id, user_id, content):

420	441	@defer.inlineCallbacks
421	442	def get_joined_groups(self, user_id):
422	443	group_ids = yield self.store.get_joined_groups(user_id)
423		defer.returnValue({"groups": group_ids})
	444	return {"groups": group_ids}
424	445
425	446	@defer.inlineCallbacks
426	447	def get_publicised_groups_for_user(self, user_id):

432	453	for app_service in self.store.get_app_services():
433	454	result.extend(app_service.get_groups_for_user(user_id))
434	455
435		defer.returnValue({"groups": result})
436		else:
437		bulk_result = yield self.transport_client.bulk_get_publicised_groups(
438		get_domain_from_id(user_id), [user_id]
439		)
	456	return {"groups": result}
	457	else:
	458	try:
	459	bulk_result = yield self.transport_client.bulk_get_publicised_groups(
	460	get_domain_from_id(user_id), [user_id]
	461	)
	462	except RequestSendFailed:
	463	raise SynapseError(502, "Failed to contact group server")
	464
440	465	result = bulk_result.get("users", {}).get(user_id)
441	466	# TODO: Verify attestations
442		defer.returnValue({"groups": result})
	467	return {"groups": result}
443	468
444	469	@defer.inlineCallbacks
445	470	def bulk_get_publicised_groups(self, user_ids, proxy=True):

474	499	for app_service in self.store.get_app_services():
475	500	results[uid].extend(app_service.get_groups_for_user(uid))
476	501
477		defer.returnValue({"users": results})
	502	return {"users": results}

+9

-9

synapse/handlers/identity.py less more

81	81	"%s is not a trusted ID server: rejecting 3pid " + "credentials",
82	82	id_server,
83	83	)
84		defer.returnValue(None)
	84	return None
85	85
86	86	try:
87	87	data = yield self.http_client.get_json(

94	94	raise e.to_synapse_error()
95	95
96	96	if "medium" in data:
97		defer.returnValue(data)
98		defer.returnValue(None)
	97	return data
	98	return None
99	99
100	100	@defer.inlineCallbacks
101	101	def bind_threepid(self, creds, mxid):

132	132	)
133	133	except CodeMessageException as e:
134	134	data = json.loads(e.msg) # XXX WAT?
135		defer.returnValue(data)
	135	return data
136	136
137	137	@defer.inlineCallbacks
138	138	def try_unbind_threepid(self, mxid, threepid):

160	160
161	161	# We don't know where to unbind, so we don't have a choice but to return
162	162	if not id_servers:
163		defer.returnValue(False)
	163	return False
164	164
165	165	changed = True
166	166	for id_server in id_servers:

168	168	mxid, threepid, id_server
169	169	)
170	170
171		defer.returnValue(changed)
	171	return changed
172	172
173	173	@defer.inlineCallbacks
174	174	def try_unbind_threepid_with_id_server(self, mxid, threepid, id_server):

223	223	id_server=id_server,
224	224	)
225	225
226		defer.returnValue(changed)
	226	return changed
227	227
228	228	@defer.inlineCallbacks
229	229	def requestEmailToken(

249	249	% (id_server, "/_matrix/identity/api/v1/validate/email/requestToken"),
250	250	params,
251	251	)
252		defer.returnValue(data)
	252	return data
253	253	except HttpResponseException as e:
254	254	logger.info("Proxied requestToken failed: %r", e)
255	255	raise e.to_synapse_error()

277	277	% (id_server, "/_matrix/identity/api/v1/validate/msisdn/requestToken"),
278	278	params,
279	279	)
280		defer.returnValue(data)
	280	return data
281	281	except HttpResponseException as e:
282	282	logger.info("Proxied requestToken failed: %r", e)
283	283	raise e.to_synapse_error()

+25

-29

synapse/handlers/initial_sync.py less more

249	249	"end": now_token.to_string(),
250	250	}
251	251
252		defer.returnValue(ret)
	252	return ret
253	253
254	254	@defer.inlineCallbacks
255	255	def room_initial_sync(self, requester, room_id, pagin_config=None):

300	300
301	301	result["account_data"] = account_data_events
302	302
303		defer.returnValue(result)
	303	return result
304	304
305	305	@defer.inlineCallbacks
306	306	def _room_initial_sync_parted(

329	329
330	330	time_now = self.clock.time_msec()
331	331
332		defer.returnValue(
333		{
334		"membership": membership,
335		"room_id": room_id,
336		"messages": {
337		"chunk": (
338		yield self._event_serializer.serialize_events(
339		messages, time_now
340		)
341		),
342		"start": start_token.to_string(),
343		"end": end_token.to_string(),
344		},
345		"state": (
346		yield self._event_serializer.serialize_events(
347		room_state.values(), time_now
348		)
	332	return {
	333	"membership": membership,
	334	"room_id": room_id,
	335	"messages": {
	336	"chunk": (
	337	yield self._event_serializer.serialize_events(messages, time_now)
349	338	),
350		"presence": [],
351		"receipts": [],
352		}
353		)
	339	"start": start_token.to_string(),
	340	"end": end_token.to_string(),
	341	},
	342	"state": (
	343	yield self._event_serializer.serialize_events(
	344	room_state.values(), time_now
	345	)
	346	),
	347	"presence": [],
	348	"receipts": [],
	349	}
354	350
355	351	@defer.inlineCallbacks
356	352	def _room_initial_sync_joined(

383	379	def get_presence():
384	380	# If presence is disabled, return an empty list
385	381	if not self.hs.config.use_presence:
386		defer.returnValue([])
	382	return []
387	383
388	384	states = yield presence_handler.get_states(
389	385	[m.user_id for m in room_members], as_event=True
390	386	)
391	387
392		defer.returnValue(states)
	388	return states
393	389
394	390	@defer.inlineCallbacks
395	391	def get_receipts():

398	394	)
399	395	if not receipts:
400	396	receipts = []
401		defer.returnValue(receipts)
	397	return receipts
402	398
403	399	presence, receipts, (messages, token) = yield make_deferred_yieldable(
404	400	defer.gatherResults(

441	437	if not is_peeking:
442	438	ret["membership"] = membership
443	439
444		defer.returnValue(ret)
	440	return ret
445	441
446	442	@defer.inlineCallbacks
447	443	def _check_in_room_or_world_readable(self, room_id, user_id):

452	448	# * The user is a guest user, and has joined the room
453	449	# else it will throw.
454	450	member_event = yield self.auth.check_user_was_in_room(room_id, user_id)
455		defer.returnValue((member_event.membership, member_event.event_id))
	451	return (member_event.membership, member_event.event_id)
456	452	return
457	453	except AuthError:
458	454	visibility = yield self.state_handler.get_current_state(

462	458	visibility
463	459	and visibility.content["history_visibility"] == "world_readable"
464	460	):
465		defer.returnValue((Membership.JOIN, None))
	461	return (Membership.JOIN, None)
466	462	return
467	463	raise AuthError(
468	464	403, "Guest access not allowed", errcode=Codes.GUEST_ACCESS_FORBIDDEN

+25

-19

synapse/handlers/message.py less more

86	86	)
87	87	data = room_state[membership_event_id].get(key)
88	88
89		defer.returnValue(data)
	89	return data
90	90
91	91	@defer.inlineCallbacks
92	92	def get_state_events(

173	173	# events, as clients won't use them.
174	174	bundle_aggregations=False,
175	175	)
176		defer.returnValue(events)
	176	return events
177	177
178	178	@defer.inlineCallbacks
179	179	def get_joined_members(self, requester, room_id):

212	212	# Loop fell through, AS has no interested users in room
213	213	raise AuthError(403, "Appservice not in room")
214	214
215		defer.returnValue(
216		{
217		user_id: {
218		"avatar_url": profile.avatar_url,
219		"display_name": profile.display_name,
220		}
221		for user_id, profile in iteritems(users_with_profile)
	215	return {
	216	user_id: {
	217	"avatar_url": profile.avatar_url,
	218	"display_name": profile.display_name,
222	219	}
223		)
	220	for user_id, profile in iteritems(users_with_profile)
	221	}
224	222
225	223
226	224	class EventCreationHandler(object):

379	377	# tolerate them in event_auth.check().
380	378	prev_state_ids = yield context.get_prev_state_ids(self.store)
381	379	prev_event_id = prev_state_ids.get((EventTypes.Member, event.sender))
382		prev_event = yield self.store.get_event(prev_event_id, allow_none=True)
	380	prev_event = (
	381	yield self.store.get_event(prev_event_id, allow_none=True)
	382	if prev_event_id
	383	else None
	384	)
383	385	if not prev_event or prev_event.membership != Membership.JOIN:
384	386	logger.warning(
385	387	(

397	399
398	400	self.validator.validate_new(event)
399	401
400		defer.returnValue((event, context))
	402	return (event, context)
401	403
402	404	def _is_exempt_from_privacy_policy(self, builder, requester):
403	405	""""Determine if an event to be sent is exempt from having to consent

424	426	@defer.inlineCallbacks
425	427	def _is_server_notices_room(self, room_id):
426	428	if self.config.server_notices_mxid is None:
427		defer.returnValue(False)
	429	return False
428	430	user_ids = yield self.store.get_users_in_room(room_id)
429		defer.returnValue(self.config.server_notices_mxid in user_ids)
	431	return self.config.server_notices_mxid in user_ids
430	432
431	433	@defer.inlineCallbacks
432	434	def assert_accepted_privacy_policy(self, requester):

506	508	event.event_id,
507	509	prev_state.event_id,
508	510	)
509		defer.returnValue(prev_state)
	511	return prev_state
510	512
511	513	yield self.handle_new_client_event(
512	514	requester=requester, event=event, context=context, ratelimit=ratelimit

522	524	"""
523	525	prev_state_ids = yield context.get_prev_state_ids(self.store)
524	526	prev_event_id = prev_state_ids.get((event.type, event.state_key))
	527	if not prev_event_id:
	528	return
525	529	prev_event = yield self.store.get_event(prev_event_id, allow_none=True)
526	530	if not prev_event:
527	531	return

530	534	prev_content = encode_canonical_json(prev_event.content)
531	535	next_content = encode_canonical_json(event.content)
532	536	if prev_content == next_content:
533		defer.returnValue(prev_event)
	537	return prev_event
534	538	return
535	539
536	540	@defer.inlineCallbacks

562	566	yield self.send_nonmember_event(
563	567	requester, event, context, ratelimit=ratelimit
564	568	)
565		defer.returnValue(event)
	569	return event
566	570
567	571	@measure_func("create_new_client_event")
568	572	@defer.inlineCallbacks

625	629
626	630	logger.debug("Created event %s", event.event_id)
627	631
628		defer.returnValue((event, context))
	632	return (event, context)
629	633
630	634	@measure_func("handle_new_client_event")
631	635	@defer.inlineCallbacks

790	794	get_prev_content=False,
791	795	allow_rejected=False,
792	796	allow_none=True,
793		check_room_id=event.room_id,
794	797	)
795	798
796	799	# we can make some additional checks now if we have the original event.
797	800	if original_event:
798	801	if original_event.type == EventTypes.Create:
799	802	raise AuthError(403, "Redacting create events is not permitted")
	803
	804	if original_event.room_id != event.room_id:
	805	raise SynapseError(400, "Cannot redact event from a different room")
800	806
801	807	prev_state_ids = yield context.get_prev_state_ids(self.store)
802	808	auth_events_ids = yield self.auth.compute_auth_events(

+6

-8

synapse/handlers/pagination.py less more

241	241	)
242	242
243	243	if not events:
244		defer.returnValue(
245		{
246		"chunk": [],
247		"start": pagin_config.from_token.to_string(),
248		"end": next_token.to_string(),
249		}
250		)
	244	return {
	245	"chunk": [],
	246	"start": pagin_config.from_token.to_string(),
	247	"end": next_token.to_string(),
	248	}
251	249
252	250	state = None
253	251	if event_filter and event_filter.lazy_load_members() and len(events) > 0:

285	283	)
286	284	)
287	285
288		defer.returnValue(chunk)
	286	return chunk

+25

-33

synapse/handlers/presence.py less more

332	332	"""Checks the presence of users that have timed out and updates as
333	333	appropriate.
334	334	"""
335		logger.info("Handling presence timeouts")
	335	logger.debug("Handling presence timeouts")
336	336	now = self.clock.time_msec()
337	337
338	338	# Fetch the list of users that may have timed out. Things may have

460	460	if affect_presence:
461	461	run_in_background(_end)
462	462
463		defer.returnValue(_user_syncing())
	463	return _user_syncing()
464	464
465	465	def get_currently_syncing_users(self):
466	466	"""Get the set of user ids that are currently syncing on this HS.

555	555	"""Get the current presence state for a user.
556	556	"""
557	557	res = yield self.current_state_for_users([user_id])
558		defer.returnValue(res[user_id])
	558	return res[user_id]
559	559
560	560	@defer.inlineCallbacks
561	561	def current_state_for_users(self, user_ids):

584	584	states.update(new)
585	585	self.user_to_current_state.update(new)
586	586
587		defer.returnValue(states)
	587	return states
588	588
589	589	@defer.inlineCallbacks
590	590	def _persist_and_notify(self, states):

680	680	def get_state(self, target_user, as_event=False):
681	681	results = yield self.get_states([target_user.to_string()], as_event=as_event)
682	682
683		defer.returnValue(results[0])
	683	return results[0]
684	684
685	685	@defer.inlineCallbacks
686	686	def get_states(self, target_user_ids, as_event=False):

702	702
703	703	now = self.clock.time_msec()
704	704	if as_event:
705		defer.returnValue(
706		[
707		{
708		"type": "m.presence",
709		"content": format_user_presence_state(state, now),
710		}
711		for state in updates
712		]
713		)
	705	return [
	706	{
	707	"type": "m.presence",
	708	"content": format_user_presence_state(state, now),
	709	}
	710	for state in updates
	711	]
714	712	else:
715		defer.returnValue(updates)
	713	return updates
716	714
717	715	@defer.inlineCallbacks
718	716	def set_state(self, target_user, state, ignore_status_msg=False):

756	754	)
757	755
758	756	if observer_room_ids & observed_room_ids:
759		defer.returnValue(True)
760
761		defer.returnValue(False)
	757	return True
	758
	759	return False
762	760
763	761	@defer.inlineCallbacks
764	762	def get_all_presence_updates(self, last_id, current_id):

777	775	# TODO(markjh): replicate the unpersisted changes.
778	776	# This could use the in-memory stores for recent changes.
779	777	rows = yield self.store.get_all_presence_updates(last_id, current_id)
780		defer.returnValue(rows)
	778	return rows
781	779
782	780	def notify_new_event(self):
783	781	"""Called when new events have happened. Handles users and servers

1033	1031	#
1034	1032	# Hence this guard where we just return nothing so that the sync
1035	1033	# doesn't return. C.f. #5503.
1036		defer.returnValue(([], max_token))
	1034	return ([], max_token)
1037	1035
1038	1036	presence = self.get_presence_handler()
1039	1037	stream_change_cache = self.store.presence_stream_cache

1067	1065	updates = yield presence.current_state_for_users(user_ids_changed)
1068	1066
1069	1067	if include_offline:
1070		defer.returnValue((list(updates.values()), max_token))
	1068	return (list(updates.values()), max_token)
1071	1069	else:
1072		defer.returnValue(
1073		(
1074		[
1075		s
1076		for s in itervalues(updates)
1077		if s.state != PresenceState.OFFLINE
1078		],
1079		max_token,
1080		)
	1070	return (
	1071	[s for s in itervalues(updates) if s.state != PresenceState.OFFLINE],
	1072	max_token,
1081	1073	)
1082	1074
1083	1075	def get_current_key(self):

1106	1098	)
1107	1099	users_interested_in.update(user_ids)
1108	1100
1109		defer.returnValue(users_interested_in)
	1101	return users_interested_in
1110	1102
1111	1103
1112	1104	def handle_timeouts(user_states, is_mine_fn, syncing_user_ids, now):

1286	1278	# Always notify self
1287	1279	users_to_states.setdefault(state.user_id, []).append(state)
1288	1280
1289		defer.returnValue((room_ids_to_states, users_to_states))
	1281	return (room_ids_to_states, users_to_states)
1290	1282
1291	1283
1292	1284	@defer.inlineCallbacks

1320	1312	host = get_domain_from_id(user_id)
1321	1313	hosts_and_states.append(([host], states))
1322	1314
1323		defer.returnValue(hosts_and_states)
	1315	return hosts_and_states

+9

-9

synapse/handlers/profile.py less more

72	72	raise SynapseError(404, "Profile was not found", Codes.NOT_FOUND)
73	73	raise
74	74
75		defer.returnValue({"displayname": displayname, "avatar_url": avatar_url})
	75	return {"displayname": displayname, "avatar_url": avatar_url}
76	76	else:
77	77	try:
78	78	result = yield self.federation.make_query(

81	81	args={"user_id": user_id},
82	82	ignore_backoff=True,
83	83	)
84		defer.returnValue(result)
	84	return result
85	85	except RequestSendFailed as e:
86	86	raise_from(SynapseError(502, "Failed to fetch profile"), e)
87	87	except HttpResponseException as e:

107	107	raise SynapseError(404, "Profile was not found", Codes.NOT_FOUND)
108	108	raise
109	109
110		defer.returnValue({"displayname": displayname, "avatar_url": avatar_url})
	110	return {"displayname": displayname, "avatar_url": avatar_url}
111	111	else:
112	112	profile = yield self.store.get_from_remote_profile_cache(user_id)
113		defer.returnValue(profile or {})
	113	return profile or {}
114	114
115	115	@defer.inlineCallbacks
116	116	def get_displayname(self, target_user):

124	124	raise SynapseError(404, "Profile was not found", Codes.NOT_FOUND)
125	125	raise
126	126
127		defer.returnValue(displayname)
	127	return displayname
128	128	else:
129	129	try:
130	130	result = yield self.federation.make_query(

138	138	except HttpResponseException as e:
139	139	raise e.to_synapse_error()
140	140
141		defer.returnValue(result["displayname"])
	141	return result["displayname"]
142	142
143	143	@defer.inlineCallbacks
144	144	def set_displayname(self, target_user, requester, new_displayname, by_admin=False):

185	185	if e.code == 404:
186	186	raise SynapseError(404, "Profile was not found", Codes.NOT_FOUND)
187	187	raise
188		defer.returnValue(avatar_url)
	188	return avatar_url
189	189	else:
190	190	try:
191	191	result = yield self.federation.make_query(

199	199	except HttpResponseException as e:
200	200	raise e.to_synapse_error()
201	201
202		defer.returnValue(result["avatar_url"])
	202	return result["avatar_url"]
203	203
204	204	@defer.inlineCallbacks
205	205	def set_avatar_url(self, target_user, requester, new_avatar_url, by_admin=False):

250	250	raise SynapseError(404, "Profile was not found", Codes.NOT_FOUND)
251	251	raise
252	252
253		defer.returnValue(response)
	253	return response
254	254
255	255	@defer.inlineCallbacks
256	256	def _update_join_states(self, requester, target_user):

+7

-7

synapse/handlers/receipts.py less more

92	92
93	93	if min_batch_id is None:
94	94	# no new receipts
95		defer.returnValue(False)
	95	return False
96	96
97	97	affected_room_ids = list(set([r.room_id for r in receipts]))
98	98

102	102	min_batch_id, max_batch_id, affected_room_ids
103	103	)
104	104
105		defer.returnValue(True)
	105	return True
106	106
107	107	@defer.inlineCallbacks
108	108	def received_client_receipt(self, room_id, receipt_type, user_id, event_id):

132	132	)
133	133
134	134	if not result:
135		defer.returnValue([])
	135	return []
136	136
137		defer.returnValue(result)
	137	return result
138	138
139	139
140	140	class ReceiptEventSource(object):

147	147	to_key = yield self.get_current_key()
148	148
149	149	if from_key == to_key:
150		defer.returnValue(([], to_key))
	150	return ([], to_key)
151	151
152	152	events = yield self.store.get_linearized_receipts_for_rooms(
153	153	room_ids, from_key=from_key, to_key=to_key
154	154	)
155	155
156		defer.returnValue((events, to_key))
	156	return (events, to_key)
157	157
158	158	def get_current_key(self, direction="f"):
159	159	return self.store.get_max_receipt_stream_id()

172	172	room_ids, from_key=from_key, to_key=to_key
173	173	)
174	174
175		defer.returnValue((events, to_key))
	175	return (events, to_key)

+8

-8

synapse/handlers/register.py less more

264	264	# Bind email to new account
265	265	yield self._register_email_threepid(user_id, threepid_dict, None, False)
266	266
267		defer.returnValue(user_id)
	267	return user_id
268	268
269	269	@defer.inlineCallbacks
270	270	def _auto_join_rooms(self, user_id):

359	359	appservice_id=service_id,
360	360	create_profile_with_displayname=user.localpart,
361	361	)
362		defer.returnValue(user_id)
	362	return user_id
363	363
364	364	@defer.inlineCallbacks
365	365	def check_recaptcha(self, ip, private_key, challenge, response):

460	460
461	461	id = self._next_generated_user_id
462	462	self._next_generated_user_id += 1
463		defer.returnValue(str(id))
	463	return str(id)
464	464
465	465	@defer.inlineCallbacks
466	466	def _validate_captcha(self, ip_addr, private_key, challenge, response):

480	480	"error_url": "http://www.recaptcha.net/recaptcha/api/challenge?"
481	481	+ "error=%s" % lines[1],
482	482	}
483		defer.returnValue(json)
	483	return json
484	484
485	485	@defer.inlineCallbacks
486	486	def _submit_captcha(self, ip_addr, private_key, challenge, response):

496	496	"response": response,
497	497	},
498	498	)
499		defer.returnValue(data)
	499	return data
500	500
501	501	@defer.inlineCallbacks
502	502	def _join_user_to_room(self, requester, room_identifier):

621	621	initial_display_name=initial_display_name,
622	622	is_guest=is_guest,
623	623	)
624		defer.returnValue((r["device_id"], r["access_token"]))
	624	return (r["device_id"], r["access_token"])
625	625
626	626	valid_until_ms = None
627	627	if self.session_lifetime is not None:

644	644	user_id, device_id=device_id, valid_until_ms=valid_until_ms
645	645	)
646	646
647		defer.returnValue((device_id, access_token))
	647	return (device_id, access_token)
648	648
649	649	@defer.inlineCallbacks
650	650	def post_registration_actions(

797	797	if ex.errcode == Codes.MISSING_PARAM:
798	798	# This will only happen if the ID server returns a malformed response
799	799	logger.info("Can't add incomplete 3pid")
800		defer.returnValue(None)
	800	return None
801	801	raise
802	802
803	803	yield self._auth_handler.add_threepid(

+8

-8

synapse/handlers/room.py less more

127	127	old_room_id,
128	128	new_version, # args for _upgrade_room
129	129	)
130		defer.returnValue(ret)
	130	return ret
131	131
132	132	@defer.inlineCallbacks
133	133	def _upgrade_room(self, requester, old_room_id, new_version):

192	192	requester, old_room_id, new_room_id, old_room_state
193	193	)
194	194
195		defer.returnValue(new_room_id)
	195	return new_room_id
196	196
197	197	@defer.inlineCallbacks
198	198	def _update_upgraded_room_pls(

670	670	result["room_alias"] = room_alias.to_string()
671	671	yield directory_handler.send_room_alias_update_event(requester, room_id)
672	672
673		defer.returnValue(result)
	673	return result
674	674
675	675	@defer.inlineCallbacks
676	676	def _send_events_for_new_room(

795	795	room_creator_user_id=creator_id,
796	796	is_public=is_public,
797	797	)
798		defer.returnValue(gen_room_id)
	798	return gen_room_id
799	799	except StoreError:
800	800	attempts += 1
801	801	raise StoreError(500, "Couldn't generate a room ID.")

838	838	event_id, get_prev_content=True, allow_none=True
839	839	)
840	840	if not event:
841		defer.returnValue(None)
	841	return None
842	842	return
843	843
844	844	filtered = yield (filter_evts([event]))

889	889
890	890	results["end"] = token.copy_and_replace("room_key", results["end"]).to_string()
891	891
892		defer.returnValue(results)
	892	return results
893	893
894	894
895	895	class RoomEventSource(object):

940	940	else:
941	941	end_key = to_key
942	942
943		defer.returnValue((events, end_key))
	943	return (events, end_key)
944	944
945	945	def get_current_key(self):
946	946	return self.store.get_room_events_max_id()

958	958	limit=config.limit,
959	959	)
960	960
961		defer.returnValue((events, next_key))
	961	return (events, next_key)

+5

-5

synapse/handlers/room_list.py less more

324	324	current_limit=since_token.current_limit - 1,
325	325	).to_token()
326	326
327		defer.returnValue(results)
	327	return results
328	328
329	329	@defer.inlineCallbacks
330	330	def _append_room_entry_to_chunk(

419	419	if join_rules_event:
420	420	join_rule = join_rules_event.content.get("join_rule", None)
421	421	if not allow_private and join_rule and join_rule != JoinRules.PUBLIC:
422		defer.returnValue(None)
	422	return None
423	423
424	424	# Return whether this room is open to federation users or not
425	425	create_event = current_state.get((EventTypes.Create, ""))

468	468	if avatar_url:
469	469	result["avatar_url"] = avatar_url
470	470
471		defer.returnValue(result)
	471	return result
472	472
473	473	@defer.inlineCallbacks
474	474	def get_remote_public_room_list(

481	481	third_party_instance_id=None,
482	482	):
483	483	if not self.enable_room_list_search:
484		defer.returnValue({"chunk": [], "total_room_count_estimate": 0})
	484	return {"chunk": [], "total_room_count_estimate": 0}
485	485
486	486	if search_filter:
487	487	# We currently don't support searching across federation, so we have

506	506	]
507	507	}
508	508
509		defer.returnValue(res)
	509	return res
510	510
511	511	def _get_remote_list_cached(
512	512	self,

+102

-26

synapse/handlers/room_member.py less more

25	25
26	26	from twisted.internet import defer
27	27
28		import synapse.server
29		import synapse.types
	28	from synapse import types
30	29	from synapse.api.constants import EventTypes, Membership
31	30	from synapse.api.errors import AuthError, Codes, HttpResponseException, SynapseError
32	31	from synapse.types import RoomID, UserID

190	189	)
191	190	if duplicate is not None:
192	191	# Discard the new event since this membership change is a no-op.
193		defer.returnValue(duplicate)
	192	return duplicate
194	193
195	194	yield self.event_creation_handler.handle_new_client_event(
196	195	requester, event, context, extra_users=[target], ratelimit=ratelimit

232	231	if prev_member_event.membership == Membership.JOIN:
233	232	yield self._user_left_room(target, room_id)
234	233
235		defer.returnValue(event)
	234	return event
236	235
237	236	@defer.inlineCallbacks
238	237	def copy_room_tags_and_direct_to_room(self, old_room_id, new_room_id, user_id):

302	301	require_consent=require_consent,
303	302	)
304	303
305		defer.returnValue(result)
	304	return result
306	305
307	306	@defer.inlineCallbacks
308	307	def _update_membership(

422	421	same_membership = old_membership == effective_membership_state
423	422	same_sender = requester.user.to_string() == old_state.sender
424	423	if same_sender and same_membership and same_content:
425		defer.returnValue(old_state)
	424	return old_state
426	425
427	426	if old_membership in ["ban", "leave"] and action == "kick":
428	427	raise AuthError(403, "The target user is not in the room")

472	471	ret = yield self._remote_join(
473	472	requester, remote_room_hosts, room_id, target, content
474	473	)
475		defer.returnValue(ret)
	474	return ret
476	475
477	476	elif effective_membership_state == Membership.LEAVE:
478	477	if not is_host_in_room:

494	493	res = yield self._remote_reject_invite(
495	494	requester, remote_room_hosts, room_id, target
496	495	)
497		defer.returnValue(res)
	496	return res
498	497
499	498	res = yield self._local_membership_update(
500	499	requester=requester,

507	506	content=content,
508	507	require_consent=require_consent,
509	508	)
510		defer.returnValue(res)
	509	return res
511	510
512	511	@defer.inlineCallbacks
513	512	def send_membership_event(

542	541	), "Sender (%s) must be same as requester (%s)" % (sender, requester.user)
543	542	assert self.hs.is_mine(sender), "Sender must be our own: %s" % (sender,)
544	543	else:
545		requester = synapse.types.create_requester(target_user)
	544	requester = types.create_requester(target_user)
546	545
547	546	prev_event = yield self.event_creation_handler.deduplicate_state_event(
548	547	event, context

595	594	"""
596	595	guest_access_id = current_state_ids.get((EventTypes.GuestAccess, ""), None)
597	596	if not guest_access_id:
598		defer.returnValue(False)
	597	return False
599	598
600	599	guest_access = yield self.store.get_event(guest_access_id)
601	600
602		defer.returnValue(
	601	return (
603	602	guest_access
604	603	and guest_access.content
605	604	and "guest_access" in guest_access.content

634	633	servers.remove(room_alias.domain)
635	634	servers.insert(0, room_alias.domain)
636	635
637		defer.returnValue((RoomID.from_string(room_id), servers))
	636	return (RoomID.from_string(room_id), servers)
638	637
639	638	@defer.inlineCallbacks
640	639	def _get_inviter(self, user_id, room_id):

642	641	user_id=user_id, room_id=room_id
643	642	)
644	643	if invite:
645		defer.returnValue(UserID.from_string(invite.sender))
	644	return UserID.from_string(invite.sender)
646	645
647	646	@defer.inlineCallbacks
648	647	def do_3pid_invite(

707	706	if "signatures" not in data:
708	707	raise AuthError(401, "No signatures on 3pid binding")
709	708	yield self._verify_any_signature(data, id_server)
710		defer.returnValue(data["mxid"])
	709	return data["mxid"]
711	710
712	711	except IOError as e:
713	712	logger.warn("Error from identity server lookup: %s" % (e,))
714		defer.returnValue(None)
	713	return None
715	714
716	715	@defer.inlineCallbacks
717	716	def _verify_any_signature(self, data, server_hostname):

903	902	if not public_keys:
904	903	public_keys.append(fallback_public_key)
905	904	display_name = data["display_name"]
906		defer.returnValue((token, public_keys, fallback_public_key, display_name))
	905	return (token, public_keys, fallback_public_key, display_name)
907	906
908	907	@defer.inlineCallbacks
909	908	def _is_host_in_room(self, current_state_ids):

912	911	create_event_id = current_state_ids.get(("m.room.create", ""))
913	912	if len(current_state_ids) == 1 and create_event_id:
914	913	# We can only get here if we're in the process of creating the room
915		defer.returnValue(True)
	914	return True
916	915
917	916	for etype, state_key in current_state_ids:
918	917	if etype != EventTypes.Member or not self.hs.is_mine_id(state_key):

924	923	continue
925	924
926	925	if event.membership == Membership.JOIN:
927		defer.returnValue(True)
928
929		defer.returnValue(False)
	926	return True
	927
	928	return False
930	929
931	930	@defer.inlineCallbacks
932	931	def _is_server_notice_room(self, room_id):
933	932	if self._server_notices_mxid is None:
934		defer.returnValue(False)
	933	return False
935	934	user_ids = yield self.store.get_users_in_room(room_id)
936		defer.returnValue(self._server_notices_mxid in user_ids)
	935	return self._server_notices_mxid in user_ids
937	936
938	937
939	938	class RoomMemberMasterHandler(RoomMemberHandler):

945	944	self.distributor.declare("user_left_room")
946	945
947	946	@defer.inlineCallbacks
	947	def _is_remote_room_too_complex(self, room_id, remote_room_hosts):
	948	"""
	949	Check if complexity of a remote room is too great.
	950
	951	Args:
	952	room_id (str)
	953	remote_room_hosts (list[str])
	954
	955	Returns: bool of whether the complexity is too great, or None
	956	if unable to be fetched
	957	"""
	958	max_complexity = self.hs.config.limit_remote_rooms.complexity
	959	complexity = yield self.federation_handler.get_room_complexity(
	960	remote_room_hosts, room_id
	961	)
	962
	963	if complexity:
	964	if complexity["v1"] > max_complexity:
	965	return True
	966	return False
	967	return None
	968
	969	@defer.inlineCallbacks
	970	def _is_local_room_too_complex(self, room_id):
	971	"""
	972	Check if the complexity of a local room is too great.
	973
	974	Args:
	975	room_id (str)
	976
	977	Returns: bool
	978	"""
	979	max_complexity = self.hs.config.limit_remote_rooms.complexity
	980	complexity = yield self.store.get_room_complexity(room_id)
	981
	982	if complexity["v1"] > max_complexity:
	983	return True
	984
	985	return False
	986
	987	@defer.inlineCallbacks
948	988	def _remote_join(self, requester, remote_room_hosts, room_id, user, content):
949	989	"""Implements RoomMemberHandler._remote_join
950	990	"""
951	991	# filter ourselves out of remote_room_hosts: do_invite_join ignores it
952	992	# and if it is the only entry we'd like to return a 404 rather than a
953	993	# 500.
954
955	994	remote_room_hosts = [
956	995	host for host in remote_room_hosts if host != self.hs.hostname
957	996	]
958	997
959	998	if len(remote_room_hosts) == 0:
960	999	raise SynapseError(404, "No known servers")
	1000
	1001	if self.hs.config.limit_remote_rooms.enabled:
	1002	# Fetch the room complexity
	1003	too_complex = yield self._is_remote_room_too_complex(
	1004	room_id, remote_room_hosts
	1005	)
	1006	if too_complex is True:
	1007	raise SynapseError(
	1008	code=400,
	1009	msg=self.hs.config.limit_remote_rooms.complexity_error,
	1010	errcode=Codes.RESOURCE_LIMIT_EXCEEDED,
	1011	)
961	1012
962	1013	# We don't do an auth check if we are doing an invite
963	1014	# join dance for now, since we're kinda implicitly checking

968	1019	)
969	1020	yield self._user_joined_room(user, room_id)
970	1021
	1022	# Check the room we just joined wasn't too large, if we didn't fetch the
	1023	# complexity of it before.
	1024	if self.hs.config.limit_remote_rooms.enabled:
	1025	if too_complex is False:
	1026	# We checked, and we're under the limit.
	1027	return
	1028
	1029	# Check again, but with the local state events
	1030	too_complex = yield self._is_local_room_too_complex(room_id)
	1031
	1032	if too_complex is False:
	1033	# We're under the limit.
	1034	return
	1035
	1036	# The room is too large. Leave.
	1037	requester = types.create_requester(user, None, False, None)
	1038	yield self.update_membership(
	1039	requester=requester, target=user, room_id=room_id, action="leave"
	1040	)
	1041	raise SynapseError(
	1042	code=400,
	1043	msg=self.hs.config.limit_remote_rooms.complexity_error,
	1044	errcode=Codes.RESOURCE_LIMIT_EXCEEDED,
	1045	)
	1046
971	1047	@defer.inlineCallbacks
972	1048	def _remote_reject_invite(self, requester, remote_room_hosts, room_id, target):
973	1049	"""Implements RoomMemberHandler._remote_reject_invite

977	1053	ret = yield fed_handler.do_remotely_reject_invite(
978	1054	remote_room_hosts, room_id, target.to_string()
979	1055	)
980		defer.returnValue(ret)
	1056	return ret
981	1057	except Exception as e:
982	1058	# if we were unable to reject the exception, just mark
983	1059	# it as rejected on our end and plough ahead.

988	1064	logger.warn("Failed to reject invite: %s", e)
989	1065
990	1066	yield self.store.locally_reject_invite(target.to_string(), room_id)
991		defer.returnValue({})
	1067	return {}
992	1068
993	1069	def _user_joined_room(self, target, room_id):
994	1070	"""Implements RoomMemberHandler._user_joined_room

+1

-1

synapse/handlers/room_member_worker.py less more

52	52
53	53	yield self._user_joined_room(user, room_id)
54	54
55		defer.returnValue(ret)
	55	return ret
56	56
57	57	def _remote_reject_invite(self, requester, remote_room_hosts, room_id, target):
58	58	"""Implements RoomMemberHandler._remote_reject_invite

+6

-8

synapse/handlers/search.py less more

68	68	# Scan through the old room for further predecessors
69	69	room_id = predecessor["room_id"]
70	70
71		defer.returnValue(historical_room_ids)
	71	return historical_room_ids
72	72
73	73	@defer.inlineCallbacks
74	74	def search(self, user, content, batch=None):

185	185	room_ids.intersection_update({batch_group_key})
186	186
187	187	if not room_ids:
188		defer.returnValue(
189		{
190		"search_categories": {
191		"room_events": {"results": [], "count": 0, "highlights": []}
192		}
	188	return {
	189	"search_categories": {
	190	"room_events": {"results": [], "count": 0, "highlights": []}
193	191	}
194		)
	192	}
195	193
196	194	rank_map = {} # event_id -> rank of event
197	195	allowed_events = []

454	452	if global_next_batch:
455	453	rooms_cat_res["next_batch"] = global_next_batch
456	454
457		defer.returnValue({"search_categories": {"room_events": rooms_cat_res}})
	455	return {"search_categories": {"room_events": rooms_cat_res}}

+4

-4

synapse/handlers/state_deltas.py less more

47	47
48	48	if not event and not prev_event:
49	49	logger.debug("Neither event exists: %r %r", prev_event_id, event_id)
50		defer.returnValue(None)
	50	return None
51	51
52	52	prev_value = None
53	53	value = None

61	61	logger.debug("prev_value: %r -> value: %r", prev_value, value)
62	62
63	63	if value == public_value and prev_value != public_value:
64		defer.returnValue(True)
	64	return True
65	65	elif value != public_value and prev_value == public_value:
66		defer.returnValue(False)
	66	return False
67	67	else:
68		defer.returnValue(None)
	68	return None

+3

-3

synapse/handlers/stats.py less more

85	85
86	86	# If still None then the initial background update hasn't happened yet
87	87	if self.pos is None:
88		defer.returnValue(None)
	88	return None
89	89
90	90	# Loop round handling deltas until we're up to date
91	91	while True:

327	327	== "world_readable"
328	328	)
329	329	):
330		defer.returnValue(True)
	330	return True
331	331	else:
332		defer.returnValue(False)
	332	return False

+88

-67

synapse/handlers/sync.py less more

262	262	timeout,
263	263	full_state,
264	264	)
265		defer.returnValue(res)
	265	return res
266	266
267	267	@defer.inlineCallbacks
268	268	def _wait_for_sync_for_user(self, sync_config, since_token, timeout, full_state):

302	302	lazy_loaded = "false"
303	303	non_empty_sync_counter.labels(sync_type, lazy_loaded).inc()
304	304
305		defer.returnValue(result)
	305	return result
306	306
307	307	def current_sync_for_user(self, sync_config, since_token=None, full_state=False):
308	308	"""Get the sync for client needed to match what the server has now.

316	316	user_id = user.to_string()
317	317	rules = yield self.store.get_push_rules_for_user(user_id)
318	318	rules = format_push_rules_for_user(user, rules)
319		defer.returnValue(rules)
	319	return rules
320	320
321	321	@defer.inlineCallbacks
322	322	def ephemeral_by_room(self, sync_result_builder, now_token, since_token=None):

377	377	event_copy = {k: v for (k, v) in iteritems(event) if k != "room_id"}
378	378	ephemeral_by_room.setdefault(room_id, []).append(event_copy)
379	379
380		defer.returnValue((now_token, ephemeral_by_room))
	380	return (now_token, ephemeral_by_room)
381	381
382	382	@defer.inlineCallbacks
383	383	def _load_filtered_recents(

425	425	recents = []
426	426
427	427	if not limited or block_all_timeline:
428		defer.returnValue(
429		TimelineBatch(events=recents, prev_batch=now_token, limited=False)
	428	return TimelineBatch(
	429	events=recents, prev_batch=now_token, limited=False
430	430	)
431	431
432	432	filtering_factor = 2

489	489
490	490	prev_batch_token = now_token.copy_and_replace("room_key", room_key)
491	491
492		defer.returnValue(
493		TimelineBatch(
494		events=recents,
495		prev_batch=prev_batch_token,
496		limited=limited or newly_joined_room,
497		)
	492	return TimelineBatch(
	493	events=recents,
	494	prev_batch=prev_batch_token,
	495	limited=limited or newly_joined_room,
498	496	)
499	497
500	498	@defer.inlineCallbacks

516	514	if event.is_state():
517	515	state_ids = state_ids.copy()
518	516	state_ids[(event.type, event.state_key)] = event.event_id
519		defer.returnValue(state_ids)
	517	return state_ids
520	518
521	519	@defer.inlineCallbacks
522	520	def get_state_at(self, room_id, stream_position, state_filter=StateFilter.all()):

548	546	else:
549	547	# no events in this room - so presumably no state
550	548	state = {}
551		defer.returnValue(state)
	549	return state
552	550
553	551	@defer.inlineCallbacks
554	552	def compute_summary(self, room_id, sync_config, batch, state, now_token):

578	576	)
579	577
580	578	if not last_events:
581		defer.returnValue(None)
	579	return None
582	580	return
583	581
584	582	last_event = last_events[-1]

610	608	if name_id:
611	609	name = yield self.store.get_event(name_id, allow_none=True)
612	610	if name and name.content.get("name"):
613		defer.returnValue(summary)
	611	return summary
614	612
615	613	if canonical_alias_id:
616	614	canonical_alias = yield self.store.get_event(
617	615	canonical_alias_id, allow_none=True
618	616	)
619	617	if canonical_alias and canonical_alias.content.get("alias"):
620		defer.returnValue(summary)
	618	return summary
621	619
622	620	me = sync_config.user.to_string()
623	621

651	649	summary["m.heroes"] = sorted([user_id for user_id in gone_user_ids])[0:5]
652	650
653	651	if not sync_config.filter_collection.lazy_load_members():
654		defer.returnValue(summary)
	652	return summary
655	653
656	654	# ensure we send membership events for heroes if needed
657	655	cache_key = (sync_config.user.to_string(), sync_config.device_id)

685	683	cache.set(s.state_key, s.event_id)
686	684	state[(EventTypes.Member, s.state_key)] = s
687	685
688		defer.returnValue(summary)
	686	return summary
689	687
690	688	def get_lazy_loaded_members_cache(self, cache_key):
691	689	cache = self.lazy_loaded_members_cache.get(cache_key)

782	780	lazy_load_members=lazy_load_members,
783	781	)
784	782	elif batch.limited:
785		state_at_timeline_start = yield self.store.get_state_ids_for_event(
786		batch.events[0].event_id, state_filter=state_filter
787		)
	783	if batch:
	784	state_at_timeline_start = yield self.store.get_state_ids_for_event(
	785	batch.events[0].event_id, state_filter=state_filter
	786	)
	787	else:
	788	# Its not clear how we get here, but empirically we do
	789	# (#5407). Logging has been added elsewhere to try and
	790	# figure out where this state comes from.
	791	state_at_timeline_start = yield self.get_state_at(
	792	room_id, stream_position=now_token, state_filter=state_filter
	793	)
788	794
789	795	# for now, we disable LL for gappy syncs - see
790	796	# https://github.com/vector-im/riot-web/issues/7211#issuecomment-419976346

804	810	room_id, stream_position=since_token, state_filter=state_filter
805	811	)
806	812
807		current_state_ids = yield self.store.get_state_ids_for_event(
808		batch.events[-1].event_id, state_filter=state_filter
809		)
	813	if batch:
	814	current_state_ids = yield self.store.get_state_ids_for_event(
	815	batch.events[-1].event_id, state_filter=state_filter
	816	)
	817	else:
	818	# Its not clear how we get here, but empirically we do
	819	# (#5407). Logging has been added elsewhere to try and
	820	# figure out where this state comes from.
	821	current_state_ids = yield self.get_state_at(
	822	room_id, stream_position=now_token, state_filter=state_filter
	823	)
810	824
811	825	state_ids = _calculate_state(
812	826	timeline_contains=timeline_state,

870	884	if state_ids:
871	885	state = yield self.store.get_events(list(state_ids.values()))
872	886
873		defer.returnValue(
874		{
875		(e.type, e.state_key): e
876		for e in sync_config.filter_collection.filter_room_state(
877		list(state.values())
878		)
879		}
880		)
	887	return {
	888	(e.type, e.state_key): e
	889	for e in sync_config.filter_collection.filter_room_state(
	890	list(state.values())
	891	)
	892	}
881	893
882	894	@defer.inlineCallbacks
883	895	def unread_notifs_for_room_id(self, room_id, sync_config):

893	905	notifs = yield self.store.get_unread_event_push_actions_by_room_for_user(
894	906	room_id, sync_config.user.to_string(), last_unread_event_id
895	907	)
896		defer.returnValue(notifs)
	908	return notifs
897	909
898	910	# There is no new information in this period, so your notification
899	911	# count is whatever it was last time.
900		defer.returnValue(None)
	912	return None
901	913
902	914	@defer.inlineCallbacks
903	915	def generate_sync_result(self, sync_config, since_token=None, full_state=False):

988	1000	"Sync result for newly joined room %s: %r", room_id, joined_room
989	1001	)
990	1002
991		defer.returnValue(
992		SyncResult(
993		presence=sync_result_builder.presence,
994		account_data=sync_result_builder.account_data,
995		joined=sync_result_builder.joined,
996		invited=sync_result_builder.invited,
997		archived=sync_result_builder.archived,
998		to_device=sync_result_builder.to_device,
999		device_lists=device_lists,
1000		groups=sync_result_builder.groups,
1001		device_one_time_keys_count=one_time_key_counts,
1002		next_batch=sync_result_builder.now_token,
1003		)
	1003	return SyncResult(
	1004	presence=sync_result_builder.presence,
	1005	account_data=sync_result_builder.account_data,
	1006	joined=sync_result_builder.joined,
	1007	invited=sync_result_builder.invited,
	1008	archived=sync_result_builder.archived,
	1009	to_device=sync_result_builder.to_device,
	1010	device_lists=device_lists,
	1011	groups=sync_result_builder.groups,
	1012	device_one_time_keys_count=one_time_key_counts,
	1013	next_batch=sync_result_builder.now_token,
1004	1014	)
1005	1015
1006	1016	@measure_func("_generate_sync_entry_for_groups")

1123	1133	# Remove any users that we still share a room with.
1124	1134	newly_left_users -= users_who_share_room
1125	1135
1126		defer.returnValue(
1127		DeviceLists(changed=users_that_have_changed, left=newly_left_users)
1128		)
	1136	return DeviceLists(changed=users_that_have_changed, left=newly_left_users)
1129	1137	else:
1130		defer.returnValue(DeviceLists(changed=[], left=[]))
	1138	return DeviceLists(changed=[], left=[])
1131	1139
1132	1140	@defer.inlineCallbacks
1133	1141	def _generate_sync_entry_for_to_device(self, sync_result_builder):

1224	1232
1225	1233	sync_result_builder.account_data = account_data_for_user
1226	1234
1227		defer.returnValue(account_data_by_room)
	1235	return account_data_by_room
1228	1236
1229	1237	@defer.inlineCallbacks
1230	1238	def _generate_sync_entry_for_presence(

1324	1332	)
1325	1333	if not tags_by_room:
1326	1334	logger.debug("no-oping sync")
1327		defer.returnValue(([], [], [], []))
	1335	return ([], [], [], [])
1328	1336
1329	1337	ignored_account_data = yield self.store.get_global_account_data_by_type_for_user(
1330	1338	"m.ignored_user_list", user_id=user_id

1387	1395
1388	1396	newly_left_users -= newly_joined_or_invited_users
1389	1397
1390		defer.returnValue(
1391		(
1392		newly_joined_rooms,
1393		newly_joined_or_invited_users,
1394		newly_left_rooms,
1395		newly_left_users,
1396		)
	1398	return (
	1399	newly_joined_rooms,
	1400	newly_joined_or_invited_users,
	1401	newly_left_rooms,
	1402	newly_left_users,
1397	1403	)
1398	1404
1399	1405	@defer.inlineCallbacks

1413	1419	)
1414	1420
1415	1421	if rooms_changed:
1416		defer.returnValue(True)
	1422	return True
1417	1423
1418	1424	stream_id = RoomStreamToken.parse_stream_token(since_token.room_key).stream
1419	1425	for room_id in sync_result_builder.joined_room_ids:
1420	1426	if self.store.has_room_changed_since(room_id, stream_id):
1421		defer.returnValue(True)
1422		defer.returnValue(False)
	1427	return True
	1428	return False
1423	1429
1424	1430	@defer.inlineCallbacks
1425	1431	def _get_rooms_changed(self, sync_result_builder, ignored_users):

1636	1642	)
1637	1643	room_entries.append(entry)
1638	1644
1639		defer.returnValue((room_entries, invited, newly_joined_rooms, newly_left_rooms))
	1645	return (room_entries, invited, newly_joined_rooms, newly_left_rooms)
1640	1646
1641	1647	@defer.inlineCallbacks
1642	1648	def _get_all_rooms(self, sync_result_builder, ignored_users):

1710	1716	)
1711	1717	)
1712	1718
1713		defer.returnValue((room_entries, invited, []))
	1719	return (room_entries, invited, [])
1714	1720
1715	1721	@defer.inlineCallbacks
1716	1722	def _generate_room_entry(

1763	1769	recents=events,
1764	1770	newly_joined_room=newly_joined,
1765	1771	)
	1772
	1773	if not batch and batch.limited:
	1774	# This resulted in #5407, which is weird, so lets log! We do it
	1775	# here as we have the maximum amount of information.
	1776	user_id = sync_result_builder.sync_config.user.to_string()
	1777	logger.info(
	1778	"Issue #5407: Found limited batch with no events. user %s, room %s,"
	1779	" sync_config %s, newly_joined %s, events %s, batch %s.",
	1780	user_id,
	1781	room_id,
	1782	sync_config,
	1783	newly_joined,
	1784	events,
	1785	batch,
	1786	)
1766	1787
1767	1788	if newly_joined:
1768	1789	# debug for https://github.com/matrix-org/synapse/issues/4422

1911	1932	joined_room_ids.add(room_id)
1912	1933
1913	1934	joined_room_ids = frozenset(joined_room_ids)
1914		defer.returnValue(joined_room_ids)
	1935	return joined_room_ids
1915	1936
1916	1937
1917	1938	def _action_has_highlight(actions):

+3

-3

synapse/handlers/typing.py less more

82	82	self._room_typing = {}
83	83
84	84	def _handle_timeouts(self):
85		logger.info("Checking for typing timeouts")
	85	logger.debug("Checking for typing timeouts")
86	86
87	87	now = self.clock.time_msec()
88	88

139	139
140	140	if was_present:
141	141	# No point sending another notification
142		defer.returnValue(None)
	142	return None
143	143
144	144	self._push_update(member=member, typing=True)
145	145

172	172	def _stopped_typing(self, member):
173	173	if member.user_id not in self._room_typing.get(member.room_id, set()):
174	174	# No point
175		defer.returnValue(None)
	175	return None
176	176
177	177	self._member_typing_until.pop(member, None)
178	178	self._member_last_federation_poke.pop(member, None)

+1

-1

synapse/handlers/user_directory.py less more

132	132
133	133	# If still None then the initial background update hasn't happened yet
134	134	if self.pos is None:
135		defer.returnValue(None)
	135	return None
136	136
137	137	# Loop round handling deltas until we're up to date
138	138	while True:

+13

-15

synapse/http/client.py less more

293	293	logger.info(
294	294	"Received response to %s %s: %s", method, redact_uri(uri), response.code
295	295	)
296		defer.returnValue(response)
	296	return response
297	297	except Exception as e:
298	298	incoming_responses_counter.labels(method, "ERR").inc()
299	299	logger.info(

344	344	body = yield make_deferred_yieldable(readBody(response))
345	345
346	346	if 200 <= response.code < 300:
347		defer.returnValue(json.loads(body))
	347	return json.loads(body)
348	348	else:
349	349	raise HttpResponseException(response.code, response.phrase, body)
350	350

384	384	body = yield make_deferred_yieldable(readBody(response))
385	385
386	386	if 200 <= response.code < 300:
387		defer.returnValue(json.loads(body))
	387	return json.loads(body)
388	388	else:
389	389	raise HttpResponseException(response.code, response.phrase, body)
390	390

409	409	ValueError: if the response was not JSON
410	410	"""
411	411	body = yield self.get_raw(uri, args, headers=headers)
412		defer.returnValue(json.loads(body))
	412	return json.loads(body)
413	413
414	414	@defer.inlineCallbacks
415	415	def put_json(self, uri, json_body, args={}, headers=None):

452	452	body = yield make_deferred_yieldable(readBody(response))
453	453
454	454	if 200 <= response.code < 300:
455		defer.returnValue(json.loads(body))
	455	return json.loads(body)
456	456	else:
457	457	raise HttpResponseException(response.code, response.phrase, body)
458	458

487	487	body = yield make_deferred_yieldable(readBody(response))
488	488
489	489	if 200 <= response.code < 300:
490		defer.returnValue(body)
	490	return body
491	491	else:
492	492	raise HttpResponseException(response.code, response.phrase, body)
493	493

544	544	except Exception as e:
545	545	raise_from(SynapseError(502, ("Failed to download remote body: %s" % e)), e)
546	546
547		defer.returnValue(
548		(
549		length,
550		resp_headers,
551		response.request.absoluteURI.decode("ascii"),
552		response.code,
553		)
	547	return (
	548	length,
	549	resp_headers,
	550	response.request.absoluteURI.decode("ascii"),
	551	response.code,
554	552	)
555	553
556	554

626	624
627	625	try:
628	626	body = yield make_deferred_yieldable(readBody(response))
629		defer.returnValue(body)
	627	return body
630	628	except PartialDownloadError as e:
631	629	# twisted dislikes google's response, no content length.
632		defer.returnValue(e.response)
	630	return e.response
633	631
634	632
635	633	def encode_urlencode_args(args):

+33

-181

synapse/http/federation/matrix_federation_agent.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14		import json
	14
15	15	import logging
16		import random
17		import time
18	16
19	17	import attr
20	18	from netaddr import IPAddress

23	21	from twisted.internet import defer
24	22	from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS
25	23	from twisted.internet.interfaces import IStreamClientEndpoint
26		from twisted.web.client import URI, Agent, HTTPConnectionPool, RedirectAgent, readBody
27		from twisted.web.http import stringToDatetime
	24	from twisted.web.client import URI, Agent, HTTPConnectionPool
28	25	from twisted.web.http_headers import Headers
29	26	from twisted.web.iweb import IAgent
30	27
31	28	from synapse.http.federation.srv_resolver import SrvResolver, pick_server_from_list
	29	from synapse.http.federation.well_known_resolver import WellKnownResolver
32	30	from synapse.logging.context import make_deferred_yieldable
33	31	from synapse.util import Clock
34		from synapse.util.caches.ttlcache import TTLCache
35		from synapse.util.metrics import Measure
36
37		# period to cache .well-known results for by default
38		WELL_KNOWN_DEFAULT_CACHE_PERIOD = 24 * 3600
39
40		# jitter to add to the .well-known default cache ttl
41		WELL_KNOWN_DEFAULT_CACHE_PERIOD_JITTER = 10 * 60
42
43		# period to cache failure to fetch .well-known for
44		WELL_KNOWN_INVALID_CACHE_PERIOD = 1 * 3600
45
46		# cap for .well-known cache period
47		WELL_KNOWN_MAX_CACHE_PERIOD = 48 * 3600
48	32
49	33	logger = logging.getLogger(__name__)
50		well_known_cache = TTLCache("well-known")
51	34
52	35
53	36	@implementer(IAgent)

63	46	tls_client_options_factory (ClientTLSOptionsFactory\|None):
64	47	factory to use for fetching client tls options, or none to disable TLS.
65	48
66		_well_known_tls_policy (IPolicyForHTTPS\|None):
67		TLS policy to use for fetching .well-known files. None to use a default
68		(browser-like) implementation.
69
70	49	_srv_resolver (SrvResolver\|None):
71	50	SRVResolver impl to use for looking up SRV records. None to use a default
72	51	implementation.

80	59	self,
81	60	reactor,
82	61	tls_client_options_factory,
83		_well_known_tls_policy=None,
84	62	_srv_resolver=None,
85		_well_known_cache=well_known_cache,
	63	_well_known_cache=None,
86	64	):
87	65	self._reactor = reactor
88	66	self._clock = Clock(reactor)

97	75	self._pool.maxPersistentPerHost = 5
98	76	self._pool.cachedConnectionTimeout = 2 * 60
99	77
100		agent_args = {}
101		if _well_known_tls_policy is not None:
102		# the param is called 'contextFactory', but actually passing a
103		# contextfactory is deprecated, and it expects an IPolicyForHTTPS.
104		agent_args["contextFactory"] = _well_known_tls_policy
105		_well_known_agent = RedirectAgent(
106		Agent(self._reactor, pool=self._pool, **agent_args)
	78	self._well_known_resolver = WellKnownResolver(
	79	self._reactor,
	80	agent=Agent(
	81	self._reactor,
	82	pool=self._pool,
	83	contextFactory=tls_client_options_factory,
	84	),
	85	well_known_cache=_well_known_cache,
107	86	)
108		self._well_known_agent = _well_known_agent
109
110		# our cache of .well-known lookup results, mapping from server name
111		# to delegated name. The values can be:
112		# `bytes`: a valid server-name
113		# `None`: there is no (valid) .well-known here
114		self._well_known_cache = _well_known_cache
115	87
116	88	@defer.inlineCallbacks
117	89	def request(self, method, uri, headers=None, bodyProducer=None):

176	148	res = yield make_deferred_yieldable(
177	149	agent.request(method, uri, headers, bodyProducer)
178	150	)
179		defer.returnValue(res)
	151	return res
180	152
181	153	@defer.inlineCallbacks
182	154	def _route_matrix_uri(self, parsed_uri, lookup_well_known=True):

204	176	port = parsed_uri.port
205	177	if port == -1:
206	178	port = 8448
207		defer.returnValue(
208		_RoutingResult(
209		host_header=parsed_uri.netloc,
210		tls_server_name=parsed_uri.host,
211		target_host=parsed_uri.host,
212		target_port=port,
213		)
	179	return _RoutingResult(
	180	host_header=parsed_uri.netloc,
	181	tls_server_name=parsed_uri.host,
	182	target_host=parsed_uri.host,
	183	target_port=port,
214	184	)
215	185
216	186	if parsed_uri.port != -1:
217	187	# there is an explicit port
218		defer.returnValue(
219		_RoutingResult(
220		host_header=parsed_uri.netloc,
221		tls_server_name=parsed_uri.host,
222		target_host=parsed_uri.host,
223		target_port=parsed_uri.port,
224		)
	188	return _RoutingResult(
	189	host_header=parsed_uri.netloc,
	190	tls_server_name=parsed_uri.host,
	191	target_host=parsed_uri.host,
	192	target_port=parsed_uri.port,
225	193	)
226	194
227	195	if lookup_well_known:
228	196	# try a .well-known lookup
229		well_known_server = yield self._get_well_known(parsed_uri.host)
	197	well_known_result = yield self._well_known_resolver.get_well_known(
	198	parsed_uri.host
	199	)
	200	well_known_server = well_known_result.delegated_server
230	201
231	202	if well_known_server:
232	203	# if we found a .well-known, start again, but don't do another

258	229	)
259	230
260	231	res = yield self._route_matrix_uri(new_uri, lookup_well_known=False)
261		defer.returnValue(res)
	232	return res
262	233
263	234	# try a SRV lookup
264	235	service_name = b"_matrix._tcp.%s" % (parsed_uri.host,)

282	253	parsed_uri.host.decode("ascii"),
283	254	)
284	255
285		defer.returnValue(
286		_RoutingResult(
287		host_header=parsed_uri.netloc,
288		tls_server_name=parsed_uri.host,
289		target_host=target_host,
290		target_port=port,
291		)
	256	return _RoutingResult(
	257	host_header=parsed_uri.netloc,
	258	tls_server_name=parsed_uri.host,
	259	target_host=target_host,
	260	target_port=port,
292	261	)
293
294		@defer.inlineCallbacks
295		def _get_well_known(self, server_name):
296		"""Attempt to fetch and parse a .well-known file for the given server
297
298		Args:
299		server_name (bytes): name of the server, from the requested url
300
301		Returns:
302		Deferred[bytes\|None]: either the new server name, from the .well-known, or
303		None if there was no .well-known file.
304		"""
305		try:
306		result = self._well_known_cache[server_name]
307		except KeyError:
308		# TODO: should we linearise so that we don't end up doing two .well-known
309		# requests for the same server in parallel?
310		with Measure(self._clock, "get_well_known"):
311		result, cache_period = yield self._do_get_well_known(server_name)
312
313		if cache_period > 0:
314		self._well_known_cache.set(server_name, result, cache_period)
315
316		defer.returnValue(result)
317
318		@defer.inlineCallbacks
319		def _do_get_well_known(self, server_name):
320		"""Actually fetch and parse a .well-known, without checking the cache
321
322		Args:
323		server_name (bytes): name of the server, from the requested url
324
325		Returns:
326		Deferred[Tuple[bytes\|None\|object],int]:
327		result, cache period, where result is one of:
328		- the new server name from the .well-known (as a `bytes`)
329		- None if there was no .well-known file.
330		- INVALID_WELL_KNOWN if the .well-known was invalid
331		"""
332		uri = b"https://%s/.well-known/matrix/server" % (server_name,)
333		uri_str = uri.decode("ascii")
334		logger.info("Fetching %s", uri_str)
335		try:
336		response = yield make_deferred_yieldable(
337		self._well_known_agent.request(b"GET", uri)
338		)
339		body = yield make_deferred_yieldable(readBody(response))
340		if response.code != 200:
341		raise Exception("Non-200 response %s" % (response.code,))
342
343		parsed_body = json.loads(body.decode("utf-8"))
344		logger.info("Response from .well-known: %s", parsed_body)
345		if not isinstance(parsed_body, dict):
346		raise Exception("not a dict")
347		if "m.server" not in parsed_body:
348		raise Exception("Missing key 'm.server'")
349		except Exception as e:
350		logger.info("Error fetching %s: %s", uri_str, e)
351
352		# add some randomness to the TTL to avoid a stampeding herd every hour
353		# after startup
354		cache_period = WELL_KNOWN_INVALID_CACHE_PERIOD
355		cache_period += random.uniform(0, WELL_KNOWN_DEFAULT_CACHE_PERIOD_JITTER)
356		defer.returnValue((None, cache_period))
357
358		result = parsed_body["m.server"].encode("ascii")
359
360		cache_period = _cache_period_from_headers(
361		response.headers, time_now=self._reactor.seconds
362		)
363		if cache_period is None:
364		cache_period = WELL_KNOWN_DEFAULT_CACHE_PERIOD
365		# add some randomness to the TTL to avoid a stampeding herd every 24 hours
366		# after startup
367		cache_period += random.uniform(0, WELL_KNOWN_DEFAULT_CACHE_PERIOD_JITTER)
368		else:
369		cache_period = min(cache_period, WELL_KNOWN_MAX_CACHE_PERIOD)
370
371		defer.returnValue((result, cache_period))
372	262
373	263
374	264	@implementer(IStreamClientEndpoint)

385	275	return self.ep.connect(protocol_factory)
386	276
387	277
388		def _cache_period_from_headers(headers, time_now=time.time):
389		cache_controls = _parse_cache_control(headers)
390
391		if b"no-store" in cache_controls:
392		return 0
393
394		if b"max-age" in cache_controls:
395		try:
396		max_age = int(cache_controls[b"max-age"])
397		return max_age
398		except ValueError:
399		pass
400
401		expires = headers.getRawHeaders(b"expires")
402		if expires is not None:
403		try:
404		expires_date = stringToDatetime(expires[-1])
405		return expires_date - time_now()
406		except ValueError:
407		# RFC7234 says 'A cache recipient MUST interpret invalid date formats,
408		# especially the value "0", as representing a time in the past (i.e.,
409		# "already expired").
410		return 0
411
412		return None
413
414
415		def _parse_cache_control(headers):
416		cache_controls = {}
417		for hdr in headers.getRawHeaders(b"cache-control", []):
418		for directive in hdr.split(b","):
419		splits = [x.strip() for x in directive.split(b"=", 1)]
420		k = splits[0].lower()
421		v = splits[1] if len(splits) > 1 else None
422		cache_controls[k] = v
423		return cache_controls
424
425
426	278	@attr.s
427	279	class _RoutingResult(object):
428	280	"""The result returned by `_route_matrix_uri`.

+4

-4

synapse/http/federation/srv_resolver.py less more

119	119	if cache_entry:
120	120	if all(s.expires > now for s in cache_entry):
121	121	servers = list(cache_entry)
122		defer.returnValue(servers)
	122	return servers
123	123
124	124	try:
125	125	answers, _, _ = yield make_deferred_yieldable(

128	128	except DNSNameError:
129	129	# TODO: cache this. We can get the SOA out of the exception, and use
130	130	# the negative-TTL value.
131		defer.returnValue([])
	131	return []
132	132	except DomainError as e:
133	133	# We failed to resolve the name (other than a NameError)
134	134	# Try something in the cache, else rereaise

137	137	logger.warn(
138	138	"Failed to resolve %r, falling back to cache. %r", service_name, e
139	139	)
140		defer.returnValue(list(cache_entry))
	140	return list(cache_entry)
141	141	else:
142	142	raise e
143	143

168	168	)
169	169
170	170	self._cache[service_name] = list(servers)
171		defer.returnValue(servers)
	171	return servers

+187

-0

synapse/http/federation/well_known_resolver.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2019 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import json
	16	import logging
	17	import random
	18	import time
	19
	20	import attr
	21
	22	from twisted.internet import defer
	23	from twisted.web.client import RedirectAgent, readBody
	24	from twisted.web.http import stringToDatetime
	25
	26	from synapse.logging.context import make_deferred_yieldable
	27	from synapse.util import Clock
	28	from synapse.util.caches.ttlcache import TTLCache
	29	from synapse.util.metrics import Measure
	30
	31	# period to cache .well-known results for by default
	32	WELL_KNOWN_DEFAULT_CACHE_PERIOD = 24 * 3600
	33
	34	# jitter to add to the .well-known default cache ttl
	35	WELL_KNOWN_DEFAULT_CACHE_PERIOD_JITTER = 10 * 60
	36
	37	# period to cache failure to fetch .well-known for
	38	WELL_KNOWN_INVALID_CACHE_PERIOD = 1 * 3600
	39
	40	# cap for .well-known cache period
	41	WELL_KNOWN_MAX_CACHE_PERIOD = 48 * 3600
	42
	43	# lower bound for .well-known cache period
	44	WELL_KNOWN_MIN_CACHE_PERIOD = 5 * 60
	45
	46	logger = logging.getLogger(__name__)
	47
	48
	49	_well_known_cache = TTLCache("well-known")
	50
	51
	52	@attr.s(slots=True, frozen=True)
	53	class WellKnownLookupResult(object):
	54	delegated_server = attr.ib()
	55
	56
	57	class WellKnownResolver(object):
	58	"""Handles well-known lookups for matrix servers.
	59	"""
	60
	61	def __init__(self, reactor, agent, well_known_cache=None):
	62	self._reactor = reactor
	63	self._clock = Clock(reactor)
	64
	65	if well_known_cache is None:
	66	well_known_cache = _well_known_cache
	67
	68	self._well_known_cache = well_known_cache
	69	self._well_known_agent = RedirectAgent(agent)
	70
	71	@defer.inlineCallbacks
	72	def get_well_known(self, server_name):
	73	"""Attempt to fetch and parse a .well-known file for the given server
	74
	75	Args:
	76	server_name (bytes): name of the server, from the requested url
	77
	78	Returns:
	79	Deferred[WellKnownLookupResult]: The result of the lookup
	80	"""
	81	try:
	82	result = self._well_known_cache[server_name]
	83	except KeyError:
	84	# TODO: should we linearise so that we don't end up doing two .well-known
	85	# requests for the same server in parallel?
	86	with Measure(self._clock, "get_well_known"):
	87	result, cache_period = yield self._do_get_well_known(server_name)
	88
	89	if cache_period > 0:
	90	self._well_known_cache.set(server_name, result, cache_period)
	91
	92	return WellKnownLookupResult(delegated_server=result)
	93
	94	@defer.inlineCallbacks
	95	def _do_get_well_known(self, server_name):
	96	"""Actually fetch and parse a .well-known, without checking the cache
	97
	98	Args:
	99	server_name (bytes): name of the server, from the requested url
	100
	101	Returns:
	102	Deferred[Tuple[bytes\|None\|object],int]:
	103	result, cache period, where result is one of:
	104	- the new server name from the .well-known (as a `bytes`)
	105	- None if there was no .well-known file.
	106	- INVALID_WELL_KNOWN if the .well-known was invalid
	107	"""
	108	uri = b"https://%s/.well-known/matrix/server" % (server_name,)
	109	uri_str = uri.decode("ascii")
	110	logger.info("Fetching %s", uri_str)
	111	try:
	112	response = yield make_deferred_yieldable(
	113	self._well_known_agent.request(b"GET", uri)
	114	)
	115	body = yield make_deferred_yieldable(readBody(response))
	116	if response.code != 200:
	117	raise Exception("Non-200 response %s" % (response.code,))
	118
	119	parsed_body = json.loads(body.decode("utf-8"))
	120	logger.info("Response from .well-known: %s", parsed_body)
	121	if not isinstance(parsed_body, dict):
	122	raise Exception("not a dict")
	123	if "m.server" not in parsed_body:
	124	raise Exception("Missing key 'm.server'")
	125	except Exception as e:
	126	logger.info("Error fetching %s: %s", uri_str, e)
	127
	128	# add some randomness to the TTL to avoid a stampeding herd every hour
	129	# after startup
	130	cache_period = WELL_KNOWN_INVALID_CACHE_PERIOD
	131	cache_period += random.uniform(0, WELL_KNOWN_DEFAULT_CACHE_PERIOD_JITTER)
	132	return (None, cache_period)
	133
	134	result = parsed_body["m.server"].encode("ascii")
	135
	136	cache_period = _cache_period_from_headers(
	137	response.headers, time_now=self._reactor.seconds
	138	)
	139	if cache_period is None:
	140	cache_period = WELL_KNOWN_DEFAULT_CACHE_PERIOD
	141	# add some randomness to the TTL to avoid a stampeding herd every 24 hours
	142	# after startup
	143	cache_period += random.uniform(0, WELL_KNOWN_DEFAULT_CACHE_PERIOD_JITTER)
	144	else:
	145	cache_period = min(cache_period, WELL_KNOWN_MAX_CACHE_PERIOD)
	146	cache_period = max(cache_period, WELL_KNOWN_MIN_CACHE_PERIOD)
	147
	148	return (result, cache_period)
	149
	150
	151	def _cache_period_from_headers(headers, time_now=time.time):
	152	cache_controls = _parse_cache_control(headers)
	153
	154	if b"no-store" in cache_controls:
	155	return 0
	156
	157	if b"max-age" in cache_controls:
	158	try:
	159	max_age = int(cache_controls[b"max-age"])
	160	return max_age
	161	except ValueError:
	162	pass
	163
	164	expires = headers.getRawHeaders(b"expires")
	165	if expires is not None:
	166	try:
	167	expires_date = stringToDatetime(expires[-1])
	168	return expires_date - time_now()
	169	except ValueError:
	170	# RFC7234 says 'A cache recipient MUST interpret invalid date formats,
	171	# especially the value "0", as representing a time in the past (i.e.,
	172	# "already expired").
	173	return 0
	174
	175	return None
	176
	177
	178	def _parse_cache_control(headers):
	179	cache_controls = {}
	180	for hdr in headers.getRawHeaders(b"cache-control", []):
	181	for directive in hdr.split(b","):
	182	splits = [x.strip() for x in directive.split(b"=", 1)]
	183	k = splits[0].lower()
	184	v = splits[1] if len(splits) > 1 else None
	185	cache_controls[k] = v
	186	return cache_controls

+8

-8

synapse/http/matrixfederationclient.py less more

157	157	response.code,
158	158	response.phrase.decode("ascii", errors="replace"),
159	159	)
160		defer.returnValue(body)
	160	return body
161	161
162	162
163	163	class MatrixFederationHttpClient(object):

255	255
256	256	response = yield self._send_request(request, **send_request_args)
257	257
258		defer.returnValue(response)
	258	return response
259	259
260	260	@defer.inlineCallbacks
261	261	def _send_request(

519	519	_flatten_response_never_received(e),
520	520	)
521	521	raise
522		defer.returnValue(response)
	522	return response
523	523
524	524	def build_auth_headers(
525	525	self, destination, method, url_bytes, content=None, destination_is=None

643	643	self.reactor, self.default_timeout, request, response
644	644	)
645	645
646		defer.returnValue(body)
	646	return body
647	647
648	648	@defer.inlineCallbacks
649	649	def post_json(

712	712	body = yield _handle_json_response(
713	713	self.reactor, _sec_timeout, request, response
714	714	)
715		defer.returnValue(body)
	715	return body
716	716
717	717	@defer.inlineCallbacks
718	718	def get_json(

777	777	self.reactor, self.default_timeout, request, response
778	778	)
779	779
780		defer.returnValue(body)
	780	return body
781	781
782	782	@defer.inlineCallbacks
783	783	def delete_json(

835	835	body = yield _handle_json_response(
836	836	self.reactor, self.default_timeout, request, response
837	837	)
838		defer.returnValue(body)
	838	return body
839	839
840	840	@defer.inlineCallbacks
841	841	def get_file(

901	901	response.phrase.decode("ascii", errors="replace"),
902	902	length,
903	903	)
904		defer.returnValue((length, headers))
	904	return (length, headers)
905	905
906	906
907	907	class _ReadBodyToFileProtocol(protocol.Protocol):

+6

-1

synapse/http/servlet.py less more

165	165	value = args[name][0]
166	166
167	167	if encoding:
168		value = value.decode(encoding)
	168	try:
	169	value = value.decode(encoding)
	170	except ValueError:
	171	raise SynapseError(
	172	400, "Query parameter %r must be %s" % (name, encoding)
	173	)
169	174
170	175	if allowed_values is not None and value not in allowed_values:
171	176	message = "Query parameter %r must be one of [%s]" % (

+537

-280

synapse/logging/opentracing.py less more

10	10	# distributed under the License is distributed on an "AS IS" BASIS,
11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13		# limitations under the License.import opentracing
	13	# limitations under the License.
14	14
15	15
16	16	# NOTE

88	88	# We start
89	89	yield we_wait
90	90	# we finish
91		defer.returnValue(something_usual_and_useful)
	91	return something_usual_and_useful
92	92
93	93	Operation names can be explicitly set for functions by using
94	94	``trace_using_operation_name`` and

112	112	# We start
113	113	yield we_wait
114	114	# we finish
115		defer.returnValue(something_usual_and_useful)
	115	return something_usual_and_useful
116	116
117	117	Contexts and carriers
118	118	---------------------

149	149	"""
150	150
151	151	import contextlib
	152	import inspect
152	153	import logging
153	154	import re
154	155	from functools import wraps
	156
	157	from canonicaljson import json
155	158
156	159	from twisted.internet import defer
157	160

172	175	logger = logging.getLogger(__name__)
173	176
174	177
175		class _DumTagNames(object):
	178	# Block everything by default
	179	# A regex which matches the server_names to expose traces for.
	180	# None means 'block everything'.
	181	_homeserver_whitelist = None
	182
	183	# Util methods
	184
	185
	186	def only_if_tracing(func):
	187	"""Executes the function only if we're tracing. Otherwise return.
	188	Assumes the function wrapped may return None"""
	189
	190	@wraps(func)
	191	def _only_if_tracing_inner(args, *kwargs):
	192	if opentracing:
	193	return func(args, *kwargs)
	194	else:
	195	return
	196
	197	return _only_if_tracing_inner
	198
	199
	200	@contextlib.contextmanager
	201	def _noop_context_manager(args, *kwargs):
	202	"""Does exactly what it says on the tin"""
	203	yield
	204
	205
	206	# Setup
	207
	208
	209	def init_tracer(config):
	210	"""Set the whitelists and initialise the JaegerClient tracer
	211
	212	Args:
	213	config (HomeserverConfig): The config used by the homeserver
	214	"""
	215	global opentracing
	216	if not config.opentracer_enabled:
	217	# We don't have a tracer
	218	opentracing = None
	219	return
	220
	221	if not opentracing or not JaegerConfig:
	222	raise ConfigError(
	223	"The server has been configured to use opentracing but opentracing is not "
	224	"installed."
	225	)
	226
	227	# Include the worker name
	228	name = config.worker_name if config.worker_name else "master"
	229
	230	# Pull out the jaeger config if it was given. Otherwise set it to something sensible.
	231	# See https://github.com/jaegertracing/jaeger-client-python/blob/master/jaeger_client/config.py
	232
	233	set_homeserver_whitelist(config.opentracer_whitelist)
	234
	235	JaegerConfig(
	236	config=config.jaeger_config,
	237	service_name="{} {}".format(config.server_name, name),
	238	scope_manager=LogContextScopeManager(config),
	239	).initialize_tracer()
	240
	241	# Set up tags to be opentracing's tags
	242	global tags
	243	tags = opentracing.tags
	244
	245
	246	# Whitelisting
	247
	248
	249	@only_if_tracing
	250	def set_homeserver_whitelist(homeserver_whitelist):
	251	"""Sets the homeserver whitelist
	252
	253	Args:
	254	homeserver_whitelist (Iterable[str]): regex of whitelisted homeservers
	255	"""
	256	global _homeserver_whitelist
	257	if homeserver_whitelist:
	258	# Makes a single regex which accepts all passed in regexes in the list
	259	_homeserver_whitelist = re.compile(
	260	"({})".format(")\|(".join(homeserver_whitelist))
	261	)
	262
	263
	264	@only_if_tracing
	265	def whitelisted_homeserver(destination):
	266	"""Checks if a destination matches the whitelist
	267
	268	Args:
	269	destination (str)
	270	"""
	271	_homeserver_whitelist
	272	if _homeserver_whitelist:
	273	return _homeserver_whitelist.match(destination)
	274	return False
	275
	276
	277	# Start spans and scopes
	278
	279	# Could use kwargs but I want these to be explicit
	280	def start_active_span(
	281	operation_name,
	282	child_of=None,
	283	references=None,
	284	tags=None,
	285	start_time=None,
	286	ignore_active_span=False,
	287	finish_on_close=True,
	288	):
	289	"""Starts an active opentracing span. Note, the scope doesn't become active
	290	until it has been entered, however, the span starts from the time this
	291	message is called.
	292	Args:
	293	See opentracing.tracer
	294	Returns:
	295	scope (Scope) or noop_context_manager
	296	"""
	297
	298	if opentracing is None:
	299	return _noop_context_manager()
	300
	301	else:
	302	# We need to enter the scope here for the logcontext to become active
	303	return opentracing.tracer.start_active_span(
	304	operation_name,
	305	child_of=child_of,
	306	references=references,
	307	tags=tags,
	308	start_time=start_time,
	309	ignore_active_span=ignore_active_span,
	310	finish_on_close=finish_on_close,
	311	)
	312
	313
	314	def start_active_span_follows_from(operation_name, contexts):
	315	if opentracing is None:
	316	return _noop_context_manager()
	317	else:
	318	references = [opentracing.follows_from(context) for context in contexts]
	319	scope = start_active_span(operation_name, references=references)
	320	return scope
	321
	322
	323	def start_active_span_from_context(
	324	headers,
	325	operation_name,
	326	references=None,
	327	tags=None,
	328	start_time=None,
	329	ignore_active_span=False,
	330	finish_on_close=True,
	331	):
	332	"""
	333	Extracts a span context from Twisted Headers.
	334	args:
	335	headers (twisted.web.http_headers.Headers)
	336
	337	For the other args see opentracing.tracer
	338
	339	returns:
	340	span_context (opentracing.span.SpanContext)
	341	"""
	342	# Twisted encodes the values as lists whereas opentracing doesn't.
	343	# So, we take the first item in the list.
	344	# Also, twisted uses byte arrays while opentracing expects strings.
	345
	346	if opentracing is None:
	347	return _noop_context_manager()
	348
	349	header_dict = {k.decode(): v[0].decode() for k, v in headers.getAllRawHeaders()}
	350	context = opentracing.tracer.extract(opentracing.Format.HTTP_HEADERS, header_dict)
	351
	352	return opentracing.tracer.start_active_span(
	353	operation_name,
	354	child_of=context,
	355	references=references,
	356	tags=tags,
	357	start_time=start_time,
	358	ignore_active_span=ignore_active_span,
	359	finish_on_close=finish_on_close,
	360	)
	361
	362
	363	def start_active_span_from_edu(
	364	edu_content,
	365	operation_name,
	366	references=[],
	367	tags=None,
	368	start_time=None,
	369	ignore_active_span=False,
	370	finish_on_close=True,
	371	):
	372	"""
	373	Extracts a span context from an edu and uses it to start a new active span
	374
	375	Args:
	376	edu_content (dict): and edu_content with a `context` field whose value is
	377	canonical json for a dict which contains opentracing information.
	378
	379	For the other args see opentracing.tracer
	380	"""
	381
	382	if opentracing is None:
	383	return _noop_context_manager()
	384
	385	carrier = json.loads(edu_content.get("context", "{}")).get("opentracing", {})
	386	context = opentracing.tracer.extract(opentracing.Format.TEXT_MAP, carrier)
	387	_references = [
	388	opentracing.child_of(span_context_from_string(x))
	389	for x in carrier.get("references", [])
	390	]
	391
	392	# For some reason jaeger decided not to support the visualization of multiple parent
	393	# spans or explicitely show references. I include the span context as a tag here as
	394	# an aid to people debugging but it's really not an ideal solution.
	395
	396	references += _references
	397
	398	scope = opentracing.tracer.start_active_span(
	399	operation_name,
	400	child_of=context,
	401	references=references,
	402	tags=tags,
	403	start_time=start_time,
	404	ignore_active_span=ignore_active_span,
	405	finish_on_close=finish_on_close,
	406	)
	407
	408	scope.span.set_tag("references", carrier.get("references", []))
	409	return scope
	410
	411
	412	# Opentracing setters for tags, logs, etc
	413
	414
	415	@only_if_tracing
	416	def set_tag(key, value):
	417	"""Sets a tag on the active span"""
	418	opentracing.tracer.active_span.set_tag(key, value)
	419
	420
	421	@only_if_tracing
	422	def log_kv(key_values, timestamp=None):
	423	"""Log to the active span"""
	424	opentracing.tracer.active_span.log_kv(key_values, timestamp)
	425
	426
	427	@only_if_tracing
	428	def set_operation_name(operation_name):
	429	"""Sets the operation name of the active span"""
	430	opentracing.tracer.active_span.set_operation_name(operation_name)
	431
	432
	433	# Injection and extraction
	434
	435
	436	@only_if_tracing
	437	def inject_active_span_twisted_headers(headers, destination):
	438	"""
	439	Injects a span context into twisted headers in-place
	440
	441	Args:
	442	headers (twisted.web.http_headers.Headers)
	443	span (opentracing.Span)
	444
	445	Returns:
	446	In-place modification of headers
	447
	448	Note:
	449	The headers set by the tracer are custom to the tracer implementation which
	450	should be unique enough that they don't interfere with any headers set by
	451	synapse or twisted. If we're still using jaeger these headers would be those
	452	here:
	453	https://github.com/jaegertracing/jaeger-client-python/blob/master/jaeger_client/constants.py
	454	"""
	455
	456	if not whitelisted_homeserver(destination):
	457	return
	458
	459	span = opentracing.tracer.active_span
	460	carrier = {}
	461	opentracing.tracer.inject(span, opentracing.Format.HTTP_HEADERS, carrier)
	462
	463	for key, value in carrier.items():
	464	headers.addRawHeaders(key, value)
	465
	466
	467	@only_if_tracing
	468	def inject_active_span_byte_dict(headers, destination):
	469	"""
	470	Injects a span context into a dict where the headers are encoded as byte
	471	strings
	472
	473	Args:
	474	headers (dict)
	475	span (opentracing.Span)
	476
	477	Returns:
	478	In-place modification of headers
	479
	480	Note:
	481	The headers set by the tracer are custom to the tracer implementation which
	482	should be unique enough that they don't interfere with any headers set by
	483	synapse or twisted. If we're still using jaeger these headers would be those
	484	here:
	485	https://github.com/jaegertracing/jaeger-client-python/blob/master/jaeger_client/constants.py
	486	"""
	487	if not whitelisted_homeserver(destination):
	488	return
	489
	490	span = opentracing.tracer.active_span
	491
	492	carrier = {}
	493	opentracing.tracer.inject(span, opentracing.Format.HTTP_HEADERS, carrier)
	494
	495	for key, value in carrier.items():
	496	headers[key.encode()] = [value.encode()]
	497
	498
	499	@only_if_tracing
	500	def inject_active_span_text_map(carrier, destination=None):
	501	"""
	502	Injects a span context into a dict
	503
	504	Args:
	505	carrier (dict)
	506	destination (str): the name of the remote server. The span context
	507	will only be injected if the destination matches the homeserver_whitelist
	508	or destination is None.
	509
	510	Returns:
	511	In-place modification of carrier
	512
	513	Note:
	514	The headers set by the tracer are custom to the tracer implementation which
	515	should be unique enough that they don't interfere with any headers set by
	516	synapse or twisted. If we're still using jaeger these headers would be those
	517	here:
	518	https://github.com/jaegertracing/jaeger-client-python/blob/master/jaeger_client/constants.py
	519	"""
	520
	521	if destination and not whitelisted_homeserver(destination):
	522	return
	523
	524	opentracing.tracer.inject(
	525	opentracing.tracer.active_span, opentracing.Format.TEXT_MAP, carrier
	526	)
	527
	528
	529	def active_span_context_as_string():
	530	"""
	531	Returns:
	532	The active span context encoded as a string.
	533	"""
	534	carrier = {}
	535	if opentracing:
	536	opentracing.tracer.inject(
	537	opentracing.tracer.active_span, opentracing.Format.TEXT_MAP, carrier
	538	)
	539	return json.dumps(carrier)
	540
	541
	542	@only_if_tracing
	543	def span_context_from_string(carrier):
	544	"""
	545	Returns:
	546	The active span context decoded from a string.
	547	"""
	548	carrier = json.loads(carrier)
	549	return opentracing.tracer.extract(opentracing.Format.TEXT_MAP, carrier)
	550
	551
	552	@only_if_tracing
	553	def extract_text_map(carrier):
	554	"""
	555	Wrapper method for opentracing's tracer.extract for TEXT_MAP.
	556	Args:
	557	carrier (dict): a dict possibly containing a span context.
	558
	559	Returns:
	560	The active span context extracted from carrier.
	561	"""
	562	return opentracing.tracer.extract(opentracing.Format.TEXT_MAP, carrier)
	563
	564
	565	# Tracing decorators
	566
	567
	568	def trace(func):
	569	"""
	570	Decorator to trace a function.
	571	Sets the operation name to that of the function's.
	572	"""
	573	if opentracing is None:
	574	return func
	575
	576	@wraps(func)
	577	def _trace_inner(self, args, *kwargs):
	578	if opentracing is None:
	579	return func(self, args, *kwargs)
	580
	581	scope = start_active_span(func.__name__)
	582	scope.__enter__()
	583
	584	try:
	585	result = func(self, args, *kwargs)
	586	if isinstance(result, defer.Deferred):
	587
	588	def call_back(result):
	589	scope.__exit__(None, None, None)
	590	return result
	591
	592	def err_back(result):
	593	scope.span.set_tag(tags.ERROR, True)
	594	scope.__exit__(None, None, None)
	595	return result
	596
	597	result.addCallbacks(call_back, err_back)
	598
	599	else:
	600	scope.__exit__(None, None, None)
	601
	602	return result
	603
	604	except Exception as e:
	605	scope.__exit__(type(e), None, e.__traceback__)
	606	raise
	607
	608	return _trace_inner
	609
	610
	611	def trace_using_operation_name(operation_name):
	612	"""Decorator to trace a function. Explicitely sets the operation_name."""
	613
	614	def trace(func):
	615	"""
	616	Decorator to trace a function.
	617	Sets the operation name to that of the function's.
	618	"""
	619	if opentracing is None:
	620	return func
	621
	622	@wraps(func)
	623	def _trace_inner(self, args, *kwargs):
	624	if opentracing is None:
	625	return func(self, args, *kwargs)
	626
	627	scope = start_active_span(operation_name)
	628	scope.__enter__()
	629
	630	try:
	631	result = func(self, args, *kwargs)
	632	if isinstance(result, defer.Deferred):
	633
	634	def call_back(result):
	635	scope.__exit__(None, None, None)
	636	return result
	637
	638	def err_back(result):
	639	scope.span.set_tag(tags.ERROR, True)
	640	scope.__exit__(None, None, None)
	641	return result
	642
	643	result.addCallbacks(call_back, err_back)
	644	else:
	645	scope.__exit__(None, None, None)
	646
	647	return result
	648
	649	except Exception as e:
	650	scope.__exit__(type(e), None, e.__traceback__)
	651	raise
	652
	653	return _trace_inner
	654
	655	return trace
	656
	657
	658	def tag_args(func):
	659	"""
	660	Tags all of the args to the active span.
	661	"""
	662
	663	if not opentracing:
	664	return func
	665
	666	@wraps(func)
	667	def _tag_args_inner(self, args, *kwargs):
	668	argspec = inspect.getargspec(func)
	669	for i, arg in enumerate(argspec.args[1:]):
	670	set_tag("ARG_" + arg, args[i])
	671	set_tag("args", args[len(argspec.args) :])
	672	set_tag("kwargs", kwargs)
	673	return func(self, args, *kwargs)
	674
	675	return _tag_args_inner
	676
	677
	678	def trace_servlet(servlet_name, func):
	679	"""Decorator which traces a serlet. It starts a span with some servlet specific
	680	tags such as the servlet_name and request information"""
	681	if not opentracing:
	682	return func
	683
	684	@wraps(func)
	685	@defer.inlineCallbacks
	686	def _trace_servlet_inner(request, args, *kwargs):
	687	with start_active_span(
	688	"incoming-client-request",
	689	tags={
	690	"request_id": request.get_request_id(),
	691	tags.SPAN_KIND: tags.SPAN_KIND_RPC_SERVER,
	692	tags.HTTP_METHOD: request.get_method(),
	693	tags.HTTP_URL: request.get_redacted_uri(),
	694	tags.PEER_HOST_IPV6: request.getClientIP(),
	695	"servlet_name": servlet_name,
	696	},
	697	):
	698	result = yield defer.maybeDeferred(func, request, args, *kwargs)
	699	return result
	700
	701	return _trace_servlet_inner
	702
	703
	704	# Helper class
	705
	706
	707	class _DummyTagNames(object):
176	708	"""wrapper of opentracings tags. We need to have them if we
177	709	want to reference them without opentracing around. Clearly they
178	710	should never actually show up in a trace. `set_tags` overwrites

204	736	SPAN_KIND_RPC_SERVER = INVALID_TAG
205	737
206	738
207		def only_if_tracing(func):
208		"""Executes the function only if we're tracing. Otherwise return.
209		Assumes the function wrapped may return None"""
210
211		@wraps(func)
212		def _only_if_tracing_inner(args, *kwargs):
213		if opentracing:
214		return func(args, *kwargs)
215		else:
216		return
217
218		return _only_if_tracing_inner
219
220
221		# A regex which matches the server_names to expose traces for.
222		# None means 'block everything'.
223		_homeserver_whitelist = None
224
225		tags = _DumTagNames
226
227
228		def init_tracer(config):
229		"""Set the whitelists and initialise the JaegerClient tracer
230
231		Args:
232		config (HomeserverConfig): The config used by the homeserver
233		"""
234		global opentracing
235		if not config.opentracer_enabled:
236		# We don't have a tracer
237		opentracing = None
238		return
239
240		if not opentracing or not JaegerConfig:
241		raise ConfigError(
242		"The server has been configured to use opentracing but opentracing is not "
243		"installed."
244		)
245
246		# Include the worker name
247		name = config.worker_name if config.worker_name else "master"
248
249		set_homeserver_whitelist(config.opentracer_whitelist)
250		jaeger_config = JaegerConfig(
251		config={"sampler": {"type": "const", "param": 1}, "logging": True},
252		service_name="{} {}".format(config.server_name, name),
253		scope_manager=LogContextScopeManager(config),
254		)
255		jaeger_config.initialize_tracer()
256
257		# Set up tags to be opentracing's tags
258		global tags
259		tags = opentracing.tags
260
261
262		@contextlib.contextmanager
263		def _noop_context_manager(args, *kwargs):
264		"""Does absolutely nothing really well. Can be entered and exited arbitrarily.
265		Good substitute for an opentracing scope."""
266		yield
267
268
269		# Could use kwargs but I want these to be explicit
270		def start_active_span(
271		operation_name,
272		child_of=None,
273		references=None,
274		tags=None,
275		start_time=None,
276		ignore_active_span=False,
277		finish_on_close=True,
278		):
279		"""Starts an active opentracing span. Note, the scope doesn't become active
280		until it has been entered, however, the span starts from the time this
281		message is called.
282		Args:
283		See opentracing.tracer
284		Returns:
285		scope (Scope) or noop_context_manager
286		"""
287		if opentracing is None:
288		return _noop_context_manager()
289		else:
290		# We need to enter the scope here for the logcontext to become active
291		return opentracing.tracer.start_active_span(
292		operation_name,
293		child_of=child_of,
294		references=references,
295		tags=tags,
296		start_time=start_time,
297		ignore_active_span=ignore_active_span,
298		finish_on_close=finish_on_close,
299		)
300
301
302		@only_if_tracing
303		def close_active_span():
304		"""Closes the active span. This will close it's logcontext if the context
305		was made for the span"""
306		opentracing.tracer.scope_manager.active.__exit__(None, None, None)
307
308
309		@only_if_tracing
310		def set_tag(key, value):
311		"""Set's a tag on the active span"""
312		opentracing.tracer.active_span.set_tag(key, value)
313
314
315		@only_if_tracing
316		def log_kv(key_values, timestamp=None):
317		"""Log to the active span"""
318		opentracing.tracer.active_span.log_kv(key_values, timestamp)
319
320
321		# Note: we don't have a get baggage items because we're trying to hide all
322		# scope and span state from synapse. I think this method may also be useless
323		# as a result
324		@only_if_tracing
325		def set_baggage_item(key, value):
326		"""Attach baggage to the active span"""
327		opentracing.tracer.active_span.set_baggage_item(key, value)
328
329
330		@only_if_tracing
331		def set_operation_name(operation_name):
332		"""Sets the operation name of the active span"""
333		opentracing.tracer.active_span.set_operation_name(operation_name)
334
335
336		@only_if_tracing
337		def set_homeserver_whitelist(homeserver_whitelist):
338		"""Sets the whitelist
339
340		Args:
341		homeserver_whitelist (iterable of strings): regex of whitelisted homeservers
342		"""
343		global _homeserver_whitelist
344		if homeserver_whitelist:
345		# Makes a single regex which accepts all passed in regexes in the list
346		_homeserver_whitelist = re.compile(
347		"({})".format(")\|(".join(homeserver_whitelist))
348		)
349
350
351		@only_if_tracing
352		def whitelisted_homeserver(destination):
353		"""Checks if a destination matches the whitelist
354		Args:
355		destination (String)"""
356		if _homeserver_whitelist:
357		return _homeserver_whitelist.match(destination)
358		return False
359
360
361		def start_active_span_from_context(
362		headers,
363		operation_name,
364		references=None,
365		tags=None,
366		start_time=None,
367		ignore_active_span=False,
368		finish_on_close=True,
369		):
370		"""
371		Extracts a span context from Twisted Headers.
372		args:
373		headers (twisted.web.http_headers.Headers)
374		returns:
375		span_context (opentracing.span.SpanContext)
376		"""
377		# Twisted encodes the values as lists whereas opentracing doesn't.
378		# So, we take the first item in the list.
379		# Also, twisted uses byte arrays while opentracing expects strings.
380		if opentracing is None:
381		return _noop_context_manager()
382
383		header_dict = {k.decode(): v[0].decode() for k, v in headers.getAllRawHeaders()}
384		context = opentracing.tracer.extract(opentracing.Format.HTTP_HEADERS, header_dict)
385
386		return opentracing.tracer.start_active_span(
387		operation_name,
388		child_of=context,
389		references=references,
390		tags=tags,
391		start_time=start_time,
392		ignore_active_span=ignore_active_span,
393		finish_on_close=finish_on_close,
394		)
395
396
397		@only_if_tracing
398		def inject_active_span_twisted_headers(headers, destination):
399		"""
400		Injects a span context into twisted headers inplace
401
402		Args:
403		headers (twisted.web.http_headers.Headers)
404		span (opentracing.Span)
405
406		Returns:
407		Inplace modification of headers
408
409		Note:
410		The headers set by the tracer are custom to the tracer implementation which
411		should be unique enough that they don't interfere with any headers set by
412		synapse or twisted. If we're still using jaeger these headers would be those
413		here:
414		https://github.com/jaegertracing/jaeger-client-python/blob/master/jaeger_client/constants.py
415		"""
416
417		if not whitelisted_homeserver(destination):
418		return
419
420		span = opentracing.tracer.active_span
421		carrier = {}
422		opentracing.tracer.inject(span, opentracing.Format.HTTP_HEADERS, carrier)
423
424		for key, value in carrier.items():
425		headers.addRawHeaders(key, value)
426
427
428		@only_if_tracing
429		def inject_active_span_byte_dict(headers, destination):
430		"""
431		Injects a span context into a dict where the headers are encoded as byte
432		strings
433
434		Args:
435		headers (dict)
436		span (opentracing.Span)
437
438		Returns:
439		Inplace modification of headers
440
441		Note:
442		The headers set by the tracer are custom to the tracer implementation which
443		should be unique enough that they don't interfere with any headers set by
444		synapse or twisted. If we're still using jaeger these headers would be those
445		here:
446		https://github.com/jaegertracing/jaeger-client-python/blob/master/jaeger_client/constants.py
447		"""
448		if not whitelisted_homeserver(destination):
449		return
450
451		span = opentracing.tracer.active_span
452
453		carrier = {}
454		opentracing.tracer.inject(span, opentracing.Format.HTTP_HEADERS, carrier)
455
456		for key, value in carrier.items():
457		headers[key.encode()] = [value.encode()]
458
459
460		def trace_servlet(servlet_name, func):
461		"""Decorator which traces a serlet. It starts a span with some servlet specific
462		tags such as the servlet_name and request information"""
463
464		@wraps(func)
465		@defer.inlineCallbacks
466		def _trace_servlet_inner(request, args, *kwargs):
467		with start_active_span_from_context(
468		request.requestHeaders,
469		"incoming-client-request",
470		tags={
471		"request_id": request.get_request_id(),
472		tags.SPAN_KIND: tags.SPAN_KIND_RPC_SERVER,
473		tags.HTTP_METHOD: request.get_method(),
474		tags.HTTP_URL: request.get_redacted_uri(),
475		tags.PEER_HOST_IPV6: request.getClientIP(),
476		"servlet_name": servlet_name,
477		},
478		):
479		result = yield defer.maybeDeferred(func, request, args, *kwargs)
480		defer.returnValue(result)
481
482		return _trace_servlet_inner
	739	tags = _DummyTagNames

+1

-1

synapse/logging/scopecontextmanager.py less more

130	130
131	131	def close(self):
132	132	if self.manager.active is not self:
133		logger.error("Tried to close a none active scope!")
	133	logger.error("Tried to close a non-active scope!")
134	134	return
135	135
136	136	if self._finish_on_close:

+1

-1

synapse/module_api/__init__.py less more

100	100	)
101	101	user_id = yield self.register_user(localpart, displayname, emails)
102	102	_, access_token = yield self.register_device(user_id)
103		defer.returnValue((user_id, access_token))
	103	return (user_id, access_token)
104	104
105	105	def register_user(self, localpart, displayname=None, emails=[]):
106	106	"""Registers a new user with given localpart and optional displayname, emails.

+9

-9

synapse/notifier.py less more

364	364	current_token = user_stream.current_token
365	365	result = yield callback(prev_token, current_token)
366	366
367		defer.returnValue(result)
	367	return result
368	368
369	369	@defer.inlineCallbacks
370	370	def get_events_for(

399	399	@defer.inlineCallbacks
400	400	def check_for_updates(before_token, after_token):
401	401	if not after_token.is_after(before_token):
402		defer.returnValue(EventStreamResult([], (from_token, from_token)))
	402	return EventStreamResult([], (from_token, from_token))
403	403
404	404	events = []
405	405	end_token = from_token

439	439	events.extend(new_events)
440	440	end_token = end_token.copy_and_replace(keyname, new_key)
441	441
442		defer.returnValue(EventStreamResult(events, (from_token, end_token)))
	442	return EventStreamResult(events, (from_token, end_token))
443	443
444	444	user_id_for_stream = user.to_string()
445	445	if is_peeking:

464	464	from_token=from_token,
465	465	)
466	466
467		defer.returnValue(result)
	467	return result
468	468
469	469	@defer.inlineCallbacks
470	470	def _get_room_ids(self, user, explicit_room_id):
471	471	joined_room_ids = yield self.store.get_rooms_for_user(user.to_string())
472	472	if explicit_room_id:
473	473	if explicit_room_id in joined_room_ids:
474		defer.returnValue(([explicit_room_id], True))
	474	return ([explicit_room_id], True)
475	475	if (yield self._is_world_readable(explicit_room_id)):
476		defer.returnValue(([explicit_room_id], False))
	476	return ([explicit_room_id], False)
477	477	raise AuthError(403, "Non-joined access not allowed")
478		defer.returnValue((joined_room_ids, True))
	478	return (joined_room_ids, True)
479	479
480	480	@defer.inlineCallbacks
481	481	def _is_world_readable(self, room_id):

483	483	room_id, EventTypes.RoomHistoryVisibility, ""
484	484	)
485	485	if state and "history_visibility" in state.content:
486		defer.returnValue(state.content["history_visibility"] == "world_readable")
	486	return state.content["history_visibility"] == "world_readable"
487	487	else:
488		defer.returnValue(False)
	488	return False
489	489
490	490	@log_function
491	491	def remove_expired_streams(self):

+7

-1

synapse/push/baserules.py less more

244	244	"key": "type",
245	245	"pattern": "m.room.tombstone",
246	246	"_id": "_tombstone",
247		}
	247	},
	248	{
	249	"kind": "event_match",
	250	"key": "state_key",
	251	"pattern": "",
	252	"_id": "_tombstone_statekey",
	253	},
248	254	],
249	255	"actions": ["notify", {"set_tweak": "highlight", "value": True}],
250	256	},

+5

-5

synapse/push/bulk_push_rule_evaluator.py less more

94	94	invited
95	95	)
96	96
97		defer.returnValue(rules_by_user)
	97	return rules_by_user
98	98
99	99	@cached()
100	100	def _get_rules_for_room(self, room_id):

133	133
134	134	pl_event = auth_events.get(POWER_KEY)
135	135
136		defer.returnValue((pl_event.content if pl_event else {}, sender_level))
	136	return (pl_event.content if pl_event else {}, sender_level)
137	137
138	138	@defer.inlineCallbacks
139	139	def action_for_event_by_user(self, event, context):

282	282	if state_group and self.state_group == state_group:
283	283	logger.debug("Using cached rules for %r", self.room_id)
284	284	self.room_push_rule_cache_metrics.inc_hits()
285		defer.returnValue(self.rules_by_user)
	285	return self.rules_by_user
286	286
287	287	with (yield self.linearizer.queue(())):
288	288	if state_group and self.state_group == state_group:
289	289	logger.debug("Using cached rules for %r", self.room_id)
290	290	self.room_push_rule_cache_metrics.inc_hits()
291		defer.returnValue(self.rules_by_user)
	291	return self.rules_by_user
292	292
293	293	self.room_push_rule_cache_metrics.inc_misses()
294	294

365	365	logger.debug(
366	366	"Returning push rules for %r %r", self.room_id, ret_rules_by_user.keys()
367	367	)
368		defer.returnValue(ret_rules_by_user)
	368	return ret_rules_by_user
369	369
370	370	@defer.inlineCallbacks
371	371	def _update_rules_with_member_event_ids(

+12

-6

synapse/push/emailpusher.py less more

233	233	return
234	234
235	235	self.last_stream_ordering = last_stream_ordering
236		yield self.store.update_pusher_last_stream_ordering_and_success(
237		self.app_id,
238		self.email,
239		self.user_id,
240		last_stream_ordering,
241		self.clock.time_msec(),
	236	pusher_still_exists = (
	237	yield self.store.update_pusher_last_stream_ordering_and_success(
	238	self.app_id,
	239	self.email,
	240	self.user_id,
	241	last_stream_ordering,
	242	self.clock.time_msec(),
	243	)
242	244	)
	245	if not pusher_still_exists:
	246	# The pusher has been deleted while we were processing, so
	247	# lets just stop and return.
	248	self.on_stop()
243	249
244	250	def seconds_until(self, ts_msec):
245	251	secs = (ts_msec - self.clock.time_msec()) / 1000

+29

-16

synapse/push/httppusher.py less more

198	198	http_push_processed_counter.inc()
199	199	self.backoff_delay = HttpPusher.INITIAL_BACKOFF_SEC
200	200	self.last_stream_ordering = push_action["stream_ordering"]
201		yield self.store.update_pusher_last_stream_ordering_and_success(
202		self.app_id,
203		self.pushkey,
204		self.user_id,
205		self.last_stream_ordering,
206		self.clock.time_msec(),
	201	pusher_still_exists = (
	202	yield self.store.update_pusher_last_stream_ordering_and_success(
	203	self.app_id,
	204	self.pushkey,
	205	self.user_id,
	206	self.last_stream_ordering,
	207	self.clock.time_msec(),
	208	)
207	209	)
	210	if not pusher_still_exists:
	211	# The pusher has been deleted while we were processing, so
	212	# lets just stop and return.
	213	self.on_stop()
	214	return
	215
208	216	if self.failing_since:
209	217	self.failing_since = None
210	218	yield self.store.update_pusher_failing_since(

233	241	)
234	242	self.backoff_delay = HttpPusher.INITIAL_BACKOFF_SEC
235	243	self.last_stream_ordering = push_action["stream_ordering"]
236		yield self.store.update_pusher_last_stream_ordering(
	244	pusher_still_exists = yield self.store.update_pusher_last_stream_ordering(
237	245	self.app_id,
238	246	self.pushkey,
239	247	self.user_id,
240	248	self.last_stream_ordering,
241	249	)
	250	if not pusher_still_exists:
	251	# The pusher has been deleted while we were processing, so
	252	# lets just stop and return.
	253	self.on_stop()
	254	return
242	255
243	256	self.failing_since = None
244	257	yield self.store.update_pusher_failing_since(

257	270	@defer.inlineCallbacks
258	271	def _process_one(self, push_action):
259	272	if "notify" not in push_action["actions"]:
260		defer.returnValue(True)
	273	return True
261	274
262	275	tweaks = push_rule_evaluator.tweaks_for_actions(push_action["actions"])
263	276	badge = yield push_tools.get_badge_count(self.hs.get_datastore(), self.user_id)
264	277
265	278	event = yield self.store.get_event(push_action["event_id"], allow_none=True)
266	279	if event is None:
267		defer.returnValue(True) # It's been redacted
	280	return True # It's been redacted
268	281	rejected = yield self.dispatch_push(event, tweaks, badge)
269	282	if rejected is False:
270		defer.returnValue(False)
	283	return False
271	284
272	285	if isinstance(rejected, list) or isinstance(rejected, tuple):
273	286	for pk in rejected:

281	294	else:
282	295	logger.info("Pushkey %s was rejected: removing", pk)
283	296	yield self.hs.remove_pusher(self.app_id, pk, self.user_id)
284		defer.returnValue(True)
	297	return True
285	298
286	299	@defer.inlineCallbacks
287	300	def _build_notification_dict(self, event, tweaks, badge):

301	314	],
302	315	}
303	316	}
304		defer.returnValue(d)
	317	return d
305	318
306	319	ctx = yield push_tools.get_context_for_event(
307	320	self.store, self.state_handler, event, self.user_id

344	357	if "name" in ctx and len(ctx["name"]) > 0:
345	358	d["notification"]["room_name"] = ctx["name"]
346	359
347		defer.returnValue(d)
	360	return d
348	361
349	362	@defer.inlineCallbacks
350	363	def dispatch_push(self, event, tweaks, badge):
351	364	notification_dict = yield self._build_notification_dict(event, tweaks, badge)
352	365	if not notification_dict:
353		defer.returnValue([])
	366	return []
354	367	try:
355	368	resp = yield self.http_client.post_json_get_json(
356	369	self.url, notification_dict

363	376	type(e),
364	377	e,
365	378	)
366		defer.returnValue(False)
	379	return False
367	380	rejected = []
368	381	if "rejected" in resp:
369	382	rejected = resp["rejected"]
370		defer.returnValue(rejected)
	383	return rejected
371	384
372	385	@defer.inlineCallbacks
373	386	def _send_badge(self, badge):

+34

-50

synapse/push/mailer.py less more

315	315	if not merge:
316	316	room_vars["notifs"].append(notifvars)
317	317
318		defer.returnValue(room_vars)
	318	return room_vars
319	319
320	320	@defer.inlineCallbacks
321	321	def get_notif_vars(self, notif, user_id, notif_event, room_state_ids):

342	342	if messagevars is not None:
343	343	ret["messages"].append(messagevars)
344	344
345		defer.returnValue(ret)
	345	return ret
346	346
347	347	@defer.inlineCallbacks
348	348	def get_message_vars(self, notif, event, room_state_ids):

378	378	if "body" in event.content:
379	379	ret["body_text_plain"] = event.content["body"]
380	380
381		defer.returnValue(ret)
	381	return ret
382	382
383	383	def add_text_message_vars(self, messagevars, event):
384	384	msgformat = event.content.get("format")

427	427	inviter_name = name_from_member_event(inviter_member_event)
428	428
429	429	if room_name is None:
430		defer.returnValue(
431		INVITE_FROM_PERSON
432		% {"person": inviter_name, "app": self.app_name}
433		)
	430	return INVITE_FROM_PERSON % {
	431	"person": inviter_name,
	432	"app": self.app_name,
	433	}
434	434	else:
435		defer.returnValue(
436		INVITE_FROM_PERSON_TO_ROOM
437		% {
438		"person": inviter_name,
439		"room": room_name,
440		"app": self.app_name,
441		}
442		)
	435	return INVITE_FROM_PERSON_TO_ROOM % {
	436	"person": inviter_name,
	437	"room": room_name,
	438	"app": self.app_name,
	439	}
443	440
444	441	sender_name = None
445	442	if len(notifs_by_room[room_id]) == 1:

453	450	sender_name = name_from_member_event(state_event)
454	451
455	452	if sender_name is not None and room_name is not None:
456		defer.returnValue(
457		MESSAGE_FROM_PERSON_IN_ROOM
458		% {
459		"person": sender_name,
460		"room": room_name,
461		"app": self.app_name,
462		}
463		)
	453	return MESSAGE_FROM_PERSON_IN_ROOM % {
	454	"person": sender_name,
	455	"room": room_name,
	456	"app": self.app_name,
	457	}
464	458	elif sender_name is not None:
465		defer.returnValue(
466		MESSAGE_FROM_PERSON
467		% {"person": sender_name, "app": self.app_name}
468		)
	459	return MESSAGE_FROM_PERSON % {
	460	"person": sender_name,
	461	"app": self.app_name,
	462	}
469	463	else:
470	464	# There's more than one notification for this room, so just
471	465	# say there are several
472	466	if room_name is not None:
473		defer.returnValue(
474		MESSAGES_IN_ROOM % {"room": room_name, "app": self.app_name}
475		)
	467	return MESSAGES_IN_ROOM % {"room": room_name, "app": self.app_name}
476	468	else:
477	469	# If the room doesn't have a name, say who the messages
478	470	# are from explicitly to avoid, "messages in the Bob room"

492	484	]
493	485	)
494	486
495		defer.returnValue(
496		MESSAGES_FROM_PERSON
497		% {
498		"person": descriptor_from_member_events(
499		member_events.values()
500		),
501		"app": self.app_name,
502		}
503		)
	487	return MESSAGES_FROM_PERSON % {
	488	"person": descriptor_from_member_events(member_events.values()),
	489	"app": self.app_name,
	490	}
504	491	else:
505	492	# Stuff's happened in multiple different rooms
506	493
507	494	# ...but we still refer to the 'reason' room which triggered the mail
508	495	if reason["room_name"] is not None:
509		defer.returnValue(
510		MESSAGES_IN_ROOM_AND_OTHERS
511		% {"room": reason["room_name"], "app": self.app_name}
512		)
	496	return MESSAGES_IN_ROOM_AND_OTHERS % {
	497	"room": reason["room_name"],
	498	"app": self.app_name,
	499	}
513	500	else:
514	501	# If the reason room doesn't have a name, say who the messages
515	502	# are from explicitly to avoid, "messages in the Bob room"

526	513	[room_state_ids[room_id][("m.room.member", s)] for s in sender_ids]
527	514	)
528	515
529		defer.returnValue(
530		MESSAGES_FROM_PERSON_AND_OTHERS
531		% {
532		"person": descriptor_from_member_events(member_events.values()),
533		"app": self.app_name,
534		}
535		)
	516	return MESSAGES_FROM_PERSON_AND_OTHERS % {
	517	"person": descriptor_from_member_events(member_events.values()),
	518	"app": self.app_name,
	519	}
536	520
537	521	def make_room_link(self, room_id):
538	522	if self.hs.config.email_riot_base_url:

+12

-13

synapse/push/presentable_names.py less more

54	54	room_state_ids[("m.room.name", "")], allow_none=True
55	55	)
56	56	if m_room_name and m_room_name.content and m_room_name.content["name"]:
57		defer.returnValue(m_room_name.content["name"])
	57	return m_room_name.content["name"]
58	58
59	59	# does it have a canonical alias?
60	60	if ("m.room.canonical_alias", "") in room_state_ids:

67	67	and canon_alias.content["alias"]
68	68	and _looks_like_an_alias(canon_alias.content["alias"])
69	69	):
70		defer.returnValue(canon_alias.content["alias"])
	70	return canon_alias.content["alias"]
71	71
72	72	# at this point we're going to need to search the state by all state keys
73	73	# for an event type, so rearrange the data structure

81	81	if alias_event and alias_event.content.get("aliases"):
82	82	the_aliases = alias_event.content["aliases"]
83	83	if len(the_aliases) > 0 and _looks_like_an_alias(the_aliases[0]):
84		defer.returnValue(the_aliases[0])
	84	return the_aliases[0]
85	85
86	86	if not fallback_to_members:
87		defer.returnValue(None)
	87	return None
88	88
89	89	my_member_event = None
90	90	if ("m.room.member", user_id) in room_state_ids:

103	103	)
104	104	if inviter_member_event:
105	105	if fallback_to_single_member:
106		defer.returnValue(
107		"Invite from %s"
108		% (name_from_member_event(inviter_member_event),)
	106	return "Invite from %s" % (
	107	name_from_member_event(inviter_member_event),
109	108	)
110	109	else:
111	110	return
112	111	else:
113		defer.returnValue("Room Invite")
	112	return "Room Invite"
114	113
115	114	# we're going to have to generate a name based on who's in the room,
116	115	# so find out who is in the room that isn't the user.

153	152	# return "Inviting %s" % (
154	153	# descriptor_from_member_events(third_party_invites)
155	154	# )
156		defer.returnValue("Inviting email address")
	155	return "Inviting email address"
157	156	else:
158		defer.returnValue(ALL_ALONE)
	157	return ALL_ALONE
159	158	else:
160		defer.returnValue(name_from_member_event(all_members[0]))
	159	return name_from_member_event(all_members[0])
161	160	else:
162		defer.returnValue(ALL_ALONE)
	161	return ALL_ALONE
163	162	elif len(other_members) == 1 and not fallback_to_single_member:
164	163	return
165	164	else:
166		defer.returnValue(descriptor_from_member_events(other_members))
	165	return descriptor_from_member_events(other_members)
167	166
168	167
169	168	def descriptor_from_member_events(member_events):

+2

-2

synapse/push/push_tools.py less more

38	38	# return one badge count per conversation, as count per
39	39	# message is so noisy as to be almost useless
40	40	badge += 1 if notifs["notify_count"] else 0
41		defer.returnValue(badge)
	41	return badge
42	42
43	43
44	44	@defer.inlineCallbacks

60	60	sender_state_event = yield store.get_event(sender_state_event_id)
61	61	ctx["sender_display_name"] = name_from_member_event(sender_state_event)
62	62
63		defer.returnValue(ctx)
	63	return ctx

+3

-3

synapse/push/pusherpool.py less more

122	122	)
123	123	pusher = yield self.start_pusher_by_id(app_id, pushkey, user_id)
124	124
125		defer.returnValue(pusher)
	125	return pusher
126	126
127	127	@defer.inlineCallbacks
128	128	def remove_pushers_by_app_id_and_pushkey_not_user(

223	223	if pusher_dict:
224	224	pusher = yield self._start_pusher(pusher_dict)
225	225
226		defer.returnValue(pusher)
	226	return pusher
227	227
228	228	@defer.inlineCallbacks
229	229	def _start_pushers(self):

292	292
293	293	p.on_started(have_notifs)
294	294
295		defer.returnValue(p)
	295	return p
296	296
297	297	@defer.inlineCallbacks
298	298	def remove_pusher(self, app_id, pushkey, user_id):

+1

-0

synapse/python_dependencies.py less more

71	71	"netaddr>=0.7.18",
72	72	"Jinja2>=2.9",
73	73	"bleach>=1.4.3",
	74	"sdnotify>=0.3",
74	75	]
75	76
76	77	CONDITIONAL_REQUIREMENTS = {

+1

-1

synapse/replication/http/_base.py less more

184	184	except RequestSendFailed as e:
185	185	raise_from(SynapseError(502, "Failed to talk to master"), e)
186	186
187		defer.returnValue(result)
	187	return result
188	188
189	189	return send_request
190	190

+5

-5

synapse/replication/http/federation.py less more

79	79
80	80	payload = {"events": event_payloads, "backfilled": backfilled}
81	81
82		defer.returnValue(payload)
	82	return payload
83	83
84	84	@defer.inlineCallbacks
85	85	def _handle_request(self, request):

112	112	event_and_contexts, backfilled
113	113	)
114	114
115		defer.returnValue((200, {}))
	115	return (200, {})
116	116
117	117
118	118	class ReplicationFederationSendEduRestServlet(ReplicationEndpoint):

155	155
156	156	result = yield self.registry.on_edu(edu_type, origin, edu_content)
157	157
158		defer.returnValue((200, result))
	158	return (200, result)
159	159
160	160
161	161	class ReplicationGetQueryRestServlet(ReplicationEndpoint):

203	203
204	204	result = yield self.registry.on_query(query_type, args)
205	205
206		defer.returnValue((200, result))
	206	return (200, result)
207	207
208	208
209	209	class ReplicationCleanRoomRestServlet(ReplicationEndpoint):

237	237	def _handle_request(self, request, room_id):
238	238	yield self.store.clean_room_for_join(room_id)
239	239
240		defer.returnValue((200, {}))
	240	return (200, {})
241	241
242	242
243	243	def register_servlets(hs, http_server):

+1

-1

synapse/replication/http/login.py less more

63	63	user_id, device_id, initial_display_name, is_guest
64	64	)
65	65
66		defer.returnValue((200, {"device_id": device_id, "access_token": access_token}))
	66	return (200, {"device_id": device_id, "access_token": access_token})
67	67
68	68
69	69	def register_servlets(hs, http_server):

+2

-2

synapse/replication/http/membership.py less more

82	82	remote_room_hosts, room_id, user_id, event_content
83	83	)
84	84
85		defer.returnValue((200, {}))
	85	return (200, {})
86	86
87	87
88	88	class ReplicationRemoteRejectInviteRestServlet(ReplicationEndpoint):

152	152	yield self.store.locally_reject_invite(user_id, room_id)
153	153	ret = {}
154	154
155		defer.returnValue((200, ret))
	155	return (200, ret)
156	156
157	157
158	158	class ReplicationUserJoinedLeftRoomRestServlet(ReplicationEndpoint):

+2

-2

synapse/replication/http/register.py less more

89	89	address=content["address"],
90	90	)
91	91
92		defer.returnValue((200, {}))
	92	return (200, {})
93	93
94	94
95	95	class ReplicationPostRegisterActionsServlet(ReplicationEndpoint):

142	142	bind_msisdn=bind_msisdn,
143	143	)
144	144
145		defer.returnValue((200, {}))
	145	return (200, {})
146	146
147	147
148	148	def register_servlets(hs, http_server):

+2

-2

synapse/replication/http/send_event.py less more

84	84	"extra_users": [u.to_string() for u in extra_users],
85	85	}
86	86
87		defer.returnValue(payload)
	87	return payload
88	88
89	89	@defer.inlineCallbacks
90	90	def _handle_request(self, request, event_id):

116	116	requester, event, context, ratelimit=ratelimit, extra_users=extra_users
117	117	)
118	118
119		defer.returnValue((200, {}))
	119	return (200, {})
120	120
121	121
122	122	def register_servlets(hs, http_server):

+6

-6

synapse/replication/tcp/streams/_base.py less more

157	157	updates, current_token = yield self.get_updates_since(self.last_token)
158	158	self.last_token = current_token
159	159
160		defer.returnValue((updates, current_token))
	160	return (updates, current_token)
161	161
162	162	@defer.inlineCallbacks
163	163	def get_updates_since(self, from_token):

171	171	sent over the replication steam.
172	172	"""
173	173	if from_token in ("NOW", "now"):
174		defer.returnValue(([], self.upto_token))
	174	return ([], self.upto_token)
175	175
176	176	current_token = self.upto_token
177	177
178	178	from_token = int(from_token)
179	179
180	180	if from_token == current_token:
181		defer.returnValue(([], current_token))
	181	return ([], current_token)
182	182
183	183	if self._LIMITED:
184	184	rows = yield self.update_function(

197	197	if self._LIMITED and len(updates) >= MAX_EVENTS_BEHIND:
198	198	raise Exception("stream %s has fallen behind" % (self.NAME))
199	199
200		defer.returnValue((updates, current_token))
	200	return (updates, current_token)
201	201
202	202	def current_token(self):
203	203	"""Gets the current token of the underlying streams. Should be provided

296	296	@defer.inlineCallbacks
297	297	def update_function(self, from_token, to_token, limit):
298	298	rows = yield self.store.get_all_push_rule_updates(from_token, to_token, limit)
299		defer.returnValue([(row[0], row[2]) for row in rows])
	299	return [(row[0], row[2]) for row in rows]
300	300
301	301
302	302	class PushersStream(Stream):

423	423	for stream_id, user_id, account_data_type, content in global_results
424	424	)
425	425
426		defer.returnValue(results)
	426	return results
427	427
428	428
429	429	class GroupServerStream(Stream):

+1

-1

synapse/replication/tcp/streams/events.py less more

133	133
134	134	all_updates = heapq.merge(event_updates, state_updates)
135	135
136		defer.returnValue(all_updates)
	136	return all_updates
137	137
138	138	@classmethod
139	139	def parse_row(cls, row):

+1

-0

synapse/res/templates/account_renewed.html less more

0

<html><body>Your account has been successfully renewed.</body><html>

+1

-0

synapse/res/templates/invalid_token.html less more

0

<html><body>Invalid renewal token.</body><html>

+33

-113

synapse/rest/admin/__init__.py less more

26	26
27	27	import synapse
28	28	from synapse.api.constants import Membership, UserTypes
29		from synapse.api.errors import AuthError, Codes, NotFoundError, SynapseError
	29	from synapse.api.errors import Codes, NotFoundError, SynapseError
30	30	from synapse.http.server import JsonResource
31	31	from synapse.http.servlet import (
32	32	RestServlet,

35	35	parse_json_object_from_request,
36	36	parse_string,
37	37	)
38		from synapse.rest.admin._base import assert_requester_is_admin, assert_user_is_admin
	38	from synapse.rest.admin._base import (
	39	assert_requester_is_admin,
	40	assert_user_is_admin,
	41	historical_admin_path_patterns,
	42	)
	43	from synapse.rest.admin.media import register_servlets_for_media_repo
39	44	from synapse.rest.admin.server_notice_servlet import SendServerNoticeServlet
40	45	from synapse.types import UserID, create_requester
41	46	from synapse.util.versionstring import get_version_string

43	48	logger = logging.getLogger(__name__)
44	49
45	50
46		def historical_admin_path_patterns(path_regex):
47		"""Returns the list of patterns for an admin endpoint, including historical ones
48
49		This is a backwards-compatibility hack. Previously, the Admin API was exposed at
50		various paths under /_matrix/client. This function returns a list of patterns
51		matching those paths (as well as the new one), so that existing scripts which rely
52		on the endpoints being available there are not broken.
53
54		Note that this should only be used for existing endpoints: new ones should just
55		register for the /_synapse/admin path.
56		"""
57		return list(
58		re.compile(prefix + path_regex)
59		for prefix in (
60		"^/_synapse/admin/v1",
61		"^/_matrix/client/api/v1/admin",
62		"^/_matrix/client/unstable/admin",
63		"^/_matrix/client/r0/admin",
64		)
65		)
66
67
68	51	class UsersRestServlet(RestServlet):
69	52	PATTERNS = historical_admin_path_patterns("/users/(?P<user_id>[^/]*)")
70	53

83	66
84	67	ret = yield self.handlers.admin_handler.get_users()
85	68
86		defer.returnValue((200, ret))
	69	return (200, ret)
87	70
88	71
89	72	class VersionServlet(RestServlet):

226	209	)
227	210
228	211	result = yield register._create_registration_details(user_id, body)
229		defer.returnValue((200, result))
	212	return (200, result)
230	213
231	214
232	215	class WhoisRestServlet(RestServlet):

251	234
252	235	ret = yield self.handlers.admin_handler.get_whois(target_user)
253	236
254		defer.returnValue((200, ret))
255
256
257		class PurgeMediaCacheRestServlet(RestServlet):
258		PATTERNS = historical_admin_path_patterns("/purge_media_cache")
259
260		def __init__(self, hs):
261		self.media_repository = hs.get_media_repository()
262		self.auth = hs.get_auth()
263
264		@defer.inlineCallbacks
265		def on_POST(self, request):
266		yield assert_requester_is_admin(self.auth, request)
267
268		before_ts = parse_integer(request, "before_ts", required=True)
269		logger.info("before_ts: %r", before_ts)
270
271		ret = yield self.media_repository.delete_old_remote_media(before_ts)
272
273		defer.returnValue((200, ret))
	237	return (200, ret)
274	238
275	239
276	240	class PurgeHistoryRestServlet(RestServlet):

355	319	room_id, token, delete_local_events=delete_local_events
356	320	)
357	321
358		defer.returnValue((200, {"purge_id": purge_id}))
	322	return (200, {"purge_id": purge_id})
359	323
360	324
361	325	class PurgeHistoryStatusRestServlet(RestServlet):

380	344	if purge_status is None:
381	345	raise NotFoundError("purge id '%s' not found" % purge_id)
382	346
383		defer.returnValue((200, purge_status.asdict()))
	347	return (200, purge_status.asdict())
384	348
385	349
386	350	class DeactivateAccountRestServlet(RestServlet):

412	376	else:
413	377	id_server_unbind_result = "no-support"
414	378
415		defer.returnValue((200, {"id_server_unbind_result": id_server_unbind_result}))
	379	return (200, {"id_server_unbind_result": id_server_unbind_result})
416	380
417	381
418	382	class ShutdownRoomRestServlet(RestServlet):

530	494	room_id, new_room_id, requester_user_id
531	495	)
532	496
533		defer.returnValue(
534		(
535		200,
536		{
537		"kicked_users": kicked_users,
538		"failed_to_kick_users": failed_to_kick_users,
539		"local_aliases": aliases_for_room,
540		"new_room_id": new_room_id,
541		},
542		)
543		)
544
545
546		class QuarantineMediaInRoom(RestServlet):
547		"""Quarantines all media in a room so that no one can download it via
548		this server.
549		"""
550
551		PATTERNS = historical_admin_path_patterns("/quarantine_media/(?P<room_id>[^/]+)")
552
553		def __init__(self, hs):
554		self.store = hs.get_datastore()
555		self.auth = hs.get_auth()
556
557		@defer.inlineCallbacks
558		def on_POST(self, request, room_id):
559		requester = yield self.auth.get_user_by_req(request)
560		yield assert_user_is_admin(self.auth, requester.user)
561
562		num_quarantined = yield self.store.quarantine_media_ids_in_room(
563		room_id, requester.user.to_string()
564		)
565
566		defer.returnValue((200, {"num_quarantined": num_quarantined}))
567
568
569		class ListMediaInRoom(RestServlet):
570		"""Lists all of the media in a given room.
571		"""
572
573		PATTERNS = historical_admin_path_patterns("/room/(?P<room_id>[^/]+)/media")
574
575		def __init__(self, hs):
576		self.store = hs.get_datastore()
577
578		@defer.inlineCallbacks
579		def on_GET(self, request, room_id):
580		requester = yield self.auth.get_user_by_req(request)
581		is_admin = yield self.auth.is_server_admin(requester.user)
582		if not is_admin:
583		raise AuthError(403, "You are not a server admin")
584
585		local_mxcs, remote_mxcs = yield self.store.get_media_mxcs_in_room(room_id)
586
587		defer.returnValue((200, {"local": local_mxcs, "remote": remote_mxcs}))
	497	return (
	498	200,
	499	{
	500	"kicked_users": kicked_users,
	501	"failed_to_kick_users": failed_to_kick_users,
	502	"local_aliases": aliases_for_room,
	503	"new_room_id": new_room_id,
	504	},
	505	)
588	506
589	507
590	508	class ResetPasswordRestServlet(RestServlet):

628	546	yield self._set_password_handler.set_password(
629	547	target_user_id, new_password, requester
630	548	)
631		defer.returnValue((200, {}))
	549	return (200, {})
632	550
633	551
634	552	class GetUsersPaginatedRestServlet(RestServlet):

670	588	logger.info("limit: %s, start: %s", limit, start)
671	589
672	590	ret = yield self.handlers.admin_handler.get_users_paginate(order, start, limit)
673		defer.returnValue((200, ret))
	591	return (200, ret)
674	592
675	593	@defer.inlineCallbacks
676	594	def on_POST(self, request, target_user_id):

698	616	logger.info("limit: %s, start: %s", limit, start)
699	617
700	618	ret = yield self.handlers.admin_handler.get_users_paginate(order, start, limit)
701		defer.returnValue((200, ret))
	619	return (200, ret)
702	620
703	621
704	622	class SearchUsersRestServlet(RestServlet):

741	659	logger.info("term: %s ", term)
742	660
743	661	ret = yield self.handlers.admin_handler.search_users(term)
744		defer.returnValue((200, ret))
	662	return (200, ret)
745	663
746	664
747	665	class DeleteGroupAdminRestServlet(RestServlet):

764	682	raise SynapseError(400, "Can only delete local groups")
765	683
766	684	yield self.group_server.delete_group(group_id, requester.user.to_string())
767		defer.returnValue((200, {}))
	685	return (200, {})
768	686
769	687
770	688	class AccountValidityRenewServlet(RestServlet):

795	713	)
796	714
797	715	res = {"expiration_ts": expiration_ts}
798		defer.returnValue((200, res))
	716	return (200, res)
799	717
800	718
801	719	########################################################################################

826	744	def register_servlets_for_client_rest_resource(hs, http_server):
827	745	"""Register only the servlets which need to be exposed on /_matrix/client/xxx"""
828	746	WhoisRestServlet(hs).register(http_server)
829		PurgeMediaCacheRestServlet(hs).register(http_server)
830	747	PurgeHistoryStatusRestServlet(hs).register(http_server)
831	748	DeactivateAccountRestServlet(hs).register(http_server)
832	749	PurgeHistoryRestServlet(hs).register(http_server)

835	752	GetUsersPaginatedRestServlet(hs).register(http_server)
836	753	SearchUsersRestServlet(hs).register(http_server)
837	754	ShutdownRoomRestServlet(hs).register(http_server)
838		QuarantineMediaInRoom(hs).register(http_server)
839		ListMediaInRoom(hs).register(http_server)
840	755	UserRegisterServlet(hs).register(http_server)
841	756	DeleteGroupAdminRestServlet(hs).register(http_server)
842	757	AccountValidityRenewServlet(hs).register(http_server)
	758
	759	# Load the media repo ones if we're using them.
	760	if hs.config.can_load_media_repo:
	761	register_servlets_for_media_repo(hs, http_server)
	762
843	763	# don't add more things here: new servlets should only be exposed on
844	764	# /_synapse/admin so should not go here. Instead register them in AdminRestResource.

+25

-0

synapse/rest/admin/_base.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14
	15	import re
	16
14	17	from twisted.internet import defer
15	18
16	19	from synapse.api.errors import AuthError
	20
	21
	22	def historical_admin_path_patterns(path_regex):
	23	"""Returns the list of patterns for an admin endpoint, including historical ones
	24
	25	This is a backwards-compatibility hack. Previously, the Admin API was exposed at
	26	various paths under /_matrix/client. This function returns a list of patterns
	27	matching those paths (as well as the new one), so that existing scripts which rely
	28	on the endpoints being available there are not broken.
	29
	30	Note that this should only be used for existing endpoints: new ones should just
	31	register for the /_synapse/admin path.
	32	"""
	33	return list(
	34	re.compile(prefix + path_regex)
	35	for prefix in (
	36	"^/_synapse/admin/v1",
	37	"^/_matrix/client/api/v1/admin",
	38	"^/_matrix/client/unstable/admin",
	39	"^/_matrix/client/r0/admin",
	40	)
	41	)
17	42
18	43
19	44	@defer.inlineCallbacks

+101

-0

synapse/rest/admin/media.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2018-2019 New Vector Ltd
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	import logging
	17
	18	from twisted.internet import defer
	19
	20	from synapse.api.errors import AuthError
	21	from synapse.http.servlet import RestServlet, parse_integer
	22	from synapse.rest.admin._base import (
	23	assert_requester_is_admin,
	24	assert_user_is_admin,
	25	historical_admin_path_patterns,
	26	)
	27
	28	logger = logging.getLogger(__name__)
	29
	30
	31	class QuarantineMediaInRoom(RestServlet):
	32	"""Quarantines all media in a room so that no one can download it via
	33	this server.
	34	"""
	35
	36	PATTERNS = historical_admin_path_patterns("/quarantine_media/(?P<room_id>[^/]+)")
	37
	38	def __init__(self, hs):
	39	self.store = hs.get_datastore()
	40	self.auth = hs.get_auth()
	41
	42	@defer.inlineCallbacks
	43	def on_POST(self, request, room_id):
	44	requester = yield self.auth.get_user_by_req(request)
	45	yield assert_user_is_admin(self.auth, requester.user)
	46
	47	num_quarantined = yield self.store.quarantine_media_ids_in_room(
	48	room_id, requester.user.to_string()
	49	)
	50
	51	return (200, {"num_quarantined": num_quarantined})
	52
	53
	54	class ListMediaInRoom(RestServlet):
	55	"""Lists all of the media in a given room.
	56	"""
	57
	58	PATTERNS = historical_admin_path_patterns("/room/(?P<room_id>[^/]+)/media")
	59
	60	def __init__(self, hs):
	61	self.store = hs.get_datastore()
	62
	63	@defer.inlineCallbacks
	64	def on_GET(self, request, room_id):
	65	requester = yield self.auth.get_user_by_req(request)
	66	is_admin = yield self.auth.is_server_admin(requester.user)
	67	if not is_admin:
	68	raise AuthError(403, "You are not a server admin")
	69
	70	local_mxcs, remote_mxcs = yield self.store.get_media_mxcs_in_room(room_id)
	71
	72	return (200, {"local": local_mxcs, "remote": remote_mxcs})
	73
	74
	75	class PurgeMediaCacheRestServlet(RestServlet):
	76	PATTERNS = historical_admin_path_patterns("/purge_media_cache")
	77
	78	def __init__(self, hs):
	79	self.media_repository = hs.get_media_repository()
	80	self.auth = hs.get_auth()
	81
	82	@defer.inlineCallbacks
	83	def on_POST(self, request):
	84	yield assert_requester_is_admin(self.auth, request)
	85
	86	before_ts = parse_integer(request, "before_ts", required=True)
	87	logger.info("before_ts: %r", before_ts)
	88
	89	ret = yield self.media_repository.delete_old_remote_media(before_ts)
	90
	91	return (200, ret)
	92
	93
	94	def register_servlets_for_media_repo(hs, http_server):
	95	"""
	96	Media repo specific APIs.
	97	"""
	98	PurgeMediaCacheRestServlet(hs).register(http_server)
	99	QuarantineMediaInRoom(hs).register(http_server)
	100	ListMediaInRoom(hs).register(http_server)

+1

-1

synapse/rest/admin/server_notice_servlet.py less more

91	91	event_content=body["content"],
92	92	)
93	93
94		defer.returnValue((200, {"event_id": event.event_id}))
	94	return (200, {"event_id": event.event_id})
95	95
96	96	def on_PUT(self, request, txn_id):
97	97	return self.txns.fetch_or_execute_request(

+8

-10

synapse/rest/client/v1/directory.py less more

53	53	dir_handler = self.handlers.directory_handler
54	54	res = yield dir_handler.get_association(room_alias)
55	55
56		defer.returnValue((200, res))
	56	return (200, res)
57	57
58	58	@defer.inlineCallbacks
59	59	def on_PUT(self, request, room_alias):

86	86	requester, room_alias, room_id, servers
87	87	)
88	88
89		defer.returnValue((200, {}))
	89	return (200, {})
90	90
91	91	@defer.inlineCallbacks
92	92	def on_DELETE(self, request, room_alias):

101	101	service.url,
102	102	room_alias.to_string(),
103	103	)
104		defer.returnValue((200, {}))
	104	return (200, {})
105	105	except InvalidClientCredentialsError:
106	106	# fallback to default user behaviour if they aren't an AS
107	107	pass

117	117	"User %s deleted alias %s", user.to_string(), room_alias.to_string()
118	118	)
119	119
120		defer.returnValue((200, {}))
	120	return (200, {})
121	121
122	122
123	123	class ClientDirectoryListServer(RestServlet):

135	135	if room is None:
136	136	raise NotFoundError("Unknown room")
137	137
138		defer.returnValue(
139		(200, {"visibility": "public" if room["is_public"] else "private"})
140		)
	138	return (200, {"visibility": "public" if room["is_public"] else "private"})
141	139
142	140	@defer.inlineCallbacks
143	141	def on_PUT(self, request, room_id):

150	148	requester, room_id, visibility
151	149	)
152	150
153		defer.returnValue((200, {}))
	151	return (200, {})
154	152
155	153	@defer.inlineCallbacks
156	154	def on_DELETE(self, request, room_id):

160	158	requester, room_id, "private"
161	159	)
162	160
163		defer.returnValue((200, {}))
	161	return (200, {})
164	162
165	163
166	164	class ClientAppserviceDirectoryListServer(RestServlet):

194	192	requester.app_service.id, network_id, room_id, visibility
195	193	)
196	194
197		defer.returnValue((200, {}))
	195	return (200, {})

+3

-3

synapse/rest/client/v1/events.py less more

66	66	is_guest=is_guest,
67	67	)
68	68
69		defer.returnValue((200, chunk))
	69	return (200, chunk)
70	70
71	71	def on_OPTIONS(self, request):
72	72	return (200, {})

90	90	time_now = self.clock.time_msec()
91	91	if event:
92	92	event = yield self._event_serializer.serialize_event(event, time_now)
93		defer.returnValue((200, event))
	93	return (200, event)
94	94	else:
95		defer.returnValue((404, "Event not found."))
	95	return (404, "Event not found.")
96	96
97	97
98	98	def register_servlets(hs, http_server):

+1

-1

synapse/rest/client/v1/initial_sync.py less more

41	41	include_archived=include_archived,
42	42	)
43	43
44		defer.returnValue((200, content))
	44	return (200, content)
45	45
46	46
47	47	def register_servlets(hs, http_server):

+7

-7

synapse/rest/client/v1/login.py less more

151	151	well_known_data = self._well_known_builder.get_well_known()
152	152	if well_known_data:
153	153	result["well_known"] = well_known_data
154		defer.returnValue((200, result))
	154	return (200, result)
155	155
156	156	@defer.inlineCallbacks
157	157	def _do_other_login(self, login_submission):

211	211	result = yield self._register_device_with_callback(
212	212	canonical_user_id, login_submission, callback_3pid
213	213	)
214		defer.returnValue(result)
	214	return result
215	215
216	216	# No password providers were able to handle this 3pid
217	217	# Check local store

240	240	result = yield self._register_device_with_callback(
241	241	canonical_user_id, login_submission, callback
242	242	)
243		defer.returnValue(result)
	243	return result
244	244
245	245	@defer.inlineCallbacks
246	246	def _register_device_with_callback(self, user_id, login_submission, callback=None):

272	272	if callback is not None:
273	273	yield callback(result)
274	274
275		defer.returnValue(result)
	275	return result
276	276
277	277	@defer.inlineCallbacks
278	278	def do_token_login(self, login_submission):

283	283	)
284	284
285	285	result = yield self._register_device_with_callback(user_id, login_submission)
286		defer.returnValue(result)
	286	return result
287	287
288	288	@defer.inlineCallbacks
289	289	def do_jwt_login(self, login_submission):

320	320	result = yield self._register_device_with_callback(
321	321	registered_user_id, login_submission
322	322	)
323		defer.returnValue(result)
	323	return result
324	324
325	325
326	326	class BaseSSORedirectServlet(RestServlet):

394	394	# even if that's being used old-http style to signal end-of-data
395	395	body = pde.response
396	396	result = yield self.handle_cas_response(request, body, client_redirect_url)
397		defer.returnValue(result)
	397	return result
398	398
399	399	def handle_cas_response(self, request, cas_response_body, client_redirect_url):
400	400	user, attributes = self.parse_cas_response(cas_response_body)

+2

-2

synapse/rest/client/v1/logout.py less more

48	48	requester.user.to_string(), requester.device_id
49	49	)
50	50
51		defer.returnValue((200, {}))
	51	return (200, {})
52	52
53	53
54	54	class LogoutAllRestServlet(RestServlet):

74	74	# .. and then delete any access tokens which weren't associated with
75	75	# devices.
76	76	yield self._auth_handler.delete_access_tokens_for_user(user_id)
77		defer.returnValue((200, {}))
	77	return (200, {})
78	78
79	79
80	80	def register_servlets(hs, http_server):

+2

-2

synapse/rest/client/v1/presence.py less more

55	55	state = yield self.presence_handler.get_state(target_user=user)
56	56	state = format_user_presence_state(state, self.clock.time_msec())
57	57
58		defer.returnValue((200, state))
	58	return (200, state)
59	59
60	60	@defer.inlineCallbacks
61	61	def on_PUT(self, request, user_id):

87	87	if self.hs.config.use_presence:
88	88	yield self.presence_handler.set_state(user, state)
89	89
90		defer.returnValue((200, {}))
	90	return (200, {})
91	91
92	92	def on_OPTIONS(self, request):
93	93	return (200, {})

+7

-7

synapse/rest/client/v1/profile.py less more

47	47	if displayname is not None:
48	48	ret["displayname"] = displayname
49	49
50		defer.returnValue((200, ret))
	50	return (200, ret)
51	51
52	52	@defer.inlineCallbacks
53	53	def on_PUT(self, request, user_id):

60	60	try:
61	61	new_name = content["displayname"]
62	62	except Exception:
63		defer.returnValue((400, "Unable to parse name"))
	63	return (400, "Unable to parse name")
64	64
65	65	yield self.profile_handler.set_displayname(user, requester, new_name, is_admin)
66	66
67		defer.returnValue((200, {}))
	67	return (200, {})
68	68
69	69	def on_OPTIONS(self, request, user_id):
70	70	return (200, {})

97	97	if avatar_url is not None:
98	98	ret["avatar_url"] = avatar_url
99	99
100		defer.returnValue((200, ret))
	100	return (200, ret)
101	101
102	102	@defer.inlineCallbacks
103	103	def on_PUT(self, request, user_id):

109	109	try:
110	110	new_name = content["avatar_url"]
111	111	except Exception:
112		defer.returnValue((400, "Unable to parse name"))
	112	return (400, "Unable to parse name")
113	113
114	114	yield self.profile_handler.set_avatar_url(user, requester, new_name, is_admin)
115	115
116		defer.returnValue((200, {}))
	116	return (200, {})
117	117
118	118	def on_OPTIONS(self, request, user_id):
119	119	return (200, {})

149	149	if avatar_url is not None:
150	150	ret["avatar_url"] = avatar_url
151	151
152		defer.returnValue((200, ret))
	152	return (200, ret)
153	153
154	154
155	155	def register_servlets(hs, http_server):

+5

-5

synapse/rest/client/v1/push_rule.py less more

68	68	if "attr" in spec:
69	69	yield self.set_rule_attr(user_id, spec, content)
70	70	self.notify_user(user_id)
71		defer.returnValue((200, {}))
	71	return (200, {})
72	72
73	73	if spec["rule_id"].startswith("."):
74	74	# Rule ids starting with '.' are reserved for server default rules.

105	105	except RuleNotFoundException as e:
106	106	raise SynapseError(400, str(e))
107	107
108		defer.returnValue((200, {}))
	108	return (200, {})
109	109
110	110	@defer.inlineCallbacks
111	111	def on_DELETE(self, request, path):

122	122	try:
123	123	yield self.store.delete_push_rule(user_id, namespaced_rule_id)
124	124	self.notify_user(user_id)
125		defer.returnValue((200, {}))
	125	return (200, {})
126	126	except StoreError as e:
127	127	if e.code == 404:
128	128	raise NotFoundError()

150	150	)
151	151
152	152	if path[0] == "":
153		defer.returnValue((200, rules))
	153	return (200, rules)
154	154	elif path[0] == "global":
155	155	result = _filter_ruleset_with_path(rules["global"], path[1:])
156		defer.returnValue((200, result))
	156	return (200, result)
157	157	else:
158	158	raise UnrecognizedRequestError()
159	159

+4

-4

synapse/rest/client/v1/pusher.py less more

61	61	if k not in allowed_keys:
62	62	del p[k]
63	63
64		defer.returnValue((200, {"pushers": pushers}))
	64	return (200, {"pushers": pushers})
65	65
66	66	def on_OPTIONS(self, _):
67	67	return 200, {}

93	93	yield self.pusher_pool.remove_pusher(
94	94	content["app_id"], content["pushkey"], user_id=user.to_string()
95	95	)
96		defer.returnValue((200, {}))
	96	return (200, {})
97	97
98	98	assert_params_in_dict(
99	99	content,

142	142
143	143	self.notifier.on_new_replication_data()
144	144
145		defer.returnValue((200, {}))
	145	return (200, {})
146	146
147	147	def on_OPTIONS(self, _):
148	148	return 200, {}

189	189	)
190	190	request.write(PushersRemoveRestServlet.SUCCESS_HTML)
191	191	finish_request(request)
192		defer.returnValue(None)
	192	return None
193	193
194	194	def on_OPTIONS(self, _):
195	195	return 200, {}

+33

-25

synapse/rest/client/v1/room.py less more

90	90	requester, self.get_room_config(request)
91	91	)
92	92
93		defer.returnValue((200, info))
	93	return (200, info)
94	94
95	95	def get_room_config(self, request):
96	96	user_supplied_config = parse_json_object_from_request(request)

172	172
173	173	if format == "event":
174	174	event = format_event_for_client_v2(data.get_dict())
175		defer.returnValue((200, event))
	175	return (200, event)
176	176	elif format == "content":
177		defer.returnValue((200, data.get_dict()["content"]))
	177	return (200, data.get_dict()["content"])
178	178
179	179	@defer.inlineCallbacks
180	180	def on_PUT(self, request, room_id, event_type, state_key, txn_id=None):

209	209	ret = {}
210	210	if event:
211	211	ret = {"event_id": event.event_id}
212		defer.returnValue((200, ret))
	212	return (200, ret)
213	213
214	214
215	215	# TODO: Needs unit testing for generic events + feedback

243	243	requester, event_dict, txn_id=txn_id
244	244	)
245	245
246		defer.returnValue((200, {"event_id": event.event_id}))
	246	return (200, {"event_id": event.event_id})
247	247
248	248	def on_GET(self, request, room_id, event_type, txn_id):
249	249	return (200, "Not implemented")

306	306	third_party_signed=content.get("third_party_signed", None),
307	307	)
308	308
309		defer.returnValue((200, {"room_id": room_id}))
	309	return (200, {"room_id": room_id})
310	310
311	311	def on_PUT(self, request, room_identifier, txn_id):
312	312	return self.txns.fetch_or_execute_request(

359	359	limit=limit, since_token=since_token
360	360	)
361	361
362		defer.returnValue((200, data))
	362	return (200, data)
363	363
364	364	@defer.inlineCallbacks
365	365	def on_POST(self, request):

404	404	network_tuple=network_tuple,
405	405	)
406	406
407		defer.returnValue((200, data))
	407	return (200, data)
408	408
409	409
410	410	# TODO: Needs unit testing

455	455	continue
456	456	chunk.append(event)
457	457
458		defer.returnValue((200, {"chunk": chunk}))
	458	return (200, {"chunk": chunk})
459	459
460	460
461	461	# deprecated in favour of /members?membership=join?

476	476	requester, room_id
477	477	)
478	478
479		defer.returnValue((200, {"joined": users_with_profile}))
	479	return (200, {"joined": users_with_profile})
480	480
481	481
482	482	# TODO: Needs better unit testing

509	509	event_filter=event_filter,
510	510	)
511	511
512		defer.returnValue((200, msgs))
	512	return (200, msgs)
513	513
514	514
515	515	# TODO: Needs unit testing

530	530	user_id=requester.user.to_string(),
531	531	is_guest=requester.is_guest,
532	532	)
533		defer.returnValue((200, events))
	533	return (200, events)
534	534
535	535
536	536	# TODO: Needs unit testing

549	549	content = yield self.initial_sync_handler.room_initial_sync(
550	550	room_id=room_id, requester=requester, pagin_config=pagination_config
551	551	)
552		defer.returnValue((200, content))
	552	return (200, content)
553	553
554	554
555	555	class RoomEventServlet(RestServlet):

567	567	@defer.inlineCallbacks
568	568	def on_GET(self, request, room_id, event_id):
569	569	requester = yield self.auth.get_user_by_req(request, allow_guest=True)
570		event = yield self.event_handler.get_event(requester.user, room_id, event_id)
	570	try:
	571	event = yield self.event_handler.get_event(
	572	requester.user, room_id, event_id
	573	)
	574	except AuthError:
	575	# This endpoint is supposed to return a 404 when the requester does
	576	# not have permission to access the event
	577	# https://matrix.org/docs/spec/client_server/r0.5.0#get-matrix-client-r0-rooms-roomid-event-eventid
	578	raise SynapseError(404, "Event not found.", errcode=Codes.NOT_FOUND)
571	579
572	580	time_now = self.clock.time_msec()
573	581	if event:
574	582	event = yield self._event_serializer.serialize_event(event, time_now)
575		defer.returnValue((200, event))
576		else:
577		defer.returnValue((404, "Event not found."))
	583	return (200, event)
	584
	585	return SynapseError(404, "Event not found.", errcode=Codes.NOT_FOUND)
578	586
579	587
580	588	class RoomEventContextServlet(RestServlet):

624	632	results["state"], time_now
625	633	)
626	634
627		defer.returnValue((200, results))
	635	return (200, results)
628	636
629	637
630	638	class RoomForgetRestServlet(TransactionRestServlet):

643	651
644	652	yield self.room_member_handler.forget(user=requester.user, room_id=room_id)
645	653
646		defer.returnValue((200, {}))
	654	return (200, {})
647	655
648	656	def on_PUT(self, request, room_id, txn_id):
649	657	return self.txns.fetch_or_execute_request(

693	701	requester,
694	702	txn_id,
695	703	)
696		defer.returnValue((200, {}))
	704	return (200, {})
697	705	return
698	706
699	707	target = requester.user

720	728	if membership_action == "join":
721	729	return_value["room_id"] = room_id
722	730
723		defer.returnValue((200, return_value))
	731	return (200, return_value)
724	732
725	733	def _has_3pid_invite_keys(self, content):
726	734	for key in {"id_server", "medium", "address"}:

762	770	txn_id=txn_id,
763	771	)
764	772
765		defer.returnValue((200, {"event_id": event.event_id}))
	773	return (200, {"event_id": event.event_id})
766	774
767	775	def on_PUT(self, request, room_id, event_id, txn_id):
768	776	return self.txns.fetch_or_execute_request(

807	815	target_user=target_user, auth_user=requester.user, room_id=room_id
808	816	)
809	817
810		defer.returnValue((200, {}))
	818	return (200, {})
811	819
812	820
813	821	class SearchRestServlet(RestServlet):

829	837	requester.user, content, batch
830	838	)
831	839
832		defer.returnValue((200, results))
	840	return (200, results)
833	841
834	842
835	843	class JoinedRoomsRestServlet(RestServlet):

845	853	requester = yield self.auth.get_user_by_req(request, allow_guest=True)
846	854
847	855	room_ids = yield self.store.get_rooms_for_user(requester.user.to_string())
848		defer.returnValue((200, {"joined_rooms": list(room_ids)}))
	856	return (200, {"joined_rooms": list(room_ids)})
849	857
850	858
851	859	def register_txn_path(servlet, regex_string, http_server, with_get=False):

+9

-11

synapse/rest/client/v1/voip.py less more

59	59	password = turnPassword
60	60
61	61	else:
62		defer.returnValue((200, {}))
	62	return (200, {})
63	63
64		defer.returnValue(
65		(
66		200,
67		{
68		"username": username,
69		"password": password,
70		"ttl": userLifetime / 1000,
71		"uris": turnUris,
72		},
73		)
	64	return (
	65	200,
	66	{
	67	"username": username,
	68	"password": password,
	69	"ttl": userLifetime / 1000,
	70	"uris": turnUris,
	71	},
74	72	)
75	73
76	74	def on_OPTIONS(self, request):

+16

-16

synapse/rest/client/v2_alpha/account.py less more

116	116	# Wrap the session id in a JSON object
117	117	ret = {"sid": sid}
118	118
119		defer.returnValue((200, ret))
	119	return (200, ret)
120	120
121	121	@defer.inlineCallbacks
122	122	def send_password_reset(self, email, client_secret, send_attempt, next_link=None):

148	148	# Check that the send_attempt is higher than previous attempts
149	149	if send_attempt <= last_send_attempt:
150	150	# If not, just return a success without sending an email
151		defer.returnValue(session_id)
	151	return session_id
152	152	else:
153	153	# An non-validated session does not exist yet.
154	154	# Generate a session id

184	184	token_expires,
185	185	)
186	186
187		defer.returnValue(session_id)
	187	return session_id
188	188
189	189
190	190	class MsisdnPasswordRequestTokenRestServlet(RestServlet):

220	220	raise SynapseError(400, "MSISDN not found", Codes.THREEPID_NOT_FOUND)
221	221
222	222	ret = yield self.identity_handler.requestMsisdnToken(**body)
223		defer.returnValue((200, ret))
	223	return (200, ret)
224	224
225	225
226	226	class PasswordResetSubmitTokenServlet(RestServlet):

278	278	request.setResponseCode(302)
279	279	request.setHeader("Location", next_link)
280	280	finish_request(request)
281		defer.returnValue(None)
	281	return None
282	282
283	283	# Otherwise show the success template
284	284	html = self.config.email_password_reset_success_html_content

294	294
295	295	request.write(html.encode("utf-8"))
296	296	finish_request(request)
297		defer.returnValue(None)
	297	return None
298	298
299	299	def load_jinja2_template(self, template_dir, template_filename, template_vars):
300	300	"""Loads a jinja2 template with variables to insert

329	329	)
330	330	response_code = 200 if valid else 400
331	331
332		defer.returnValue((response_code, {"success": valid}))
	332	return (response_code, {"success": valid})
333	333
334	334
335	335	class PasswordRestServlet(RestServlet):

398	398
399	399	yield self._set_password_handler.set_password(user_id, new_password, requester)
400	400
401		defer.returnValue((200, {}))
	401	return (200, {})
402	402
403	403	def on_OPTIONS(self, _):
404	404	return 200, {}

433	433	yield self._deactivate_account_handler.deactivate_account(
434	434	requester.user.to_string(), erase
435	435	)
436		defer.returnValue((200, {}))
	436	return (200, {})
437	437
438	438	yield self.auth_handler.validate_user_via_ui_auth(
439	439	requester, body, self.hs.get_ip_from_request(request)

446	446	else:
447	447	id_server_unbind_result = "no-support"
448	448
449		defer.returnValue((200, {"id_server_unbind_result": id_server_unbind_result}))
	449	return (200, {"id_server_unbind_result": id_server_unbind_result})
450	450
451	451
452	452	class EmailThreepidRequestTokenRestServlet(RestServlet):

480	480	raise SynapseError(400, "Email is already in use", Codes.THREEPID_IN_USE)
481	481
482	482	ret = yield self.identity_handler.requestEmailToken(**body)
483		defer.returnValue((200, ret))
	483	return (200, ret)
484	484
485	485
486	486	class MsisdnThreepidRequestTokenRestServlet(RestServlet):

515	515	raise SynapseError(400, "MSISDN is already in use", Codes.THREEPID_IN_USE)
516	516
517	517	ret = yield self.identity_handler.requestMsisdnToken(**body)
518		defer.returnValue((200, ret))
	518	return (200, ret)
519	519
520	520
521	521	class ThreepidRestServlet(RestServlet):

535	535
536	536	threepids = yield self.datastore.user_get_threepids(requester.user.to_string())
537	537
538		defer.returnValue((200, {"threepids": threepids}))
	538	return (200, {"threepids": threepids})
539	539
540	540	@defer.inlineCallbacks
541	541	def on_POST(self, request):

567	567	logger.debug("Binding threepid %s to %s", threepid, user_id)
568	568	yield self.identity_handler.bind_threepid(threePidCreds, user_id)
569	569
570		defer.returnValue((200, {}))
	570	return (200, {})
571	571
572	572
573	573	class ThreepidDeleteRestServlet(RestServlet):

602	602	else:
603	603	id_server_unbind_result = "no-support"
604	604
605		defer.returnValue((200, {"id_server_unbind_result": id_server_unbind_result}))
	605	return (200, {"id_server_unbind_result": id_server_unbind_result})
606	606
607	607
608	608	class WhoamiRestServlet(RestServlet):

616	616	def on_GET(self, request):
617	617	requester = yield self.auth.get_user_by_req(request)
618	618
619		defer.returnValue((200, {"user_id": requester.user.to_string()}))
	619	return (200, {"user_id": requester.user.to_string()})
620	620
621	621
622	622	def register_servlets(hs, http_server):

+4

-4

synapse/rest/client/v2_alpha/account_data.py less more

54	54
55	55	self.notifier.on_new_event("account_data_key", max_id, users=[user_id])
56	56
57		defer.returnValue((200, {}))
	57	return (200, {})
58	58
59	59	@defer.inlineCallbacks
60	60	def on_GET(self, request, user_id, account_data_type):

69	69	if event is None:
70	70	raise NotFoundError("Account data not found")
71	71
72		defer.returnValue((200, event))
	72	return (200, event)
73	73
74	74
75	75	class RoomAccountDataServlet(RestServlet):

111	111
112	112	self.notifier.on_new_event("account_data_key", max_id, users=[user_id])
113	113
114		defer.returnValue((200, {}))
	114	return (200, {})
115	115
116	116	@defer.inlineCallbacks
117	117	def on_GET(self, request, user_id, room_id, account_data_type):

126	126	if event is None:
127	127	raise NotFoundError("Room account data not found")
128	128
129		defer.returnValue((200, event))
	129	return (200, event)
130	130
131	131
132	132	def register_servlets(hs, http_server):

+15

-6

synapse/rest/client/v2_alpha/account_validity.py less more

41	41	self.hs = hs
42	42	self.account_activity_handler = hs.get_account_validity_handler()
43	43	self.auth = hs.get_auth()
	44	self.success_html = hs.config.account_validity.account_renewed_html_content
	45	self.failure_html = hs.config.account_validity.invalid_token_html_content
44	46
45	47	@defer.inlineCallbacks
46	48	def on_GET(self, request):

48	50	raise SynapseError(400, "Missing renewal token")
49	51	renewal_token = request.args[b"token"][0]
50	52
51		yield self.account_activity_handler.renew_account(renewal_token.decode("utf8"))
	53	token_valid = yield self.account_activity_handler.renew_account(
	54	renewal_token.decode("utf8")
	55	)
52	56
53		request.setResponseCode(200)
	57	if token_valid:
	58	status_code = 200
	59	response = self.success_html
	60	else:
	61	status_code = 404
	62	response = self.failure_html
	63
	64	request.setResponseCode(status_code)
54	65	request.setHeader(b"Content-Type", b"text/html; charset=utf-8")
55		request.setHeader(
56		b"Content-Length", b"%d" % (len(AccountValidityRenewServlet.SUCCESS_HTML),)
57		)
58		request.write(AccountValidityRenewServlet.SUCCESS_HTML)
	66	request.setHeader(b"Content-Length", b"%d" % (len(response),))
	67	request.write(response.encode("utf8"))
59	68	finish_request(request)
60	69	defer.returnValue(None)
61	70

+2

-2

synapse/rest/client/v2_alpha/auth.py less more

206	206	request.write(html_bytes)
207	207	finish_request(request)
208	208
209		defer.returnValue(None)
	209	return None
210	210	elif stagetype == LoginType.TERMS:
211	211	if ("session" not in request.args or len(request.args["session"])) == 0:
212	212	raise SynapseError(400, "No session supplied")

238	238
239	239	request.write(html_bytes)
240	240	finish_request(request)
241		defer.returnValue(None)
	241	return None
242	242	else:
243	243	raise SynapseError(404, "Unknown auth stage type")
244	244

+1

-1

synapse/rest/client/v2_alpha/capabilities.py less more

57	57	"m.change_password": {"enabled": change_password},
58	58	}
59	59	}
60		defer.returnValue((200, response))
	60	return (200, response)
61	61
62	62
63	63	def register_servlets(hs, http_server):

+5

-5

synapse/rest/client/v2_alpha/devices.py less more

47	47	devices = yield self.device_handler.get_devices_by_user(
48	48	requester.user.to_string()
49	49	)
50		defer.returnValue((200, {"devices": devices}))
	50	return (200, {"devices": devices})
51	51
52	52
53	53	class DeleteDevicesRestServlet(RestServlet):

90	90	yield self.device_handler.delete_devices(
91	91	requester.user.to_string(), body["devices"]
92	92	)
93		defer.returnValue((200, {}))
	93	return (200, {})
94	94
95	95
96	96	class DeviceRestServlet(RestServlet):

113	113	device = yield self.device_handler.get_device(
114	114	requester.user.to_string(), device_id
115	115	)
116		defer.returnValue((200, device))
	116	return (200, device)
117	117
118	118	@interactive_auth_handler
119	119	@defer.inlineCallbacks

136	136	)
137	137
138	138	yield self.device_handler.delete_device(requester.user.to_string(), device_id)
139		defer.returnValue((200, {}))
	139	return (200, {})
140	140
141	141	@defer.inlineCallbacks
142	142	def on_PUT(self, request, device_id):

146	146	yield self.device_handler.update_device(
147	147	requester.user.to_string(), device_id, body
148	148	)
149		defer.returnValue((200, {}))
	149	return (200, {})
150	150
151	151
152	152	def register_servlets(hs, http_server):

+2

-2

synapse/rest/client/v2_alpha/filter.py less more

55	55	user_localpart=target_user.localpart, filter_id=filter_id
56	56	)
57	57
58		defer.returnValue((200, filter.get_filter_json()))
	58	return (200, filter.get_filter_json())
59	59	except (KeyError, StoreError):
60	60	raise SynapseError(400, "No such filter", errcode=Codes.NOT_FOUND)
61	61

88	88	user_localpart=target_user.localpart, user_filter=content
89	89	)
90	90
91		defer.returnValue((200, {"filter_id": str(filter_id)}))
	91	return (200, {"filter_id": str(filter_id)})
92	92
93	93
94	94	def register_servlets(hs, http_server):

+32

-32

synapse/rest/client/v2_alpha/groups.py less more

46	46	group_id, requester_user_id
47	47	)
48	48
49		defer.returnValue((200, group_description))
	49	return (200, group_description)
50	50
51	51	@defer.inlineCallbacks
52	52	def on_POST(self, request, group_id):

58	58	group_id, requester_user_id, content
59	59	)
60	60
61		defer.returnValue((200, {}))
	61	return (200, {})
62	62
63	63
64	64	class GroupSummaryServlet(RestServlet):

82	82	group_id, requester_user_id
83	83	)
84	84
85		defer.returnValue((200, get_group_summary))
	85	return (200, get_group_summary)
86	86
87	87
88	88	class GroupSummaryRoomsCatServlet(RestServlet):

119	119	content=content,
120	120	)
121	121
122		defer.returnValue((200, resp))
	122	return (200, resp)
123	123
124	124	@defer.inlineCallbacks
125	125	def on_DELETE(self, request, group_id, category_id, room_id):

130	130	group_id, requester_user_id, room_id=room_id, category_id=category_id
131	131	)
132	132
133		defer.returnValue((200, resp))
	133	return (200, resp)
134	134
135	135
136	136	class GroupCategoryServlet(RestServlet):

156	156	group_id, requester_user_id, category_id=category_id
157	157	)
158	158
159		defer.returnValue((200, category))
	159	return (200, category)
160	160
161	161	@defer.inlineCallbacks
162	162	def on_PUT(self, request, group_id, category_id):

168	168	group_id, requester_user_id, category_id=category_id, content=content
169	169	)
170	170
171		defer.returnValue((200, resp))
	171	return (200, resp)
172	172
173	173	@defer.inlineCallbacks
174	174	def on_DELETE(self, request, group_id, category_id):

179	179	group_id, requester_user_id, category_id=category_id
180	180	)
181	181
182		defer.returnValue((200, resp))
	182	return (200, resp)
183	183
184	184
185	185	class GroupCategoriesServlet(RestServlet):

203	203	group_id, requester_user_id
204	204	)
205	205
206		defer.returnValue((200, category))
	206	return (200, category)
207	207
208	208
209	209	class GroupRoleServlet(RestServlet):

227	227	group_id, requester_user_id, role_id=role_id
228	228	)
229	229
230		defer.returnValue((200, category))
	230	return (200, category)
231	231
232	232	@defer.inlineCallbacks
233	233	def on_PUT(self, request, group_id, role_id):

239	239	group_id, requester_user_id, role_id=role_id, content=content
240	240	)
241	241
242		defer.returnValue((200, resp))
	242	return (200, resp)
243	243
244	244	@defer.inlineCallbacks
245	245	def on_DELETE(self, request, group_id, role_id):

250	250	group_id, requester_user_id, role_id=role_id
251	251	)
252	252
253		defer.returnValue((200, resp))
	253	return (200, resp)
254	254
255	255
256	256	class GroupRolesServlet(RestServlet):

274	274	group_id, requester_user_id
275	275	)
276	276
277		defer.returnValue((200, category))
	277	return (200, category)
278	278
279	279
280	280	class GroupSummaryUsersRoleServlet(RestServlet):

311	311	content=content,
312	312	)
313	313
314		defer.returnValue((200, resp))
	314	return (200, resp)
315	315
316	316	@defer.inlineCallbacks
317	317	def on_DELETE(self, request, group_id, role_id, user_id):

322	322	group_id, requester_user_id, user_id=user_id, role_id=role_id
323	323	)
324	324
325		defer.returnValue((200, resp))
	325	return (200, resp)
326	326
327	327
328	328	class GroupRoomServlet(RestServlet):

346	346	group_id, requester_user_id
347	347	)
348	348
349		defer.returnValue((200, result))
	349	return (200, result)
350	350
351	351
352	352	class GroupUsersServlet(RestServlet):

370	370	group_id, requester_user_id
371	371	)
372	372
373		defer.returnValue((200, result))
	373	return (200, result)
374	374
375	375
376	376	class GroupInvitedUsersServlet(RestServlet):

394	394	group_id, requester_user_id
395	395	)
396	396
397		defer.returnValue((200, result))
	397	return (200, result)
398	398
399	399
400	400	class GroupSettingJoinPolicyServlet(RestServlet):

419	419	group_id, requester_user_id, content
420	420	)
421	421
422		defer.returnValue((200, result))
	422	return (200, result)
423	423
424	424
425	425	class GroupCreateServlet(RestServlet):

449	449	group_id, requester_user_id, content
450	450	)
451	451
452		defer.returnValue((200, result))
	452	return (200, result)
453	453
454	454
455	455	class GroupAdminRoomsServlet(RestServlet):

476	476	group_id, requester_user_id, room_id, content
477	477	)
478	478
479		defer.returnValue((200, result))
	479	return (200, result)
480	480
481	481	@defer.inlineCallbacks
482	482	def on_DELETE(self, request, group_id, room_id):

487	487	group_id, requester_user_id, room_id
488	488	)
489	489
490		defer.returnValue((200, result))
	490	return (200, result)
491	491
492	492
493	493	class GroupAdminRoomsConfigServlet(RestServlet):

515	515	group_id, requester_user_id, room_id, config_key, content
516	516	)
517	517
518		defer.returnValue((200, result))
	518	return (200, result)
519	519
520	520
521	521	class GroupAdminUsersInviteServlet(RestServlet):

545	545	group_id, user_id, requester_user_id, config
546	546	)
547	547
548		defer.returnValue((200, result))
	548	return (200, result)
549	549
550	550
551	551	class GroupAdminUsersKickServlet(RestServlet):

572	572	group_id, user_id, requester_user_id, content
573	573	)
574	574
575		defer.returnValue((200, result))
	575	return (200, result)
576	576
577	577
578	578	class GroupSelfLeaveServlet(RestServlet):

597	597	group_id, requester_user_id, requester_user_id, content
598	598	)
599	599
600		defer.returnValue((200, result))
	600	return (200, result)
601	601
602	602
603	603	class GroupSelfJoinServlet(RestServlet):

622	622	group_id, requester_user_id, content
623	623	)
624	624
625		defer.returnValue((200, result))
	625	return (200, result)
626	626
627	627
628	628	class GroupSelfAcceptInviteServlet(RestServlet):

647	647	group_id, requester_user_id, content
648	648	)
649	649
650		defer.returnValue((200, result))
	650	return (200, result)
651	651
652	652
653	653	class GroupSelfUpdatePublicityServlet(RestServlet):

671	671	publicise = content["publicise"]
672	672	yield self.store.update_group_publicity(group_id, requester_user_id, publicise)
673	673
674		defer.returnValue((200, {}))
	674	return (200, {})
675	675
676	676
677	677	class PublicisedGroupsForUserServlet(RestServlet):

693	693
694	694	result = yield self.groups_handler.get_publicised_groups_for_user(user_id)
695	695
696		defer.returnValue((200, result))
	696	return (200, result)
697	697
698	698
699	699	class PublicisedGroupsForUsersServlet(RestServlet):

718	718
719	719	result = yield self.groups_handler.bulk_get_publicised_groups(user_ids)
720	720
721		defer.returnValue((200, result))
	721	return (200, result)
722	722
723	723
724	724	class GroupsForUserServlet(RestServlet):

740	740
741	741	result = yield self.groups_handler.get_joined_groups(requester_user_id)
742	742
743		defer.returnValue((200, result))
	743	return (200, result)
744	744
745	745
746	746	def register_servlets(hs, http_server):

+4

-4

synapse/rest/client/v2_alpha/keys.py less more

94	94	result = yield self.e2e_keys_handler.upload_keys_for_user(
95	95	user_id, device_id, body
96	96	)
97		defer.returnValue((200, result))
	97	return (200, result)
98	98
99	99
100	100	class KeyQueryServlet(RestServlet):

148	148	timeout = parse_integer(request, "timeout", 10 * 1000)
149	149	body = parse_json_object_from_request(request)
150	150	result = yield self.e2e_keys_handler.query_devices(body, timeout)
151		defer.returnValue((200, result))
	151	return (200, result)
152	152
153	153
154	154	class KeyChangesServlet(RestServlet):

188	188
189	189	results = yield self.device_handler.get_user_ids_changed(user_id, from_token)
190	190
191		defer.returnValue((200, results))
	191	return (200, results)
192	192
193	193
194	194	class OneTimeKeyServlet(RestServlet):

223	223	timeout = parse_integer(request, "timeout", 10 * 1000)
224	224	body = parse_json_object_from_request(request)
225	225	result = yield self.e2e_keys_handler.claim_one_time_keys(body, timeout)
226		defer.returnValue((200, result))
	226	return (200, result)
227	227
228	228
229	229	def register_servlets(hs, http_server):

+1

-3

synapse/rest/client/v2_alpha/notifications.py less more

87	87	returned_push_actions.append(returned_pa)
88	88	next_token = str(pa["stream_ordering"])
89	89
90		defer.returnValue(
91		(200, {"notifications": returned_push_actions, "next_token": next_token})
92		)
	90	return (200, {"notifications": returned_push_actions, "next_token": next_token})
93	91
94	92
95	93	def register_servlets(hs, http_server):

+8

-10

synapse/rest/client/v2_alpha/openid.py less more

82	82
83	83	yield self.store.insert_open_id_token(token, ts_valid_until_ms, user_id)
84	84
85		defer.returnValue(
86		(
87		200,
88		{
89		"access_token": token,
90		"token_type": "Bearer",
91		"matrix_server_name": self.server_name,
92		"expires_in": self.EXPIRES_MS / 1000,
93		},
94		)
	85	return (
	86	200,
	87	{
	88	"access_token": token,
	89	"token_type": "Bearer",
	90	"matrix_server_name": self.server_name,
	91	"expires_in": self.EXPIRES_MS / 1000,
	92	},
95	93	)
96	94
97	95

+1

-1

synapse/rest/client/v2_alpha/read_marker.py less more

58	58	event_id=read_marker_event_id,
59	59	)
60	60
61		defer.returnValue((200, {}))
	61	return (200, {})
62	62
63	63
64	64	def register_servlets(hs, http_server):

+1

-1

synapse/rest/client/v2_alpha/receipts.py less more

51	51	room_id, receipt_type, user_id=requester.user.to_string(), event_id=event_id
52	52	)
53	53
54		defer.returnValue((200, {}))
	54	return (200, {})
55	55
56	56
57	57	def register_servlets(hs, http_server):

+18

-20

synapse/rest/client/v2_alpha/register.py less more

94	94	raise SynapseError(400, "Email is already in use", Codes.THREEPID_IN_USE)
95	95
96	96	ret = yield self.identity_handler.requestEmailToken(**body)
97		defer.returnValue((200, ret))
	97	return (200, ret)
98	98
99	99
100	100	class MsisdnRegisterRequestTokenRestServlet(RestServlet):

137	137	)
138	138
139	139	ret = yield self.identity_handler.requestMsisdnToken(**body)
140		defer.returnValue((200, ret))
	140	return (200, ret)
141	141
142	142
143	143	class UsernameAvailabilityRestServlet(RestServlet):

177	177
178	178	yield self.registration_handler.check_username(username)
179	179
180		defer.returnValue((200, {"available": True}))
	180	return (200, {"available": True})
181	181
182	182
183	183	class RegisterRestServlet(RestServlet):

229	229
230	230	if kind == b"guest":
231	231	ret = yield self._do_guest_registration(body, address=client_addr)
232		defer.returnValue(ret)
	232	return ret
233	233	return
234	234	elif kind != b"user":
235	235	raise UnrecognizedRequestError(

281	281	result = yield self._do_appservice_registration(
282	282	desired_username, access_token, body
283	283	)
284		defer.returnValue((200, result)) # we throw for non 200 responses
	284	return (200, result) # we throw for non 200 responses
285	285	return
286	286
287	287	# for either shared secret or regular registration, downcase the

300	300	result = yield self._do_shared_secret_registration(
301	301	desired_username, desired_password, body
302	302	)
303		defer.returnValue((200, result)) # we throw for non 200 responses
	303	return (200, result) # we throw for non 200 responses
304	304	return
305	305
306	306	# == Normal User Registration == (everyone else)

499	499	bind_msisdn=params.get("bind_msisdn"),
500	500	)
501	501
502		defer.returnValue((200, return_dict))
	502	return (200, return_dict)
503	503
504	504	def on_OPTIONS(self, _):
505	505	return 200, {}

509	509	user_id = yield self.registration_handler.appservice_register(
510	510	username, as_token
511	511	)
512		defer.returnValue((yield self._create_registration_details(user_id, body)))
	512	return (yield self._create_registration_details(user_id, body))
513	513
514	514	@defer.inlineCallbacks
515	515	def _do_shared_secret_registration(self, username, password, body):

545	545	)
546	546
547	547	result = yield self._create_registration_details(user_id, body)
548		defer.returnValue(result)
	548	return result
549	549
550	550	@defer.inlineCallbacks
551	551	def _create_registration_details(self, user_id, params):

569	569	)
570	570
571	571	result.update({"access_token": access_token, "device_id": device_id})
572		defer.returnValue(result)
	572	return result
573	573
574	574	@defer.inlineCallbacks
575	575	def _do_guest_registration(self, params, address=None):

587	587	user_id, device_id, initial_display_name, is_guest=True
588	588	)
589	589
590		defer.returnValue(
591		(
592		200,
593		{
594		"user_id": user_id,
595		"device_id": device_id,
596		"access_token": access_token,
597		"home_server": self.hs.hostname,
598		},
599		)
	590	return (
	591	200,
	592	{
	593	"user_id": user_id,
	594	"device_id": device_id,
	595	"access_token": access_token,
	596	"home_server": self.hs.hostname,
	597	},
600	598	)
601	599
602	600

+4

-4

synapse/rest/client/v2_alpha/relations.py less more

117	117	requester, event_dict=event_dict, txn_id=txn_id
118	118	)
119	119
120		defer.returnValue((200, {"event_id": event.event_id}))
	120	return (200, {"event_id": event.event_id})
121	121
122	122
123	123	class RelationPaginationServlet(RestServlet):

197	197	return_value["chunk"] = events
198	198	return_value["original_event"] = original_event
199	199
200		defer.returnValue((200, return_value))
	200	return (200, return_value)
201	201
202	202
203	203	class RelationAggregationPaginationServlet(RestServlet):

269	269	to_token=to_token,
270	270	)
271	271
272		defer.returnValue((200, pagination_chunk.to_dict()))
	272	return (200, pagination_chunk.to_dict())
273	273
274	274
275	275	class RelationAggregationGroupPaginationServlet(RestServlet):

355	355	return_value = result.to_dict()
356	356	return_value["chunk"] = events
357	357
358		defer.returnValue((200, return_value))
	358	return (200, return_value)
359	359
360	360
361	361	def register_servlets(hs, http_server):

+1

-1

synapse/rest/client/v2_alpha/report_event.py less more

71	71	received_ts=self.clock.time_msec(),
72	72	)
73	73
74		defer.returnValue((200, {}))
	74	return (200, {})
75	75
76	76
77	77	def register_servlets(hs, http_server):

+7

-7

synapse/rest/client/v2_alpha/room_keys.py less more

134	134	body = {"rooms": {room_id: body}}
135	135
136	136	yield self.e2e_room_keys_handler.upload_room_keys(user_id, version, body)
137		defer.returnValue((200, {}))
	137	return (200, {})
138	138
139	139	@defer.inlineCallbacks
140	140	def on_GET(self, request, room_id, session_id):

217	217	else:
218	218	room_keys = room_keys["rooms"][room_id]
219	219
220		defer.returnValue((200, room_keys))
	220	return (200, room_keys)
221	221
222	222	@defer.inlineCallbacks
223	223	def on_DELETE(self, request, room_id, session_id):

241	241	yield self.e2e_room_keys_handler.delete_room_keys(
242	242	user_id, version, room_id, session_id
243	243	)
244		defer.returnValue((200, {}))
	244	return (200, {})
245	245
246	246
247	247	class RoomKeysNewVersionServlet(RestServlet):

292	292	info = parse_json_object_from_request(request)
293	293
294	294	new_version = yield self.e2e_room_keys_handler.create_version(user_id, info)
295		defer.returnValue((200, {"version": new_version}))
	295	return (200, {"version": new_version})
296	296
297	297	# we deliberately don't have a PUT /version, as these things really should
298	298	# be immutable to avoid people footgunning

337	337	except SynapseError as e:
338	338	if e.code == 404:
339	339	raise SynapseError(404, "No backup found", Codes.NOT_FOUND)
340		defer.returnValue((200, info))
	340	return (200, info)
341	341
342	342	@defer.inlineCallbacks
343	343	def on_DELETE(self, request, version):

357	357	user_id = requester.user.to_string()
358	358
359	359	yield self.e2e_room_keys_handler.delete_version(user_id, version)
360		defer.returnValue((200, {}))
	360	return (200, {})
361	361
362	362	@defer.inlineCallbacks
363	363	def on_PUT(self, request, version):

391	391	)
392	392
393	393	yield self.e2e_room_keys_handler.update_version(user_id, version, info)
394		defer.returnValue((200, {}))
	394	return (200, {})
395	395
396	396
397	397	def register_servlets(hs, http_server):

+1

-1

synapse/rest/client/v2_alpha/room_upgrade_rest_servlet.py less more

79	79
80	80	ret = {"replacement_room": new_room_id}
81	81
82		defer.returnValue((200, ret))
	82	return (200, ret)
83	83
84	84
85	85	def register_servlets(hs, http_server):

+1

-1

synapse/rest/client/v2_alpha/sendtodevice.py less more

59	59	)
60	60
61	61	response = (200, {})
62		defer.returnValue(response)
	62	return response
63	63
64	64
65	65	def register_servlets(hs, http_server):

+22

-26

synapse/rest/client/v2_alpha/sync.py less more

173	173	time_now, sync_result, requester.access_token_id, filter
174	174	)
175	175
176		defer.returnValue((200, response_content))
	176	return (200, response_content)
177	177
178	178	@defer.inlineCallbacks
179	179	def encode_response(self, time_now, sync_result, access_token_id, filter):

204	204	event_formatter,
205	205	)
206	206
207		defer.returnValue(
208		{
209		"account_data": {"events": sync_result.account_data},
210		"to_device": {"events": sync_result.to_device},
211		"device_lists": {
212		"changed": list(sync_result.device_lists.changed),
213		"left": list(sync_result.device_lists.left),
214		},
215		"presence": SyncRestServlet.encode_presence(
216		sync_result.presence, time_now
217		),
218		"rooms": {"join": joined, "invite": invited, "leave": archived},
219		"groups": {
220		"join": sync_result.groups.join,
221		"invite": sync_result.groups.invite,
222		"leave": sync_result.groups.leave,
223		},
224		"device_one_time_keys_count": sync_result.device_one_time_keys_count,
225		"next_batch": sync_result.next_batch.to_string(),
226		}
227		)
	207	return {
	208	"account_data": {"events": sync_result.account_data},
	209	"to_device": {"events": sync_result.to_device},
	210	"device_lists": {
	211	"changed": list(sync_result.device_lists.changed),
	212	"left": list(sync_result.device_lists.left),
	213	},
	214	"presence": SyncRestServlet.encode_presence(sync_result.presence, time_now),
	215	"rooms": {"join": joined, "invite": invited, "leave": archived},
	216	"groups": {
	217	"join": sync_result.groups.join,
	218	"invite": sync_result.groups.invite,
	219	"leave": sync_result.groups.leave,
	220	},
	221	"device_one_time_keys_count": sync_result.device_one_time_keys_count,
	222	"next_batch": sync_result.next_batch.to_string(),
	223	}
228	224
229	225	@staticmethod
230	226	def encode_presence(events, time_now):

272	268	event_formatter=event_formatter,
273	269	)
274	270
275		defer.returnValue(joined)
	271	return joined
276	272
277	273	@defer.inlineCallbacks
278	274	def encode_invited(self, rooms, time_now, token_id, event_formatter):

308	304	invited_state.append(invite)
309	305	invited[room.room_id] = {"invite_state": {"events": invited_state}}
310	306
311		defer.returnValue(invited)
	307	return invited
312	308
313	309	@defer.inlineCallbacks
314	310	def encode_archived(self, rooms, time_now, token_id, event_fields, event_formatter):

341	337	event_formatter=event_formatter,
342	338	)
343	339
344		defer.returnValue(joined)
	340	return joined
345	341
346	342	@defer.inlineCallbacks
347	343	def encode_room(

413	409	result["unread_notifications"] = room.unread_notifications
414	410	result["summary"] = room.summary
415	411
416		defer.returnValue(result)
	412	return result
417	413
418	414
419	415	def register_servlets(hs, http_server):

+3

-3

synapse/rest/client/v2_alpha/tags.py less more

44	44
45	45	tags = yield self.store.get_tags_for_room(user_id, room_id)
46	46
47		defer.returnValue((200, {"tags": tags}))
	47	return (200, {"tags": tags})
48	48
49	49
50	50	class TagServlet(RestServlet):

75	75
76	76	self.notifier.on_new_event("account_data_key", max_id, users=[user_id])
77	77
78		defer.returnValue((200, {}))
	78	return (200, {})
79	79
80	80	@defer.inlineCallbacks
81	81	def on_DELETE(self, request, user_id, room_id, tag):

87	87
88	88	self.notifier.on_new_event("account_data_key", max_id, users=[user_id])
89	89
90		defer.returnValue((200, {}))
	90	return (200, {})
91	91
92	92
93	93	def register_servlets(hs, http_server):

+5

-5

synapse/rest/client/v2_alpha/thirdparty.py less more

39	39	yield self.auth.get_user_by_req(request, allow_guest=True)
40	40
41	41	protocols = yield self.appservice_handler.get_3pe_protocols()
42		defer.returnValue((200, protocols))
	42	return (200, protocols)
43	43
44	44
45	45	class ThirdPartyProtocolServlet(RestServlet):

59	59	only_protocol=protocol
60	60	)
61	61	if protocol in protocols:
62		defer.returnValue((200, protocols[protocol]))
	62	return (200, protocols[protocol])
63	63	else:
64		defer.returnValue((404, {"error": "Unknown protocol"}))
	64	return (404, {"error": "Unknown protocol"})
65	65
66	66
67	67	class ThirdPartyUserServlet(RestServlet):

84	84	ThirdPartyEntityKind.USER, protocol, fields
85	85	)
86	86
87		defer.returnValue((200, results))
	87	return (200, results)
88	88
89	89
90	90	class ThirdPartyLocationServlet(RestServlet):

107	107	ThirdPartyEntityKind.LOCATION, protocol, fields
108	108	)
109	109
110		defer.returnValue((200, results))
	110	return (200, results)
111	111
112	112
113	113	def register_servlets(hs, http_server):

+2

-2

synapse/rest/client/v2_alpha/user_directory.py less more

59	59	user_id = requester.user.to_string()
60	60
61	61	if not self.hs.config.user_directory_search_enabled:
62		defer.returnValue((200, {"limited": False, "results": []}))
	62	return (200, {"limited": False, "results": []})
63	63
64	64	body = parse_json_object_from_request(request)
65	65

75	75	user_id, search_term, limit
76	76	)
77	77
78		defer.returnValue((200, results))
	78	return (200, results)
79	79
80	80
81	81	def register_servlets(hs, http_server):

+15

-11

synapse/rest/media/v1/media_repository.py less more

32	32	RequestSendFailed,
33	33	SynapseError,
34	34	)
	35	from synapse.config._base import ConfigError
35	36	from synapse.logging.context import defer_to_thread
36	37	from synapse.metrics.background_process_metrics import run_as_background_process
37	38	from synapse.util.async_helpers import Linearizer

170	171
171	172	yield self._generate_thumbnails(None, media_id, media_id, media_type)
172	173
173		defer.returnValue("mxc://%s/%s" % (self.server_name, media_id))
	174	return "mxc://%s/%s" % (self.server_name, media_id)
174	175
175	176	@defer.inlineCallbacks
176	177	def get_local_media(self, request, media_id, name):

281	282	with responder:
282	283	pass
283	284
284		defer.returnValue(media_info)
	285	return media_info
285	286
286	287	@defer.inlineCallbacks
287	288	def _get_remote_media_impl(self, server_name, media_id):

316	317
317	318	responder = yield self.media_storage.fetch_media(file_info)
318	319	if responder:
319		defer.returnValue((responder, media_info))
	320	return (responder, media_info)
320	321
321	322	# Failed to find the file anywhere, lets download it.
322	323
323	324	media_info = yield self._download_remote_file(server_name, media_id, file_id)
324	325
325	326	responder = yield self.media_storage.fetch_media(file_info)
326		defer.returnValue((responder, media_info))
	327	return (responder, media_info)
327	328
328	329	@defer.inlineCallbacks
329	330	def _download_remote_file(self, server_name, media_id, file_id):

420	421
421	422	yield self._generate_thumbnails(server_name, media_id, file_id, media_type)
422	423
423		defer.returnValue(media_info)
	424	return media_info
424	425
425	426	def _get_thumbnail_requirements(self, media_type):
426	427	return self.thumbnail_requirements.get(media_type, ())

499	500	media_id, t_width, t_height, t_type, t_method, t_len
500	501	)
501	502
502		defer.returnValue(output_path)
	503	return output_path
503	504
504	505	@defer.inlineCallbacks
505	506	def generate_remote_exact_thumbnail(

553	554	t_len,
554	555	)
555	556
556		defer.returnValue(output_path)
	557	return output_path
557	558
558	559	@defer.inlineCallbacks
559	560	def _generate_thumbnails(

666	667	media_id, t_width, t_height, t_type, t_method, t_len
667	668	)
668	669
669		defer.returnValue({"width": m_width, "height": m_height})
	670	return {"width": m_width, "height": m_height}
670	671
671	672	@defer.inlineCallbacks
672	673	def delete_old_remote_media(self, before_ts):

703	704	yield self.store.delete_remote_media(origin, media_id)
704	705	deleted += 1
705	706
706		defer.returnValue({"deleted": deleted})
	707	return {"deleted": deleted}
707	708
708	709
709	710	class MediaRepositoryResource(Resource):

752	753	"""
753	754
754	755	def __init__(self, hs):
755		Resource.__init__(self)
756
	756	# If we're not configured to use it, raise if we somehow got here.
	757	if not hs.config.can_load_media_repo:
	758	raise ConfigError("Synapse is not configured to use a media repo.")
	759
	760	super().__init__()
757	761	media_repo = hs.get_media_repository()
758	762
759	763	self.putChild(b"upload", UploadResource(hs, media_repo))

+7

-7

synapse/rest/media/v1/media_storage.py less more

68	68	)
69	69	yield finish_cb()
70	70
71		defer.returnValue(fname)
	71	return fname
72	72
73	73	@contextlib.contextmanager
74	74	def store_into_file(self, file_info):

142	142	path = self._file_info_to_path(file_info)
143	143	local_path = os.path.join(self.local_media_directory, path)
144	144	if os.path.exists(local_path):
145		defer.returnValue(FileResponder(open(local_path, "rb")))
	145	return FileResponder(open(local_path, "rb"))
146	146
147	147	for provider in self.storage_providers:
148	148	res = yield provider.fetch(path, file_info)
149	149	if res:
150		defer.returnValue(res)
151
152		defer.returnValue(None)
	150	return res
	151
	152	return None
153	153
154	154	@defer.inlineCallbacks
155	155	def ensure_media_is_in_local_cache(self, file_info):

165	165	path = self._file_info_to_path(file_info)
166	166	local_path = os.path.join(self.local_media_directory, path)
167	167	if os.path.exists(local_path):
168		defer.returnValue(local_path)
	168	return local_path
169	169
170	170	dirname = os.path.dirname(local_path)
171	171	if not os.path.exists(dirname):

180	180	)
181	181	yield res.write_to_consumer(consumer)
182	182	yield consumer.wait()
183		defer.returnValue(local_path)
	183	return local_path
184	184
185	185	raise Exception("file could not be found")
186	186

+16

-18

synapse/rest/media/v1/preview_url_resource.py less more

181	181	og = cache_result["og"]
182	182	if isinstance(og, six.text_type):
183	183	og = og.encode("utf8")
184		defer.returnValue(og)
	184	return og
185	185	return
186	186
187	187	media_info = yield self._download_url(url, user)

283	283	media_info["created_ts"],
284	284	)
285	285
286		defer.returnValue(jsonog)
	286	return jsonog
287	287
288	288	@defer.inlineCallbacks
289	289	def _download_url(self, url, user):

353	353	# therefore not expire it.
354	354	raise
355	355
356		defer.returnValue(
357		{
358		"media_type": media_type,
359		"media_length": length,
360		"download_name": download_name,
361		"created_ts": time_now_ms,
362		"filesystem_id": file_id,
363		"filename": fname,
364		"uri": uri,
365		"response_code": code,
366		# FIXME: we should calculate a proper expiration based on the
367		# Cache-Control and Expire headers. But for now, assume 1 hour.
368		"expires": 60 * 60 * 1000,
369		"etag": headers["ETag"][0] if "ETag" in headers else None,
370		}
371		)
	356	return {
	357	"media_type": media_type,
	358	"media_length": length,
	359	"download_name": download_name,
	360	"created_ts": time_now_ms,
	361	"filesystem_id": file_id,
	362	"filename": fname,
	363	"uri": uri,
	364	"response_code": code,
	365	# FIXME: we should calculate a proper expiration based on the
	366	# Cache-Control and Expire headers. But for now, assume 1 hour.
	367	"expires": 60 * 60 * 1000,
	368	"etag": headers["ETag"][0] if "ETag" in headers else None,
	369	}
372	370
373	371	def _start_expire_url_cache_data(self):
374	372	return run_as_background_process(

+1

-1

synapse/server_notices/resource_limits_server_notices.py less more

192	192	if event_id in referenced_events:
193	193	referenced_events.remove(event.event_id)
194	194
195		defer.returnValue((currently_blocked, referenced_events))
	195	return (currently_blocked, referenced_events)

+3

-3

synapse/server_notices/server_notices_manager.py less more

85	85	res = yield self._event_creation_handler.create_and_send_nonmember_event(
86	86	requester, event_dict, ratelimit=False
87	87	)
88		defer.returnValue(res)
	88	return res
89	89
90	90	@cachedInlineCallbacks()
91	91	def get_notice_room_for_user(self, user_id):

119	119	# we found a room which our user shares with the system notice
120	120	# user
121	121	logger.info("Using room %s", room.room_id)
122		defer.returnValue(room.room_id)
	122	return room.room_id
123	123
124	124	# apparently no existing notice room: create a new one
125	125	logger.info("Creating server notices room for %s", user_id)

157	157	self._notifier.on_new_event("account_data_key", max_id, users=[user_id])
158	158
159	159	logger.info("Created server notices room %s for %s", room_id, user_id)
160		defer.returnValue(room_id)
	160	return room_id

+18

-20

synapse/state/__init__.py less more

134	134	event = None
135	135	if event_id:
136	136	event = yield self.store.get_event(event_id, allow_none=True)
137		defer.returnValue(event)
	137	return event
138	138	return
139	139
140	140	state_map = yield self.store.get_events(

144	144	key: state_map[e_id] for key, e_id in iteritems(state) if e_id in state_map
145	145	}
146	146
147		defer.returnValue(state)
	147	return state
148	148
149	149	@defer.inlineCallbacks
150	150	def get_current_state_ids(self, room_id, latest_event_ids=None):

168	168	ret = yield self.resolve_state_groups_for_events(room_id, latest_event_ids)
169	169	state = ret.state
170	170
171		defer.returnValue(state)
	171	return state
172	172
173	173	@defer.inlineCallbacks
174	174	def get_current_users_in_room(self, room_id, latest_event_ids=None):

188	188	logger.debug("calling resolve_state_groups from get_current_users_in_room")
189	189	entry = yield self.resolve_state_groups_for_events(room_id, latest_event_ids)
190	190	joined_users = yield self.store.get_joined_users_from_state(room_id, entry)
191		defer.returnValue(joined_users)
	191	return joined_users
192	192
193	193	@defer.inlineCallbacks
194	194	def get_current_hosts_in_room(self, room_id, latest_event_ids=None):

197	197	logger.debug("calling resolve_state_groups from get_current_hosts_in_room")
198	198	entry = yield self.resolve_state_groups_for_events(room_id, latest_event_ids)
199	199	joined_hosts = yield self.store.get_joined_hosts(room_id, entry)
200		defer.returnValue(joined_hosts)
	200	return joined_hosts
201	201
202	202	@defer.inlineCallbacks
203	203	def compute_event_context(self, event, old_state=None):

240	240	prev_state_ids=prev_state_ids,
241	241	)
242	242
243		defer.returnValue(context)
	243	return context
244	244
245	245	if old_state:
246	246	# We already have the state, so we don't need to calculate it.

274	274	prev_state_ids=prev_state_ids,
275	275	)
276	276
277		defer.returnValue(context)
	277	return context
278	278
279	279	logger.debug("calling resolve_state_groups from compute_event_context")
280	280

342	342	delta_ids=delta_ids,
343	343	)
344	344
345		defer.returnValue(context)
	345	return context
346	346
347	347	@defer.inlineCallbacks
348	348	def resolve_state_groups_for_events(self, room_id, event_ids):

367	367	state_groups_ids = yield self.store.get_state_groups_ids(room_id, event_ids)
368	368
369	369	if len(state_groups_ids) == 0:
370		defer.returnValue(_StateCacheEntry(state={}, state_group=None))
	370	return _StateCacheEntry(state={}, state_group=None)
371	371	elif len(state_groups_ids) == 1:
372	372	name, state_list = list(state_groups_ids.items()).pop()
373	373
374	374	prev_group, delta_ids = yield self.store.get_state_group_delta(name)
375	375
376		defer.returnValue(
377		_StateCacheEntry(
378		state=state_list,
379		state_group=name,
380		prev_group=prev_group,
381		delta_ids=delta_ids,
382		)
	376	return _StateCacheEntry(
	377	state=state_list,
	378	state_group=name,
	379	prev_group=prev_group,
	380	delta_ids=delta_ids,
383	381	)
384	382
385	383	room_version = yield self.store.get_room_version(room_id)

391	389	None,
392	390	state_res_store=StateResolutionStore(self.store),
393	391	)
394		defer.returnValue(result)
	392	return result
395	393
396	394	@defer.inlineCallbacks
397	395	def resolve_events(self, room_version, state_sets, event):

414	412
415	413	new_state = {key: state_map[ev_id] for key, ev_id in iteritems(new_state)}
416	414
417		defer.returnValue(new_state)
	415	return new_state
418	416
419	417
420	418	class StateResolutionHandler(object):

478	476	if self._state_cache is not None:
479	477	cache = self._state_cache.get(group_names, None)
480	478	if cache:
481		defer.returnValue(cache)
	479	return cache
482	480
483	481	logger.info(
484	482	"Resolving state for %s with %d groups", room_id, len(state_groups_ids)

524	522	if self._state_cache is not None:
525	523	self._state_cache[group_names] = cache
526	524
527		defer.returnValue(cache)
	525	return cache
528	526
529	527
530	528	def _make_state_cache_entry(new_state, state_groups_ids):

+3

-5

synapse/state/v1.py less more

54	54	a map from (type, state_key) to event_id.
55	55	"""
56	56	if len(state_sets) == 1:
57		defer.returnValue(state_sets[0])
	57	return state_sets[0]
58	58
59	59	unconflicted_state, conflicted_state = _seperate(state_sets)
60	60

96	96	state_map_new = yield state_map_factory(new_needed_events)
97	97	state_map.update(state_map_new)
98	98
99		defer.returnValue(
100		_resolve_with_state(
101		unconflicted_state, conflicted_state, auth_events, state_map
102		)
	99	return _resolve_with_state(
	100	unconflicted_state, conflicted_state, auth_events, state_map
103	101	)
104	102
105	103

+13

-13

synapse/state/v2.py less more

62	62	unconflicted_state, conflicted_state = _seperate(state_sets)
63	63
64	64	if not conflicted_state:
65		defer.returnValue(unconflicted_state)
	65	return unconflicted_state
66	66
67	67	logger.debug("%d conflicted state entries", len(conflicted_state))
68	68	logger.debug("Calculating auth chain difference")

136	136
137	137	logger.debug("done")
138	138
139		defer.returnValue(resolved_state)
	139	return resolved_state
140	140
141	141
142	142	@defer.inlineCallbacks

167	167	aev = yield _get_event(aid, event_map, state_res_store)
168	168	if (aev.type, aev.state_key) == (EventTypes.Create, ""):
169	169	if aev.content.get("creator") == event.sender:
170		defer.returnValue(100)
	170	return 100
171	171	break
172		defer.returnValue(0)
	172	return 0
173	173
174	174	level = pl.content.get("users", {}).get(event.sender)
175	175	if level is None:
176	176	level = pl.content.get("users_default", 0)
177	177
178	178	if level is None:
179		defer.returnValue(0)
	179	return 0
180	180	else:
181		defer.returnValue(int(level))
	181	return int(level)
182	182
183	183
184	184	@defer.inlineCallbacks

223	223	intersection = set(auth_sets[0]).intersection(*auth_sets[1:])
224	224	union = set().union(*auth_sets)
225	225
226		defer.returnValue(union - intersection)
	226	return union - intersection
227	227
228	228
229	229	def _seperate(state_sets):

342	342	it = lexicographical_topological_sort(graph, key=_get_power_order)
343	343	sorted_events = list(it)
344	344
345		defer.returnValue(sorted_events)
	345	return sorted_events
346	346
347	347
348	348	@defer.inlineCallbacks

395	395	except AuthError:
396	396	pass
397	397
398		defer.returnValue(resolved_state)
	398	return resolved_state
399	399
400	400
401	401	@defer.inlineCallbacks

438	438
439	439	event_ids.sort(key=lambda ev_id: order_map[ev_id])
440	440
441		defer.returnValue(event_ids)
	441	return event_ids
442	442
443	443
444	444	@defer.inlineCallbacks

461	461	while event:
462	462	depth = mainline_map.get(event.event_id)
463	463	if depth is not None:
464		defer.returnValue(depth)
	464	return depth
465	465
466	466	auth_events = event.auth_event_ids()
467	467	event = None

473	473	break
474	474
475	475	# Didn't find a power level auth event, so we just return 0
476		defer.returnValue(0)
	476	return 0
477	477
478	478
479	479	@defer.inlineCallbacks

492	492	if event_id not in event_map:
493	493	events = yield state_res_store.get_events([event_id], allow_rejected=True)
494	494	event_map.update(events)
495		defer.returnValue(event_map[event_id])
	495	return event_map[event_id]
496	496
497	497
498	498	def lexicographical_topological_sort(graph, key):

+2

-2

synapse/static/index.html less more

47	47	</div>
48	48	<h1>It works! Synapse is running</h1>
49	49	<p>Your Synapse server is listening on this port and is ready for messages.</p>
50		<p>To use this server you'll need <a href="https://matrix.org/docs/projects/try-matrix-now.html#clients" target="_blank">a Matrix client</a>.
	50	<p>To use this server you'll need <a href="https://matrix.org/docs/projects/try-matrix-now.html#clients" target="_blank" rel="noopener noreferrer">a Matrix client</a>.
51	51	</p>
52	52	<p>Welcome to the Matrix universe :)</p>
53	53	<hr>
54	54	<p>
55	55	<small>
56		<a href="https://matrix.org" target="_blank">
	56	<a href="https://matrix.org" target="_blank" rel="noopener noreferrer">
57	57	matrix.org
58	58	</a>
59	59	</small>

+4

-4

synapse/storage/__init__.py less more

468	468	return self._simple_select_list(
469	469	table="users",
470	470	keyvalues={},
471		retcols=["name", "password_hash", "is_guest", "admin"],
	471	retcols=["name", "password_hash", "is_guest", "admin", "user_type"],
472	472	desc="get_users",
473	473	)
474	474

493	493	orderby=order,
494	494	start=start,
495	495	limit=limit,
496		retcols=["name", "password_hash", "is_guest", "admin"],
	496	retcols=["name", "password_hash", "is_guest", "admin", "user_type"],
497	497	)
498	498	count = yield self.runInteraction("get_users_paginate", self.get_user_count_txn)
499	499	retval = {"users": users, "total": count}
500		defer.returnValue(retval)
	500	return retval
501	501
502	502	def search_users(self, term):
503	503	"""Function to search users list for one or more users with

513	513	table="users",
514	514	term=term,
515	515	col="name",
516		retcols=["name", "password_hash", "is_guest", "admin"],
	516	retcols=["name", "password_hash", "is_guest", "admin", "user_type"],
517	517	desc="search_users",
518	518	)
519	519

+23

-9

synapse/storage/_base.py less more

85	85	class LoggingTransaction(object):
86	86	"""An object that almost-transparently proxies for the 'txn' object
87	87	passed to the constructor. Adds logging and metrics to the .execute()
88		method."""
	88	method.
	89
	90	Args:
	91	txn: The database transcation object to wrap.
	92	name (str): The name of this transactions for logging.
	93	database_engine (Sqlite3Engine\|PostgresEngine)
	94	after_callbacks(list\|None): A list that callbacks will be appended to
	95	that have been added by `call_after` which should be run on
	96	successful completion of the transaction. None indicates that no
	97	callbacks should be allowed to be scheduled to run.
	98	exception_callbacks(list\|None): A list that callbacks will be appended
	99	to that have been added by `call_on_exception` which should be run
	100	if transaction ends with an error. None indicates that no callbacks
	101	should be allowed to be scheduled to run.
	102	"""
89	103
90	104	__slots__ = [
91	105	"txn",

96	110	]
97	111
98	112	def __init__(
99		self, txn, name, database_engine, after_callbacks, exception_callbacks
	113	self, txn, name, database_engine, after_callbacks=None, exception_callbacks=None
100	114	):
101	115	object.__setattr__(self, "txn", txn)
102	116	object.__setattr__(self, "name", name)

498	512	after_callback(after_args, *after_kwargs)
499	513	raise
500	514
501		defer.returnValue(result)
	515	return result
502	516
503	517	@defer.inlineCallbacks
504	518	def runWithConnection(self, func, args, *kwargs):

538	552	with PreserveLoggingContext():
539	553	result = yield self._db_pool.runWithConnection(inner_func, args, *kwargs)
540	554
541		defer.returnValue(result)
	555	return result
542	556
543	557	@staticmethod
544	558	def cursor_to_dict(cursor):

600	614	# a cursor after we receive an error from the db.
601	615	if not or_ignore:
602	616	raise
603		defer.returnValue(False)
604		defer.returnValue(True)
	617	return False
	618	return True
605	619
606	620	@staticmethod
607	621	def _simple_insert_txn(txn, table, values):

693	707	insertion_values,
694	708	lock=lock,
695	709	)
696		defer.returnValue(result)
	710	return result
697	711	except self.database_engine.module.IntegrityError as e:
698	712	attempts += 1
699	713	if attempts >= 5:

1106	1120	results = []
1107	1121
1108	1122	if not iterable:
1109		defer.returnValue(results)
	1123	return results
1110	1124
1111	1125	# iterables can not be sliced, so convert it to a list first
1112	1126	it_list = list(iterable)

1127	1141
1128	1142	results.extend(rows)
1129	1143
1130		defer.returnValue(results)
	1144	return results
1131	1145
1132	1146	@classmethod
1133	1147	def _simple_select_many_txn(cls, txn, table, column, iterable, keyvalues, retcols):

+7

-9

synapse/storage/account_data.py less more

110	110	)
111	111
112	112	if result:
113		defer.returnValue(json.loads(result))
	113	return json.loads(result)
114	114	else:
115		defer.returnValue(None)
	115	return None
116	116
117	117	@cached(num_args=2)
118	118	def get_account_data_for_room(self, user_id, room_id):

263	263	on_invalidate=cache_context.invalidate,
264	264	)
265	265	if not ignored_account_data:
266		defer.returnValue(False)
267
268		defer.returnValue(
269		ignored_user_id in ignored_account_data.get("ignored_users", {})
270		)
	266	return False
	267
	268	return ignored_user_id in ignored_account_data.get("ignored_users", {})
271	269
272	270
273	271	class AccountDataStore(AccountDataWorkerStore):

331	329	)
332	330
333	331	result = self._account_data_id_gen.get_current_token()
334		defer.returnValue(result)
	332	return result
335	333
336	334	@defer.inlineCallbacks
337	335	def add_account_data_for_user(self, user_id, account_data_type, content):

372	370	)
373	371
374	372	result = self._account_data_id_gen.get_current_token()
375		defer.returnValue(result)
	373	return result
376	374
377	375	def _update_max_stream_id(self, next_id):
378	376	"""Update the max stream_id

+6

-8

synapse/storage/appservice.py less more

144	144	for service in as_list:
145	145	if service.id == res["as_id"]:
146	146	services.append(service)
147		defer.returnValue(services)
	147	return services
148	148
149	149	@defer.inlineCallbacks
150	150	def get_appservice_state(self, service):

163	163	desc="get_appservice_state",
164	164	)
165	165	if result:
166		defer.returnValue(result.get("state"))
	166	return result.get("state")
167	167	return
168		defer.returnValue(None)
	168	return None
169	169
170	170	def set_appservice_state(self, service, state):
171	171	"""Set the application service state.

297	297	)
298	298
299	299	if not entry:
300		defer.returnValue(None)
	300	return None
301	301
302	302	event_ids = json.loads(entry["event_ids"])
303	303
304	304	events = yield self.get_events_as_list(event_ids)
305	305
306		defer.returnValue(
307		AppServiceTransaction(service=service, id=entry["txn_id"], events=events)
308		)
	306	return AppServiceTransaction(service=service, id=entry["txn_id"], events=events)
309	307
310	308	def _get_last_txn(self, txn, service_id):
311	309	txn.execute(

359	357
360	358	events = yield self.get_events_as_list(event_ids)
361	359
362		defer.returnValue((upper_bound, events))
	360	return (upper_bound, events)
363	361
364	362
365	363	class ApplicationServiceTransactionStore(ApplicationServiceTransactionWorkerStore):

+11

-11

synapse/storage/background_updates.py less more

114	114	" Unscheduling background update task."
115	115	)
116	116	self._all_done = True
117		defer.returnValue(None)
	117	return None
118	118
119	119	@defer.inlineCallbacks
120	120	def has_completed_background_updates(self):

126	126	# if we've previously determined that there is nothing left to do, that
127	127	# is easy
128	128	if self._all_done:
129		defer.returnValue(True)
	129	return True
130	130
131	131	# obviously, if we have things in our queue, we're not done.
132	132	if self._background_update_queue:
133		defer.returnValue(False)
	133	return False
134	134
135	135	# otherwise, check if there are updates to be run. This is important,
136	136	# as we may be running on a worker which doesn't perform the bg updates

143	143	)
144	144	if not updates:
145	145	self._all_done = True
146		defer.returnValue(True)
147
148		defer.returnValue(False)
	146	return True
	147
	148	return False
149	149
150	150	@defer.inlineCallbacks
151	151	def do_next_background_update(self, desired_duration_ms):

172	172
173	173	if not self._background_update_queue:
174	174	# no work left to do
175		defer.returnValue(None)
	175	return None
176	176
177	177	# pop from the front, and add back to the back
178	178	update_name = self._background_update_queue.pop(0)
179	179	self._background_update_queue.append(update_name)
180	180
181	181	res = yield self._do_background_update(update_name, desired_duration_ms)
182		defer.returnValue(res)
	182	return res
183	183
184	184	@defer.inlineCallbacks
185	185	def _do_background_update(self, update_name, desired_duration_ms):

230	230
231	231	performance.update(items_updated, duration_ms)
232	232
233		defer.returnValue(len(self._background_update_performance))
	233	return len(self._background_update_performance)
234	234
235	235	def register_background_update_handler(self, update_name, update_handler):
236	236	"""Register a handler for doing a background update.

265	265	@defer.inlineCallbacks
266	266	def noop_update(progress, batch_size):
267	267	yield self._end_background_update(update_name)
268		defer.returnValue(1)
	268	return 1
269	269
270	270	self.register_background_update_handler(update_name, noop_update)
271	271

369	369	logger.info("Adding index %s to %s", index_name, table)
370	370	yield self.runWithConnection(runner)
371	371	yield self._end_background_update(update_name)
372		defer.returnValue(1)
	372	return 1
373	373
374	374	self.register_background_update_handler(update_name, updater)
375	375

+13

-15

synapse/storage/client_ips.py less more

103	103
104	104	yield self.runWithConnection(f)
105	105	yield self._end_background_update("user_ips_drop_nonunique_index")
106		defer.returnValue(1)
	106	return 1
107	107
108	108	@defer.inlineCallbacks
109	109	def _analyze_user_ip(self, progress, batch_size):

120	120
121	121	yield self._end_background_update("user_ips_analyze")
122	122
123		defer.returnValue(1)
	123	return 1
124	124
125	125	@defer.inlineCallbacks
126	126	def _remove_user_ip_dupes(self, progress, batch_size):

290	290	if last:
291	291	yield self._end_background_update("user_ips_remove_dupes")
292	292
293		defer.returnValue(batch_size)
	293	return batch_size
294	294
295	295	@defer.inlineCallbacks
296	296	def insert_client_ip(

400	400	"device_id": did,
401	401	"last_seen": last_seen,
402	402	}
403		defer.returnValue(ret)
	403	return ret
404	404
405	405	@classmethod
406	406	def _get_last_client_ip_by_device_txn(cls, txn, user_id, device_id, retcols):

460	460	((row["access_token"], row["ip"]), (row["user_agent"], row["last_seen"]))
461	461	for row in rows
462	462	)
463		defer.returnValue(
464		list(
465		{
466		"access_token": access_token,
467		"ip": ip,
468		"user_agent": user_agent,
469		"last_seen": last_seen,
470		}
471		for (access_token, ip), (user_agent, last_seen) in iteritems(results)
472		)
473		)
	463	return list(
	464	{
	465	"access_token": access_token,
	466	"ip": ip,
	467	"user_agent": user_agent,
	468	"last_seen": last_seen,
	469	}
	470	for (access_token, ip), (user_agent, last_seen) in iteritems(results)
	471	)

+5

-5

synapse/storage/deviceinbox.py less more

91	91	user_id, last_deleted_stream_id
92	92	)
93	93	if not has_changed:
94		defer.returnValue(0)
	94	return 0
95	95
96	96	def delete_messages_for_device_txn(txn):
97	97	sql = (

114	114	last_deleted_stream_id, up_to_stream_id
115	115	)
116	116
117		defer.returnValue(count)
	117	return count
118	118
119	119	def get_new_device_msgs_for_remote(
120	120	self, destination, last_stream_id, current_stream_id, limit

262	262	destination, stream_id
263	263	)
264	264
265		defer.returnValue(self._device_inbox_id_gen.get_current_token())
	265	return self._device_inbox_id_gen.get_current_token()
266	266
267	267	@defer.inlineCallbacks
268	268	def add_messages_from_remote_to_device_inbox(

311	311	for user_id in local_messages_by_user_then_device.keys():
312	312	self._device_inbox_stream_cache.entity_has_changed(user_id, stream_id)
313	313
314		defer.returnValue(stream_id)
	314	return stream_id
315	315
316	316	def _add_messages_to_local_device_inbox_txn(
317	317	self, txn, stream_id, messages_by_user_then_device

425	425
426	426	yield self._end_background_update(self.DEVICE_INBOX_STREAM_ID)
427	427
428		defer.returnValue(1)
	428	return 1

+16

-16

synapse/storage/devices.py less more

70	70	desc="get_devices_by_user",
71	71	)
72	72
73		defer.returnValue({d["device_id"]: d for d in devices})
	73	return {d["device_id"]: d for d in devices}
74	74
75	75	@defer.inlineCallbacks
76	76	def get_devices_by_remote(self, destination, from_stream_id, limit):

87	87	destination, int(from_stream_id)
88	88	)
89	89	if not has_changed:
90		defer.returnValue((now_stream_id, []))
	90	return (now_stream_id, [])
91	91
92	92	# We retrieve n+1 devices from the list of outbound pokes where n is
93	93	# our outbound device update limit. We then check if the very last

110	110
111	111	# Return an empty list if there are no updates
112	112	if not updates:
113		defer.returnValue((now_stream_id, []))
	113	return (now_stream_id, [])
114	114
115	115	# if we have exceeded the limit, we need to exclude any results with the
116	116	# same stream_id as the last row.

146	146	# skip that stream_id and return an empty list, and continue with the next
147	147	# stream_id next time.
148	148	if not query_map:
149		defer.returnValue((stream_id_cutoff, []))
	149	return (stream_id_cutoff, [])
150	150
151	151	results = yield self._get_device_update_edus_by_remote(
152	152	destination, from_stream_id, query_map
153	153	)
154	154
155		defer.returnValue((now_stream_id, results))
	155	return (now_stream_id, results)
156	156
157	157	def _get_devices_by_remote_txn(
158	158	self, txn, destination, from_stream_id, now_stream_id, limit

231	231
232	232	results.append(result)
233	233
234		defer.returnValue(results)
	234	return results
235	235
236	236	def _get_last_device_update_for_remote_user(
237	237	self, destination, user_id, from_stream_id

329	329	else:
330	330	results[user_id] = yield self._get_cached_devices_for_user(user_id)
331	331
332		defer.returnValue((user_ids_not_in_cache, results))
	332	return (user_ids_not_in_cache, results)
333	333
334	334	@cachedInlineCallbacks(num_args=2, tree=True)
335	335	def _get_cached_user_device(self, user_id, device_id):

339	339	retcol="content",
340	340	desc="_get_cached_user_device",
341	341	)
342		defer.returnValue(db_to_json(content))
	342	return db_to_json(content)
343	343
344	344	@cachedInlineCallbacks()
345	345	def _get_cached_devices_for_user(self, user_id):

349	349	retcols=("device_id", "content"),
350	350	desc="_get_cached_devices_for_user",
351	351	)
352		defer.returnValue(
353		{device["device_id"]: db_to_json(device["content"]) for device in devices}
354		)
	352	return {
	353	device["device_id"]: db_to_json(device["content"]) for device in devices
	354	}
355	355
356	356	def get_devices_with_keys_by_user(self, user_id):
357	357	"""Get all devices (with any device keys) for a user

481	481	results = {user_id: None for user_id in user_ids}
482	482	results.update({row["user_id"]: row["stream_id"] for row in rows})
483	483
484		defer.returnValue(results)
	484	return results
485	485
486	486
487	487	class DeviceStore(DeviceWorkerStore, BackgroundUpdateStore):

542	542	"""
543	543	key = (user_id, device_id)
544	544	if self.device_id_exists_cache.get(key, None):
545		defer.returnValue(False)
	545	return False
546	546
547	547	try:
548	548	inserted = yield self._simple_insert(

556	556	or_ignore=True,
557	557	)
558	558	self.device_id_exists_cache.prefill(key, True)
559		defer.returnValue(inserted)
	559	return inserted
560	560	except Exception as e:
561	561	logger.error(
562	562	"store_device with device_id=%s(%r) user_id=%s(%r)"

779	779	hosts,
780	780	stream_id,
781	781	)
782		defer.returnValue(stream_id)
	782	return stream_id
783	783
784	784	def _add_device_change_txn(self, txn, user_id, device_ids, hosts, stream_id):
785	785	now = self._clock.time_msec()

888	888
889	889	yield self.runWithConnection(f)
890	890	yield self._end_background_update(DROP_DEVICE_LIST_STREAMS_NON_UNIQUE_INDEXES)
891		defer.returnValue(1)
	891	return 1

+5

-5

synapse/storage/directory.py less more

45	45	)
46	46
47	47	if not room_id:
48		defer.returnValue(None)
	48	return None
49	49	return
50	50
51	51	servers = yield self._simple_select_onecol(

56	56	)
57	57
58	58	if not servers:
59		defer.returnValue(None)
	59	return None
60	60	return
61	61
62		defer.returnValue(RoomAliasMapping(room_id, room_alias.to_string(), servers))
	62	return RoomAliasMapping(room_id, room_alias.to_string(), servers)
63	63
64	64	def get_room_alias_creator(self, room_alias):
65	65	return self._simple_select_one_onecol(

124	124	raise SynapseError(
125	125	409, "Room alias %s already exists" % room_alias.to_string()
126	126	)
127		defer.returnValue(ret)
	127	return ret
128	128
129	129	@defer.inlineCallbacks
130	130	def delete_room_alias(self, room_alias):

132	132	"delete_room_alias", self._delete_room_alias_txn, room_alias
133	133	)
134	134
135		defer.returnValue(room_id)
	135	return room_id
136	136
137	137	def _delete_room_alias_txn(self, txn, room_alias):
138	138	txn.execute(

+3

-3

synapse/storage/e2e_room_keys.py less more

60	60
61	61	row["session_data"] = json.loads(row["session_data"])
62	62
63		defer.returnValue(row)
	63	return row
64	64
65	65	@defer.inlineCallbacks
66	66	def set_e2e_room_key(self, user_id, version, room_id, session_id, room_key):

117	117	try:
118	118	version = int(version)
119	119	except ValueError:
120		defer.returnValue({"rooms": {}})
	120	return {"rooms": {}}
121	121
122	122	keyvalues = {"user_id": user_id, "version": version}
123	123	if room_id:

150	150	"session_data": json.loads(row["session_data"]),
151	151	}
152	152
153		defer.returnValue(sessions)
	153	return sessions
154	154
155	155	@defer.inlineCallbacks
156	156	def delete_e2e_room_keys(self, user_id, version, room_id=None, session_id=None):

+3

-5

synapse/storage/end_to_end_keys.py less more

40	40	dict containing "key_json", "device_display_name".
41	41	"""
42	42	if not query_list:
43		defer.returnValue({})
	43	return {}
44	44
45	45	results = yield self.runInteraction(
46	46	"get_e2e_device_keys",

54	54	for device_id, device_info in iteritems(device_keys):
55	55	device_info["keys"] = db_to_json(device_info.pop("key_json"))
56	56
57		defer.returnValue(results)
	57	return results
58	58
59	59	def _get_e2e_device_keys_txn(
60	60	self, txn, query_list, include_all_devices=False, include_deleted_devices=False

129	129	desc="add_e2e_one_time_keys_check",
130	130	)
131	131
132		defer.returnValue(
133		{(row["algorithm"], row["key_id"]): row["key_json"] for row in rows}
134		)
	132	return {(row["algorithm"], row["key_id"]): row["key_json"] for row in rows}
135	133
136	134	@defer.inlineCallbacks
137	135	def add_e2e_one_time_keys(self, user_id, device_id, time_now, new_keys):

+6

-6

synapse/storage/event_federation.py less more

130	130	)
131	131
132	132	if not rows:
133		defer.returnValue(0)
	133	return 0
134	134	else:
135		defer.returnValue(max(row["depth"] for row in rows))
	135	return max(row["depth"] for row in rows)
136	136
137	137	def _get_oldest_events_in_room_txn(self, txn, room_id):
138	138	return self._simple_select_onecol_txn(

168	168	# make sure that we don't completely ignore the older events.
169	169	res = res[0:5] + random.sample(res[5:], 5)
170	170
171		defer.returnValue(res)
	171	return res
172	172
173	173	def get_latest_event_ids_and_hashes_in_room(self, room_id):
174	174	"""

410	410	limit,
411	411	)
412	412	events = yield self.get_events_as_list(ids)
413		defer.returnValue(events)
	413	return events
414	414
415	415	def _get_missing_events(self, txn, room_id, earliest_events, latest_events, limit):
416	416

462	462	desc="get_successor_events",
463	463	)
464	464
465		defer.returnValue([row["event_id"] for row in rows])
	465	return [row["event_id"] for row in rows]
466	466
467	467
468	468	class EventFederationStore(EventFederationWorkerStore):

653	653	if not result:
654	654	yield self._end_background_update(self.EVENT_AUTH_STATE_ONLY)
655	655
656		defer.returnValue(batch_size)
	656	return batch_size

+8

-10

synapse/storage/event_push_actions.py less more

78	78	db_conn.cursor(),
79	79	name="_find_stream_orderings_for_times_txn",
80	80	database_engine=self.database_engine,
81		after_callbacks=[],
82		exception_callbacks=[],
83	81	)
84	82	self._find_stream_orderings_for_times_txn(cur)
85	83	cur.close()

101	99	user_id,
102	100	last_read_event_id,
103	101	)
104		defer.returnValue(ret)
	102	return ret
105	103
106	104	def _get_unread_counts_by_receipt_txn(
107	105	self, txn, room_id, user_id, last_read_event_id

179	177	return [r[0] for r in txn]
180	178
181	179	ret = yield self.runInteraction("get_push_action_users_in_range", f)
182		defer.returnValue(ret)
	180	return ret
183	181
184	182	@defer.inlineCallbacks
185	183	def get_unread_push_actions_for_user_in_range_for_http(

280	278
281	279	# Take only up to the limit. We have to stop at the limit because
282	280	# one of the subqueries may have hit the limit.
283		defer.returnValue(notifs[:limit])
	281	return notifs[:limit]
284	282
285	283	@defer.inlineCallbacks
286	284	def get_unread_push_actions_for_user_in_range_for_email(

381	379	notifs.sort(key=lambda r: -(r["received_ts"] or 0))
382	380
383	381	# Now return the first `limit`
384		defer.returnValue(notifs[:limit])
	382	return notifs[:limit]
385	383
386	384	def get_if_maybe_push_in_range_for_user(self, user_id, min_stream_ordering):
387	385	"""A fast check to see if there might be something to push for the

478	476	keyvalues={"event_id": event_id},
479	477	desc="remove_push_actions_from_staging",
480	478	)
481		defer.returnValue(res)
	479	return res
482	480	except Exception:
483	481	# this method is called from an exception handler, so propagating
484	482	# another exception here really isn't helpful - there's nothing

733	731	push_actions = yield self.runInteraction("get_push_actions_for_user", f)
734	732	for pa in push_actions:
735	733	pa["actions"] = _deserialize_action(pa["actions"], pa["highlight"])
736		defer.returnValue(push_actions)
	734	return push_actions
737	735
738	736	@defer.inlineCallbacks
739	737	def get_time_of_last_push_action_before(self, stream_ordering):

750	748	return txn.fetchone()
751	749
752	750	result = yield self.runInteraction("get_time_of_last_push_action_before", f)
753		defer.returnValue(result[0] if result else None)
	751	return result[0] if result else None
754	752
755	753	@defer.inlineCallbacks
756	754	def get_latest_push_action_stream_ordering(self):

759	757	return txn.fetchone()
760	758
761	759	result = yield self.runInteraction("get_latest_push_action_stream_ordering", f)
762		defer.returnValue(result[0] or 0)
	760	return result[0] or 0
763	761
764	762	def _remove_push_actions_for_event_id_txn(self, txn, room_id, event_id):
765	763	# Sad that we have to blow away the cache for the whole room here

+182

-168

synapse/storage/events.py less more

222	222	except self.database_engine.module.IntegrityError:
223	223	logger.exception("IntegrityError, retrying.")
224	224	res = yield func(self, args, delete_existing=True, *kwargs)
225		defer.returnValue(res)
	225	return res
226	226
227	227	return f
228	228

308	308
309	309	max_persisted_id = yield self._stream_id_gen.get_current_token()
310	310
311		defer.returnValue(max_persisted_id)
	311	return max_persisted_id
312	312
313	313	@defer.inlineCallbacks
314	314	@log_function

333	333	yield make_deferred_yieldable(deferred)
334	334
335	335	max_persisted_id = yield self._stream_id_gen.get_current_token()
336		defer.returnValue((event.internal_metadata.stream_ordering, max_persisted_id))
	336	return (event.internal_metadata.stream_ordering, max_persisted_id)
337	337
338	338	def _maybe_start_persisting(self, room_id):
339	339	@defer.inlineCallbacks

363	363	if not events_and_contexts:
364	364	return
365	365
366		if backfilled:
367		stream_ordering_manager = self._backfill_id_gen.get_next_mult(
368		len(events_and_contexts)
369		)
370		else:
371		stream_ordering_manager = self._stream_id_gen.get_next_mult(
372		len(events_and_contexts)
373		)
374
375		with stream_ordering_manager as stream_orderings:
376		for (event, context), stream in zip(events_and_contexts, stream_orderings):
377		event.internal_metadata.stream_ordering = stream
378
379		chunks = [
380		events_and_contexts[x : x + 100]
381		for x in range(0, len(events_and_contexts), 100)
382		]
383
384		for chunk in chunks:
385		# We can't easily parallelize these since different chunks
386		# might contain the same event. :(
387
388		# NB: Assumes that we are only persisting events for one room
389		# at a time.
390
391		# map room_id->list[event_ids] giving the new forward
392		# extremities in each room
393		new_forward_extremeties = {}
394
395		# map room_id->(type,state_key)->event_id tracking the full
396		# state in each room after adding these events.
397		# This is simply used to prefill the get_current_state_ids
398		# cache
399		current_state_for_room = {}
400
401		# map room_id->(to_delete, to_insert) where to_delete is a list
402		# of type/state keys to remove from current state, and to_insert
403		# is a map (type,key)->event_id giving the state delta in each
404		# room
405		state_delta_for_room = {}
406
407		if not backfilled:
408		with Measure(self._clock, "_calculate_state_and_extrem"):
409		# Work out the new "current state" for each room.
410		# We do this by working out what the new extremities are and then
411		# calculating the state from that.
412		events_by_room = {}
413		for event, context in chunk:
414		events_by_room.setdefault(event.room_id, []).append(
415		(event, context)
	366	chunks = [
	367	events_and_contexts[x : x + 100]
	368	for x in range(0, len(events_and_contexts), 100)
	369	]
	370
	371	for chunk in chunks:
	372	# We can't easily parallelize these since different chunks
	373	# might contain the same event. :(
	374
	375	# NB: Assumes that we are only persisting events for one room
	376	# at a time.
	377
	378	# map room_id->list[event_ids] giving the new forward
	379	# extremities in each room
	380	new_forward_extremeties = {}
	381
	382	# map room_id->(type,state_key)->event_id tracking the full
	383	# state in each room after adding these events.
	384	# This is simply used to prefill the get_current_state_ids
	385	# cache
	386	current_state_for_room = {}
	387
	388	# map room_id->(to_delete, to_insert) where to_delete is a list
	389	# of type/state keys to remove from current state, and to_insert
	390	# is a map (type,key)->event_id giving the state delta in each
	391	# room
	392	state_delta_for_room = {}
	393
	394	if not backfilled:
	395	with Measure(self._clock, "_calculate_state_and_extrem"):
	396	# Work out the new "current state" for each room.
	397	# We do this by working out what the new extremities are and then
	398	# calculating the state from that.
	399	events_by_room = {}
	400	for event, context in chunk:
	401	events_by_room.setdefault(event.room_id, []).append(
	402	(event, context)
	403	)
	404
	405	for room_id, ev_ctx_rm in iteritems(events_by_room):
	406	latest_event_ids = yield self.get_latest_event_ids_in_room(
	407	room_id
	408	)
	409	new_latest_event_ids = yield self._calculate_new_extremities(
	410	room_id, ev_ctx_rm, latest_event_ids
	411	)
	412
	413	latest_event_ids = set(latest_event_ids)
	414	if new_latest_event_ids == latest_event_ids:
	415	# No change in extremities, so no change in state
	416	continue
	417
	418	# there should always be at least one forward extremity.
	419	# (except during the initial persistence of the send_join
	420	# results, in which case there will be no existing
	421	# extremities, so we'll `continue` above and skip this bit.)
	422	assert new_latest_event_ids, "No forward extremities left!"
	423
	424	new_forward_extremeties[room_id] = new_latest_event_ids
	425
	426	len_1 = (
	427	len(latest_event_ids) == 1
	428	and len(new_latest_event_ids) == 1
	429	)
	430	if len_1:
	431	all_single_prev_not_state = all(
	432	len(event.prev_event_ids()) == 1
	433	and not event.is_state()
	434	for event, ctx in ev_ctx_rm
416	435	)
417
418		for room_id, ev_ctx_rm in iteritems(events_by_room):
419		latest_event_ids = yield self.get_latest_event_ids_in_room(
420		room_id
	436	# Don't bother calculating state if they're just
	437	# a long chain of single ancestor non-state events.
	438	if all_single_prev_not_state:
	439	continue
	440
	441	state_delta_counter.inc()
	442	if len(new_latest_event_ids) == 1:
	443	state_delta_single_event_counter.inc()
	444
	445	# This is a fairly handwavey check to see if we could
	446	# have guessed what the delta would have been when
	447	# processing one of these events.
	448	# What we're interested in is if the latest extremities
	449	# were the same when we created the event as they are
	450	# now. When this server creates a new event (as opposed
	451	# to receiving it over federation) it will use the
	452	# forward extremities as the prev_events, so we can
	453	# guess this by looking at the prev_events and checking
	454	# if they match the current forward extremities.
	455	for ev, _ in ev_ctx_rm:
	456	prev_event_ids = set(ev.prev_event_ids())
	457	if latest_event_ids == prev_event_ids:
	458	state_delta_reuse_delta_counter.inc()
	459	break
	460
	461	logger.info("Calculating state delta for room %s", room_id)
	462	with Measure(
	463	self._clock, "persist_events.get_new_state_after_events"
	464	):
	465	res = yield self._get_new_state_after_events(
	466	room_id,
	467	ev_ctx_rm,
	468	latest_event_ids,
	469	new_latest_event_ids,
421	470	)
422		new_latest_event_ids = yield self._calculate_new_extremities(
423		room_id, ev_ctx_rm, latest_event_ids
424		)
425
426		latest_event_ids = set(latest_event_ids)
427		if new_latest_event_ids == latest_event_ids:
428		# No change in extremities, so no change in state
429		continue
430
431		# there should always be at least one forward extremity.
432		# (except during the initial persistence of the send_join
433		# results, in which case there will be no existing
434		# extremities, so we'll `continue` above and skip this bit.)
435		assert new_latest_event_ids, "No forward extremities left!"
436
437		new_forward_extremeties[room_id] = new_latest_event_ids
438
439		len_1 = (
440		len(latest_event_ids) == 1
441		and len(new_latest_event_ids) == 1
442		)
443		if len_1:
444		all_single_prev_not_state = all(
445		len(event.prev_event_ids()) == 1
446		and not event.is_state()
447		for event, ctx in ev_ctx_rm
	471	current_state, delta_ids = res
	472
	473	# If either are not None then there has been a change,
	474	# and we need to work out the delta (or use that
	475	# given)
	476	if delta_ids is not None:
	477	# If there is a delta we know that we've
	478	# only added or replaced state, never
	479	# removed keys entirely.
	480	state_delta_for_room[room_id] = ([], delta_ids)
	481	elif current_state is not None:
	482	with Measure(
	483	self._clock, "persist_events.calculate_state_delta"
	484	):
	485	delta = yield self._calculate_state_delta(
	486	room_id, current_state
448	487	)
449		# Don't bother calculating state if they're just
450		# a long chain of single ancestor non-state events.
451		if all_single_prev_not_state:
452		continue
453
454		state_delta_counter.inc()
455		if len(new_latest_event_ids) == 1:
456		state_delta_single_event_counter.inc()
457
458		# This is a fairly handwavey check to see if we could
459		# have guessed what the delta would have been when
460		# processing one of these events.
461		# What we're interested in is if the latest extremities
462		# were the same when we created the event as they are
463		# now. When this server creates a new event (as opposed
464		# to receiving it over federation) it will use the
465		# forward extremities as the prev_events, so we can
466		# guess this by looking at the prev_events and checking
467		# if they match the current forward extremities.
468		for ev, _ in ev_ctx_rm:
469		prev_event_ids = set(ev.prev_event_ids())
470		if latest_event_ids == prev_event_ids:
471		state_delta_reuse_delta_counter.inc()
472		break
473
474		logger.info("Calculating state delta for room %s", room_id)
475		with Measure(
476		self._clock, "persist_events.get_new_state_after_events"
477		):
478		res = yield self._get_new_state_after_events(
479		room_id,
480		ev_ctx_rm,
481		latest_event_ids,
482		new_latest_event_ids,
483		)
484		current_state, delta_ids = res
485
486		# If either are not None then there has been a change,
487		# and we need to work out the delta (or use that
488		# given)
489		if delta_ids is not None:
490		# If there is a delta we know that we've
491		# only added or replaced state, never
492		# removed keys entirely.
493		state_delta_for_room[room_id] = ([], delta_ids)
494		elif current_state is not None:
495		with Measure(
496		self._clock, "persist_events.calculate_state_delta"
497		):
498		delta = yield self._calculate_state_delta(
499		room_id, current_state
500		)
501		state_delta_for_room[room_id] = delta
502
503		# If we have the current_state then lets prefill
504		# the cache with it.
505		if current_state is not None:
506		current_state_for_room[room_id] = current_state
	488	state_delta_for_room[room_id] = delta
	489
	490	# If we have the current_state then lets prefill
	491	# the cache with it.
	492	if current_state is not None:
	493	current_state_for_room[room_id] = current_state
	494
	495	# We want to calculate the stream orderings as late as possible, as
	496	# we only notify after all events with a lesser stream ordering have
	497	# been persisted. I.e. if we spend 10s inside the with block then
	498	# that will delay all subsequent events from being notified about.
	499	# Hence why we do it down here rather than wrapping the entire
	500	# function.
	501	#
	502	# Its safe to do this after calculating the state deltas etc as we
	503	# only need to protect the persistence of the events. This is to
	504	# ensure that queries of the form "fetch events since X" don't
	505	# return events and stream positions after events that are still in
	506	# flight, as otherwise subsequent requests "fetch event since Y"
	507	# will not return those events.
	508	#
	509	# Note: Multiple instances of this function cannot be in flight at
	510	# the same time for the same room.
	511	if backfilled:
	512	stream_ordering_manager = self._backfill_id_gen.get_next_mult(
	513	len(chunk)
	514	)
	515	else:
	516	stream_ordering_manager = self._stream_id_gen.get_next_mult(len(chunk))
	517
	518	with stream_ordering_manager as stream_orderings:
	519	for (event, context), stream in zip(chunk, stream_orderings):
	520	event.internal_metadata.stream_ordering = stream
507	521
508	522	yield self.runInteraction(
509	523	"persist_events",

594	608	stale = latest_event_ids & result
595	609	stale_forward_extremities_counter.observe(len(stale))
596	610
597		defer.returnValue(result)
	611	return result
598	612
599	613	@defer.inlineCallbacks
600	614	def _get_events_which_are_prevs(self, event_ids):

632	646	"_get_events_which_are_prevs", _get_events_which_are_prevs_txn, chunk
633	647	)
634	648
635		defer.returnValue(results)
	649	return results
636	650
637	651	@defer.inlineCallbacks
638	652	def _get_prevs_before_rejected(self, event_ids):

694	708	"_get_prevs_before_rejected", _get_prevs_before_rejected_txn, chunk
695	709	)
696	710
697		defer.returnValue(existing_prevs)
	711	return existing_prevs
698	712
699	713	@defer.inlineCallbacks
700	714	def _get_new_state_after_events(

795	809	# If they old and new groups are the same then we don't need to do
796	810	# anything.
797	811	if old_state_groups == new_state_groups:
798		defer.returnValue((None, None))
	812	return (None, None)
799	813
800	814	if len(new_state_groups) == 1 and len(old_state_groups) == 1:
801	815	# If we're going from one state group to another, lets check if

812	826	# the current state in memory then lets also return that,
813	827	# but it doesn't matter if we don't.
814	828	new_state = state_groups_map.get(new_state_group)
815		defer.returnValue((new_state, delta_ids))
	829	return (new_state, delta_ids)
816	830
817	831	# Now that we have calculated new_state_groups we need to get
818	832	# their state IDs so we can resolve to a single state set.

824	838	if len(new_state_groups) == 1:
825	839	# If there is only one state group, then we know what the current
826	840	# state is.
827		defer.returnValue((state_groups_map[new_state_groups.pop()], None))
	841	return (state_groups_map[new_state_groups.pop()], None)
828	842
829	843	# Ok, we need to defer to the state handler to resolve our state sets.
830	844

853	867	state_res_store=StateResolutionStore(self),
854	868	)
855	869
856		defer.returnValue((res.state, None))
	870	return (res.state, None)
857	871
858	872	@defer.inlineCallbacks
859	873	def _calculate_state_delta(self, room_id, current_state):

876	890	if ev_id != existing_state.get(key)
877	891	}
878	892
879		defer.returnValue((to_delete, to_insert))
	893	return (to_delete, to_insert)
880	894
881	895	@log_function
882	896	def _persist_events_txn(

916	930
917	931	min_stream_order = events_and_contexts[0][0].internal_metadata.stream_ordering
918	932	max_stream_order = events_and_contexts[-1][0].internal_metadata.stream_ordering
919
920		self._update_current_state_txn(txn, state_delta_for_room, min_stream_order)
921	933
922	934	self._update_forward_extremities_txn(
923	935	txn,

991	1003	all_events_and_contexts=all_events_and_contexts,
992	1004	backfilled=backfilled,
993	1005	)
	1006
	1007	# We call this last as it assumes we've inserted the events into
	1008	# room_memberships, where applicable.
	1009	self._update_current_state_txn(txn, state_delta_for_room, min_stream_order)
994	1010
995	1011	def _update_current_state_txn(self, txn, state_delta_by_room, stream_id):
996	1012	for room_id, current_state_tuple in iteritems(state_delta_by_room):

1061	1077	),
1062	1078	)
1063	1079
1064		self._simple_insert_many_txn(
1065		txn,
1066		table="current_state_events",
1067		values=[
1068		{
1069		"event_id": ev_id,
1070		"room_id": room_id,
1071		"type": key[0],
1072		"state_key": key[1],
1073		}
	1080	# We include the membership in the current state table, hence we do
	1081	# a lookup when we insert. This assumes that all events have already
	1082	# been inserted into room_memberships.
	1083	txn.executemany(
	1084	"""INSERT INTO current_state_events
	1085	(room_id, type, state_key, event_id, membership)
	1086	VALUES (?, ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?))
	1087	""",
	1088	[
	1089	(room_id, key[0], key[1], ev_id, ev_id)
1074	1090	for key, ev_id in iteritems(to_insert)
1075	1091	],
1076	1092	)

1561	1577	return count
1562	1578
1563	1579	ret = yield self.runInteraction("count_messages", _count_messages)
1564		defer.returnValue(ret)
	1580	return ret
1565	1581
1566	1582	@defer.inlineCallbacks
1567	1583	def count_daily_sent_messages(self):

1582	1598	return count
1583	1599
1584	1600	ret = yield self.runInteraction("count_daily_sent_messages", _count_messages)
1585		defer.returnValue(ret)
	1601	return ret
1586	1602
1587	1603	@defer.inlineCallbacks
1588	1604	def count_daily_active_rooms(self):

1597	1613	return count
1598	1614
1599	1615	ret = yield self.runInteraction("count_daily_active_rooms", _count)
1600		defer.returnValue(ret)
	1616	return ret
1601	1617
1602	1618	def get_current_backfill_token(self):
1603	1619	"""The current minimum token that backfilled events have reached"""

2180	2196	"""
2181	2197	to_1, so_1 = yield self._get_event_ordering(event_id1)
2182	2198	to_2, so_2 = yield self._get_event_ordering(event_id2)
2183		defer.returnValue((to_1, so_1) > (to_2, so_2))
	2199	return (to_1, so_1) > (to_2, so_2)
2184	2200
2185	2201	@cachedInlineCallbacks(max_entries=5000)
2186	2202	def _get_event_ordering(self, event_id):

2194	2210	if not res:
2195	2211	raise SynapseError(404, "Could not find event %s" % (event_id,))
2196	2212
2197		defer.returnValue(
2198		(int(res["topological_ordering"]), int(res["stream_ordering"]))
2199		)
	2213	return (int(res["topological_ordering"]), int(res["stream_ordering"]))
2200	2214
2201	2215	def get_all_updated_current_state_deltas(self, from_token, to_token, limit):
2202	2216	def get_all_updated_current_state_deltas_txn(txn):

+3

-3

synapse/storage/events_bg_updates.py less more

134	134	if not result:
135	135	yield self._end_background_update(self.EVENT_FIELDS_SENDER_URL_UPDATE_NAME)
136	136
137		defer.returnValue(result)
	137	return result
138	138
139	139	@defer.inlineCallbacks
140	140	def _background_reindex_origin_server_ts(self, progress, batch_size):

211	211	if not result:
212	212	yield self._end_background_update(self.EVENT_ORIGIN_SERVER_TS_NAME)
213	213
214		defer.returnValue(result)
	214	return result
215	215
216	216	@defer.inlineCallbacks
217	217	def _cleanup_extremities_bg_update(self, progress, batch_size):

395	395	"_cleanup_extremities_bg_update_drop_table", _drop_table_txn
396	396	)
397	397
398		defer.returnValue(num_handled)
	398	return num_handled

+126

-112

synapse/storage/events_worker.py less more

28	28	from synapse.events import FrozenEvent, event_type_from_format_version # noqa: F401
29	29	from synapse.events.snapshot import EventContext # noqa: F401
30	30	from synapse.events.utils import prune_event
31		from synapse.logging.context import (
32		LoggingContext,
33		PreserveLoggingContext,
34		make_deferred_yieldable,
35		run_in_background,
36		)
	31	from synapse.logging.context import LoggingContext, PreserveLoggingContext
37	32	from synapse.metrics.background_process_metrics import run_as_background_process
38	33	from synapse.types import get_domain_from_id
39	34	from synapse.util import batch_iter

138	133	If there is a mismatch, behave as per allow_none.
139	134
140	135	Returns:
141		Deferred : A FrozenEvent.
142		"""
	136	Deferred[EventBase\|None]
	137	"""
	138	if not isinstance(event_id, str):
	139	raise TypeError("Invalid event event_id %r" % (event_id,))
	140
143	141	events = yield self.get_events_as_list(
144	142	[event_id],
145	143	check_redacted=check_redacted,

156	154	if event is None and not allow_none:
157	155	raise NotFoundError("Could not find event %s" % (event_id,))
158	156
159		defer.returnValue(event)
	157	return event
160	158
161	159	@defer.inlineCallbacks
162	160	def get_events(

186	184	allow_rejected=allow_rejected,
187	185	)
188	186
189		defer.returnValue({e.event_id: e for e in events})
	187	return {e.event_id: e for e in events}
190	188
191	189	@defer.inlineCallbacks
192	190	def get_events_as_list(

216	214	"""
217	215
218	216	if not event_ids:
219		defer.returnValue([])
	217	return []
220	218
221	219	# there may be duplicates so we cast the list to a set
222	220	event_entry_map = yield self._get_events_from_cache_or_db(

312	310	event.unsigned["prev_content"] = prev.content
313	311	event.unsigned["prev_sender"] = prev.sender
314	312
315		defer.returnValue(events)
	313	return events
316	314
317	315	@defer.inlineCallbacks
318	316	def _get_events_from_cache_or_db(self, event_ids, allow_rejected=False):

338	336	log_ctx = LoggingContext.current_context()
339	337	log_ctx.record_event_fetch(len(missing_events_ids))
340	338
341		# Note that _enqueue_events is also responsible for turning db rows
	339	# Note that _get_events_from_db is also responsible for turning db rows
342	340	# into FrozenEvents (via _get_event_from_row), which involves seeing if
343	341	# the events have been redacted, and if so pulling the redaction event out
344	342	# of the database to check it.
345	343	#
346		# _enqueue_events is a bit of a rubbish name but naming is hard.
347		missing_events = yield self._enqueue_events(
	344	missing_events = yield self._get_events_from_db(
348	345	missing_events_ids, allow_rejected=allow_rejected
349	346	)
350	347

417	414	The fetch requests. Each entry consists of a list of event
418	415	ids to be fetched, and a deferred to be completed once the
419	416	events have been fetched.
	417
	418	The deferreds are callbacked with a dictionary mapping from event id
	419	to event row. Note that it may well contain additional events that
	420	were not part of this request.
420	421	"""
421	422	with Measure(self._clock, "_fetch_event_list"):
422	423	try:
423		event_id_lists = list(zip(*event_list))[0]
424		event_ids = [item for sublist in event_id_lists for item in sublist]
	424	events_to_fetch = set(
	425	event_id for events, _ in event_list for event_id in events
	426	)
425	427
426	428	row_dict = self._new_transaction(
427		conn, "do_fetch", [], [], self._fetch_event_rows, event_ids
	429	conn, "do_fetch", [], [], self._fetch_event_rows, events_to_fetch
428	430	)
429	431
430	432	# We only want to resolve deferreds from the main thread
431		def fire(lst, res):
432		for ids, d in lst:
433		if not d.called:
434		try:
435		with PreserveLoggingContext():
436		d.callback([res[i] for i in ids if i in res])
437		except Exception:
438		logger.exception("Failed to callback")
	433	def fire():
	434	for _, d in event_list:
	435	d.callback(row_dict)
439	436
440	437	with PreserveLoggingContext():
441		self.hs.get_reactor().callFromThread(fire, event_list, row_dict)
	438	self.hs.get_reactor().callFromThread(fire)
442	439	except Exception as e:
443	440	logger.exception("do_fetch")
444	441

453	450	self.hs.get_reactor().callFromThread(fire, event_list, e)
454	451
455	452	@defer.inlineCallbacks
456		def _enqueue_events(self, events, allow_rejected=False):
	453	def _get_events_from_db(self, event_ids, allow_rejected=False):
	454	"""Fetch a bunch of events from the database.
	455
	456	Returned events will be added to the cache for future lookups.
	457
	458	Args:
	459	event_ids (Iterable[str]): The event_ids of the events to fetch
	460	allow_rejected (bool): Whether to include rejected events
	461
	462	Returns:
	463	Deferred[Dict[str, _EventCacheEntry]]:
	464	map from event id to result. May return extra events which
	465	weren't asked for.
	466	"""
	467	fetched_events = {}
	468	events_to_fetch = event_ids
	469
	470	while events_to_fetch:
	471	row_map = yield self._enqueue_events(events_to_fetch)
	472
	473	# we need to recursively fetch any redactions of those events
	474	redaction_ids = set()
	475	for event_id in events_to_fetch:
	476	row = row_map.get(event_id)
	477	fetched_events[event_id] = row
	478	if row:
	479	redaction_ids.update(row["redactions"])
	480
	481	events_to_fetch = redaction_ids.difference(fetched_events.keys())
	482	if events_to_fetch:
	483	logger.debug("Also fetching redaction events %s", events_to_fetch)
	484
	485	# build a map from event_id to EventBase
	486	event_map = {}
	487	for event_id, row in fetched_events.items():
	488	if not row:
	489	continue
	490	assert row["event_id"] == event_id
	491
	492	rejected_reason = row["rejected_reason"]
	493
	494	if not allow_rejected and rejected_reason:
	495	continue
	496
	497	d = json.loads(row["json"])
	498	internal_metadata = json.loads(row["internal_metadata"])
	499
	500	format_version = row["format_version"]
	501	if format_version is None:
	502	# This means that we stored the event before we had the concept
	503	# of a event format version, so it must be a V1 event.
	504	format_version = EventFormatVersions.V1
	505
	506	original_ev = event_type_from_format_version(format_version)(
	507	event_dict=d,
	508	internal_metadata_dict=internal_metadata,
	509	rejected_reason=rejected_reason,
	510	)
	511
	512	event_map[event_id] = original_ev
	513
	514	# finally, we can decide whether each one nededs redacting, and build
	515	# the cache entries.
	516	result_map = {}
	517	for event_id, original_ev in event_map.items():
	518	redactions = fetched_events[event_id]["redactions"]
	519	redacted_event = self._maybe_redact_event_row(
	520	original_ev, redactions, event_map
	521	)
	522
	523	cache_entry = _EventCacheEntry(
	524	event=original_ev, redacted_event=redacted_event
	525	)
	526
	527	self._get_event_cache.prefill((event_id,), cache_entry)
	528	result_map[event_id] = cache_entry
	529
	530	return result_map
	531
	532	@defer.inlineCallbacks
	533	def _enqueue_events(self, events):
457	534	"""Fetches events from the database using the _event_fetch_list. This
458	535	allows batch and bulk fetching of events - it allows us to fetch events
459	536	without having to create a new transaction for each request for events.
460		"""
461		if not events:
462		defer.returnValue({})
	537
	538	Args:
	539	events (Iterable[str]): events to be fetched.
	540
	541	Returns:
	542	Deferred[Dict[str, Dict]]: map from event id to row data from the database.
	543	May contain events that weren't requested.
	544	"""
463	545
464	546	events_d = defer.Deferred()
465	547	with self._event_fetch_lock:

478	560	"fetch_events", self.runWithConnection, self._do_fetch
479	561	)
480	562
481		logger.debug("Loading %d events", len(events))
	563	logger.debug("Loading %d events: %s", len(events), events)
482	564	with PreserveLoggingContext():
483		rows = yield events_d
484		logger.debug("Loaded %d events (%d rows)", len(events), len(rows))
485
486		if not allow_rejected:
487		rows[:] = [r for r in rows if r["rejected_reason"] is None]
488
489		res = yield make_deferred_yieldable(
490		defer.gatherResults(
491		[
492		run_in_background(
493		self._get_event_from_row,
494		row["internal_metadata"],
495		row["json"],
496		row["redactions"],
497		rejected_reason=row["rejected_reason"],
498		format_version=row["format_version"],
499		)
500		for row in rows
501		],
502		consumeErrors=True,
503		)
504		)
505
506		defer.returnValue({e.event.event_id: e for e in res if e})
	565	row_map = yield events_d
	566	logger.debug("Loaded %d events (%d rows)", len(events), len(row_map))
	567
	568	return row_map
507	569
508	570	def _fetch_event_rows(self, txn, event_ids):
509	571	"""Fetch event rows from the database

576	638
577	639	return event_dict
578	640
579		@defer.inlineCallbacks
580		def _get_event_from_row(
581		self, internal_metadata, js, redactions, format_version, rejected_reason=None
582		):
583		"""Parse an event row which has been read from the database
584
585		Args:
586		internal_metadata (str): json-encoded internal_metadata column
587		js (str): json-encoded event body from event_json
588		redactions (list[str]): a list of the events which claim to have redacted
589		this event, from the redactions table
590		format_version: (str): the 'format_version' column
591		rejected_reason (str\|None): the reason this event was rejected, if any
592
593		Returns:
594		_EventCacheEntry
595		"""
596		with Measure(self._clock, "_get_event_from_row"):
597		d = json.loads(js)
598		internal_metadata = json.loads(internal_metadata)
599
600		if format_version is None:
601		# This means that we stored the event before we had the concept
602		# of a event format version, so it must be a V1 event.
603		format_version = EventFormatVersions.V1
604
605		original_ev = event_type_from_format_version(format_version)(
606		event_dict=d,
607		internal_metadata_dict=internal_metadata,
608		rejected_reason=rejected_reason,
609		)
610
611		redacted_event = yield self._maybe_redact_event_row(original_ev, redactions)
612
613		cache_entry = _EventCacheEntry(
614		event=original_ev, redacted_event=redacted_event
615		)
616
617		self._get_event_cache.prefill((original_ev.event_id,), cache_entry)
618
619		defer.returnValue(cache_entry)
620
621		@defer.inlineCallbacks
622		def _maybe_redact_event_row(self, original_ev, redactions):
	641	def _maybe_redact_event_row(self, original_ev, redactions, event_map):
623	642	"""Given an event object and a list of possible redacting event ids,
624	643	determine whether to honour any of those redactions and if so return a redacted
625	644	event.

627	646	Args:
628	647	original_ev (EventBase):
629	648	redactions (iterable[str]): list of event ids of potential redaction events
	649	event_map (dict[str, EventBase]): other events which have been fetched, in
	650	which we can look up the redaaction events. Map from event id to event.
630	651
631	652	Returns:
632	653	Deferred[EventBase\|None]: if the event should be redacted, a pruned

636	657	# we choose to ignore redactions of m.room.create events.
637	658	return None
638	659
639		if original_ev.type == "m.room.redaction":
640		# ... and redaction events
641		return None
642
643		redaction_map = yield self._get_events_from_cache_or_db(redactions)
644
645	660	for redaction_id in redactions:
646		redaction_entry = redaction_map.get(redaction_id)
647		if not redaction_entry:
	661	redaction_event = event_map.get(redaction_id)
	662	if not redaction_event or redaction_event.rejected_reason:
648	663	# we don't have the redaction event, or the redaction event was not
649	664	# authorized.
650	665	logger.debug(

654	669	)
655	670	continue
656	671
657		redaction_event = redaction_entry.event
658	672	if redaction_event.room_id != original_ev.room_id:
659	673	logger.debug(
660	674	"%s was redacted by %s but redaction was in a different room!",

709	723	desc="have_events_in_timeline",
710	724	)
711	725
712		defer.returnValue(set(r["event_id"] for r in rows))
	726	return set(r["event_id"] for r in rows)
713	727
714	728	@defer.inlineCallbacks
715	729	def have_seen_events(self, event_ids):

735	749	input_iterator = iter(event_ids)
736	750	for chunk in iter(lambda: list(itertools.islice(input_iterator, 100)), []):
737	751	yield self.runInteraction("have_seen_events", have_seen_events_txn, chunk)
738		defer.returnValue(results)
	752	return results
739	753
740	754	def get_seen_events_with_rejections(self, event_ids):
741	755	"""Given a list of event ids, check if we rejected them.

846	860	# it.
847	861	complexity_v1 = round(state_events / 500, 2)
848	862
849		defer.returnValue({"v1": complexity_v1})
	863	return {"v1": complexity_v1}

+1

-3

synapse/storage/filtering.py less more

13	13	# limitations under the License.
14	14
15	15	from canonicaljson import encode_canonical_json
16
17		from twisted.internet import defer
18	16
19	17	from synapse.api.errors import Codes, SynapseError
20	18	from synapse.util.caches.descriptors import cachedInlineCallbacks

40	38	desc="get_user_filter",
41	39	)
42	40
43		defer.returnValue(db_to_json(def_json))
	41	return db_to_json(def_json)
44	42
45	43	def add_user_filter(self, user_localpart, user_filter):
46	44	def_json = encode_canonical_json(user_filter)

+18

-22

synapse/storage/group_server.py less more

306	306	desc="get_group_categories",
307	307	)
308	308
309		defer.returnValue(
310		{
311		row["category_id"]: {
312		"is_public": row["is_public"],
313		"profile": json.loads(row["profile"]),
314		}
315		for row in rows
	309	return {
	310	row["category_id"]: {
	311	"is_public": row["is_public"],
	312	"profile": json.loads(row["profile"]),
316	313	}
317		)
	314	for row in rows
	315	}
318	316
319	317	@defer.inlineCallbacks
320	318	def get_group_category(self, group_id, category_id):

327	325
328	326	category["profile"] = json.loads(category["profile"])
329	327
330		defer.returnValue(category)
	328	return category
331	329
332	330	def upsert_group_category(self, group_id, category_id, profile, is_public):
333	331	"""Add/update room category for group

369	367	desc="get_group_roles",
370	368	)
371	369
372		defer.returnValue(
373		{
374		row["role_id"]: {
375		"is_public": row["is_public"],
376		"profile": json.loads(row["profile"]),
377		}
378		for row in rows
	370	return {
	371	row["role_id"]: {
	372	"is_public": row["is_public"],
	373	"profile": json.loads(row["profile"]),
379	374	}
380		)
	375	for row in rows
	376	}
381	377
382	378	@defer.inlineCallbacks
383	379	def get_group_role(self, group_id, role_id):

390	386
391	387	role["profile"] = json.loads(role["profile"])
392	388
393		defer.returnValue(role)
	389	return role
394	390
395	391	def upsert_group_role(self, group_id, role_id, profile, is_public):
396	392	"""Add/remove user role

959	955	_register_user_group_membership_txn,
960	956	next_id,
961	957	)
962		defer.returnValue(res)
	958	return res
963	959
964	960	@defer.inlineCallbacks
965	961	def create_group(

1056	1052
1057	1053	now = int(self._clock.time_msec())
1058	1054	if row and now < row["valid_until_ms"]:
1059		defer.returnValue(json.loads(row["attestation_json"]))
1060
1061		defer.returnValue(None)
	1055	return json.loads(row["attestation_json"])
	1056
	1057	return None
1062	1058
1063	1059	def get_joined_groups(self, user_id):
1064	1060	return self._simple_select_onecol(

+1

-1

synapse/storage/monthly_active_users.py less more

172	172	)
173	173	if user_id:
174	174	count = count + 1
175		defer.returnValue(count)
	175	return count
176	176
177	177	@defer.inlineCallbacks
178	178	def upsert_monthly_active_user(self, user_id):

+1

-1

synapse/storage/prepare_database.py less more

26	26
27	27	# Remember to update this number every time a change is made to database
28	28	# schema files, so the users will be informed on server restarts.
29		SCHEMA_VERSION = 55
	29	SCHEMA_VERSION = 56
30	30
31	31	dir_path = os.path.abspath(os.path.dirname(__file__))
32	32

+2

-4

synapse/storage/presence.py less more

89	89	presence_states,
90	90	)
91	91
92		defer.returnValue(
93		(stream_orderings[-1], self._presence_id_gen.get_current_token())
94		)
	92	return (stream_orderings[-1], self._presence_id_gen.get_current_token())
95	93
96	94	def _update_presence_txn(self, txn, stream_orderings, presence_states):
97	95	for stream_id, state in zip(stream_orderings, presence_states):

179	177	for row in rows:
180	178	row["currently_active"] = bool(row["currently_active"])
181	179
182		defer.returnValue({row["user_id"]: UserPresenceState(**row) for row in rows})
	180	return {row["user_id"]: UserPresenceState(**row) for row in rows}
183	181
184	182	def get_current_presence_token(self):
185	183	return self._presence_id_gen.get_current_token()

+5

-7

synapse/storage/profile.py less more

33	33	except StoreError as e:
34	34	if e.code == 404:
35	35	# no match
36		defer.returnValue(ProfileInfo(None, None))
	36	return ProfileInfo(None, None)
37	37	return
38	38	else:
39	39	raise
40	40
41		defer.returnValue(
42		ProfileInfo(
43		avatar_url=profile["avatar_url"], display_name=profile["displayname"]
44		)
	41	return ProfileInfo(
	42	avatar_url=profile["avatar_url"], display_name=profile["displayname"]
45	43	)
46	44
47	45	def get_profile_displayname(self, user_localpart):

167	165	)
168	166
169	167	if res:
170		defer.returnValue(True)
	168	return True
171	169
172	170	res = yield self._simple_select_one_onecol(
173	171	table="group_invites",

178	176	)
179	177
180	178	if res:
181		defer.returnValue(True)
	179	return True

+8

-10

synapse/storage/push_rule.py less more

119	119
120	120	rules = _load_rules(rows, enabled_map)
121	121
122		defer.returnValue(rules)
	122	return rules
123	123
124	124	@cachedInlineCallbacks(max_entries=5000)
125	125	def get_push_rules_enabled_for_user(self, user_id):

129	129	retcols=("user_name", "rule_id", "enabled"),
130	130	desc="get_push_rules_enabled_for_user",
131	131	)
132		defer.returnValue(
133		{r["rule_id"]: False if r["enabled"] == 0 else True for r in results}
134		)
	132	return {r["rule_id"]: False if r["enabled"] == 0 else True for r in results}
135	133
136	134	def have_push_rules_changed_for_user(self, user_id, last_id):
137	135	if not self.push_rules_stream_cache.has_entity_changed(user_id, last_id):

159	157	)
160	158	def bulk_get_push_rules(self, user_ids):
161	159	if not user_ids:
162		defer.returnValue({})
	160	return {}
163	161
164	162	results = {user_id: [] for user_id in user_ids}
165	163

181	179	for user_id, rules in results.items():
182	180	results[user_id] = _load_rules(rules, enabled_map_by_user.get(user_id, {}))
183	181
184		defer.returnValue(results)
	182	return results
185	183
186	184	@defer.inlineCallbacks
187	185	def move_push_rule_from_room_to_room(self, new_room_id, user_id, rule):

252	250	result = yield self._bulk_get_push_rules_for_room(
253	251	event.room_id, state_group, current_state_ids, event=event
254	252	)
255		defer.returnValue(result)
	253	return result
256	254
257	255	@cachedInlineCallbacks(num_args=2, cache_context=True)
258	256	def _bulk_get_push_rules_for_room(

311	309
312	310	rules_by_user = {k: v for k, v in rules_by_user.items() if v is not None}
313	311
314		defer.returnValue(rules_by_user)
	312	return rules_by_user
315	313
316	314	@cachedList(
317	315	cached_method_name="get_push_rules_enabled_for_user",

321	319	)
322	320	def bulk_get_push_rules_enabled(self, user_ids):
323	321	if not user_ids:
324		defer.returnValue({})
	322	return {}
325	323
326	324	results = {user_id: {} for user_id in user_ids}
327	325

335	333	for row in rows:
336	334	enabled = bool(row["enabled"])
337	335	results.setdefault(row["user_name"], {})[row["rule_id"]] = enabled
338		defer.returnValue(results)
	336	return results
339	337
340	338
341	339	class PushRuleStore(PushRulesWorkerStore):

+27

-13

synapse/storage/pusher.py less more

62	62	ret = yield self._simple_select_one_onecol(
63	63	"pushers", {"user_name": user_id}, "id", allow_none=True
64	64	)
65		defer.returnValue(ret is not None)
	65	return ret is not None
66	66
67	67	def get_pushers_by_app_id_and_pushkey(self, app_id, pushkey):
68	68	return self.get_pushers_by({"app_id": app_id, "pushkey": pushkey})

94	94	],
95	95	desc="get_pushers_by",
96	96	)
97		defer.returnValue(self._decode_pushers_rows(ret))
	97	return self._decode_pushers_rows(ret)
98	98
99	99	@defer.inlineCallbacks
100	100	def get_all_pushers(self):

105	105	return self._decode_pushers_rows(rows)
106	106
107	107	rows = yield self.runInteraction("get_all_pushers", get_pushers)
108		defer.returnValue(rows)
	108	return rows
109	109
110	110	def get_all_updated_pushers(self, last_id, current_id, limit):
111	111	if last_id == current_id:

204	204	result = {user_id: False for user_id in user_ids}
205	205	result.update({r["user_name"]: True for r in rows})
206	206
207		defer.returnValue(result)
	207	return result
208	208
209	209
210	210	class PusherStore(PusherWorkerStore):

307	307	def update_pusher_last_stream_ordering_and_success(
308	308	self, app_id, pushkey, user_id, last_stream_ordering, last_success
309	309	):
310		yield self._simple_update_one(
311		"pushers",
312		{"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
313		{
	310	"""Update the last stream ordering position we've processed up to for
	311	the given pusher.
	312
	313	Args:
	314	app_id (str)
	315	pushkey (str)
	316	last_stream_ordering (int)
	317	last_success (int)
	318
	319	Returns:
	320	Deferred[bool]: True if the pusher still exists; False if it has been deleted.
	321	"""
	322	updated = yield self._simple_update(
	323	table="pushers",
	324	keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
	325	updatevalues={
314	326	"last_stream_ordering": last_stream_ordering,
315	327	"last_success": last_success,
316	328	},
317	329	desc="update_pusher_last_stream_ordering_and_success",
318	330	)
319	331
	332	return bool(updated)
	333
320	334	@defer.inlineCallbacks
321	335	def update_pusher_failing_since(self, app_id, pushkey, user_id, failing_since):
322		yield self._simple_update_one(
323		"pushers",
324		{"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
325		{"failing_since": failing_since},
	336	yield self._simple_update(
	337	table="pushers",
	338	keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
	339	updatevalues={"failing_since": failing_since},
326	340	desc="update_pusher_failing_since",
327	341	)
328	342

342	356	"throttle_ms": row["throttle_ms"],
343	357	}
344	358
345		defer.returnValue(params_by_room)
	359	return params_by_room
346	360
347	361	@defer.inlineCallbacks
348	362	def set_throttle_params(self, pusher_id, room_id, params):

+16

-20

synapse/storage/receipts.py less more

57	57	@cachedInlineCallbacks()
58	58	def get_users_with_read_receipts_in_room(self, room_id):
59	59	receipts = yield self.get_receipts_for_room(room_id, "m.read")
60		defer.returnValue(set(r["user_id"] for r in receipts))
	60	return set(r["user_id"] for r in receipts)
61	61
62	62	@cached(num_args=2)
63	63	def get_receipts_for_room(self, room_id, receipt_type):

91	91	desc="get_receipts_for_user",
92	92	)
93	93
94		defer.returnValue({row["room_id"]: row["event_id"] for row in rows})
	94	return {row["room_id"]: row["event_id"] for row in rows}
95	95
96	96	@defer.inlineCallbacks
97	97	def get_receipts_for_user_with_orderings(self, user_id, receipt_type):

109	109	return txn.fetchall()
110	110
111	111	rows = yield self.runInteraction("get_receipts_for_user_with_orderings", f)
112		defer.returnValue(
113		{
114		row[0]: {
115		"event_id": row[1],
116		"topological_ordering": row[2],
117		"stream_ordering": row[3],
118		}
119		for row in rows
	112	return {
	113	row[0]: {
	114	"event_id": row[1],
	115	"topological_ordering": row[2],
	116	"stream_ordering": row[3],
120	117	}
121		)
	118	for row in rows
	119	}
122	120
123	121	@defer.inlineCallbacks
124	122	def get_linearized_receipts_for_rooms(self, room_ids, to_key, from_key=None):

146	144	room_ids, to_key, from_key=from_key
147	145	)
148	146
149		defer.returnValue([ev for res in results.values() for ev in res])
	147	return [ev for res in results.values() for ev in res]
150	148
151	149	def get_linearized_receipts_for_room(self, room_id, to_key, from_key=None):
152	150	"""Get receipts for a single room for sending to clients.

196	194	rows = yield self.runInteraction("get_linearized_receipts_for_room", f)
197	195
198	196	if not rows:
199		defer.returnValue([])
	197	return []
200	198
201	199	content = {}
202	200	for row in rows:

204	202	row["user_id"]
205	203	] = json.loads(row["data"])
206	204
207		defer.returnValue(
208		[{"type": "m.receipt", "room_id": room_id, "content": content}]
209		)
	205	return [{"type": "m.receipt", "room_id": room_id, "content": content}]
210	206
211	207	@cachedList(
212	208	cached_method_name="_get_linearized_receipts_for_room",

216	212	)
217	213	def _get_linearized_receipts_for_rooms(self, room_ids, to_key, from_key=None):
218	214	if not room_ids:
219		defer.returnValue({})
	215	return {}
220	216
221	217	def f(txn):
222	218	if from_key:

263	259	room_id: [results[room_id]] if room_id in results else []
264	260	for room_id in room_ids
265	261	}
266		defer.returnValue(results)
	262	return results
267	263
268	264	def get_all_updated_receipts(self, last_id, current_id, limit=None):
269	265	if last_id == current_id:

467	463	)
468	464
469	465	if event_ts is None:
470		defer.returnValue(None)
	466	return None
471	467
472	468	now = self._clock.time_msec()
473	469	logger.debug(

481	477
482	478	max_persisted_id = self._receipts_id_gen.get_current_token()
483	479
484		defer.returnValue((stream_id, max_persisted_id))
	480	return (stream_id, max_persisted_id)
485	481
486	482	def insert_graph_receipt(self, room_id, receipt_type, user_id, event_ids, data):
487	483	return self.runInteraction(

+37

-37

synapse/storage/registration.py less more

74	74
75	75	info = yield self.get_user_by_id(user_id)
76	76	if not info:
77		defer.returnValue(False)
	77	return False
78	78
79	79	now = self.clock.time_msec()
80	80	trial_duration_ms = self.config.mau_trial_days * 24 * 60 * 60 * 1000
81	81	is_trial = (now - info["creation_ts"] * 1000) < trial_duration_ms
82		defer.returnValue(is_trial)
	82	return is_trial
83	83
84	84	@cached()
85	85	def get_user_by_access_token(self, token):

114	114	allow_none=True,
115	115	desc="get_expiration_ts_for_user",
116	116	)
117		defer.returnValue(res)
	117	return res
118	118
119	119	@defer.inlineCallbacks
120	120	def set_account_validity_for_user(

189	189	desc="get_user_from_renewal_token",
190	190	)
191	191
192		defer.returnValue(res)
	192	return res
193	193
194	194	@defer.inlineCallbacks
195	195	def get_renewal_token_for_user(self, user_id):

208	208	desc="get_renewal_token_for_user",
209	209	)
210	210
211		defer.returnValue(res)
	211	return res
212	212
213	213	@defer.inlineCallbacks
214	214	def get_users_expiring_soon(self):

236	236	self.config.account_validity.renew_at,
237	237	)
238	238
239		defer.returnValue(res)
	239	return res
240	240
241	241	@defer.inlineCallbacks
242	242	def set_renewal_mail_status(self, user_id, email_sent):

279	279	desc="is_server_admin",
280	280	)
281	281
282		defer.returnValue(res if res else False)
	282	return res if res else False
283	283
284	284	def _query_for_auth(self, txn, token):
285	285	sql = (

310	310	res = yield self.runInteraction(
311	311	"is_support_user", self.is_support_user_txn, user_id
312	312	)
313		defer.returnValue(res)
	313	return res
314	314
315	315	def is_support_user_txn(self, txn, user_id):
316	316	res = self._simple_select_one_onecol_txn(

348	348	return 0
349	349
350	350	ret = yield self.runInteraction("count_users", _count_users)
351		defer.returnValue(ret)
	351	return ret
352	352
353	353	def count_daily_user_type(self):
354	354	"""

394	394	return count
395	395
396	396	ret = yield self.runInteraction("count_users", _count_users)
397		defer.returnValue(ret)
	397	return ret
398	398
399	399	@defer.inlineCallbacks
400	400	def find_next_generated_user_id_localpart(self):

424	424	if i not in found:
425	425	return i
426	426
427		defer.returnValue(
	427	return (
428	428	(
429	429	yield self.runInteraction(
430	430	"find_next_generated_user_id", _find_next_generated_user_id

446	446	user_id = yield self.runInteraction(
447	447	"get_user_id_by_threepid", self.get_user_id_by_threepid_txn, medium, address
448	448	)
449		defer.returnValue(user_id)
	449	return user_id
450	450
451	451	def get_user_id_by_threepid_txn(self, txn, medium, address):
452	452	"""Returns user id from threepid

486	486	["medium", "address", "validated_at", "added_at"],
487	487	"user_get_threepids",
488	488	)
489		defer.returnValue(ret)
	489	return ret
490	490
491	491	def user_delete_threepid(self, user_id, medium, address):
492	492	return self._simple_delete(

567	567	retcol="id_server",
568	568	desc="get_id_servers_user_bound",
569	569	)
	570
	571	@cachedInlineCallbacks()
	572	def get_user_deactivated_status(self, user_id):
	573	"""Retrieve the value for the `deactivated` property for the provided user.
	574
	575	Args:
	576	user_id (str): The ID of the user to retrieve the status for.
	577
	578	Returns:
	579	defer.Deferred(bool): The requested value.
	580	"""
	581
	582	res = yield self._simple_select_one_onecol(
	583	table="users",
	584	keyvalues={"name": user_id},
	585	retcol="deactivated",
	586	desc="get_user_deactivated_status",
	587	)
	588
	589	# Convert the integer into a boolean.
	590	return res == 1
570	591
571	592
572	593	class RegistrationStore(

676	697	if end:
677	698	yield self._end_background_update("users_set_deactivated_flag")
678	699
679		defer.returnValue(batch_size)
	700	return batch_size
680	701
681	702	@defer.inlineCallbacks
682	703	def add_access_token_to_user(self, user_id, token, device_id, valid_until_ms):

956	977	desc="is_guest",
957	978	)
958	979
959		defer.returnValue(res if res else False)
	980	return res if res else False
960	981
961	982	def add_user_pending_deactivation(self, user_id):
962	983	"""

1023	1044
1024	1045	yield self._end_background_update("user_threepids_grandfather")
1025	1046
1026		defer.returnValue(1)
	1047	return 1
1027	1048
1028	1049	def get_threepid_validation_session(
1029	1050	self, medium, client_secret, address=None, sid=None, validated=True

1316	1337	user_id,
1317	1338	deactivated,
1318	1339	)
1319
1320		@cachedInlineCallbacks()
1321		def get_user_deactivated_status(self, user_id):
1322		"""Retrieve the value for the `deactivated` property for the provided user.
1323
1324		Args:
1325		user_id (str): The ID of the user to retrieve the status for.
1326
1327		Returns:
1328		defer.Deferred(bool): The requested value.
1329		"""
1330
1331		res = yield self._simple_select_one_onecol(
1332		table="users",
1333		keyvalues={"name": user_id},
1334		retcol="deactivated",
1335		desc="get_user_deactivated_status",
1336		)
1337
1338		# Convert the integer into a boolean.
1339		defer.returnValue(res == 1)

+1

-3

synapse/storage/relations.py less more

15	15	import logging
16	16
17	17	import attr
18
19		from twisted.internet import defer
20	18
21	19	from synapse.api.constants import RelationTypes
22	20	from synapse.api.errors import SynapseError

362	360	return
363	361
364	362	edit_event = yield self.get_event(edit_id, allow_none=True)
365		defer.returnValue(edit_event)
	363	return edit_event
366	364
367	365	def has_user_annotated_event(self, parent_id, event_type, aggregation_key, sender):
368	366	"""Check if a user has already annotated an event with the same key

+4

-6

synapse/storage/room.py less more

192	192	)
193	193
194	194	if row:
195		defer.returnValue(
196		RatelimitOverride(
197		messages_per_second=row["messages_per_second"],
198		burst_count=row["burst_count"],
199		)
	195	return RatelimitOverride(
	196	messages_per_second=row["messages_per_second"],
	197	burst_count=row["burst_count"],
200	198	)
201	199	else:
202		defer.returnValue(None)
	200	return None
203	201
204	202
205	203	class RoomStore(RoomWorkerStore, SearchStore):

+277

-77

synapse/storage/roommember.py less more

23	23	from twisted.internet import defer
24	24
25	25	from synapse.api.constants import EventTypes, Membership
	26	from synapse.metrics.background_process_metrics import run_as_background_process
	27	from synapse.storage._base import LoggingTransaction
26	28	from synapse.storage.events_worker import EventsWorkerStore
27	29	from synapse.types import get_domain_from_id
28	30	from synapse.util.async_helpers import Linearizer

52	54	MemberSummary = namedtuple("MemberSummary", ("members", "count"))
53	55
54	56	_MEMBERSHIP_PROFILE_UPDATE_NAME = "room_membership_profile_update"
	57	_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME = "current_state_events_membership"
55	58
56	59
57	60	class RoomMemberWorkerStore(EventsWorkerStore):
	61	def __init__(self, db_conn, hs):
	62	super(RoomMemberWorkerStore, self).__init__(db_conn, hs)
	63
	64	# Is the current_state_events.membership up to date? Or is the
	65	# background update still running?
	66	self._current_state_events_membership_up_to_date = False
	67
	68	txn = LoggingTransaction(
	69	db_conn.cursor(),
	70	name="_check_safe_current_state_events_membership_updated",
	71	database_engine=self.database_engine,
	72	)
	73	self._check_safe_current_state_events_membership_updated_txn(txn)
	74	txn.close()
	75
	76	def _check_safe_current_state_events_membership_updated_txn(self, txn):
	77	"""Checks if it is safe to assume the new current_state_events
	78	membership column is up to date
	79	"""
	80
	81	pending_update = self._simple_select_one_txn(
	82	txn,
	83	table="background_updates",
	84	keyvalues={"update_name": _CURRENT_STATE_MEMBERSHIP_UPDATE_NAME},
	85	retcols=["update_name"],
	86	allow_none=True,
	87	)
	88
	89	self._current_state_events_membership_up_to_date = not pending_update
	90
	91	# If the update is still running, reschedule to run.
	92	if pending_update:
	93	self._clock.call_later(
	94	15.0,
	95	run_as_background_process,
	96	"_check_safe_current_state_events_membership_updated",
	97	self.runInteraction,
	98	"_check_safe_current_state_events_membership_updated",
	99	self._check_safe_current_state_events_membership_updated_txn,
	100	)
	101
58	102	@cachedInlineCallbacks(max_entries=100000, iterable=True, cache_context=True)
59	103	def get_hosts_in_room(self, room_id, cache_context):
60	104	"""Returns the set of all hosts currently in the room

63	107	room_id, on_invalidate=cache_context.invalidate
64	108	)
65	109	hosts = frozenset(get_domain_from_id(user_id) for user_id in user_ids)
66		defer.returnValue(hosts)
	110	return hosts
67	111
68	112	@cached(max_entries=100000, iterable=True)
69	113	def get_users_in_room(self, room_id):
70	114	def f(txn):
71		sql = (
72		"SELECT m.user_id FROM room_memberships as m"
73		" INNER JOIN current_state_events as c"
74		" ON m.event_id = c.event_id "
75		" AND m.room_id = c.room_id "
76		" AND m.user_id = c.state_key"
77		" WHERE c.type = 'm.room.member' AND c.room_id = ? AND m.membership = ?"
78		)
	115	# If we can assume current_state_events.membership is up to date
	116	# then we can avoid a join, which is a Very Good Thing given how
	117	# frequently this function gets called.
	118	if self._current_state_events_membership_up_to_date:
	119	sql = """
	120	SELECT state_key FROM current_state_events
	121	WHERE type = 'm.room.member' AND room_id = ? AND membership = ?
	122	"""
	123	else:
	124	sql = """
	125	SELECT state_key FROM room_memberships as m
	126	INNER JOIN current_state_events as c
	127	ON m.event_id = c.event_id
	128	AND m.room_id = c.room_id
	129	AND m.user_id = c.state_key
	130	WHERE c.type = 'm.room.member' AND c.room_id = ? AND m.membership = ?
	131	"""
79	132
80	133	txn.execute(sql, (room_id, Membership.JOIN))
81	134	return [to_ascii(r[0]) for r in txn]

97	150	# first get counts.
98	151	# We do this all in one transaction to keep the cache small.
99	152	# FIXME: get rid of this when we have room_stats
100		sql = """
101		SELECT count(*), m.membership FROM room_memberships as m
102		INNER JOIN current_state_events as c
103		ON m.event_id = c.event_id
104		AND m.room_id = c.room_id
105		AND m.user_id = c.state_key
106		WHERE c.type = 'm.room.member' AND c.room_id = ?
107		GROUP BY m.membership
108		"""
	153
	154	# If we can assume current_state_events.membership is up to date
	155	# then we can avoid a join, which is a Very Good Thing given how
	156	# frequently this function gets called.
	157	if self._current_state_events_membership_up_to_date:
	158	# Note, rejected events will have a null membership field, so
	159	# we we manually filter them out.
	160	sql = """
	161	SELECT count(*), membership FROM current_state_events
	162	WHERE type = 'm.room.member' AND room_id = ?
	163	AND membership IS NOT NULL
	164	GROUP BY membership
	165	"""
	166	else:
	167	sql = """
	168	SELECT count(*), m.membership FROM room_memberships as m
	169	INNER JOIN current_state_events as c
	170	ON m.event_id = c.event_id
	171	AND m.room_id = c.room_id
	172	AND m.user_id = c.state_key
	173	WHERE c.type = 'm.room.member' AND c.room_id = ?
	174	GROUP BY m.membership
	175	"""
109	176
110	177	txn.execute(sql, (room_id,))
111	178	res = {}

114	181
115	182	# we order by membership and then fairly arbitrarily by event_id so
116	183	# heroes are consistent
117		sql = """
118		SELECT m.user_id, m.membership, m.event_id
119		FROM room_memberships as m
120		INNER JOIN current_state_events as c
121		ON m.event_id = c.event_id
122		AND m.room_id = c.room_id
123		AND m.user_id = c.state_key
124		WHERE c.type = 'm.room.member' AND c.room_id = ?
125		ORDER BY
126		CASE m.membership WHEN ? THEN 1 WHEN ? THEN 2 ELSE 3 END ASC,
127		m.event_id ASC
128		LIMIT ?
129		"""
	184	if self._current_state_events_membership_up_to_date:
	185	# Note, rejected events will have a null membership field, so
	186	# we we manually filter them out.
	187	sql = """
	188	SELECT state_key, membership, event_id
	189	FROM current_state_events
	190	WHERE type = 'm.room.member' AND room_id = ?
	191	AND membership IS NOT NULL
	192	ORDER BY
	193	CASE membership WHEN ? THEN 1 WHEN ? THEN 2 ELSE 3 END ASC,
	194	event_id ASC
	195	LIMIT ?
	196	"""
	197	else:
	198	sql = """
	199	SELECT c.state_key, m.membership, c.event_id
	200	FROM room_memberships as m
	201	INNER JOIN current_state_events as c USING (room_id, event_id)
	202	WHERE c.type = 'm.room.member' AND c.room_id = ?
	203	ORDER BY
	204	CASE m.membership WHEN ? THEN 1 WHEN ? THEN 2 ELSE 3 END ASC,
	205	c.event_id ASC
	206	LIMIT ?
	207	"""
130	208
131	209	# 6 is 5 (number of heroes) plus 1, in case one of them is the calling user.
132	210	txn.execute(sql, (room_id, Membership.JOIN, Membership.INVITE, 6))

188	266	invites = yield self.get_invited_rooms_for_user(user_id)
189	267	for invite in invites:
190	268	if invite.room_id == room_id:
191		defer.returnValue(invite)
192		defer.returnValue(None)
193
	269	return invite
	270	return None
	271
	272	@defer.inlineCallbacks
194	273	def get_rooms_for_user_where_membership_is(self, user_id, membership_list):
195	274	""" Get all the rooms for this user where the membership for this user
196	275	matches one in the membership list.
	276
	277	Filters out forgotten rooms.
197	278
198	279	Args:
199	280	user_id (str): The user ID.
200	281	membership_list (list): A list of synapse.api.constants.Membership
201	282	values which the user must be in.
	283
202	284	Returns:
203		A list of dictionary objects, with room_id, membership and sender
204		defined.
	285	Deferred[list[RoomsForUser]]
205	286	"""
206	287	if not membership_list:
207	288	return defer.succeed(None)
208	289
209		return self.runInteraction(
	290	rooms = yield self.runInteraction(
210	291	"get_rooms_for_user_where_membership_is",
211	292	self._get_rooms_for_user_where_membership_is_txn,
212	293	user_id,
213	294	membership_list,
214	295	)
215	296
	297	# Now we filter out forgotten rooms
	298	forgotten_rooms = yield self.get_forgotten_rooms_for_user(user_id)
	299	return [room for room in rooms if room.room_id not in forgotten_rooms]
	300
216	301	def _get_rooms_for_user_where_membership_is_txn(
217	302	self, txn, user_id, membership_list
218	303	):

222	307
223	308	results = []
224	309	if membership_list:
225		where_clause = "user_id = ? AND (%s) AND forgotten = 0" % (
226		" OR ".join(["membership = ?" for _ in membership_list]),
227		)
228
229		args = [user_id]
230		args.extend(membership_list)
231
232		sql = (
233		"SELECT m.room_id, m.sender, m.membership, m.event_id, e.stream_ordering"
234		" FROM current_state_events as c"
235		" INNER JOIN room_memberships as m"
236		" ON m.event_id = c.event_id"
237		" INNER JOIN events as e"
238		" ON e.event_id = c.event_id"
239		" AND m.room_id = c.room_id"
240		" AND m.user_id = c.state_key"
241		" WHERE c.type = 'm.room.member' AND %s"
242		) % (where_clause,)
243
244		txn.execute(sql, args)
	310	if self._current_state_events_membership_up_to_date:
	311	sql = """
	312	SELECT room_id, e.sender, c.membership, event_id, e.stream_ordering
	313	FROM current_state_events AS c
	314	INNER JOIN events AS e USING (room_id, event_id)
	315	WHERE
	316	c.type = 'm.room.member'
	317	AND state_key = ?
	318	AND c.membership IN (%s)
	319	""" % (
	320	",".join("?" * len(membership_list))
	321	)
	322	else:
	323	sql = """
	324	SELECT room_id, e.sender, m.membership, event_id, e.stream_ordering
	325	FROM current_state_events AS c
	326	INNER JOIN room_memberships AS m USING (room_id, event_id)
	327	INNER JOIN events AS e USING (room_id, event_id)
	328	WHERE
	329	c.type = 'm.room.member'
	330	AND state_key = ?
	331	AND m.membership IN (%s)
	332	""" % (
	333	",".join("?" * len(membership_list))
	334	)
	335
	336	txn.execute(sql, (user_id, *membership_list))
245	337	results = [RoomsForUser(**r) for r in self.cursor_to_dict(txn)]
246	338
247	339	if do_invite:

282	374	rooms = yield self.get_rooms_for_user_where_membership_is(
283	375	user_id, membership_list=[Membership.JOIN]
284	376	)
285		defer.returnValue(
286		frozenset(
287		GetRoomsForUserWithStreamOrdering(r.room_id, r.stream_ordering)
288		for r in rooms
289		)
	377	return frozenset(
	378	GetRoomsForUserWithStreamOrdering(r.room_id, r.stream_ordering)
	379	for r in rooms
290	380	)
291	381
292	382	@defer.inlineCallbacks

296	386	rooms = yield self.get_rooms_for_user_with_stream_ordering(
297	387	user_id, on_invalidate=on_invalidate
298	388	)
299		defer.returnValue(frozenset(r.room_id for r in rooms))
	389	return frozenset(r.room_id for r in rooms)
300	390
301	391	@cachedInlineCallbacks(max_entries=500000, cache_context=True, iterable=True)
302	392	def get_users_who_share_room_with_user(self, user_id, cache_context):

313	403	)
314	404	user_who_share_room.update(user_ids)
315	405
316		defer.returnValue(user_who_share_room)
	406	return user_who_share_room
317	407
318	408	@defer.inlineCallbacks
319	409	def get_joined_users_from_context(self, event, context):

329	419	result = yield self._get_joined_users_from_context(
330	420	event.room_id, state_group, current_state_ids, event=event, context=context
331	421	)
332		defer.returnValue(result)
	422	return result
333	423
334	424	def get_joined_users_from_state(self, room_id, state_entry):
335	425	state_group = state_entry.state_group

443	533	avatar_url=to_ascii(event.content.get("avatar_url", None)),
444	534	)
445	535
446		defer.returnValue(users_in_room)
	536	return users_in_room
447	537
448	538	@cachedInlineCallbacks(max_entries=10000)
449	539	def is_host_joined(self, room_id, host):

452	542
453	543	sql = """
454	544	SELECT state_key FROM current_state_events AS c
455		INNER JOIN room_memberships USING (event_id)
456		WHERE membership = 'join'
	545	INNER JOIN room_memberships AS m USING (event_id)
	546	WHERE m.membership = 'join'
457	547	AND type = 'm.room.member'
458	548	AND c.room_id = ?
459	549	AND state_key LIKE ?

468	558	rows = yield self._execute("is_host_joined", None, sql, room_id, like_clause)
469	559
470	560	if not rows:
471		defer.returnValue(False)
	561	return False
472	562
473	563	user_id = rows[0][0]
474	564	if get_domain_from_id(user_id) != host:
475	565	# This can only happen if the host name has something funky in it
476	566	raise Exception("Invalid host name")
477	567
478		defer.returnValue(True)
	568	return True
479	569
480	570	@cachedInlineCallbacks()
481	571	def was_host_joined(self, room_id, host):

508	598	rows = yield self._execute("was_host_joined", None, sql, room_id, like_clause)
509	599
510	600	if not rows:
511		defer.returnValue(False)
	601	return False
512	602
513	603	user_id = rows[0][0]
514	604	if get_domain_from_id(user_id) != host:
515	605	# This can only happen if the host name has something funky in it
516	606	raise Exception("Invalid host name")
517	607
518		defer.returnValue(True)
	608	return True
519	609
520	610	def get_joined_hosts(self, room_id, state_entry):
521	611	state_group = state_entry.state_group

542	632	cache = self._get_joined_hosts_cache(room_id)
543	633	joined_hosts = yield cache.get_destinations(state_entry)
544	634
545		defer.returnValue(joined_hosts)
	635	return joined_hosts
546	636
547	637	@cached(max_entries=10000)
548	638	def _get_joined_hosts_cache(self, room_id):

572	662	return rows[0][0]
573	663
574	664	count = yield self.runInteraction("did_forget_membership", f)
575		defer.returnValue(count == 0)
	665	return count == 0
	666
	667	@cached()
	668	def get_forgotten_rooms_for_user(self, user_id):
	669	"""Gets all rooms the user has forgotten.
	670
	671	Args:
	672	user_id (str)
	673
	674	Returns:
	675	Deferred[set[str]]
	676	"""
	677
	678	def _get_forgotten_rooms_for_user_txn(txn):
	679	# This is a slightly convoluted query that first looks up all rooms
	680	# that the user has forgotten in the past, then rechecks that list
	681	# to see if any have subsequently been updated. This is done so that
	682	# we can use a partial index on `forgotten = 1` on the assumption
	683	# that few users will actually forget many rooms.
	684	#
	685	# Note that a room is considered "forgotten" if all membership
	686	# events for that user and room have the forgotten field set (as
	687	# when a user forgets a room we update all rows for that user and
	688	# room, not just the current one).
	689	sql = """
	690	SELECT room_id, (
	691	SELECT count(*) FROM room_memberships
	692	WHERE room_id = m.room_id AND user_id = m.user_id AND forgotten = 0
	693	) AS count
	694	FROM room_memberships AS m
	695	WHERE user_id = ? AND forgotten = 1
	696	GROUP BY room_id, user_id;
	697	"""
	698	txn.execute(sql, (user_id,))
	699	return set(row[0] for row in txn if row[1] == 0)
	700
	701	return self.runInteraction(
	702	"get_forgotten_rooms_for_user", _get_forgotten_rooms_for_user_txn
	703	)
576	704
577	705	@defer.inlineCallbacks
578	706	def get_rooms_user_has_been_in(self, user_id):

600	728	super(RoomMemberStore, self).__init__(db_conn, hs)
601	729	self.register_background_update_handler(
602	730	_MEMBERSHIP_PROFILE_UPDATE_NAME, self._background_add_membership_profile
	731	)
	732	self.register_background_update_handler(
	733	_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME,
	734	self._background_current_state_membership,
	735	)
	736	self.register_background_index_update(
	737	"room_membership_forgotten_idx",
	738	index_name="room_memberships_user_room_forgotten",
	739	table="room_memberships",
	740	columns=["user_id", "room_id"],
	741	where_clause="forgotten = 1",
603	742	)
604	743
605	744	def _store_room_members_txn(self, txn, events, backfilled):

702	841	txn.execute(sql, (user_id, room_id))
703	842
704	843	self._invalidate_cache_and_stream(txn, self.did_forget, (user_id, room_id))
	844	self._invalidate_cache_and_stream(
	845	txn, self.get_forgotten_rooms_for_user, (user_id,)
	846	)
705	847
706	848	return self.runInteraction("forget_membership", f)
707	849

778	920	if not result:
779	921	yield self._end_background_update(_MEMBERSHIP_PROFILE_UPDATE_NAME)
780	922
781		defer.returnValue(result)
	923	return result
	924
	925	@defer.inlineCallbacks
	926	def _background_current_state_membership(self, progress, batch_size):
	927	"""Update the new membership column on current_state_events.
	928
	929	This works by iterating over all rooms in alphebetical order.
	930	"""
	931
	932	def _background_current_state_membership_txn(txn, last_processed_room):
	933	processed = 0
	934	while processed < batch_size:
	935	txn.execute(
	936	"""
	937	SELECT MIN(room_id) FROM current_state_events WHERE room_id > ?
	938	""",
	939	(last_processed_room,),
	940	)
	941	row = txn.fetchone()
	942	if not row or not row[0]:
	943	return processed, True
	944
	945	next_room, = row
	946
	947	sql = """
	948	UPDATE current_state_events
	949	SET membership = (
	950	SELECT membership FROM room_memberships
	951	WHERE event_id = current_state_events.event_id
	952	)
	953	WHERE room_id = ?
	954	"""
	955	txn.execute(sql, (next_room,))
	956	processed += txn.rowcount
	957
	958	last_processed_room = next_room
	959
	960	self._background_update_progress_txn(
	961	txn,
	962	_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME,
	963	{"last_processed_room": last_processed_room},
	964	)
	965
	966	return processed, False
	967
	968	# If we haven't got a last processed room then just use the empty
	969	# string, which will compare before all room IDs correctly.
	970	last_processed_room = progress.get("last_processed_room", "")
	971
	972	row_count, finished = yield self.runInteraction(
	973	"_background_current_state_membership_update",
	974	_background_current_state_membership_txn,
	975	last_processed_room,
	976	)
	977
	978	if finished:
	979	yield self._end_background_update(_CURRENT_STATE_MEMBERSHIP_UPDATE_NAME)
	980
	981	return row_count
782	982
783	983
784	984	class _JoinedHostsCache(object):

806	1006	state_entry(synapse.state._StateCacheEntry)
807	1007	"""
808	1008	if state_entry.state_group == self.state_group:
809		defer.returnValue(frozenset(self.hosts_to_joined_users))
	1009	return frozenset(self.hosts_to_joined_users)
810	1010
811	1011	with (yield self.linearizer.queue(())):
812	1012	if state_entry.state_group == self.state_group:

843	1043	else:
844	1044	self.state_group = object()
845	1045	self._len = sum(len(v) for v in itervalues(self.hosts_to_joined_users))
846		defer.returnValue(frozenset(self.hosts_to_joined_users))
	1046	return frozenset(self.hosts_to_joined_users)
847	1047
848	1048	def __len__(self):
849	1049	return self._len

+22

-0

synapse/storage/schema/delta/56/current_state_events_membership.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- We add membership to current state so that we don't need to join against
	16	-- room_memberships, which can be surprisingly costly (we do such queries
	17	-- very frequently).
	18	-- This will be null for non-membership events and the content.membership key
	19	-- for membership events. (Will also be null for membership events until the
	20	-- background update job has finished).
	21	ALTER TABLE current_state_events ADD membership TEXT;

+24

-0

synapse/storage/schema/delta/56/current_state_events_membership_mk2.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- We add membership to current state so that we don't need to join against
	16	-- room_memberships, which can be surprisingly costly (we do such queries
	17	-- very frequently).
	18	-- This will be null for non-membership events and the content.membership key
	19	-- for membership events. (Will also be null for membership events until the
	20	-- background update job has finished).
	21
	22	INSERT INTO background_updates (update_name, progress_json) VALUES
	23	('current_state_events_membership', '{}');

+18

-0

synapse/storage/schema/delta/56/room_membership_idx.sql less more

	0	/* Copyright 2019 The Matrix.org Foundation C.I.C.
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- Adds an index on room_memberships for fetching all forgotten rooms for a user
	16	INSERT INTO background_updates (update_name, progress_json) VALUES
	17	('room_membership_forgotten_idx', '{}');

+26

-30

synapse/storage/search.py less more

165	165	if not result:
166	166	yield self._end_background_update(self.EVENT_SEARCH_UPDATE_NAME)
167	167
168		defer.returnValue(result)
	168	return result
169	169
170	170	@defer.inlineCallbacks
171	171	def _background_reindex_gin_search(self, progress, batch_size):

208	208	yield self.runWithConnection(create_index)
209	209
210	210	yield self._end_background_update(self.EVENT_SEARCH_USE_GIN_POSTGRES_NAME)
211		defer.returnValue(1)
	211	return 1
212	212
213	213	@defer.inlineCallbacks
214	214	def _background_reindex_search_order(self, progress, batch_size):

286	286	if not finished:
287	287	yield self._end_background_update(self.EVENT_SEARCH_ORDER_UPDATE_NAME)
288	288
289		defer.returnValue(num_rows)
	289	return num_rows
290	290
291	291	def store_event_search_txn(self, txn, event, key, value):
292	292	"""Add event to the search table

453	453
454	454	count = sum(row["count"] for row in count_results if row["room_id"] in room_ids)
455	455
456		defer.returnValue(
457		{
458		"results": [
459		{"event": event_map[r["event_id"]], "rank": r["rank"]}
460		for r in results
461		if r["event_id"] in event_map
462		],
463		"highlights": highlights,
464		"count": count,
465		}
466		)
	456	return {
	457	"results": [
	458	{"event": event_map[r["event_id"]], "rank": r["rank"]}
	459	for r in results
	460	if r["event_id"] in event_map
	461	],
	462	"highlights": highlights,
	463	"count": count,
	464	}
467	465
468	466	@defer.inlineCallbacks
469	467	def search_rooms(self, room_ids, search_term, keys, limit, pagination_token=None):

598	596
599	597	count = sum(row["count"] for row in count_results if row["room_id"] in room_ids)
600	598
601		defer.returnValue(
602		{
603		"results": [
604		{
605		"event": event_map[r["event_id"]],
606		"rank": r["rank"],
607		"pagination_token": "%s,%s"
608		% (r["origin_server_ts"], r["stream_ordering"]),
609		}
610		for r in results
611		if r["event_id"] in event_map
612		],
613		"highlights": highlights,
614		"count": count,
615		}
616		)
	599	return {
	600	"results": [
	601	{
	602	"event": event_map[r["event_id"]],
	603	"rank": r["rank"],
	604	"pagination_token": "%s,%s"
	605	% (r["origin_server_ts"], r["stream_ordering"]),
	606	}
	607	for r in results
	608	if r["event_id"] in event_map
	609	],
	610	"highlights": highlights,
	611	"count": count,
	612	}
617	613
618	614	def _find_highlights_in_postgres(self, search_query, events):
619	615	"""Given a list of events and a search term, return a list of words

+1

-1

synapse/storage/signatures.py less more

58	58	for e_id, h in hashes.items()
59	59	}
60	60
61		defer.returnValue(list(hashes.items()))
	61	return list(hashes.items())
62	62
63	63	def _get_event_reference_hashes_txn(self, txn, event_id):
64	64	"""Get all the hashes for a given PDU.

+32

-30

synapse/storage/state.py less more

421	421
422	422	# Retrieve the room's create event
423	423	create_event = yield self.get_create_event_for_room(room_id)
424		defer.returnValue(create_event.content.get("room_version", "1"))
	424	return create_event.content.get("room_version", "1")
425	425
426	426	@defer.inlineCallbacks
427	427	def get_room_predecessor(self, room_id):

441	441	create_event = yield self.get_create_event_for_room(room_id)
442	442
443	443	# Return predecessor if present
444		defer.returnValue(create_event.content.get("predecessor", None))
	444	return create_event.content.get("predecessor", None)
445	445
446	446	@defer.inlineCallbacks
447	447	def get_create_event_for_room(self, room_id):

465	465
466	466	# Retrieve the room's create event and return
467	467	create_event = yield self.get_event(create_id)
468		defer.returnValue(create_event)
	468	return create_event
469	469
470	470	@cached(max_entries=100000, iterable=True)
471	471	def get_current_state_ids(self, room_id):

509	509	event ID.
510	510	"""
511	511
	512	where_clause, where_args = state_filter.make_sql_filter_clause()
	513
	514	if not where_clause:
	515	# We delegate to the cached version
	516	return self.get_current_state_ids(room_id)
	517
512	518	def _get_filtered_current_state_ids_txn(txn):
513	519	results = {}
514	520	sql = """
515	521	SELECT type, state_key, event_id FROM current_state_events
516	522	WHERE room_id = ?
517	523	"""
518
519		where_clause, where_args = state_filter.make_sql_filter_clause()
520	524
521	525	if where_clause:
522	526	sql += " AND (%s)" % (where_clause,)

558	562	if not event:
559	563	return
560	564
561		defer.returnValue(event.content.get("canonical_alias"))
	565	return event.content.get("canonical_alias")
562	566
563	567	@cached(max_entries=10000, iterable=True)
564	568	def get_state_group_delta(self, state_group):

608	612	dict of state_group_id -> (dict of (type, state_key) -> event id)
609	613	"""
610	614	if not event_ids:
611		defer.returnValue({})
	615	return {}
612	616
613	617	event_to_groups = yield self._get_state_group_for_events(event_ids)
614	618
615	619	groups = set(itervalues(event_to_groups))
616	620	group_to_state = yield self._get_state_for_groups(groups)
617	621
618		defer.returnValue(group_to_state)
	622	return group_to_state
619	623
620	624	@defer.inlineCallbacks
621	625	def get_state_ids_for_group(self, state_group):

629	633	"""
630	634	group_to_state = yield self._get_state_for_groups((state_group,))
631	635
632		defer.returnValue(group_to_state[state_group])
	636	return group_to_state[state_group]
633	637
634	638	@defer.inlineCallbacks
635	639	def get_state_groups(self, room_id, event_ids):

640	644	dict of state_group_id -> list of state events.
641	645	"""
642	646	if not event_ids:
643		defer.returnValue({})
	647	return {}
644	648
645	649	group_to_ids = yield self.get_state_groups_ids(room_id, event_ids)
646	650

653	657	get_prev_content=False,
654	658	)
655	659
656		defer.returnValue(
657		{
658		group: [
659		state_event_map[v]
660		for v in itervalues(event_id_map)
661		if v in state_event_map
662		]
663		for group, event_id_map in iteritems(group_to_ids)
664		}
665		)
	660	return {
	661	group: [
	662	state_event_map[v]
	663	for v in itervalues(event_id_map)
	664	if v in state_event_map
	665	]
	666	for group, event_id_map in iteritems(group_to_ids)
	667	}
666	668
667	669	@defer.inlineCallbacks
668	670	def _get_state_groups_from_groups(self, groups, state_filter):

689	691	)
690	692	results.update(res)
691	693
692		defer.returnValue(results)
	694	return results
693	695
694	696	def _get_state_groups_from_groups_txn(
695	697	self, txn, groups, state_filter=StateFilter.all()

824	826	for event_id, group in iteritems(event_to_groups)
825	827	}
826	828
827		defer.returnValue({event: event_to_state[event] for event in event_ids})
	829	return {event: event_to_state[event] for event in event_ids}
828	830
829	831	@defer.inlineCallbacks
830	832	def get_state_ids_for_events(self, event_ids, state_filter=StateFilter.all()):

850	852	for event_id, group in iteritems(event_to_groups)
851	853	}
852	854
853		defer.returnValue({event: event_to_state[event] for event in event_ids})
	855	return {event: event_to_state[event] for event in event_ids}
854	856
855	857	@defer.inlineCallbacks
856	858	def get_state_for_event(self, event_id, state_filter=StateFilter.all()):

866	868	A deferred dict from (type, state_key) -> state_event
867	869	"""
868	870	state_map = yield self.get_state_for_events([event_id], state_filter)
869		defer.returnValue(state_map[event_id])
	871	return state_map[event_id]
870	872
871	873	@defer.inlineCallbacks
872	874	def get_state_ids_for_event(self, event_id, state_filter=StateFilter.all()):

882	884	A deferred dict from (type, state_key) -> state_event
883	885	"""
884	886	state_map = yield self.get_state_ids_for_events([event_id], state_filter)
885		defer.returnValue(state_map[event_id])
	887	return state_map[event_id]
886	888
887	889	@cached(max_entries=50000)
888	890	def _get_state_group_for_event(self, event_id):

912	914	desc="_get_state_group_for_events",
913	915	)
914	916
915		defer.returnValue({row["event_id"]: row["state_group"] for row in rows})
	917	return {row["event_id"]: row["state_group"] for row in rows}
916	918
917	919	def _get_state_for_group_using_cache(self, cache, group, state_filter):
918	920	"""Checks if group is in cache. See `_get_state_for_groups`

992	994	incomplete_groups = incomplete_groups_m \| incomplete_groups_nm
993	995
994	996	if not incomplete_groups:
995		defer.returnValue(state)
	997	return state
996	998
997	999	cache_sequence_nm = self._state_group_cache.sequence
998	1000	cache_sequence_m = self._state_group_members_cache.sequence

1019	1021	# everything we need from the database anyway.
1020	1022	state[group] = state_filter.filter_state(group_state_dict)
1021	1023
1022		defer.returnValue(state)
	1024	return state
1023	1025
1024	1026	def _get_state_for_groups_using_cache(self, groups, cache, state_filter):
1025	1027	"""Gets the state at each of a list of state groups, optionally

1493	1495	self.STATE_GROUP_DEDUPLICATION_UPDATE_NAME
1494	1496	)
1495	1497
1496		defer.returnValue(result * BATCH_SIZE_SCALE_FACTOR)
	1498	return result * BATCH_SIZE_SCALE_FACTOR
1497	1499
1498	1500	@defer.inlineCallbacks
1499	1501	def _background_index_state(self, progress, batch_size):

1523	1525
1524	1526	yield self._end_background_update(self.STATE_GROUP_INDEX_UPDATE_NAME)
1525	1527
1526		defer.returnValue(1)
	1528	return 1

+20

-18

synapse/storage/stats.py less more

65	65
66	66	if not self.stats_enabled:
67	67	yield self._end_background_update("populate_stats_createtables")
68		defer.returnValue(1)
	68	return 1
69	69
70	70	# Get all the rooms that we want to process.
71	71	def _make_staging_area(txn):

119	119	self.get_earliest_token_for_room_stats.invalidate_all()
120	120
121	121	yield self._end_background_update("populate_stats_createtables")
122		defer.returnValue(1)
	122	return 1
123	123
124	124	@defer.inlineCallbacks
125	125	def _populate_stats_cleanup(self, progress, batch_size):

128	128	"""
129	129	if not self.stats_enabled:
130	130	yield self._end_background_update("populate_stats_cleanup")
131		defer.returnValue(1)
	131	return 1
132	132
133	133	position = yield self._simple_select_one_onecol(
134	134	TEMP_TABLE + "_position", None, "position"

142	142	yield self.runInteraction("populate_stats_cleanup", _delete_staging_area)
143	143
144	144	yield self._end_background_update("populate_stats_cleanup")
145		defer.returnValue(1)
	145	return 1
146	146
147	147	@defer.inlineCallbacks
148	148	def _populate_stats_process_rooms(self, progress, batch_size):
149	149
150	150	if not self.stats_enabled:
151	151	yield self._end_background_update("populate_stats_process_rooms")
152		defer.returnValue(1)
	152	return 1
153	153
154	154	# If we don't have progress filed, delete everything.
155	155	if not progress:

185	185	# No more rooms -- complete the transaction.
186	186	if not rooms_to_work_on:
187	187	yield self._end_background_update("populate_stats_process_rooms")
188		defer.returnValue(1)
	188	return 1
189	189
190	190	logger.info(
191	191	"Processing the next %d rooms of %d remaining",

210	210	avatar_id = current_state_ids.get((EventTypes.RoomAvatar, ""))
211	211	canonical_alias_id = current_state_ids.get((EventTypes.CanonicalAlias, ""))
212	212
	213	event_ids = [
	214	join_rules_id,
	215	history_visibility_id,
	216	encryption_id,
	217	name_id,
	218	topic_id,
	219	avatar_id,
	220	canonical_alias_id,
	221	]
	222
213	223	state_events = yield self.get_events(
214		[
215		join_rules_id,
216		history_visibility_id,
217		encryption_id,
218		name_id,
219		topic_id,
220		avatar_id,
221		canonical_alias_id,
222		]
	224	[ev for ev in event_ids if ev is not None]
223	225	)
224	226
225	227	def _get_or_none(event_id, arg):

302	304
303	305	if processed_event_count > batch_size:
304	306	# Don't process any more rooms, we've hit our batch size.
305		defer.returnValue(processed_event_count)
306
307		defer.returnValue(processed_event_count)
	307	return processed_event_count
	308
	309	return processed_event_count
308	310
309	311	def delete_all_stats(self):
310	312	"""

+21

-23

synapse/storage/stream.py less more

299	299	)
300	300
301	301	if not room_ids:
302		defer.returnValue({})
	302	return {}
303	303
304	304	results = {}
305	305	room_ids = list(room_ids)

322	322	)
323	323	results.update(dict(zip(rm_ids, res)))
324	324
325		defer.returnValue(results)
	325	return results
326	326
327	327	def get_rooms_that_changed(self, room_ids, from_key):
328	328	"""Given a list of rooms and a token, return rooms where there may have

363	363	the chunk of events returned.
364	364	"""
365	365	if from_key == to_key:
366		defer.returnValue(([], from_key))
	366	return ([], from_key)
367	367
368	368	from_id = RoomStreamToken.parse_stream_token(from_key).stream
369	369	to_id = RoomStreamToken.parse_stream_token(to_key).stream

373	373	)
374	374
375	375	if not has_changed:
376		defer.returnValue(([], from_key))
	376	return ([], from_key)
377	377
378	378	def f(txn):
379	379	sql = (

406	406	# get.
407	407	key = from_key
408	408
409		defer.returnValue((ret, key))
	409	return (ret, key)
410	410
411	411	@defer.inlineCallbacks
412	412	def get_membership_changes_for_user(self, user_id, from_key, to_key):

414	414	to_id = RoomStreamToken.parse_stream_token(to_key).stream
415	415
416	416	if from_key == to_key:
417		defer.returnValue([])
	417	return []
418	418
419	419	if from_id:
420	420	has_changed = self._membership_stream_cache.has_entity_changed(
421	421	user_id, int(from_id)
422	422	)
423	423	if not has_changed:
424		defer.returnValue([])
	424	return []
425	425
426	426	def f(txn):
427	427	sql = (

446	446
447	447	self._set_before_and_after(ret, rows, topo_order=False)
448	448
449		defer.returnValue(ret)
	449	return ret
450	450
451	451	@defer.inlineCallbacks
452	452	def get_recent_events_for_room(self, room_id, limit, end_token):

476	476
477	477	self._set_before_and_after(events, rows)
478	478
479		defer.returnValue((events, token))
	479	return (events, token)
480	480
481	481	@defer.inlineCallbacks
482	482	def get_recent_event_ids_for_room(self, room_id, limit, end_token):

495	495	"""
496	496	# Allow a zero limit here, and no-op.
497	497	if limit == 0:
498		defer.returnValue(([], end_token))
	498	return ([], end_token)
499	499
500	500	end_token = RoomStreamToken.parse(end_token)
501	501

510	510	# We want to return the results in ascending order.
511	511	rows.reverse()
512	512
513		defer.returnValue((rows, token))
	513	return (rows, token)
514	514
515	515	def get_room_event_after_stream_ordering(self, room_id, stream_ordering):
516	516	"""Gets details of the first event in a room at or after a stream ordering

548	548	"""
549	549	token = yield self.get_room_max_stream_ordering()
550	550	if room_id is None:
551		defer.returnValue("s%d" % (token,))
	551	return "s%d" % (token,)
552	552	else:
553	553	topo = yield self.runInteraction(
554	554	"_get_max_topological_txn", self._get_max_topological_txn, room_id
555	555	)
556		defer.returnValue("t%d-%d" % (topo, token))
	556	return "t%d-%d" % (topo, token)
557	557
558	558	def get_stream_token_for_event(self, event_id):
559	559	"""The stream token for an event

673	673	[e for e in results["after"]["event_ids"]], get_prev_content=True
674	674	)
675	675
676		defer.returnValue(
677		{
678		"events_before": events_before,
679		"events_after": events_after,
680		"start": results["before"]["token"],
681		"end": results["after"]["token"],
682		}
683		)
	676	return {
	677	"events_before": events_before,
	678	"events_after": events_after,
	679	"start": results["before"]["token"],
	680	"end": results["after"]["token"],
	681	}
684	682
685	683	def _get_events_around_txn(
686	684	self, txn, room_id, event_id, before_limit, after_limit, event_filter

784	782
785	783	events = yield self.get_events_as_list(event_ids)
786	784
787		defer.returnValue((upper_bound, events))
	785	return (upper_bound, events)
788	786
789	787	def get_federation_out_pos(self, typ):
790	788	return self._simple_select_one_onecol(

938	936
939	937	self._set_before_and_after(events, rows)
940	938
941		defer.returnValue((events, token))
	939	return (events, token)
942	940
943	941
944	942	class StreamStore(StreamWorkerStore):

+6

-6

synapse/storage/tags.py less more

65	65	room_id string, tag string and content string.
66	66	"""
67	67	if last_id == current_id:
68		defer.returnValue([])
	68	return []
69	69
70	70	def get_all_updated_tags_txn(txn):
71	71	sql = (

106	106	)
107	107	results.extend(tags)
108	108
109		defer.returnValue(results)
	109	return results
110	110
111	111	@defer.inlineCallbacks
112	112	def get_updated_tags(self, user_id, stream_id):

134	134	user_id, int(stream_id)
135	135	)
136	136	if not changed:
137		defer.returnValue({})
	137	return {}
138	138
139	139	room_ids = yield self.runInteraction("get_updated_tags", get_updated_tags_txn)
140	140

144	144	for room_id in room_ids:
145	145	results[room_id] = tags_by_room.get(room_id, {})
146	146
147		defer.returnValue(results)
	147	return results
148	148
149	149	def get_tags_for_room(self, user_id, room_id):
150	150	"""Get all the tags for the given room

193	193	self.get_tags_for_user.invalidate((user_id,))
194	194
195	195	result = self._account_data_id_gen.get_current_token()
196		defer.returnValue(result)
	196	return result
197	197
198	198	@defer.inlineCallbacks
199	199	def remove_tag_from_room(self, user_id, room_id, tag):

216	216	self.get_tags_for_user.invalidate((user_id,))
217	217
218	218	result = self._account_data_id_gen.get_current_token()
219		defer.returnValue(result)
	219	return result
220	220
221	221	def _update_revision_txn(self, txn, user_id, room_id, next_id):
222	222	"""Update the latest revision of the tags for the given user and room.

+22

-2

synapse/storage/transactions.py less more

146	146
147	147	result = self._destination_retry_cache.get(destination, SENTINEL)
148	148	if result is not SENTINEL:
149		defer.returnValue(result)
	149	return result
150	150
151	151	result = yield self.runInteraction(
152	152	"get_destination_retry_timings",

157	157	# We don't hugely care about race conditions between getting and
158	158	# invalidating the cache, since we time out fairly quickly anyway.
159	159	self._destination_retry_cache[destination] = result
160		defer.returnValue(result)
	160	return result
161	161
162	162	def _get_destination_retry_timings(self, txn, destination):
163	163	result = self._simple_select_one_txn(

195	195	def _set_destination_retry_timings(
196	196	self, txn, destination, retry_last_ts, retry_interval
197	197	):
	198
	199	if self.database_engine.can_native_upsert:
	200	# Upsert retry time interval if retry_interval is zero (i.e. we're
	201	# resetting it) or greater than the existing retry interval.
	202
	203	sql = """
	204	INSERT INTO destinations (destination, retry_last_ts, retry_interval)
	205	VALUES (?, ?, ?)
	206	ON CONFLICT (destination) DO UPDATE SET
	207	retry_last_ts = EXCLUDED.retry_last_ts,
	208	retry_interval = EXCLUDED.retry_interval
	209	WHERE
	210	EXCLUDED.retry_interval = 0
	211	OR destinations.retry_interval < EXCLUDED.retry_interval
	212	"""
	213
	214	txn.execute(sql, (destination, retry_last_ts, retry_interval))
	215
	216	return
	217
198	218	self.database_engine.lock_table(txn, "destinations")
199	219
200	220	# We need to be careful here as the data may have changed from under us

+21

-21

synapse/storage/user_directory.py less more

108	108	yield self._simple_insert(TEMP_TABLE + "_position", {"position": new_pos})
109	109
110	110	yield self._end_background_update("populate_user_directory_createtables")
111		defer.returnValue(1)
	111	return 1
112	112
113	113	@defer.inlineCallbacks
114	114	def _populate_user_directory_cleanup(self, progress, batch_size):

130	130	)
131	131
132	132	yield self._end_background_update("populate_user_directory_cleanup")
133		defer.returnValue(1)
	133	return 1
134	134
135	135	@defer.inlineCallbacks
136	136	def _populate_user_directory_process_rooms(self, progress, batch_size):

176	176	# No more rooms -- complete the transaction.
177	177	if not rooms_to_work_on:
178	178	yield self._end_background_update("populate_user_directory_process_rooms")
179		defer.returnValue(1)
	179	return 1
180	180
181	181	logger.info(
182	182	"Processing the next %d rooms of %d remaining"

256	256
257	257	if processed_event_count > batch_size:
258	258	# Don't process any more rooms, we've hit our batch size.
259		defer.returnValue(processed_event_count)
260
261		defer.returnValue(processed_event_count)
	259	return processed_event_count
	260
	261	return processed_event_count
262	262
263	263	@defer.inlineCallbacks
264	264	def _populate_user_directory_process_users(self, progress, batch_size):

267	267	"""
268	268	if not self.hs.config.user_directory_search_all_users:
269	269	yield self._end_background_update("populate_user_directory_process_users")
270		defer.returnValue(1)
	270	return 1
271	271
272	272	def _get_next_batch(txn):
273	273	sql = "SELECT user_id FROM %s LIMIT %s" % (

297	297	# No more users -- complete the transaction.
298	298	if not users_to_work_on:
299	299	yield self._end_background_update("populate_user_directory_process_users")
300		defer.returnValue(1)
	300	return 1
301	301
302	302	logger.info(
303	303	"Processing the next %d users of %d remaining"

321	321	progress,
322	322	)
323	323
324		defer.returnValue(len(users_to_work_on))
	324	return len(users_to_work_on)
325	325
326	326	@defer.inlineCallbacks
327	327	def is_room_world_readable_or_publicly_joinable(self, room_id):

343	343	join_rule_ev = yield self.get_event(join_rules_id, allow_none=True)
344	344	if join_rule_ev:
345	345	if join_rule_ev.content.get("join_rule") == JoinRules.PUBLIC:
346		defer.returnValue(True)
	346	return True
347	347
348	348	hist_vis_id = current_state_ids.get((EventTypes.RoomHistoryVisibility, ""))
349	349	if hist_vis_id:
350	350	hist_vis_ev = yield self.get_event(hist_vis_id, allow_none=True)
351	351	if hist_vis_ev:
352	352	if hist_vis_ev.content.get("history_visibility") == "world_readable":
353		defer.returnValue(True)
354
355		defer.returnValue(False)
	353	return True
	354
	355	return False
356	356
357	357	def update_profile_in_user_dir(self, user_id, display_name, avatar_url):
358	358	"""

498	498	user_ids = set(user_ids_share_pub)
499	499	user_ids.update(user_ids_share_priv)
500	500
501		defer.returnValue(user_ids)
	501	return user_ids
502	502
503	503	def add_users_who_share_private_room(self, room_id, user_id_tuples):
504	504	"""Insert entries into the users_who_share_private_rooms table. The first

608	608
609	609	users = set(pub_rows)
610	610	users.update(rows)
611		defer.returnValue(list(users))
	611	return list(users)
612	612
613	613	@defer.inlineCallbacks
614	614	def get_rooms_in_common_for_users(self, user_id, other_user_id):

617	617	sql = """
618	618	SELECT room_id FROM (
619	619	SELECT c.room_id FROM current_state_events AS c
620		INNER JOIN room_memberships USING (event_id)
	620	INNER JOIN room_memberships AS m USING (event_id)
621	621	WHERE type = 'm.room.member'
622		AND membership = 'join'
	622	AND m.membership = 'join'
623	623	AND state_key = ?
624	624	) AS f1 INNER JOIN (
625	625	SELECT c.room_id FROM current_state_events AS c
626		INNER JOIN room_memberships USING (event_id)
	626	INNER JOIN room_memberships AS m USING (event_id)
627	627	WHERE type = 'm.room.member'
628		AND membership = 'join'
	628	AND m.membership = 'join'
629	629	AND state_key = ?
630	630	) f2 USING (room_id)
631	631	"""

634	634	"get_rooms_in_common_for_users", None, sql, user_id, other_user_id
635	635	)
636	636
637		defer.returnValue([room_id for room_id, in rows])
	637	return [room_id for room_id, in rows]
638	638
639	639	def delete_all_from_user_dir(self):
640	640	"""Delete the entire user directory

781	781
782	782	limited = len(results) > limit
783	783
784		defer.returnValue({"limited": limited, "results": results})
	784	return {"limited": limited, "results": results}
785	785
786	786
787	787	def _parse_query_sqlite(search_term):

+2

-3

synapse/storage/user_erasure_store.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14
14	15	import operator
15
16		from twisted.internet import defer
17	16
18	17	from synapse.storage._base import SQLBaseStore
19	18	from synapse.util.caches.descriptors import cached, cachedList

66	65
67	66	erased_users = yield self.runInteraction("are_users_erased", _get_erased_users)
68	67	res = dict((u, u in erased_users) for u in user_ids)
69		defer.returnValue(res)
	68	return res
70	69
71	70
72	71	class UserErasureStore(UserErasureWorkerStore):

+2

-2

synapse/streams/events.py less more

55	55	device_list_key=device_list_key,
56	56	groups_key=groups_key,
57	57	)
58		defer.returnValue(token)
	58	return token
59	59
60	60	@defer.inlineCallbacks
61	61	def get_current_token_for_pagination(self):

79	79	device_list_key=0,
80	80	groups_key=0,
81	81	)
82		defer.returnValue(token)
	82	return token

+5

-3

synapse/util/__init__.py less more

48	48	with context.PreserveLoggingContext():
49	49	self._reactor.callLater(seconds, d.callback, seconds)
50	50	res = yield d
51		defer.returnValue(res)
	51	return res
52	52
53	53	def time(self):
54	54	"""Returns the current system time in seconds since epoch."""

58	58	"""Returns the current system time in miliseconds since epoch."""
59	59	return int(self.time() * 1000)
60	60
61		def looping_call(self, f, msec):
	61	def looping_call(self, f, msec, args, *kwargs):
62	62	"""Call a function repeatedly.
63	63
64	64	Waits `msec` initially before calling `f` for the first time.

69	69	Args:
70	70	f(function): The function to call repeatedly.
71	71	msec(float): How long to wait between calls in milliseconds.
	72	*args: Postional arguments to pass to function.
	73	**kwargs: Key arguments to pass to function.
72	74	"""
73		call = task.LoopingCall(f)
	75	call = task.LoopingCall(f, args, *kwargs)
74	76	call.clock = self._reactor
75	77	d = call.start(msec / 1000.0, now=False)
76	78	d.addErrback(log_failure, "Looping call died", consumeErrors=False)

+2

-2

synapse/util/async_helpers.py less more

365	365	new_defer.callback(None)
366	366	self.key_to_current_readers.get(key, set()).discard(new_defer)
367	367
368		defer.returnValue(_ctx_manager())
	368	return _ctx_manager()
369	369
370	370	@defer.inlineCallbacks
371	371	def write(self, key):

395	395	if self.key_to_current_writer[key] == new_defer:
396	396	self.key_to_current_writer.pop(key)
397	397
398		defer.returnValue(_ctx_manager())
	398	return _ctx_manager()
399	399
400	400
401	401	def _cancelled_to_timed_out_error(value, timeout):

+16

-1

synapse/util/caches/__init__.py less more

0	0	# -- coding: utf-8 --
1	1	# Copyright 2015, 2016 OpenMarket Ltd
	2	# Copyright 2019 The Matrix.org Foundation C.I.C.
2	3	#
3	4	# Licensed under the Apache License, Version 2.0 (the "License");
4	5	# you may not use this file except in compliance with the License.

50	51	response_cache_total = Gauge("synapse_util_caches_response_cache:total", "", ["name"])
51	52
52	53
53		def register_cache(cache_type, cache_name, cache):
	54	def register_cache(cache_type, cache_name, cache, collect_callback=None):
	55	"""Register a cache object for metric collection.
	56
	57	Args:
	58	cache_type (str):
	59	cache_name (str): name of the cache
	60	cache (object): cache itself
	61	collect_callback (callable\|None): if not None, a function which is called during
	62	metric collection to update additional metrics.
	63
	64	Returns:
	65	CacheMetric: an object which provides inc_{hits,misses,evictions} methods
	66	"""
54	67
55	68	# Check if the metric is already registered. Unregister it, if so.
56	69	# This usually happens during tests, as at runtime these caches are

89	102	cache_hits.labels(cache_name).set(self.hits)
90	103	cache_evicted.labels(cache_name).set(self.evicted_size)
91	104	cache_total.labels(cache_name).set(self.hits + self.misses)
	105	if collect_callback:
	106	collect_callback()
92	107	except Exception as e:
93	108	logger.warn("Error calculating metrics for %s: %s", cache_name, e)
94	109	raise

+61

-35

synapse/util/caches/descriptors.py less more

18	18	import threading
19	19	from collections import namedtuple
20	20
21		import six
22		from six import itervalues, string_types
	21	from six import itervalues
	22
	23	from prometheus_client import Gauge
23	24
24	25	from twisted.internet import defer
25	26

29	30	from synapse.util.caches import get_cache_factor_for
30	31	from synapse.util.caches.lrucache import LruCache
31	32	from synapse.util.caches.treecache import TreeCache, iterate_tree_cache_entry
32		from synapse.util.stringutils import to_ascii
33	33
34	34	from . import register_cache
35	35
36	36	logger = logging.getLogger(__name__)
37	37
	38
	39	cache_pending_metric = Gauge(
	40	"synapse_util_caches_cache_pending",
	41	"Number of lookups currently pending for this cache",
	42	["name"],
	43	)
38	44
39	45	_CacheSentinel = object()
40	46

81	87	self.name = name
82	88	self.keylen = keylen
83	89	self.thread = None
84		self.metrics = register_cache("cache", name, self.cache)
	90	self.metrics = register_cache(
	91	"cache",
	92	name,
	93	self.cache,
	94	collect_callback=self._metrics_collection_callback,
	95	)
85	96
86	97	def _on_evicted(self, evicted_count):
87	98	self.metrics.inc_evictions(evicted_count)
	99
	100	def _metrics_collection_callback(self):
	101	cache_pending_metric.labels(self.name).set(len(self._pending_deferred_cache))
88	102
89	103	def check_thread(self):
90	104	expected_thread = self.thread

107	121	update_metrics (bool): whether to update the cache hit rate metrics
108	122
109	123	Returns:
110		Either a Deferred or the raw result
	124	Either an ObservableDeferred or the raw result
111	125	"""
112	126	callbacks = [callback] if callback else []
113	127	val = self._pending_deferred_cache.get(key, _CacheSentinel)

131	145	return default
132	146
133	147	def set(self, key, value, callback=None):
	148	if not isinstance(value, defer.Deferred):
	149	raise TypeError("not a Deferred")
	150
134	151	callbacks = [callback] if callback else []
135	152	self.check_thread()
136		entry = CacheEntry(deferred=value, callbacks=callbacks)
	153	observable = ObservableDeferred(value, consumeErrors=True)
	154	observer = defer.maybeDeferred(observable.observe)
	155	entry = CacheEntry(deferred=observable, callbacks=callbacks)
137	156
138	157	existing_entry = self._pending_deferred_cache.pop(key, None)
139	158	if existing_entry:

141	160
142	161	self._pending_deferred_cache[key] = entry
143	162
144		def shuffle(result):
	163	def compare_and_pop():
	164	"""Check if our entry is still the one in _pending_deferred_cache, and
	165	if so, pop it.
	166
	167	Returns true if the entries matched.
	168	"""
145	169	existing_entry = self._pending_deferred_cache.pop(key, None)
146	170	if existing_entry is entry:
	171	return True
	172
	173	# oops, the _pending_deferred_cache has been updated since
	174	# we started our query, so we are out of date.
	175	#
	176	# Better put back whatever we took out. (We do it this way
	177	# round, rather than peeking into the _pending_deferred_cache
	178	# and then removing on a match, to make the common case faster)
	179	if existing_entry is not None:
	180	self._pending_deferred_cache[key] = existing_entry
	181
	182	return False
	183
	184	def cb(result):
	185	if compare_and_pop():
147	186	self.cache.set(key, result, entry.callbacks)
148	187	else:
149		# oops, the _pending_deferred_cache has been updated since
150		# we started our query, so we are out of date.
151		#
152		# Better put back whatever we took out. (We do it this way
153		# round, rather than peeking into the _pending_deferred_cache
154		# and then removing on a match, to make the common case faster)
155		if existing_entry is not None:
156		self._pending_deferred_cache[key] = existing_entry
157
158	188	# we're not going to put this entry into the cache, so need
159	189	# to make sure that the invalidation callbacks are called.
160	190	# That was probably done when _pending_deferred_cache was

162	192	# `invalidate` being previously called, in which case it may
163	193	# not have been. Either way, let's double-check now.
164	194	entry.invalidate()
165		return result
166
167		entry.deferred.addCallback(shuffle)
	195
	196	def eb(_fail):
	197	compare_and_pop()
	198	entry.invalidate()
	199
	200	# once the deferred completes, we can move the entry from the
	201	# _pending_deferred_cache to the real cache.
	202	#
	203	observer.addCallbacks(cb, eb)
	204	return observable
168	205
169	206	def prefill(self, key, value, callback=None):
170	207	callbacks = [callback] if callback else []

288	325	def foo(self, key, cache_context):
289	326	r1 = yield self.bar1(key, on_invalidate=cache_context.invalidate)
290	327	r2 = yield self.bar2(key, on_invalidate=cache_context.invalidate)
291		defer.returnValue(r1 + r2)
	328	return r1 + r2
292	329
293	330	Args:
294	331	num_args (int): number of positional arguments (excluding ``self`` and

397	434
398	435	ret.addErrback(onErr)
399	436
400		# If our cache_key is a string on py2, try to convert to ascii
401		# to save a bit of space in large caches. Py3 does this
402		# internally automatically.
403		if six.PY2 and isinstance(cache_key, string_types):
404		cache_key = to_ascii(cache_key)
405
406		result_d = ObservableDeferred(ret, consumeErrors=True)
407		cache.set(cache_key, result_d, callback=invalidate_callback)
	437	result_d = cache.set(cache_key, ret, callback=invalidate_callback)
408	438	observer = result_d.observe()
409	439
410		if isinstance(observer, defer.Deferred):
411		return make_deferred_yieldable(observer)
412		else:
413		return observer
	440	return make_deferred_yieldable(observer)
414	441
415	442	if self.num_args == 1:
416	443	wrapped.invalidate = lambda key: cache.invalidate(key[0])

526	553	missing.add(arg)
527	554
528	555	if missing:
529		# we need an observable deferred for each entry in the list,
	556	# we need a deferred for each entry in the list,
530	557	# which we put in the cache. Each deferred resolves with the
531	558	# relevant result for that key.
532	559	deferreds_map = {}

534	561	deferred = defer.Deferred()
535	562	deferreds_map[arg] = deferred
536	563	key = arg_to_cache_key(arg)
537		observable = ObservableDeferred(deferred)
538		cache.set(key, observable, callback=invalidate_callback)
	564	cache.set(key, deferred, callback=invalidate_callback)
539	565
540	566	def complete_all(res):
541	567	# the wrapped function has completed. It returns a

+1

-1

synapse/util/caches/response_cache.py less more

120	120	@defer.inlineCallbacks
121	121	def handle_request(request):
122	122	# etc
123		defer.returnValue(result)
	123	return result
124	124
125	125	result = yield response_cache.wrap(
126	126	key,

+1

-1

synapse/util/metrics.py less more

66	66	def measured_func(self, args, *kwargs):
67	67	with Measure(self.clock, name):
68	68	r = yield func(self, args, *kwargs)
69		defer.returnValue(r)
	69	return r
70	70
71	71	return measured_func
72	72

+7

-9

synapse/util/retryutils.py less more

94	94	# maximum backoff even though it might only have been down briefly
95	95	backoff_on_failure = not ignore_backoff
96	96
97		defer.returnValue(
98		RetryDestinationLimiter(
99		destination,
100		clock,
101		store,
102		retry_interval,
103		backoff_on_failure=backoff_on_failure,
104		**kwargs
105		)
	97	return RetryDestinationLimiter(
	98	destination,
	99	clock,
	100	store,
	101	retry_interval,
	102	backoff_on_failure=backoff_on_failure,
	103	**kwargs
106	104	)
107	105
108	106

+21

-2

synapse/util/versionstring.py less more

21	21
22	22
23	23	def get_version_string(module):
	24	"""Given a module calculate a git-aware version string for it.
	25
	26	If called on a module not in a git checkout will return `__verison__`.
	27
	28	Args:
	29	module (module)
	30
	31	Returns:
	32	str
	33	"""
	34
	35	cached_version = getattr(module, "_synapse_version_string_cache", None)
	36	if cached_version:
	37	return cached_version
	38
	39	version_string = module.__version__
	40
24	41	try:
25	42	null = open(os.devnull, "w")
26	43	cwd = os.path.dirname(os.path.abspath(module.__file__))

79	96	s for s in (git_branch, git_tag, git_commit, git_dirty) if s
80	97	)
81	98
82		return "%s (%s)" % (module.__version__, git_version)
	99	version_string = "%s (%s)" % (module.__version__, git_version)
83	100	except Exception as e:
84	101	logger.info("Failed to check for git repository: %s", e)
85	102
86		return module.__version__
	103	module._synapse_version_string_cache = version_string
	104
	105	return version_string

+4

-4

synapse/visibility.py less more

207	207	filtered_events = filter(operator.truth, filtered_events)
208	208
209	209	# we turn it into a list before returning it.
210		defer.returnValue(list(filtered_events))
	210	return list(filtered_events)
211	211
212	212
213	213	@defer.inlineCallbacks

316	316	elif redact:
317	317	to_return.append(prune_event(e))
318	318
319		defer.returnValue(to_return)
	319	return to_return
320	320
321	321	# If there are no erased users then we can just return the given list
322	322	# of events without having to copy it.
323		defer.returnValue(events)
	323	return events
324	324
325	325	# Ok, so we're dealing with events that have non-trivial visibility
326	326	# rules, so we need to also get the memberships of the room.

383	383	elif redact:
384	384	to_return.append(prune_event(e))
385	385
386		defer.returnValue(to_return)
	386	return to_return

+3

-32

tests/crypto/test_keyring.py less more

85	85	getattr(LoggingContext.current_context(), "request", None), expected
86	86	)
87	87
88		def test_wait_for_previous_lookups(self):
89		kr = keyring.Keyring(self.hs)
90
91		lookup_1_deferred = defer.Deferred()
92		lookup_2_deferred = defer.Deferred()
93
94		# we run the lookup in a logcontext so that the patched inlineCallbacks can check
95		# it is doing the right thing with logcontexts.
96		wait_1_deferred = run_in_context(
97		kr.wait_for_previous_lookups, {"server1": lookup_1_deferred}
98		)
99
100		# there were no previous lookups, so the deferred should be ready
101		self.successResultOf(wait_1_deferred)
102
103		# set off another wait. It should block because the first lookup
104		# hasn't yet completed.
105		wait_2_deferred = run_in_context(
106		kr.wait_for_previous_lookups, {"server1": lookup_2_deferred}
107		)
108
109		self.assertFalse(wait_2_deferred.called)
110
111		# let the first lookup complete (in the sentinel context)
112		lookup_1_deferred.callback(None)
113
114		# now the second wait should complete.
115		self.successResultOf(wait_2_deferred)
116
117	88	def test_verify_json_objects_for_server_awaits_previous_requests(self):
118	89	key1 = signedjson.key.generate_signing_key(1)
119	90

135	106	self.assertEquals(LoggingContext.current_context().request, "11")
136	107	with PreserveLoggingContext():
137	108	yield persp_deferred
138		defer.returnValue(persp_resp)
	109	return persp_resp
139	110
140	111	self.http_client.post_json.side_effect = get_perspectives
141	112

582	553	# logs.
583	554	ctx.request = "testctx"
584	555	rv = yield f(args, *kwargs)
585		defer.returnValue(rv)
	556	return rv
586	557
587	558
588	559	def _verify_json_for_server(kr, *args):

593	564	@defer.inlineCallbacks
594	565	def v():
595	566	rv1 = yield kr.verify_json_for_server(*args)
596		defer.returnValue(rv1)
	567	return rv1
597	568
598	569	return run_in_context(v)

+74

-3

tests/federation/test_complexity.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
	15	from mock import Mock
	16
15	17	from twisted.internet import defer
16	18
	19	from synapse.api.errors import Codes, SynapseError
17	20	from synapse.config.ratelimiting import FederationRateLimitConfig
18	21	from synapse.federation.transport import server
19	22	from synapse.rest import admin
20	23	from synapse.rest.client.v1 import login, room
	24	from synapse.types import UserID
21	25	from synapse.util.ratelimitutils import FederationRateLimiter
22	26
23	27	from tests import unittest

32	36	]
33	37
34	38	def default_config(self, name="test"):
35		config = super(RoomComplexityTests, self).default_config(name=name)
36		config["limit_large_remote_room_joins"] = True
37		config["limit_large_remote_room_complexity"] = 0.05
	39	config = super().default_config(name=name)
	40	config["limit_remote_rooms"] = {"enabled": True, "complexity": 0.05}
38	41	return config
39	42
40	43	def prepare(self, reactor, clock, homeserver):

87	90	self.assertEquals(200, channel.code)
88	91	complexity = channel.json_body["v1"]
89	92	self.assertEqual(complexity, 1.23)
	93
	94	def test_join_too_large(self):
	95
	96	u1 = self.register_user("u1", "pass")
	97
	98	handler = self.hs.get_room_member_handler()
	99	fed_transport = self.hs.get_federation_transport_client()
	100
	101	# Mock out some things, because we don't want to test the whole join
	102	fed_transport.client.get_json = Mock(return_value=defer.succeed({"v1": 9999}))
	103	handler.federation_handler.do_invite_join = Mock(return_value=defer.succeed(1))
	104
	105	d = handler._remote_join(
	106	None,
	107	["otherserver.example"],
	108	"roomid",
	109	UserID.from_string(u1),
	110	{"membership": "join"},
	111	)
	112
	113	self.pump()
	114
	115	# The request failed with a SynapseError saying the resource limit was
	116	# exceeded.
	117	f = self.get_failure(d, SynapseError)
	118	self.assertEqual(f.value.code, 400, f.value)
	119	self.assertEqual(f.value.errcode, Codes.RESOURCE_LIMIT_EXCEEDED)
	120
	121	def test_join_too_large_once_joined(self):
	122
	123	u1 = self.register_user("u1", "pass")
	124	u1_token = self.login("u1", "pass")
	125
	126	# Ok, this might seem a bit weird -- I want to test that we actually
	127	# leave the room, but I don't want to simulate two servers. So, we make
	128	# a local room, which we say we're joining remotely, even if there's no
	129	# remote, because we mock that out. Then, we'll leave the (actually
	130	# local) room, which will be propagated over federation in a real
	131	# scenario.
	132	room_1 = self.helper.create_room_as(u1, tok=u1_token)
	133
	134	handler = self.hs.get_room_member_handler()
	135	fed_transport = self.hs.get_federation_transport_client()
	136
	137	# Mock out some things, because we don't want to test the whole join
	138	fed_transport.client.get_json = Mock(return_value=defer.succeed(None))
	139	handler.federation_handler.do_invite_join = Mock(return_value=defer.succeed(1))
	140
	141	# Artificially raise the complexity
	142	self.hs.get_datastore().get_current_state_event_counts = lambda x: defer.succeed(
	143	600
	144	)
	145
	146	d = handler._remote_join(
	147	None,
	148	["otherserver.example"],
	149	room_1,
	150	UserID.from_string(u1),
	151	{"membership": "join"},
	152	)
	153
	154	self.pump()
	155
	156	# The request failed with a SynapseError saying the resource limit was
	157	# exceeded.
	158	f = self.get_failure(d, SynapseError)
	159	self.assertEqual(f.value.code, 400)
	160	self.assertEqual(f.value.errcode, Codes.RESOURCE_LIMIT_EXCEEDED)

+2

-2

tests/handlers/test_register.py less more

43	43	hs_config["max_mau_value"] = 50
44	44	hs_config["limit_usage_by_mau"] = True
45	45
46		hs = self.setup_test_homeserver(config=hs_config, expire_access_token=True)
	46	hs = self.setup_test_homeserver(config=hs_config)
47	47	return hs
48	48
49	49	def prepare(self, reactor, clock, hs):

282	282	user, requester, displayname, by_admin=True
283	283	)
284	284
285		defer.returnValue((user_id, token))
	285	return (user_id, token)

+28

-29

tests/http/federation/test_matrix_federation_agent.py less more

24	24	from twisted.internet.protocol import Factory
25	25	from twisted.protocols.tls import TLSMemoryBIOFactory
26	26	from twisted.web._newclient import ResponseNeverReceived
	27	from twisted.web.client import Agent
27	28	from twisted.web.http import HTTPChannel
28	29	from twisted.web.http_headers import Headers
29	30	from twisted.web.iweb import IPolicyForHTTPS
30	31
31	32	from synapse.config.homeserver import HomeServerConfig
32	33	from synapse.crypto.context_factory import ClientTLSOptionsFactory
33		from synapse.http.federation.matrix_federation_agent import (
34		MatrixFederationAgent,
	34	from synapse.http.federation.matrix_federation_agent import MatrixFederationAgent
	35	from synapse.http.federation.srv_resolver import Server
	36	from synapse.http.federation.well_known_resolver import (
	37	WellKnownResolver,
35	38	_cache_period_from_headers,
36	39	)
37		from synapse.http.federation.srv_resolver import Server
38	40	from synapse.logging.context import LoggingContext
39	41	from synapse.util.caches.ttlcache import TTLCache
40	42

74	76
75	77	config_dict = default_config("test", parse=False)
76	78	config_dict["federation_custom_ca_list"] = [get_test_ca_cert_file()]
77		# config_dict["trusted_key_servers"] = []
78	79
79	80	self._config = config = HomeServerConfig()
80	81	config.parse_config_dict(config_dict, "", "")
81	82
	83	self.tls_factory = ClientTLSOptionsFactory(config)
82	84	self.agent = MatrixFederationAgent(
83	85	reactor=self.reactor,
84		tls_client_options_factory=ClientTLSOptionsFactory(config),
85		_well_known_tls_policy=TrustingTLSPolicyForHTTPS(),
	86	tls_client_options_factory=self.tls_factory,
86	87	_srv_resolver=self.mock_resolver,
87	88	_well_known_cache=self.well_known_cache,
88	89	)

144	145
145	146	try:
146	147	fetch_res = yield fetch_d
147		defer.returnValue(fetch_res)
	148	return fetch_res
148	149	except Exception as e:
149	150	logger.info("Fetch of %s failed: %s", uri.decode("ascii"), e)
150	151	raise

690	691	not signed by a CA
691	692	"""
692	693
693		# we use the same test server as the other tests, but use an agent
694		# with _well_known_tls_policy left to the default, which will not
695		# trust it (since the presented cert is signed by a test CA)
	694	# we use the same test server as the other tests, but use an agent with
	695	# the config left to the default, which will not trust it (since the
	696	# presented cert is signed by a test CA)
696	697
697	698	self.mock_resolver.resolve_service.side_effect = lambda _: []
698	699	self.reactor.lookups["testserv"] = "1.2.3.4"
699	700
	701	config = default_config("test", parse=True)
	702
700	703	agent = MatrixFederationAgent(
701	704	reactor=self.reactor,
702		tls_client_options_factory=ClientTLSOptionsFactory(self._config),
	705	tls_client_options_factory=ClientTLSOptionsFactory(config),
703	706	_srv_resolver=self.mock_resolver,
704	707	_well_known_cache=self.well_known_cache,
705	708	)

927	930	self.reactor.pump((0.1,))
928	931	self.successResultOf(test_d)
929	932
930		@defer.inlineCallbacks
931		def do_get_well_known(self, serv):
932		try:
933		result = yield self.agent._get_well_known(serv)
934		logger.info("Result from well-known fetch: %s", result)
935		except Exception as e:
936		logger.warning("Error fetching well-known: %s", e)
937		raise
938		defer.returnValue(result)
939
940	933	def test_well_known_cache(self):
	934	well_known_resolver = WellKnownResolver(
	935	self.reactor,
	936	Agent(self.reactor, contextFactory=self.tls_factory),
	937	well_known_cache=self.well_known_cache,
	938	)
	939
941	940	self.reactor.lookups["testserv"] = "1.2.3.4"
942	941
943		fetch_d = self.do_get_well_known(b"testserv")
	942	fetch_d = well_known_resolver.get_well_known(b"testserv")
944	943
945	944	# there should be an attempt to connect on port 443 for the .well-known
946	945	clients = self.reactor.tcpClients

952	951	well_known_server = self._handle_well_known_connection(
953	952	client_factory,
954	953	expected_sni=b"testserv",
955		response_headers={b"Cache-Control": b"max-age=10"},
	954	response_headers={b"Cache-Control": b"max-age=1000"},
956	955	content=b'{ "m.server": "target-server" }',
957	956	)
958	957
959	958	r = self.successResultOf(fetch_d)
960		self.assertEqual(r, b"target-server")
	959	self.assertEqual(r.delegated_server, b"target-server")
961	960
962	961	# close the tcp connection
963	962	well_known_server.loseConnection()
964	963
965	964	# repeat the request: it should hit the cache
966		fetch_d = self.do_get_well_known(b"testserv")
	965	fetch_d = well_known_resolver.get_well_known(b"testserv")
967	966	r = self.successResultOf(fetch_d)
968		self.assertEqual(r, b"target-server")
	967	self.assertEqual(r.delegated_server, b"target-server")
969	968
970	969	# expire the cache
971		self.reactor.pump((10.0,))
	970	self.reactor.pump((1000.0,))
972	971
973	972	# now it should connect again
974		fetch_d = self.do_get_well_known(b"testserv")
	973	fetch_d = well_known_resolver.get_well_known(b"testserv")
975	974
976	975	self.assertEqual(len(clients), 1)
977	976	(host, port, client_factory, _timeout, _bindAddress) = clients.pop(0)

985	984	)
986	985
987	986	r = self.successResultOf(fetch_d)
988		self.assertEqual(r, b"other-server")
	987	self.assertEqual(r.delegated_server, b"other-server")
989	988
990	989
991	990	class TestCachePeriodFromHeaders(TestCase):

+1

-1

tests/http/federation/test_srv_resolver.py less more

60	60	# should have restored our context
61	61	self.assertIs(LoggingContext.current_context(), ctx)
62	62
63		defer.returnValue(result)
	63	return result
64	64
65	65	test_d = do_lookup()
66	66	self.assertNoResult(test_d)

+1

-1

tests/http/test_fedclient.py less more

67	67
68	68	try:
69	69	fetch_res = yield fetch_d
70		defer.returnValue(fetch_res)
	70	return fetch_res
71	71	finally:
72	72	check_logcontext(context)
73	73

+1

-1

tests/rest/client/test_transactions.py less more

45	45	@defer.inlineCallbacks
46	46	def cb():
47	47	yield Clock(reactor).sleep(0)
48		defer.returnValue("yay")
	48	return "yay"
49	49
50	50	@defer.inlineCallbacks
51	51	def test():

+37

-0

tests/rest/client/v2_alpha/test_register.py less more

322	322	"renew_at": 172800000, # Time in ms for 2 days
323	323	"renew_by_email_enabled": True,
324	324	"renew_email_subject": "Renew your account",
	325	"account_renewed_html_path": "account_renewed.html",
	326	"invalid_token_html_path": "invalid_token.html",
325	327	}
326	328
327	329	# Email config.

372	374	self.render(request)
373	375	self.assertEquals(channel.result["code"], b"200", channel.result)
374	376
	377	# Check that we're getting HTML back.
	378	content_type = None
	379	for header in channel.result.get("headers", []):
	380	if header[0] == b"Content-Type":
	381	content_type = header[1]
	382	self.assertEqual(content_type, b"text/html; charset=utf-8", channel.result)
	383
	384	# Check that the HTML we're getting is the one we expect on a successful renewal.
	385	expected_html = self.hs.config.account_validity.account_renewed_html_content
	386	self.assertEqual(
	387	channel.result["body"], expected_html.encode("utf8"), channel.result
	388	)
	389
375	390	# Move 3 days forward. If the renewal failed, every authed request with
376	391	# our access token should be denied from now, otherwise they should
377	392	# succeed.

379	394	request, channel = self.make_request(b"GET", "/sync", access_token=tok)
380	395	self.render(request)
381	396	self.assertEquals(channel.result["code"], b"200", channel.result)
	397
	398	def test_renewal_invalid_token(self):
	399	# Hit the renewal endpoint with an invalid token and check that it behaves as
	400	# expected, i.e. that it responds with 404 Not Found and the correct HTML.
	401	url = "/_matrix/client/unstable/account_validity/renew?token=123"
	402	request, channel = self.make_request(b"GET", url)
	403	self.render(request)
	404	self.assertEquals(channel.result["code"], b"404", channel.result)
	405
	406	# Check that we're getting HTML back.
	407	content_type = None
	408	for header in channel.result.get("headers", []):
	409	if header[0] == b"Content-Type":
	410	content_type = header[1]
	411	self.assertEqual(content_type, b"text/html; charset=utf-8", channel.result)
	412
	413	# Check that the HTML we're getting is the one we expect when using an
	414	# invalid/unknown token.
	415	expected_html = self.hs.config.account_validity.invalid_token_html_content
	416	self.assertEqual(
	417	channel.result["body"], expected_html.encode("utf8"), channel.result
	418	)
382	419
383	420	def test_manual_email_send(self):
384	421	self.email_attempts = []

+1

-1

tests/server_notices/test_resource_limits_server_notices.py less more

35	35	"room_name": "Server Notices",
36	36	}
37	37
38		hs = self.setup_test_homeserver(config=hs_config, expire_access_token=True)
	38	hs = self.setup_test_homeserver(config=hs_config)
39	39	return hs
40	40
41	41	def prepare(self, reactor, clock, hs):

+2

-2

tests/storage/test_background_update.py less more

42	42	"test_update",
43	43	progress,
44	44	)
45		defer.returnValue(count)
	45	return count
46	46
47	47	self.update_handler.side_effect = update
48	48

59	59	@defer.inlineCallbacks
60	60	def update(progress, count):
61	61	yield self.store._end_background_update("test_update")
62		defer.returnValue(count)
	62	return count
63	63
64	64	self.update_handler.side_effect = update
65	65	self.update_handler.reset_mock()

+129

-57

tests/storage/test_redaction.py less more

0	0	# -- coding: utf-8 --
1	1	# Copyright 2014-2016 OpenMarket Ltd
	2	# Copyright 2019 The Matrix.org Foundation C.I.C.
2	3	#
3	4	# Licensed under the Apache License, Version 2.0 (the "License");
4	5	# you may not use this file except in compliance with the License.

22	23	from synapse.types import RoomID, UserID
23	24
24	25	from tests import unittest
25		from tests.utils import create_room, setup_test_homeserver
26
27
28		class RedactionTestCase(unittest.TestCase):
29		@defer.inlineCallbacks
30		def setUp(self):
31		hs = yield setup_test_homeserver(
32		self.addCleanup, resource_for_federation=Mock(), http_client=None
33		)
34
	26	from tests.utils import create_room
	27
	28
	29	class RedactionTestCase(unittest.HomeserverTestCase):
	30	def make_homeserver(self, reactor, clock):
	31	return self.setup_test_homeserver(
	32	resource_for_federation=Mock(), http_client=None
	33	)
	34
	35	def prepare(self, reactor, clock, hs):
35	36	self.store = hs.get_datastore()
36	37	self.event_builder_factory = hs.get_event_builder_factory()
37	38	self.event_creation_handler = hs.get_event_creation_handler()

41	42
42	43	self.room1 = RoomID.from_string("!abc123:test")
43	44
44		yield create_room(hs, self.room1.to_string(), self.u_alice.to_string())
	45	self.get_success(
	46	create_room(hs, self.room1.to_string(), self.u_alice.to_string())
	47	)
45	48
46	49	self.depth = 1
47	50
48		@defer.inlineCallbacks
49	51	def inject_room_member(
50	52	self, room, user, membership, replaces_state=None, extra_content={}
51	53	):

62	64	},
63	65	)
64	66
65		event, context = yield self.event_creation_handler.create_new_client_event(
66		builder
67		)
68
69		yield self.store.persist_event(event, context)
70
71		defer.returnValue(event)
72
73		@defer.inlineCallbacks
	67	event, context = self.get_success(
	68	self.event_creation_handler.create_new_client_event(builder)
	69	)
	70
	71	self.get_success(self.store.persist_event(event, context))
	72
	73	return event
	74
74	75	def inject_message(self, room, user, body):
75	76	self.depth += 1
76	77

85	86	},
86	87	)
87	88
88		event, context = yield self.event_creation_handler.create_new_client_event(
89		builder
90		)
91
92		yield self.store.persist_event(event, context)
93
94		defer.returnValue(event)
95
96		@defer.inlineCallbacks
	89	event, context = self.get_success(
	90	self.event_creation_handler.create_new_client_event(builder)
	91	)
	92
	93	self.get_success(self.store.persist_event(event, context))
	94
	95	return event
	96
97	97	def inject_redaction(self, room, event_id, user, reason):
98	98	builder = self.event_builder_factory.for_room_version(
99	99	RoomVersions.V1,

107	107	},
108	108	)
109	109
110		event, context = yield self.event_creation_handler.create_new_client_event(
111		builder
112		)
113
114		yield self.store.persist_event(event, context)
115
116		@defer.inlineCallbacks
	110	event, context = self.get_success(
	111	self.event_creation_handler.create_new_client_event(builder)
	112	)
	113
	114	self.get_success(self.store.persist_event(event, context))
	115
117	116	def test_redact(self):
118		yield self.inject_room_member(self.room1, self.u_alice, Membership.JOIN)
119
120		msg_event = yield self.inject_message(self.room1, self.u_alice, "t")
	117	self.get_success(
	118	self.inject_room_member(self.room1, self.u_alice, Membership.JOIN)
	119	)
	120
	121	msg_event = self.get_success(self.inject_message(self.room1, self.u_alice, "t"))
121	122
122	123	# Check event has not been redacted:
123		event = yield self.store.get_event(msg_event.event_id)
	124	event = self.get_success(self.store.get_event(msg_event.event_id))
124	125
125	126	self.assertObjectHasAttributes(
126	127	{

135	136
136	137	# Redact event
137	138	reason = "Because I said so"
138		yield self.inject_redaction(
139		self.room1, msg_event.event_id, self.u_alice, reason
140		)
141
142		event = yield self.store.get_event(msg_event.event_id)
	139	self.get_success(
	140	self.inject_redaction(self.room1, msg_event.event_id, self.u_alice, reason)
	141	)
	142
	143	event = self.get_success(self.store.get_event(msg_event.event_id))
143	144
144	145	self.assertEqual(msg_event.event_id, event.event_id)
145	146

163	164	event.unsigned["redacted_because"],
164	165	)
165	166
166		@defer.inlineCallbacks
167	167	def test_redact_join(self):
168		yield self.inject_room_member(self.room1, self.u_alice, Membership.JOIN)
169
170		msg_event = yield self.inject_room_member(
171		self.room1, self.u_bob, Membership.JOIN, extra_content={"blue": "red"}
172		)
173
174		event = yield self.store.get_event(msg_event.event_id)
	168	self.get_success(
	169	self.inject_room_member(self.room1, self.u_alice, Membership.JOIN)
	170	)
	171
	172	msg_event = self.get_success(
	173	self.inject_room_member(
	174	self.room1, self.u_bob, Membership.JOIN, extra_content={"blue": "red"}
	175	)
	176	)
	177
	178	event = self.get_success(self.store.get_event(msg_event.event_id))
175	179
176	180	self.assertObjectHasAttributes(
177	181	{

186	190
187	191	# Redact event
188	192	reason = "Because I said so"
189		yield self.inject_redaction(
190		self.room1, msg_event.event_id, self.u_alice, reason
	193	self.get_success(
	194	self.inject_redaction(self.room1, msg_event.event_id, self.u_alice, reason)
191	195	)
192	196
193	197	# Check redaction
194	198
195		event = yield self.store.get_event(msg_event.event_id)
	199	event = self.get_success(self.store.get_event(msg_event.event_id))
196	200
197	201	self.assertTrue("redacted_because" in event.unsigned)
198	202

213	217	},
214	218	event.unsigned["redacted_because"],
215	219	)
	220
	221	def test_circular_redaction(self):
	222	redaction_event_id1 = "$redaction1_id:test"
	223	redaction_event_id2 = "$redaction2_id:test"
	224
	225	class EventIdManglingBuilder:
	226	def __init__(self, base_builder, event_id):
	227	self._base_builder = base_builder
	228	self._event_id = event_id
	229
	230	@defer.inlineCallbacks
	231	def build(self, prev_event_ids):
	232	built_event = yield self._base_builder.build(prev_event_ids)
	233	built_event.event_id = self._event_id
	234	built_event._event_dict["event_id"] = self._event_id
	235	return built_event
	236
	237	@property
	238	def room_id(self):
	239	return self._base_builder.room_id
	240
	241	event_1, context_1 = self.get_success(
	242	self.event_creation_handler.create_new_client_event(
	243	EventIdManglingBuilder(
	244	self.event_builder_factory.for_room_version(
	245	RoomVersions.V1,
	246	{
	247	"type": EventTypes.Redaction,
	248	"sender": self.u_alice.to_string(),
	249	"room_id": self.room1.to_string(),
	250	"content": {"reason": "test"},
	251	"redacts": redaction_event_id2,
	252	},
	253	),
	254	redaction_event_id1,
	255	)
	256	)
	257	)
	258
	259	self.get_success(self.store.persist_event(event_1, context_1))
	260
	261	event_2, context_2 = self.get_success(
	262	self.event_creation_handler.create_new_client_event(
	263	EventIdManglingBuilder(
	264	self.event_builder_factory.for_room_version(
	265	RoomVersions.V1,
	266	{
	267	"type": EventTypes.Redaction,
	268	"sender": self.u_alice.to_string(),
	269	"room_id": self.room1.to_string(),
	270	"content": {"reason": "test"},
	271	"redacts": redaction_event_id1,
	272	},
	273	),
	274	redaction_event_id2,
	275	)
	276	)
	277	)
	278	self.get_success(self.store.persist_event(event_2, context_2))
	279
	280	# fetch one of the redactions
	281	fetched = self.get_success(self.store.get_event(redaction_event_id1))
	282
	283	# it should have been redacted
	284	self.assertEqual(fetched.unsigned["redacted_by"], redaction_event_id2)
	285	self.assertEqual(
	286	fetched.unsigned["redacted_because"].event_id, redaction_event_id2
	287	)

+37

-2

tests/storage/test_roommember.py less more

19	19
20	20	from synapse.api.constants import EventTypes, Membership
21	21	from synapse.api.room_versions import RoomVersions
22		from synapse.types import RoomID, UserID
	22	from synapse.types import Requester, RoomID, UserID
23	23
24	24	from tests import unittest
25	25	from tests.utils import create_room, setup_test_homeserver

66	66
67	67	yield self.store.persist_event(event, context)
68	68
69		defer.returnValue(event)
	69	return event
70	70
71	71	@defer.inlineCallbacks
72	72	def test_one_member(self):

83	83	)
84	84	],
85	85	)
	86
	87
	88	class CurrentStateMembershipUpdateTestCase(unittest.HomeserverTestCase):
	89	def prepare(self, reactor, clock, homeserver):
	90	self.store = homeserver.get_datastore()
	91	self.room_creator = homeserver.get_room_creation_handler()
	92
	93	def test_can_rerun_update(self):
	94	# First make sure we have completed all updates.
	95	while not self.get_success(self.store.has_completed_background_updates()):
	96	self.get_success(self.store.do_next_background_update(100), by=0.1)
	97
	98	# Now let's create a room, which will insert a membership
	99	user = UserID("alice", "test")
	100	requester = Requester(user, None, False, None, None)
	101	self.get_success(self.room_creator.create_room(requester, {}))
	102
	103	# Register the background update to run again.
	104	self.get_success(
	105	self.store._simple_insert(
	106	table="background_updates",
	107	values={
	108	"update_name": "current_state_events_membership",
	109	"progress_json": "{}",
	110	"depends_on": None,
	111	},
	112	)
	113	)
	114
	115	# ... and tell the DataStore that it hasn't finished all updates yet
	116	self.store._all_done = False
	117
	118	# Now let's actually drive the updates to completion
	119	while not self.get_success(self.store.has_completed_background_updates()):
	120	self.get_success(self.store.do_next_background_update(100), by=0.1)

+1

-1

tests/storage/test_state.py less more

64	64
65	65	yield self.store.persist_event(event, context)
66	66
67		defer.returnValue(event)
	67	return event
68	68
69	69	def assertStateMapEqual(self, s1, s2):
70	70	for t in s1:

+3

-3

tests/test_visibility.py less more

138	138	builder
139	139	)
140	140	yield self.hs.get_datastore().persist_event(event, context)
141		defer.returnValue(event)
	141	return event
142	142
143	143	@defer.inlineCallbacks
144	144	def inject_room_member(self, user_id, membership="join", extra_content={}):

160	160	)
161	161
162	162	yield self.hs.get_datastore().persist_event(event, context)
163		defer.returnValue(event)
	163	return event
164	164
165	165	@defer.inlineCallbacks
166	166	def inject_message(self, user_id, content=None):

181	181	)
182	182
183	183	yield self.hs.get_datastore().persist_event(event, context)
184		defer.returnValue(event)
	184	return event
185	185
186	186	@defer.inlineCallbacks
187	187	def test_large_room(self):

+0

-6

tests/unittest.py less more

22	22
23	23	from canonicaljson import json
24	24
25		import twisted
26		import twisted.logger
27	25	from twisted.internet.defer import Deferred, succeed
28	26	from twisted.python.threadpool import ThreadPool
29	27	from twisted.trial import unittest

79	77
80	78	@around(self)
81	79	def setUp(orig):
82		# enable debugging of delayed calls - this means that we get a
83		# traceback when a unit test exits leaving things on the reactor.
84		twisted.internet.base.DelayedCall.debug = True
85
86	80	# if we're not starting in the sentinel logcontext, then to be honest
87	81	# all future bets are off.
88	82	if LoggingContext.current_context() is not LoggingContext.sentinel:

+91

-7

tests/util/caches/test_descriptors.py less more

26	26	make_deferred_yieldable,
27	27	)
28	28	from synapse.util.caches import descriptors
	29	from synapse.util.caches.descriptors import cached
29	30
30	31	from tests import unittest
31	32

54	55	d2 = defer.Deferred()
55	56	cache.set("key2", d2, partial(record_callback, 1))
56	57
57		# lookup should return the deferreds
58		self.assertIs(cache.get("key1"), d1)
59		self.assertIs(cache.get("key2"), d2)
	58	# lookup should return observable deferreds
	59	self.assertFalse(cache.get("key1").has_called())
	60	self.assertFalse(cache.get("key2").has_called())
60	61
61	62	# let one of the lookups complete
62	63	d2.callback("result2")
	64
	65	# for now at least, the cache will return real results rather than an
	66	# observabledeferred
63	67	self.assertEqual(cache.get("key2"), "result2")
64	68
65	69	# now do the invalidation

144	148	r = yield obj.fn(2, 5)
145	149	self.assertEqual(r, "chips")
146	150	obj.mock.assert_not_called()
	151
	152	def test_cache_with_sync_exception(self):
	153	"""If the wrapped function throws synchronously, things should continue to work
	154	"""
	155
	156	class Cls(object):
	157	@cached()
	158	def fn(self, arg1):
	159	raise SynapseError(100, "mai spoon iz too big!!1")
	160
	161	obj = Cls()
	162
	163	# this should fail immediately
	164	d = obj.fn(1)
	165	self.failureResultOf(d, SynapseError)
	166
	167	# ... leaving the cache empty
	168	self.assertEqual(len(obj.fn.cache.cache), 0)
	169
	170	# and a second call should result in a second exception
	171	d = obj.fn(1)
	172	self.failureResultOf(d, SynapseError)
147	173
148	174	def test_cache_logcontexts(self):
149	175	"""Check that logcontexts are set and restored correctly when

158	184	def inner_fn():
159	185	with PreserveLoggingContext():
160	186	yield complete_lookup
161		defer.returnValue(1)
	187	return 1
162	188
163	189	return inner_fn()
164	190

168	194	c1.name = "c1"
169	195	r = yield obj.fn(1)
170	196	self.assertEqual(LoggingContext.current_context(), c1)
171		defer.returnValue(r)
	197	return r
172	198
173	199	def check_result(r):
174	200	self.assertEqual(r, 1)

221	247
222	248	self.assertEqual(LoggingContext.current_context(), c1)
223	249
	250	# the cache should now be empty
	251	self.assertEqual(len(obj.fn.cache.cache), 0)
	252
224	253	obj = Cls()
225	254
226	255	# set off a deferred which will do a cache lookup

266	295	r = yield obj.fn(2, 3)
267	296	self.assertEqual(r, "chips")
268	297	obj.mock.assert_not_called()
	298
	299	def test_cache_iterable(self):
	300	class Cls(object):
	301	def __init__(self):
	302	self.mock = mock.Mock()
	303
	304	@descriptors.cached(iterable=True)
	305	def fn(self, arg1, arg2):
	306	return self.mock(arg1, arg2)
	307
	308	obj = Cls()
	309
	310	obj.mock.return_value = ["spam", "eggs"]
	311	r = obj.fn(1, 2)
	312	self.assertEqual(r, ["spam", "eggs"])
	313	obj.mock.assert_called_once_with(1, 2)
	314	obj.mock.reset_mock()
	315
	316	# a call with different params should call the mock again
	317	obj.mock.return_value = ["chips"]
	318	r = obj.fn(1, 3)
	319	self.assertEqual(r, ["chips"])
	320	obj.mock.assert_called_once_with(1, 3)
	321	obj.mock.reset_mock()
	322
	323	# the two values should now be cached
	324	self.assertEqual(len(obj.fn.cache.cache), 3)
	325
	326	r = obj.fn(1, 2)
	327	self.assertEqual(r, ["spam", "eggs"])
	328	r = obj.fn(1, 3)
	329	self.assertEqual(r, ["chips"])
	330	obj.mock.assert_not_called()
	331
	332	def test_cache_iterable_with_sync_exception(self):
	333	"""If the wrapped function throws synchronously, things should continue to work
	334	"""
	335
	336	class Cls(object):
	337	@descriptors.cached(iterable=True)
	338	def fn(self, arg1):
	339	raise SynapseError(100, "mai spoon iz too big!!1")
	340
	341	obj = Cls()
	342
	343	# this should fail immediately
	344	d = obj.fn(1)
	345	self.failureResultOf(d, SynapseError)
	346
	347	# ... leaving the cache empty
	348	self.assertEqual(len(obj.fn.cache.cache), 0)
	349
	350	# and a second call should result in a second exception
	351	d = obj.fn(1)
	352	self.failureResultOf(d, SynapseError)
269	353
270	354
271	355	class CachedListDescriptorTestCase(unittest.TestCase):

285	369	# we want this to behave like an asynchronous function
286	370	yield run_on_reactor()
287	371	assert LoggingContext.current_context().request == "c1"
288		defer.returnValue(self.mock(args1, arg2))
	372	return self.mock(args1, arg2)
289	373
290	374	with LoggingContext() as c1:
291	375	c1.request = "c1"

333	417	def list_fn(self, args1, arg2):
334	418	# we want this to behave like an asynchronous function
335	419	yield run_on_reactor()
336		defer.returnValue(self.mock(args1, arg2))
	420	return self.mock(args1, arg2)
337	421
338	422	obj = Cls()
339	423	invalidate0 = mock.Mock()

+3

-4

tests/utils.py less more

125	125	"enable_registration": True,
126	126	"enable_registration_captcha": False,
127	127	"macaroon_secret_key": "not even a little secret",
128		"expire_access_token": False,
129	128	"trusted_third_party_id_servers": [],
130	129	"room_invite_state_types": [],
131	130	"password_providers": [],

360	359	if fed:
361	360	register_federation_servlets(hs, fed)
362	361
363		defer.returnValue(hs)
	362	return hs
364	363
365	364
366	365	def register_federation_servlets(hs, resource):

464	463	args = [urlparse.unquote(u) for u in matcher.groups()]
465	464
466	465	(code, response) = yield func(mock_request, *args)
467		defer.returnValue((code, response))
	466	return (code, response)
468	467	except CodeMessageException as e:
469		defer.returnValue((e.code, cs_error(e.msg, code=e.errcode)))
	468	return (e.code, cs_error(e.msg, code=e.errcode))
470	469
471	470	raise KeyError("No event can handle %s" % path)
472	471