Commit da91dde24c0eb0fc5c596cb1135c1d8574f8e8c3 - matrix-synapse

+7

-4

.travis.yml less more

22	22	- develop
23	23	- /^release-v/
24	24
	25	# When running the tox environments that call Twisted Trial, we can pass the -j
	26	# flag to run the tests concurrently. We set this to 2 for CPU bound tests
	27	# (SQLite) and 4 for I/O bound tests (PostgreSQL).
25	28	matrix:
26	29	fast_finish: true
27	30	include:

32	35	env: TOX_ENV="pep8,check_isort"
33	36
34	37	- python: 2.7
35		env: TOX_ENV=py27
	38	env: TOX_ENV=py27 TRIAL_FLAGS="-j 2"
36	39
37	40	- python: 2.7
38		env: TOX_ENV=py27-old
	41	env: TOX_ENV=py27-old TRIAL_FLAGS="-j 2"
39	42
40	43	- python: 2.7
41	44	env: TOX_ENV=py27-postgres TRIAL_FLAGS="-j 4"

43	46	- postgresql
44	47
45	48	- python: 3.5
46		env: TOX_ENV=py35
	49	env: TOX_ENV=py35 TRIAL_FLAGS="-j 2"
47	50
48	51	- python: 3.6
49		env: TOX_ENV=py36
	52	env: TOX_ENV=py36 TRIAL_FLAGS="-j 2"
50	53
51	54	- python: 3.6
52	55	env: TOX_ENV=py36-postgres TRIAL_FLAGS="-j 4"

+61

-0

CHANGES.md less more

	0	Synapse 0.33.9 (2018-11-19)
	1	===========================
	2
	3	No significant changes.
	4
	5
	6	Synapse 0.33.9rc1 (2018-11-14)
	7	==============================
	8
	9	Features
	10	--------
	11
	12	- Include flags to optionally add `m.login.terms` to the registration flow when consent tracking is enabled. ([\#4004](https://github.com/matrix-org/synapse/issues/4004), [\#4133](https://github.com/matrix-org/synapse/issues/4133), [\#4142](https://github.com/matrix-org/synapse/issues/4142), [\#4184](https://github.com/matrix-org/synapse/issues/4184))
	13	- Support for replacing rooms with new ones ([\#4091](https://github.com/matrix-org/synapse/issues/4091), [\#4099](https://github.com/matrix-org/synapse/issues/4099), [\#4100](https://github.com/matrix-org/synapse/issues/4100), [\#4101](https://github.com/matrix-org/synapse/issues/4101))
	14
	15
	16	Bugfixes
	17	--------
	18
	19	- Fix exceptions when using the email mailer on Python 3. ([\#4095](https://github.com/matrix-org/synapse/issues/4095))
	20	- Fix e2e key backup with more than 9 backup versions ([\#4113](https://github.com/matrix-org/synapse/issues/4113))
	21	- Searches that request profile info now no longer fail with a 500. ([\#4122](https://github.com/matrix-org/synapse/issues/4122))
	22	- fix return code of empty key backups ([\#4123](https://github.com/matrix-org/synapse/issues/4123))
	23	- If the typing stream ID goes backwards (as on a worker when the master restarts), the worker's typing handler will no longer erroneously report rooms containing new typing events. ([\#4127](https://github.com/matrix-org/synapse/issues/4127))
	24	- Fix table lock of device_lists_remote_cache which could freeze the application ([\#4132](https://github.com/matrix-org/synapse/issues/4132))
	25	- Fix exception when using state res v2 algorithm ([\#4135](https://github.com/matrix-org/synapse/issues/4135))
	26	- Generating the user consent URI no longer fails on Python 3. ([\#4140](https://github.com/matrix-org/synapse/issues/4140), [\#4163](https://github.com/matrix-org/synapse/issues/4163))
	27	- Loading URL previews from the DB cache on Postgres will no longer cause Unicode type errors when responding to the request, and URL previews will no longer fail if the remote server returns a Content-Type header with the chartype in quotes. ([\#4157](https://github.com/matrix-org/synapse/issues/4157))
	28	- The hash_password script now works on Python 3. ([\#4161](https://github.com/matrix-org/synapse/issues/4161))
	29	- Fix noop checks when updating device keys, reducing spurious device list update notifications. ([\#4164](https://github.com/matrix-org/synapse/issues/4164))
	30
	31
	32	Deprecations and Removals
	33	-------------------------
	34
	35	- The disused and un-specced identicon generator has been removed. ([\#4106](https://github.com/matrix-org/synapse/issues/4106))
	36	- The obsolete and non-functional /pull federation endpoint has been removed. ([\#4118](https://github.com/matrix-org/synapse/issues/4118))
	37	- The deprecated v1 key exchange endpoints have been removed. ([\#4119](https://github.com/matrix-org/synapse/issues/4119))
	38	- Synapse will no longer fetch keys using the fallback deprecated v1 key exchange method and will now always use v2. ([\#4120](https://github.com/matrix-org/synapse/issues/4120))
	39
	40
	41	Internal Changes
	42	----------------
	43
	44	- Fix build of Docker image with docker-compose ([\#3778](https://github.com/matrix-org/synapse/issues/3778))
	45	- Delete unreferenced state groups during history purge ([\#4006](https://github.com/matrix-org/synapse/issues/4006))
	46	- The "Received rdata" log messages on workers is now logged at DEBUG, not INFO. ([\#4108](https://github.com/matrix-org/synapse/issues/4108))
	47	- Reduce replication traffic for device lists ([\#4109](https://github.com/matrix-org/synapse/issues/4109))
	48	- Fix `synapse_replication_tcp_protocol_*_commands` metric label to be full command name, rather than just the first character ([\#4110](https://github.com/matrix-org/synapse/issues/4110))
	49	- Log some bits about room creation ([\#4121](https://github.com/matrix-org/synapse/issues/4121))
	50	- Fix `tox` failure on old systems ([\#4124](https://github.com/matrix-org/synapse/issues/4124))
	51	- Add STATE_V2_TEST room version ([\#4128](https://github.com/matrix-org/synapse/issues/4128))
	52	- Clean up event accesses and tests ([\#4137](https://github.com/matrix-org/synapse/issues/4137))
	53	- The default logging config will now set an explicit log file encoding of UTF-8. ([\#4138](https://github.com/matrix-org/synapse/issues/4138))
	54	- Add helpers functions for getting prev and auth events of an event ([\#4139](https://github.com/matrix-org/synapse/issues/4139))
	55	- Add some tests for the HTTP pusher. ([\#4149](https://github.com/matrix-org/synapse/issues/4149))
	56	- add purge_history.sh and purge_remote_media.sh scripts to contrib/ ([\#4155](https://github.com/matrix-org/synapse/issues/4155))
	57	- HTTP tests have been refactored to contain less boilerplate. ([\#4156](https://github.com/matrix-org/synapse/issues/4156))
	58	- Drop incoming events from federation for unknown rooms ([\#4165](https://github.com/matrix-org/synapse/issues/4165))
	59
	60
0	61	Synapse 0.33.8 (2018-11-01)
1	62	===========================
2	63

+5

-3

contrib/docker/docker-compose.yml less more

5	5	services:
6	6
7	7	synapse:
8		build: ../..
	8	build:
	9	context: ../..
	10	dockerfile: docker/Dockerfile
9	11	image: docker.io/matrixdotorg/synapse:latest
10		# Since snyapse does not retry to connect to the database, restart upon
	12	# Since synapse does not retry to connect to the database, restart upon
11	13	# failure
12	14	restart: unless-stopped
13	15	# See the readme for a full documentation of the environment settings

46	48	# You may store the database tables in a local folder..
47	49	- ./schemas:/var/lib/postgresql/data
48	50	# .. or store them on some high performance storage for better results
49		# - /path/to/ssd/storage:/var/lib/postfesql/data
	51	# - /path/to/ssd/storage:/var/lib/postgresql/data

+16

-0

contrib/purge_api/README.md less more

	0	Purge history API examples
	1	==========================
	2
	3	# `purge_history.sh`
	4
	5	A bash file, that uses the [purge history API](/docs/admin_api/README.rst) to
	6	purge all messages in a list of rooms up to a certain event. You can select a
	7	timeframe or a number of messages that you want to keep in the room.
	8
	9	Just configure the variables DOMAIN, ADMIN, ROOMS_ARRAY and TIME at the top of
	10	the script.
	11
	12	# `purge_remote_media.sh`
	13
	14	A bash file, that uses the [purge history API](/docs/admin_api/README.rst) to
	15	purge all old cached remote media.

+141

-0

contrib/purge_api/purge_history.sh less more

	0	#!/bin/bash
	1
	2	# this script will use the api:
	3	# https://github.com/matrix-org/synapse/blob/master/docs/admin_api/purge_history_api.rst
	4	#
	5	# It will purge all messages in a list of rooms up to a cetrain event
	6
	7	###################################################################################################
	8	# define your domain and admin user
	9	###################################################################################################
	10	# add this user as admin in your home server:
	11	DOMAIN=yourserver.tld
	12	# add this user as admin in your home server:
	13	ADMIN="@you_admin_username:$DOMAIN"
	14
	15	API_URL="$DOMAIN:8008/_matrix/client/r0"
	16
	17	###################################################################################################
	18	#choose the rooms to prune old messages from (add a free comment at the end)
	19	###################################################################################################
	20	# the room_id's you can get e.g. from your Riot clients "View Source" button on each message
	21	ROOMS_ARRAY=(
	22	'!DgvjtOljKujDBrxyHk:matrix.org#riot:matrix.org'
	23	'!QtykxKocfZaZOUrTwp:matrix.org#Matrix HQ'
	24	)
	25
	26	# ALTERNATIVELY:
	27	# you can select all the rooms that are not encrypted and loop over the result:
	28	# SELECT room_id FROM rooms WHERE room_id NOT IN (SELECT DISTINCT room_id FROM events WHERE type ='m.room.encrypted')
	29	# or
	30	# select all rooms with at least 100 members:
	31	# SELECT q.room_id FROM (select count(*) as numberofusers, room_id FROM current_state_events WHERE type ='m.room.member'
	32	# GROUP BY room_id) AS q LEFT JOIN room_aliases a ON q.room_id=a.room_id WHERE q.numberofusers > 100 ORDER BY numberofusers desc
	33
	34	###################################################################################################
	35	# evaluate the EVENT_ID before which should be pruned
	36	###################################################################################################
	37	# choose a time before which the messages should be pruned:
	38	TIME='12 months ago'
	39	# ALTERNATIVELY:
	40	# a certain time:
	41	# TIME='2016-08-31 23:59:59'
	42
	43	# creates a timestamp from the given time string:
	44	UNIX_TIMESTAMP=$(date +%s%3N --date='TZ="UTC+2" '"$TIME")
	45
	46	# ALTERNATIVELY:
	47	# prune all messages that are older than 1000 messages ago:
	48	# LAST_MESSAGES=1000
	49	# SQL_GET_EVENT="SELECT event_id from events WHERE type='m.room.message' AND room_id ='$ROOM' ORDER BY received_ts DESC LIMIT 1 offset $(($LAST_MESSAGES - 1))"
	50
	51	# ALTERNATIVELY:
	52	# select the EVENT_ID manually:
	53	#EVENT_ID='$1471814088343495zpPNI:matrix.org' # an example event from 21st of Aug 2016 by Matthew
	54
	55	###################################################################################################
	56	# make the admin user a server admin in the database with
	57	###################################################################################################
	58	# psql -A -t --dbname=synapse -c "UPDATE users SET admin=1 WHERE name LIKE '$ADMIN'"
	59
	60	###################################################################################################
	61	# database function
	62	###################################################################################################
	63	sql (){
	64	# for sqlite3:
	65	#sqlite3 homeserver.db "pragma busy_timeout=20000;$1" \| awk '{print $2}'
	66	# for postgres:
	67	psql -A -t --dbname=synapse -c "$1" \| grep -v 'Pager'
	68	}
	69
	70	###################################################################################################
	71	# get an access token
	72	###################################################################################################
	73	# for example externally by watching Riot in your browser's network inspector
	74	# or internally on the server locally, use this:
	75	TOKEN=$(sql "SELECT token FROM access_tokens WHERE user_id='$ADMIN' ORDER BY id DESC LIMIT 1")
	76	AUTH="Authorization: Bearer $TOKEN"
	77
	78	###################################################################################################
	79	# check, if your TOKEN works. For example this works:
	80	###################################################################################################
	81	# $ curl --header "$AUTH" "$API_URL/rooms/$ROOM/state/m.room.power_levels"
	82
	83	###################################################################################################
	84	# finally start pruning the room:
	85	###################################################################################################
	86	POSTDATA='{"delete_local_events":"true"}' # this will really delete local events, so the messages in the room really disappear unless they are restored by remote federation
	87
	88	for ROOM in "${ROOMS_ARRAY[@]}"; do
	89	echo "########################################### $(date) ################# "
	90	echo "pruning room: $ROOM ..."
	91	ROOM=${ROOM%#*}
	92	#set -x
	93	echo "check for alias in db..."
	94	# for postgres:
	95	sql "SELECT * FROM room_aliases WHERE room_id='$ROOM'"
	96	echo "get event..."
	97	# for postgres:
	98	EVENT_ID=$(sql "SELECT event_id FROM events WHERE type='m.room.message' AND received_ts<'$UNIX_TIMESTAMP' AND room_id='$ROOM' ORDER BY received_ts DESC LIMIT 1;")
	99	if [ "$EVENT_ID" == "" ]; then
	100	echo "no event $TIME"
	101	else
	102	echo "event: $EVENT_ID"
	103	SLEEP=2
	104	set -x
	105	# call purge
	106	OUT=$(curl --header "$AUTH" -s -d $POSTDATA POST "$API_URL/admin/purge_history/$ROOM/$EVENT_ID")
	107	PURGE_ID=$(echo "$OUT" \|grep purge_id\|cut -d'"' -f4 )
	108	if [ "$PURGE_ID" == "" ]; then
	109	# probably the history purge is already in progress for $ROOM
	110	: "continuing with next room"
	111	else
	112	while : ; do
	113	# get status of purge and sleep longer each time if still active
	114	sleep $SLEEP
	115	STATUS=$(curl --header "$AUTH" -s GET "$API_URL/admin/purge_history_status/$PURGE_ID" \|grep status\|cut -d'"' -f4)
	116	: "$ROOM --> Status: $STATUS"
	117	[[ "$STATUS" == "active" ]] \|\| break
	118	SLEEP=$((SLEEP + 1))
	119	done
	120	fi
	121	set +x
	122	sleep 1
	123	fi
	124	done
	125
	126
	127	###################################################################################################
	128	# additionally
	129	###################################################################################################
	130	# to benefit from pruning large amounts of data, you need to call VACUUM to free the unused space.
	131	# This can take a very long time (hours) and the client have to be stopped while you do so:
	132	# $ synctl stop
	133	# $ sqlite3 -line homeserver.db "vacuum;"
	134	# $ synctl start
	135
	136	# This could be set, so you don't need to prune every time after deleting some rows:
	137	# $ sqlite3 homeserver.db "PRAGMA auto_vacuum = FULL;"
	138	# be cautious, it could make the database somewhat slow if there are a lot of deletions
	139
	140	exit

+54

-0

contrib/purge_api/purge_remote_media.sh less more

	0	#!/bin/bash
	1
	2	DOMAIN=yourserver.tld
	3	# add this user as admin in your home server:
	4	ADMIN="@you_admin_username:$DOMAIN"
	5
	6	API_URL="$DOMAIN:8008/_matrix/client/r0"
	7
	8	# choose a time before which the messages should be pruned:
	9	# TIME='2016-08-31 23:59:59'
	10	TIME='12 months ago'
	11
	12	# creates a timestamp from the given time string:
	13	UNIX_TIMESTAMP=$(date +%s%3N --date='TZ="UTC+2" '"$TIME")
	14
	15
	16	###################################################################################################
	17	# database function
	18	###################################################################################################
	19	sql (){
	20	# for sqlite3:
	21	#sqlite3 homeserver.db "pragma busy_timeout=20000;$1" \| awk '{print $2}'
	22	# for postgres:
	23	psql -A -t --dbname=synapse -c "$1" \| grep -v 'Pager'
	24	}
	25
	26	###############################################################################
	27	# make the admin user a server admin in the database with
	28	###############################################################################
	29	# sql "UPDATE users SET admin=1 WHERE name LIKE '$ADMIN'"
	30
	31	###############################################################################
	32	# get an access token
	33	###############################################################################
	34	# for example externally by watching Riot in your browser's network inspector
	35	# or internally on the server locally, use this:
	36	TOKEN=$(sql "SELECT token FROM access_tokens WHERE user_id='$ADMIN' ORDER BY id DESC LIMIT 1")
	37
	38	###############################################################################
	39	# check, if your TOKEN works. For example this works:
	40	###############################################################################
	41	# curl --header "Authorization: Bearer $TOKEN" "$API_URL/rooms/$ROOM/state/m.room.power_levels"
	42
	43	###############################################################################
	44	# optional check size before
	45	###############################################################################
	46	# echo calculate used storage before ...
	47	# du -shc ../.synapse/media_store/*
	48
	49	###############################################################################
	50	# finally start pruning media:
	51	###############################################################################
	52	set -x # for debugging the generated string
	53	curl --header "Authorization: Bearer $TOKEN" -v POST "$API_URL/admin/purge_media_cache/?before_ts=$UNIX_TIMESTAMP"

+38

-1

docs/consent_tracking.md less more

30	30	template - currently this must always be `en` (for "English");
31	31	internationalisation support is intended for the future.
32	32
33		The template for the policy itself should be versioned and named according to
	33	The template for the policy itself should be versioned and named according to
34	34	the version: for example `1.0.html`. The version of the policy which the user
35	35	has agreed to is stored in the database.
36	36

84	84	an error "Missing string query parameter 'u'". It is now possible to manually
85	85	construct URIs where users can give their consent.
86	86
	87	### Enabling consent tracking at registration
	88
	89	1. Add the following to your configuration:
	90
	91	```yaml
	92	user_consent:
	93	require_at_registration: true
	94	policy_name: "Privacy Policy" # or whatever you'd like to call the policy
	95	```
	96
	97	2. In your consent templates, make use of the `public_version` variable to
	98	see if an unauthenticated user is viewing the page. This is typically
	99	wrapped around the form that would be used to actually agree to the document:
	100
	101	```
	102	{% if not public_version %}
	103	<!-- The variables used here are only provided when the 'u' param is given to the homeserver -->
	104	<form method="post" action="consent">
	105	<input type="hidden" name="v" value="{{version}}"/>
	106	<input type="hidden" name="u" value="{{user}}"/>
	107	<input type="hidden" name="h" value="{{userhmac}}"/>
	108	<input type="submit" value="Sure thing!"/>
	109	</form>
	110	{% endif %}
	111	```
	112
	113	3. Restart Synapse to apply the changes.
	114
	115	Visiting `https://<server>/_matrix/consent` should now give you a view of the privacy
	116	document. This is what users will be able to see when registering for accounts.
	117
87	118	### Constructing the consent URI
88	119
89	120	It may be useful to manually construct the "consent URI" for a given user - for

103	134
104	135	This should result in a URI which looks something like:
105	136	`https://<server>/_matrix/consent?u=<user>&h=68a152465a4d...`.
	137
	138
	139	Note that not providing a `u` parameter will be interpreted as wanting to view
	140	the document from an unauthenticated perspective, such as prior to registration.
	141	Therefore, the `h` parameter is not required in this scenario. To enable this
	142	behaviour, set `require_at_registration` to `true` in your `user_consent` config.
106	143
107	144
108	145	Sending users a server notice asking them to agree to the policy

+9

-6

docs/privacy_policy_templates/en/1.0.html less more

11	11	<p>
12	12	All your base are belong to us.
13	13	</p>
14		<form method="post" action="consent">
15		<input type="hidden" name="v" value="{{version}}"/>
16		<input type="hidden" name="u" value="{{user}}"/>
17		<input type="hidden" name="h" value="{{userhmac}}"/>
18		<input type="submit" value="Sure thing!"/>
19		</form>
	14	{% if not public_version %}
	15	<!-- The variables used here are only provided when the 'u' param is given to the homeserver -->
	16	<form method="post" action="consent">
	17	<input type="hidden" name="v" value="{{version}}"/>
	18	<input type="hidden" name="u" value="{{user}}"/>
	19	<input type="hidden" name="h" value="{{userhmac}}"/>
	20	<input type="submit" value="Sure thing!"/>
	21	</form>
	22	{% endif %}
20	23	{% endif %}
21	24	</body>
22	25	</html>

+0

-19

jenkins/prepare_synapse.sh less more

13	13
14	14	# set up the virtualenv
15	15	tox -e py27 --notest -v
16
17		TOX_BIN=$TOX_DIR/py27/bin
18
19		# cryptography 2.2 requires setuptools >= 18.5.
20		#
21		# older versions of virtualenv (?) give us a virtualenv with the same version
22		# of setuptools as is installed on the system python (and tox runs virtualenv
23		# under python3, so we get the version of setuptools that is installed on that).
24		#
25		# anyway, make sure that we have a recent enough setuptools.
26		$TOX_BIN/pip install 'setuptools>=18.5'
27
28		# we also need a semi-recent version of pip, because old ones fail to install
29		# the "enum34" dependency of cryptography.
30		$TOX_BIN/pip install 'pip>=10'
31
32		{ python synapse/python_dependencies.py
33		echo lxml
34		} \| xargs $TOX_BIN/pip install

+34

-7

scripts/hash_password less more

2	2	import argparse
3	3	import getpass
4	4	import sys
	5	import unicodedata
5	6
6	7	import bcrypt
7	8	import yaml
8	9
9		bcrypt_rounds=12
	10	bcrypt_rounds = 12
10	11	password_pepper = ""
	12
11	13
12	14	def prompt_for_pass():
13	15	password = getpass.getpass("Password: ")

22	24
23	25	return password
24	26
	27
25	28	if __name__ == "__main__":
26	29	parser = argparse.ArgumentParser(
27		description="Calculate the hash of a new password, so that passwords"
28		" can be reset")
	30	description=(
	31	"Calculate the hash of a new password, so that passwords can be reset"
	32	)
	33	)
29	34	parser.add_argument(
30		"-p", "--password",
	35	"-p",
	36	"--password",
31	37	default=None,
32	38	help="New password for user. Will prompt if omitted.",
33	39	)
34	40	parser.add_argument(
35		"-c", "--config",
	41	"-c",
	42	"--config",
36	43	type=argparse.FileType('r'),
37		help="Path to server config file. Used to read in bcrypt_rounds and password_pepper.",
	44	help=(
	45	"Path to server config file. "
	46	"Used to read in bcrypt_rounds and password_pepper."
	47	),
38	48	)
39	49
40	50	args = parser.parse_args()

48	58	if not password:
49	59	password = prompt_for_pass()
50	60
51		print bcrypt.hashpw(password + password_pepper, bcrypt.gensalt(bcrypt_rounds))
	61	# On Python 2, make sure we decode it to Unicode before we normalise it
	62	if isinstance(password, bytes):
	63	try:
	64	password = password.decode(sys.stdin.encoding)
	65	except UnicodeDecodeError:
	66	print(
	67	"ERROR! Your password is not decodable using your terminal encoding (%s)."
	68	% (sys.stdin.encoding,)
	69	)
	70
	71	pw = unicodedata.normalize("NFKC", password)
	72
	73	hashed = bcrypt.hashpw(
	74	pw.encode('utf8') + password_pepper.encode("utf8"),
	75	bcrypt.gensalt(bcrypt_rounds),
	76	).decode('ascii')
	77
	78	print(hashed)

+7

-2

scripts-dev/federation_client.py less more

153	153	s = requests.Session()
154	154	s.mount("matrix://", MatrixConnectionAdapter())
155	155
	156	headers = {"Host": destination, "Authorization": authorization_headers[0]}
	157
	158	if method == "POST":
	159	headers["Content-Type"] = "application/json"
	160
156	161	result = s.request(
157	162	method=method,
158	163	url=dest,
159		headers={"Host": destination, "Authorization": authorization_headers[0]},
	164	headers=headers,
160	165	verify=False,
161	166	data=content,
162	167	)

202	207	parser.add_argument(
203	208	"-X",
204	209	"--method",
205		help="HTTP method to use for the request. Defaults to GET if --data is"
	210	help="HTTP method to use for the request. Defaults to GET if --body is"
206	211	"unspecified, POST if it is.",
207	212	)
208	213

+0

-39

~~scripts-dev/make_identicons.pl~~ less more

0		#!/usr/bin/env perl
1
2		use strict;
3		use warnings;
4
5		use DBI;
6		use DBD::SQLite;
7		use JSON;
8		use Getopt::Long;
9
10		my $db; # = "homeserver.db";
11		my $server = "http://localhost:8008";
12		my $size = 320;
13
14		GetOptions("db\|d=s", \$db,
15		"server\|s=s", \$server,
16		"width\|w=i", \$size) or usage();
17
18		usage() unless $db;
19
20		my $dbh = DBI->connect("dbi:SQLite:dbname=$db","","") \|\| die $DBI::errstr;
21
22		my $res = $dbh->selectall_arrayref("select token, name from access_tokens, users where access_tokens.user_id = users.id group by user_id") \|\| die $DBI::errstr;
23
24		foreach (@$res) {
25		my ($token, $mxid) = ($_->[0], $_->[1]);
26		my ($user_id) = ($mxid =~ m/@(.*):/);
27		my ($url) = $dbh->selectrow_array("select avatar_url from profiles where user_id=?", undef, $user_id);
28		if (!$url \|\| $url =~ /#auto$/) {
29		`curl -s -o tmp.png "$server/_matrix/media/v1/identicon?name=${mxid}&width=$size&height=$size"`;
30		my $json = `curl -s -X POST -H "Content-Type: image/png" -T "tmp.png" $server/_matrix/media/v1/upload?access_token=$token`;
31		my $content_uri = from_json($json)->{content_uri};
32		`curl -X PUT -H "Content-Type: application/json" --data '{ "avatar_url": "${content_uri}#auto"}' $server/_matrix/client/api/v1/profile/${mxid}/avatar_url?access_token=$token`;
33		}
34		}
35
36		sub usage {
37		die "usage: ./make-identicons.pl\n\t-d database [e.g. homeserver.db]\n\t-s homeserver (default: http://localhost:8008)\n\t-w identicon size in pixels (default 320)";
38		}⏎

+1

-1

synapse/__init__.py less more

26	26	except ImportError:
27	27	pass
28	28
29		__version__ = "0.33.8"
	29	__version__ = "0.33.9"

+8

-1

synapse/api/constants.py less more

50	50	EMAIL_IDENTITY = u"m.login.email.identity"
51	51	MSISDN = u"m.login.msisdn"
52	52	RECAPTCHA = u"m.login.recaptcha"
	53	TERMS = u"m.login.terms"
53	54	DUMMY = u"m.login.dummy"
54	55
55	56	# Only for C/S API v1

60	61	class EventTypes(object):
61	62	Member = "m.room.member"
62	63	Create = "m.room.create"
	64	Tombstone = "m.room.tombstone"
63	65	JoinRules = "m.room.join_rules"
64	66	PowerLevels = "m.room.power_levels"
65	67	Aliases = "m.room.aliases"

100	102	class RoomVersions(object):
101	103	V1 = "1"
102	104	VDH_TEST = "vdh-test-version"
	105	STATE_V2_TEST = "state-v2-test"
103	106
104	107
105	108	# the version we will give rooms which are created on this server

107	110
108	111	# vdh-test-version is a placeholder to get room versioning support working and tested
109	112	# until we have a working v2.
110		KNOWN_ROOM_VERSIONS = {RoomVersions.V1, RoomVersions.VDH_TEST}
	113	KNOWN_ROOM_VERSIONS = {
	114	RoomVersions.V1,
	115	RoomVersions.VDH_TEST,
	116	RoomVersions.STATE_V2_TEST,
	117	}
111	118
112	119	ServerNoticeMsgType = "m.server_notice"
113	120	ServerNoticeLimitReached = "m.server_notice.usage_limit_reached"

+0

-1

synapse/api/urls.py less more

27	27	STATIC_PREFIX = "/_matrix/static"
28	28	WEB_CLIENT_PREFIX = "/_matrix/client"
29	29	CONTENT_REPO_PREFIX = "/_matrix/content"
30		SERVER_KEY_PREFIX = "/_matrix/key/v1"
31	30	SERVER_KEY_V2_PREFIX = "/_matrix/key/v2"
32	31	MEDIA_PREFIX = "/_matrix/media/r0"
33	32	LEGACY_MEDIA_PREFIX = "/_matrix/media/v1"

+1

-6

synapse/app/homeserver.py less more

36	36	FEDERATION_PREFIX,
37	37	LEGACY_MEDIA_PREFIX,
38	38	MEDIA_PREFIX,
39		SERVER_KEY_PREFIX,
40	39	SERVER_KEY_V2_PREFIX,
41	40	STATIC_PREFIX,
42	41	WEB_CLIENT_PREFIX,

58	57	from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
59	58	from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
60	59	from synapse.rest import ClientRestResource
61		from synapse.rest.key.v1.server_key_resource import LocalKey
62	60	from synapse.rest.key.v2 import KeyApiV2Resource
63	61	from synapse.rest.media.v0.content_repository import ContentRepoResource
64	62	from synapse.server import HomeServer

235	233	)
236	234
237	235	if name in ["keys", "federation"]:
238		resources.update({
239		SERVER_KEY_PREFIX: LocalKey(self),
240		SERVER_KEY_V2_PREFIX: KeyApiV2Resource(self),
241		})
	236	resources[SERVER_KEY_V2_PREFIX] = KeyApiV2Resource(self)
242	237
243	238	if name == "webclient":
244	239	resources[WEB_CLIENT_PREFIX] = build_resource_for_web_client(self)

+14

-0

synapse/app/synchrotron.py less more

225	225	class SynchrotronTyping(object):
226	226	def __init__(self, hs):
227	227	self._latest_room_serial = 0
	228	self._reset()
	229
	230	def _reset(self):
	231	"""
	232	Reset the typing handler's data caches.
	233	"""
	234	# map room IDs to serial numbers
228	235	self._room_serials = {}
	236	# map room IDs to sets of users currently typing
229	237	self._room_typing = {}
230	238
231	239	def stream_positions(self):

235	243	return {"typing": self._latest_room_serial}
236	244
237	245	def process_replication_rows(self, token, rows):
	246	if self._latest_room_serial > token:
	247	# The master has gone backwards. To prevent inconsistent data, just
	248	# clear everything.
	249	self._reset()
	250
	251	# Set the latest serial token to whatever the server gave us.
238	252	self._latest_room_serial = token
239	253
240	254	for row in rows:

+18

-0

synapse/config/consent_config.py less more

41	41	# until the user consents to the privacy policy. The value of the setting is
42	42	# used as the text of the error.
43	43	#
	44	# 'require_at_registration', if enabled, will add a step to the registration
	45	# process, similar to how captcha works. Users will be required to accept the
	46	# policy before their account is created.
	47	#
	48	# 'policy_name' is the display name of the policy users will see when registering
	49	# for an account. Has no effect unless `require_at_registration` is enabled.
	50	# Defaults to "Privacy Policy".
	51	#
44	52	# user_consent:
45	53	# template_dir: res/templates/privacy
46	54	# version: 1.0

53	61	# block_events_error: >-
54	62	# To continue using this homeserver you must review and agree to the
55	63	# terms and conditions at %(consent_uri)s
	64	# require_at_registration: False
	65	# policy_name: Privacy Policy
56	66	#
57	67	"""
58	68

66	76	self.user_consent_server_notice_content = None
67	77	self.user_consent_server_notice_to_guests = False
68	78	self.block_events_without_consent_error = None
	79	self.user_consent_at_registration = False
	80	self.user_consent_policy_name = "Privacy Policy"
69	81
70	82	def read_config(self, config):
71	83	consent_config = config.get("user_consent")

82	94	self.user_consent_server_notice_to_guests = bool(consent_config.get(
83	95	"send_server_notice_to_guests", False,
84	96	))
	97	self.user_consent_at_registration = bool(consent_config.get(
	98	"require_at_registration", False,
	99	))
	100	self.user_consent_policy_name = consent_config.get(
	101	"policy_name", "Privacy Policy",
	102	)
85	103
86	104	def default_config(self, **kwargs):
87	105	return DEFAULT_CONFIG

+1

-0

synapse/config/logger.py less more

49	49	maxBytes: 104857600
50	50	backupCount: 10
51	51	filters: [context]
	52	encoding: utf8
52	53	console:
53	54	class: logging.StreamHandler
54	55	formatter: precise

+5

-3

synapse/crypto/keyclient.py less more

14	14
15	15	import logging
16	16
	17	from six.moves import urllib
	18
17	19	from canonicaljson import json
18	20
19	21	from twisted.internet import defer, reactor

27	29
28	30	logger = logging.getLogger(__name__)
29	31
30		KEY_API_V1 = b"/_matrix/key/v1/"
	32	KEY_API_V2 = "/_matrix/key/v2/server/%s"
31	33
32	34
33	35	@defer.inlineCallbacks
34		def fetch_server_key(server_name, tls_client_options_factory, path=KEY_API_V1):
	36	def fetch_server_key(server_name, tls_client_options_factory, key_id):
35	37	"""Fetch the keys for a remote server."""
36	38
37	39	factory = SynapseKeyClientFactory()
38		factory.path = path
	40	factory.path = KEY_API_V2 % (urllib.parse.quote(key_id), )
39	41	factory.host = server_name
40	42	endpoint = matrix_federation_endpoint(
41	43	reactor, server_name, tls_client_options_factory, timeout=30

+7

-103

synapse/crypto/keyring.py less more

0	0	# -- coding: utf-8 --
1	1	# Copyright 2014-2016 OpenMarket Ltd
2		# Copyright 2017 New Vector Ltd.
	2	# Copyright 2017, 2018 New Vector Ltd.
3	3	#
4	4	# Licensed under the Apache License, Version 2.0 (the "License");
5	5	# you may not use this file except in compliance with the License.

16	16	import hashlib
17	17	import logging
18	18	from collections import namedtuple
19
20		from six.moves import urllib
21	19
22	20	from signedjson.key import (
23	21	decode_verify_key_bytes,

394	392
395	393	@defer.inlineCallbacks
396	394	def get_keys_from_server(self, server_name_and_key_ids):
397		@defer.inlineCallbacks
398		def get_key(server_name, key_ids):
399		keys = None
400		try:
401		keys = yield self.get_server_verify_key_v2_direct(
402		server_name, key_ids
403		)
404		except Exception as e:
405		logger.info(
406		"Unable to get key %r for %r directly: %s %s",
407		key_ids, server_name,
408		type(e).__name__, str(e),
409		)
410
411		if not keys:
412		keys = yield self.get_server_verify_key_v1_direct(
413		server_name, key_ids
414		)
415
416		keys = {server_name: keys}
417
418		defer.returnValue(keys)
419
420	395	results = yield logcontext.make_deferred_yieldable(defer.gatherResults(
421	396	[
422		run_in_background(get_key, server_name, key_ids)
	397	run_in_background(
	398	self.get_server_verify_key_v2_direct,
	399	server_name,
	400	key_ids,
	401	)
423	402	for server_name, key_ids in server_name_and_key_ids
424	403	],
425	404	consumeErrors=True,

524	503	continue
525	504
526	505	(response, tls_certificate) = yield fetch_server_key(
527		server_name, self.hs.tls_client_options_factory,
528		path=("/_matrix/key/v2/server/%s" % (
529		urllib.parse.quote(requested_key_id),
530		)).encode("ascii"),
	506	server_name, self.hs.tls_client_options_factory, requested_key_id
531	507	)
532	508
533	509	if (u"signatures" not in response

655	631	results[server_name] = response_keys
656	632
657	633	defer.returnValue(results)
658
659		@defer.inlineCallbacks
660		def get_server_verify_key_v1_direct(self, server_name, key_ids):
661		"""Finds a verification key for the server with one of the key ids.
662		Args:
663		server_name (str): The name of the server to fetch a key for.
664		keys_ids (list of str): The key_ids to check for.
665		"""
666
667		# Try to fetch the key from the remote server.
668
669		(response, tls_certificate) = yield fetch_server_key(
670		server_name, self.hs.tls_client_options_factory
671		)
672
673		# Check the response.
674
675		x509_certificate_bytes = crypto.dump_certificate(
676		crypto.FILETYPE_ASN1, tls_certificate
677		)
678
679		if ("signatures" not in response
680		or server_name not in response["signatures"]):
681		raise KeyLookupError("Key response not signed by remote server")
682
683		if "tls_certificate" not in response:
684		raise KeyLookupError("Key response missing TLS certificate")
685
686		tls_certificate_b64 = response["tls_certificate"]
687
688		if encode_base64(x509_certificate_bytes) != tls_certificate_b64:
689		raise KeyLookupError("TLS certificate doesn't match")
690
691		# Cache the result in the datastore.
692
693		time_now_ms = self.clock.time_msec()
694
695		verify_keys = {}
696		for key_id, key_base64 in response["verify_keys"].items():
697		if is_signing_algorithm_supported(key_id):
698		key_bytes = decode_base64(key_base64)
699		verify_key = decode_verify_key_bytes(key_id, key_bytes)
700		verify_key.time_added = time_now_ms
701		verify_keys[key_id] = verify_key
702
703		for key_id in response["signatures"][server_name]:
704		if key_id not in response["verify_keys"]:
705		raise KeyLookupError(
706		"Key response must include verification keys for all"
707		" signatures"
708		)
709		if key_id in verify_keys:
710		verify_signed_json(
711		response,
712		server_name,
713		verify_keys[key_id]
714		)
715
716		yield self.store.store_server_certificate(
717		server_name,
718		server_name,
719		time_now_ms,
720		tls_certificate,
721		)
722
723		yield self.store_keys(
724		server_name=server_name,
725		from_server=server_name,
726		verify_keys=verify_keys,
727		)
728
729		defer.returnValue(verify_keys)
730	634
731	635	def store_keys(self, server_name, from_server, verify_keys):
732	636	"""Store a collection of verify keys for a given server

+2

-2

synapse/event_auth.py less more

199	199	membership = event.content["membership"]
200	200
201	201	# Check if this is the room creator joining:
202		if len(event.prev_events) == 1 and Membership.JOIN == membership:
	202	if len(event.prev_event_ids()) == 1 and Membership.JOIN == membership:
203	203	# Get room creation event:
204	204	key = (EventTypes.Create, "", )
205	205	create = auth_events.get(key)
206		if create and event.prev_events[0][0] == create.event_id:
	206	if create and event.prev_event_ids()[0] == create.event_id:
207	207	if create.content["creator"] == event.state_key:
208	208	return
209	209

+18

-0

synapse/events/__init__.py less more

158	158	def keys(self):
159	159	return six.iterkeys(self._event_dict)
160	160
	161	def prev_event_ids(self):
	162	"""Returns the list of prev event IDs. The order matches the order
	163	specified in the event, though there is no meaning to it.
	164
	165	Returns:
	166	list[str]: The list of event IDs of this event's prev_events
	167	"""
	168	return [e for e, _ in self.prev_events]
	169
	170	def auth_event_ids(self):
	171	"""Returns the list of auth event IDs. The order matches the order
	172	specified in the event, though there is no meaning to it.
	173
	174	Returns:
	175	list[str]: The list of event IDs of this event's auth_events
	176	"""
	177	return [e for e, _ in self.auth_events]
	178
161	179
162	180	class FrozenEvent(EventBase):
163	181	def __init__(self, event_dict, internal_metadata_dict={}, rejected_reason=None):

+23

-6

synapse/federation/federation_server.py less more

161	161	p["age_ts"] = request_time - int(p["age"])
162	162	del p["age"]
163	163
	164	# We try and pull out an event ID so that if later checks fail we
	165	# can log something sensible. We don't mandate an event ID here in
	166	# case future event formats get rid of the key.
	167	possible_event_id = p.get("event_id", "<Unknown>")
	168
	169	# Now we get the room ID so that we can check that we know the
	170	# version of the room.
	171	room_id = p.get("room_id")
	172	if not room_id:
	173	logger.info(
	174	"Ignoring PDU as does not have a room_id. Event ID: %s",
	175	possible_event_id,
	176	)
	177	continue
	178
	179	try:
	180	# In future we will actually use the room version to parse the
	181	# PDU into an event.
	182	yield self.store.get_room_version(room_id)
	183	except NotFoundError:
	184	logger.info("Ignoring PDU for unknown room_id: %s", room_id)
	185	continue
	186
164	187	event = event_from_pdu_json(p)
165		room_id = event.room_id
166	188	pdus_by_room.setdefault(room_id, []).append(event)
167	189
168	190	pdu_results = {}

321	343	)
322	344	else:
323	345	defer.returnValue((404, ""))
324
325		@defer.inlineCallbacks
326		@log_function
327		def on_pull_request(self, origin, versions):
328		raise NotImplementedError("Pull transactions not implemented")
329	346
330	347	@defer.inlineCallbacks
331	348	def on_query_request(self, query_type, args):

+1

-3

synapse/federation/transaction_queue.py less more

182	182	# banned then it won't receive the event because it won't
183	183	# be in the room after the ban.
184	184	destinations = yield self.state.get_current_hosts_in_room(
185		event.room_id, latest_event_ids=[
186		prev_id for prev_id, _ in event.prev_events
187		],
	185	event.room_id, latest_event_ids=event.prev_event_ids(),
188	186	)
189	187	except Exception:
190	188	logger.exception(

+0

-9

synapse/federation/transport/server.py less more

359	359	raise
360	360
361	361	defer.returnValue((code, response))
362
363
364		class FederationPullServlet(BaseFederationServlet):
365		PATH = "/pull/"
366
367		# This is for when someone asks us for everything since version X
368		def on_GET(self, origin, content, query):
369		return self.handler.on_pull_request(query["origin"][0], query["v"])
370	362
371	363
372	364	class FederationEventServlet(BaseFederationServlet):

1260	1252
1261	1253	FEDERATION_SERVLET_CLASSES = (
1262	1254	FederationSendServlet,
1263		FederationPullServlet,
1264	1255	FederationEventServlet,
1265	1256	FederationStateServlet,
1266	1257	FederationStateIdsServlet,

+0

-3

synapse/federation/units.py less more

116	116	"Require 'transaction_id' to construct a Transaction"
117	117	)
118	118
119		for p in pdus:
120		p.transaction_id = kwargs["transaction_id"]
121
122	119	kwargs["pdus"] = [p.get_pdu_json() for p in pdus]
123	120
124	121	return Transaction(**kwargs)

+21

-0

synapse/handlers/auth.py less more

58	58	LoginType.EMAIL_IDENTITY: self._check_email_identity,
59	59	LoginType.MSISDN: self._check_msisdn,
60	60	LoginType.DUMMY: self._check_dummy_auth,
	61	LoginType.TERMS: self._check_terms_auth,
61	62	}
62	63	self.bcrypt_rounds = hs.config.bcrypt_rounds
63	64

430	431	def _check_dummy_auth(self, authdict, _):
431	432	return defer.succeed(True)
432	433
	434	def _check_terms_auth(self, authdict, _):
	435	return defer.succeed(True)
	436
433	437	@defer.inlineCallbacks
434	438	def _check_threepid(self, medium, authdict):
435	439	if 'threepid_creds' not in authdict:

461	465	def _get_params_recaptcha(self):
462	466	return {"public_key": self.hs.config.recaptcha_public_key}
463	467
	468	def _get_params_terms(self):
	469	return {
	470	"policies": {
	471	"privacy_policy": {
	472	"version": self.hs.config.user_consent_version,
	473	"en": {
	474	"name": self.hs.config.user_consent_policy_name,
	475	"url": "%s/_matrix/consent?v=%s" % (
	476	self.hs.config.public_baseurl,
	477	self.hs.config.user_consent_version,
	478	),
	479	},
	480	},
	481	},
	482	}
	483
464	484	def _auth_dict_for_flows(self, flows, session):
465	485	public_flows = []
466	486	for f in flows:

468	488
469	489	get_params = {
470	490	LoginType.RECAPTCHA: self._get_params_recaptcha,
	491	LoginType.TERMS: self._get_params_terms,
471	492	}
472	493
473	494	params = {}

+29

-7

synapse/handlers/directory.py less more

137	137	)
138	138
139	139	@defer.inlineCallbacks
140		def delete_association(self, requester, room_alias):
141		# association deletion for human users
142
	140	def delete_association(self, requester, room_alias, send_event=True):
	141	"""Remove an alias from the directory
	142
	143	(this is only meant for human users; AS users should call
	144	delete_appservice_association)
	145
	146	Args:
	147	requester (Requester):
	148	room_alias (RoomAlias):
	149	send_event (bool): Whether to send an updated m.room.aliases event.
	150	Note that, if we delete the canonical alias, we will always attempt
	151	to send an m.room.canonical_alias event
	152
	153	Returns:
	154	Deferred[unicode]: room id that the alias used to point to
	155
	156	Raises:
	157	NotFoundError: if the alias doesn't exist
	158
	159	AuthError: if the user doesn't have perms to delete the alias (ie, the user
	160	is neither the creator of the alias, nor a server admin.
	161
	162	SynapseError: if the alias belongs to an AS
	163	"""
143	164	user_id = requester.user.to_string()
144	165
145	166	try:

167	188	room_id = yield self._delete_association(room_alias)
168	189
169	190	try:
170		yield self.send_room_alias_update_event(
171		requester,
172		room_id
173		)
	191	if send_event:
	192	yield self.send_room_alias_update_event(
	193	requester,
	194	room_id
	195	)
174	196
175	197	yield self._update_canonical_alias(
176	198	requester,

+15

-7

synapse/handlers/e2e_room_keys.py less more

18	18
19	19	from twisted.internet import defer
20	20
21		from synapse.api.errors import RoomKeysVersionError, StoreError, SynapseError
	21	from synapse.api.errors import NotFoundError, RoomKeysVersionError, StoreError
22	22	from synapse.util.async_helpers import Linearizer
23	23
24	24	logger = logging.getLogger(__name__)

54	54	room_id(string): room ID to get keys for, for None to get keys for all rooms
55	55	session_id(string): session ID to get keys for, for None to get keys for all
56	56	sessions
	57	Raises:
	58	NotFoundError: if the backup version does not exist
57	59	Returns:
58	60	A deferred list of dicts giving the session_data and message metadata for
59	61	these room keys.

62	64	# we deliberately take the lock to get keys so that changing the version
63	65	# works atomically
64	66	with (yield self._upload_linearizer.queue(user_id)):
	67	# make sure the backup version exists
	68	try:
	69	yield self.store.get_e2e_room_keys_version_info(user_id, version)
	70	except StoreError as e:
	71	if e.code == 404:
	72	raise NotFoundError("Unknown backup version")
	73	else:
	74	raise
	75
65	76	results = yield self.store.get_e2e_room_keys(
66	77	user_id, version, room_id, session_id
67	78	)
68
69		if results['rooms'] == {}:
70		raise SynapseError(404, "No room_keys found")
71	79
72	80	defer.returnValue(results)
73	81

119	127	}
120	128
121	129	Raises:
122		SynapseError: with code 404 if there are no versions defined
	130	NotFoundError: if there are no versions defined
123	131	RoomKeysVersionError: if the uploaded version is not the current version
124	132	"""
125	133

133	141	version_info = yield self.store.get_e2e_room_keys_version_info(user_id)
134	142	except StoreError as e:
135	143	if e.code == 404:
136		raise SynapseError(404, "Version '%s' not found" % (version,))
	144	raise NotFoundError("Version '%s' not found" % (version,))
137	145	else:
138	146	raise
139	147

147	155	raise RoomKeysVersionError(current_version=version_info['version'])
148	156	except StoreError as e:
149	157	if e.code == 404:
150		raise SynapseError(404, "Version '%s' not found" % (version,))
	158	raise NotFoundError("Version '%s' not found" % (version,))
151	159	else:
152	160	raise
153	161

+80

-117

synapse/handlers/federation.py less more

201	201	self.room_queues[room_id].append((pdu, origin))
202	202	return
203	203
204		# If we're no longer in the room just ditch the event entirely. This
205		# is probably an old server that has come back and thinks we're still
206		# in the room (or we've been rejoined to the room by a state reset).
	204	# If we're not in the room just ditch the event entirely. This is
	205	# probably an old server that has come back and thinks we're still in
	206	# the room (or we've been rejoined to the room by a state reset).
207	207	#
208		# If we were never in the room then maybe our database got vaped and
209		# we should check if we are in fact in the room. If we are then we
210		# can magically rejoin the room.
	208	# Note that if we were never in the room then we would have already
	209	# dropped the event, since we wouldn't know the room version.
211	210	is_in_room = yield self.auth.check_host_in_room(
212	211	room_id,
213	212	self.server_name
214	213	)
215	214	if not is_in_room:
216		was_in_room = yield self.store.was_host_joined(
217		pdu.room_id, self.server_name,
218		)
219		if was_in_room:
220		logger.info(
221		"[%s %s] Ignoring PDU from %s as we've left the room",
222		room_id, event_id, origin,
223		)
224		defer.returnValue(None)
	215	logger.info(
	216	"[%s %s] Ignoring PDU from %s as we're not in the room",
	217	room_id, event_id, origin,
	218	)
	219	defer.returnValue(None)
225	220
226	221	state = None
227	222	auth_chain = []

238	233	room_id, event_id, min_depth,
239	234	)
240	235
241		prevs = {e_id for e_id, _ in pdu.prev_events}
	236	prevs = set(pdu.prev_event_ids())
242	237	seen = yield self.store.have_seen_events(prevs)
243	238
244	239	if min_depth and pdu.depth < min_depth:

556	551	room_id, event_id, event,
557	552	)
558	553
559		# FIXME (erikj): Awful hack to make the case where we are not currently
560		# in the room work
561		# If state and auth_chain are None, then we don't need to do this check
562		# as we already know we have enough state in the DB to handle this
563		# event.
564		if state and auth_chain and not event.internal_metadata.is_outlier():
565		is_in_room = yield self.auth.check_host_in_room(
566		room_id,
567		self.server_name
568		)
569		else:
570		is_in_room = True
571
572		if not is_in_room:
	554	event_ids = set()
	555	if state:
	556	event_ids \|= {e.event_id for e in state}
	557	if auth_chain:
	558	event_ids \|= {e.event_id for e in auth_chain}
	559
	560	seen_ids = yield self.store.have_seen_events(event_ids)
	561
	562	if state and auth_chain is not None:
	563	# If we have any state or auth_chain given to us by the replication
	564	# layer, then we should handle them (if we haven't before.)
	565
	566	event_infos = []
	567
	568	for e in itertools.chain(auth_chain, state):
	569	if e.event_id in seen_ids:
	570	continue
	571	e.internal_metadata.outlier = True
	572	auth_ids = e.auth_event_ids()
	573	auth = {
	574	(e.type, e.state_key): e for e in auth_chain
	575	if e.event_id in auth_ids or e.type == EventTypes.Create
	576	}
	577	event_infos.append({
	578	"event": e,
	579	"auth_events": auth,
	580	})
	581	seen_ids.add(e.event_id)
	582
573	583	logger.info(
574		"[%s %s] Got event for room we're not in",
575		room_id, event_id,
576		)
577
578		try:
579		yield self._persist_auth_tree(
580		origin, auth_chain, state, event
581		)
582		except AuthError as e:
583		raise FederationError(
584		"ERROR",
585		e.code,
586		e.msg,
587		affected=event_id,
588		)
589
590		else:
591		event_ids = set()
592		if state:
593		event_ids \|= {e.event_id for e in state}
594		if auth_chain:
595		event_ids \|= {e.event_id for e in auth_chain}
596
597		seen_ids = yield self.store.have_seen_events(event_ids)
598
599		if state and auth_chain is not None:
600		# If we have any state or auth_chain given to us by the replication
601		# layer, then we should handle them (if we haven't before.)
602
603		event_infos = []
604
605		for e in itertools.chain(auth_chain, state):
606		if e.event_id in seen_ids:
607		continue
608		e.internal_metadata.outlier = True
609		auth_ids = [e_id for e_id, _ in e.auth_events]
610		auth = {
611		(e.type, e.state_key): e for e in auth_chain
612		if e.event_id in auth_ids or e.type == EventTypes.Create
613		}
614		event_infos.append({
615		"event": e,
616		"auth_events": auth,
617		})
618		seen_ids.add(e.event_id)
619
620		logger.info(
621		"[%s %s] persisting newly-received auth/state events %s",
622		room_id, event_id, [e["event"].event_id for e in event_infos]
623		)
624		yield self._handle_new_events(origin, event_infos)
625
626		try:
627		context = yield self._handle_new_event(
628		origin,
629		event,
630		state=state,
631		)
632		except AuthError as e:
633		raise FederationError(
634		"ERROR",
635		e.code,
636		e.msg,
637		affected=event.event_id,
638		)
	584	"[%s %s] persisting newly-received auth/state events %s",
	585	room_id, event_id, [e["event"].event_id for e in event_infos]
	586	)
	587	yield self._handle_new_events(origin, event_infos)
	588
	589	try:
	590	context = yield self._handle_new_event(
	591	origin,
	592	event,
	593	state=state,
	594	)
	595	except AuthError as e:
	596	raise FederationError(
	597	"ERROR",
	598	e.code,
	599	e.msg,
	600	affected=event.event_id,
	601	)
639	602
640	603	room = yield self.store.get_room(room_id)
641	604

725	688	edges = [
726	689	ev.event_id
727	690	for ev in events
728		if set(e_id for e_id, _ in ev.prev_events) - event_ids
	691	if set(ev.prev_event_ids()) - event_ids
729	692	]
730	693
731	694	logger.info(

752	715	required_auth = set(
753	716	a_id
754	717	for event in events + list(state_events.values()) + list(auth_events.values())
755		for a_id, _ in event.auth_events
	718	for a_id in event.auth_event_ids()
756	719	)
757	720	auth_events.update({
758	721	e_id: event_map[e_id] for e_id in required_auth if e_id in event_map

768	731	auth_events.update(ret_events)
769	732
770	733	required_auth.update(
771		a_id for event in ret_events.values() for a_id, _ in event.auth_events
	734	a_id for event in ret_events.values() for a_id in event.auth_event_ids()
772	735	)
773	736	missing_auth = required_auth - set(auth_events)
774	737

795	758	required_auth.update(
796	759	a_id
797	760	for event in results if event
798		for a_id, _ in event.auth_events
	761	for a_id in event.auth_event_ids()
799	762	)
800	763	missing_auth = required_auth - set(auth_events)
801	764

815	778	"auth_events": {
816	779	(auth_events[a_id].type, auth_events[a_id].state_key):
817	780	auth_events[a_id]
818		for a_id, _ in a.auth_events
	781	for a_id in a.auth_event_ids()
819	782	if a_id in auth_events
820	783	}
821	784	})

827	790	"auth_events": {
828	791	(auth_events[a_id].type, auth_events[a_id].state_key):
829	792	auth_events[a_id]
830		for a_id, _ in event_map[e_id].auth_events
	793	for a_id in event_map[e_id].auth_event_ids()
831	794	if a_id in auth_events
832	795	}
833	796	})

1040	1003	Raises:
1041	1004	SynapseError if the event does not pass muster
1042	1005	"""
1043		if len(ev.prev_events) > 20:
	1006	if len(ev.prev_event_ids()) > 20:
1044	1007	logger.warn("Rejecting event %s which has %i prev_events",
1045		ev.event_id, len(ev.prev_events))
	1008	ev.event_id, len(ev.prev_event_ids()))
1046	1009	raise SynapseError(
1047	1010	http_client.BAD_REQUEST,
1048	1011	"Too many prev_events",
1049	1012	)
1050	1013
1051		if len(ev.auth_events) > 10:
	1014	if len(ev.auth_event_ids()) > 10:
1052	1015	logger.warn("Rejecting event %s which has %i auth_events",
1053		ev.event_id, len(ev.auth_events))
	1016	ev.event_id, len(ev.auth_event_ids()))
1054	1017	raise SynapseError(
1055	1018	http_client.BAD_REQUEST,
1056	1019	"Too many auth_events",

1075	1038	def on_event_auth(self, event_id):
1076	1039	event = yield self.store.get_event(event_id)
1077	1040	auth = yield self.store.get_auth_chain(
1078		[auth_id for auth_id, _ in event.auth_events],
	1041	[auth_id for auth_id in event.auth_event_ids()],
1079	1042	include_given=True
1080	1043	)
1081	1044	defer.returnValue([e for e in auth])

1697	1660
1698	1661	missing_auth_events = set()
1699	1662	for e in itertools.chain(auth_events, state, [event]):
1700		for e_id, _ in e.auth_events:
	1663	for e_id in e.auth_event_ids():
1701	1664	if e_id not in event_map:
1702	1665	missing_auth_events.add(e_id)
1703	1666

1716	1679	for e in itertools.chain(auth_events, state, [event]):
1717	1680	auth_for_e = {
1718	1681	(event_map[e_id].type, event_map[e_id].state_key): event_map[e_id]
1719		for e_id, _ in e.auth_events
	1682	for e_id in e.auth_event_ids()
1720	1683	if e_id in event_map
1721	1684	}
1722	1685	if create_event:

1784	1747
1785	1748	# This is a hack to fix some old rooms where the initial join event
1786	1749	# didn't reference the create event in its auth events.
1787		if event.type == EventTypes.Member and not event.auth_events:
1788		if len(event.prev_events) == 1 and event.depth < 5:
	1750	if event.type == EventTypes.Member and not event.auth_event_ids():
	1751	if len(event.prev_event_ids()) == 1 and event.depth < 5:
1789	1752	c = yield self.store.get_event(
1790		event.prev_events[0][0],
	1753	event.prev_event_ids()[0],
1791	1754	allow_none=True,
1792	1755	)
1793	1756	if c and c.type == EventTypes.Create:

1834	1797
1835	1798	# Now get the current auth_chain for the event.
1836	1799	local_auth_chain = yield self.store.get_auth_chain(
1837		[auth_id for auth_id, _ in event.auth_events],
	1800	[auth_id for auth_id in event.auth_event_ids()],
1838	1801	include_given=True
1839	1802	)
1840	1803

1890	1853	"""
1891	1854	# Check if we have all the auth events.
1892	1855	current_state = set(e.event_id for e in auth_events.values())
1893		event_auth_events = set(e_id for e_id, _ in event.auth_events)
	1856	event_auth_events = set(event.auth_event_ids())
1894	1857
1895	1858	if event.is_state():
1896	1859	event_key = (event.type, event.state_key)

1934	1897	continue
1935	1898
1936	1899	try:
1937		auth_ids = [e_id for e_id, _ in e.auth_events]
	1900	auth_ids = e.auth_event_ids()
1938	1901	auth = {
1939	1902	(e.type, e.state_key): e for e in remote_auth_chain
1940	1903	if e.event_id in auth_ids or e.type == EventTypes.Create

1955	1918	pass
1956	1919
1957	1920	have_events = yield self.store.get_seen_events_with_rejections(
1958		[e_id for e_id, _ in event.auth_events]
	1921	event.auth_event_ids()
1959	1922	)
1960	1923	seen_events = set(have_events.keys())
1961	1924	except Exception:

2057	2020	continue
2058	2021
2059	2022	try:
2060		auth_ids = [e_id for e_id, _ in ev.auth_events]
	2023	auth_ids = ev.auth_event_ids()
2061	2024	auth = {
2062	2025	(e.type, e.state_key): e
2063	2026	for e in result["auth_chain"]

2249	2212	missing_remote_ids = [e.event_id for e in missing_remotes]
2250	2213	base_remote_rejected = list(missing_remotes)
2251	2214	for e in missing_remotes:
2252		for e_id, _ in e.auth_events:
	2215	for e_id in e.auth_event_ids():
2253	2216	if e_id in missing_remote_ids:
2254	2217	try:
2255	2218	base_remote_rejected.remove(e)

+3

-0

synapse/handlers/message.py less more

426	426
427	427	if event.is_state():
428	428	prev_state = yield self.deduplicate_state_event(event, context)
	429	logger.info(
	430	"Not bothering to persist duplicate state event %s", event.event_id,
	431	)
429	432	if prev_state is not None:
430	433	defer.returnValue(prev_state)
431	434

+4

-5

synapse/handlers/register.py less more

49	49	self._auth_handler = hs.get_auth_handler()
50	50	self.profile_handler = hs.get_profile_handler()
51	51	self.user_directory_handler = hs.get_user_directory_handler()
52		self.room_creation_handler = self.hs.get_room_creation_handler()
53	52	self.captcha_client = CaptchaServerHttpClient(hs)
54	53
55	54	self._next_generated_user_id = None

240	239	else:
241	240	# create room expects the localpart of the room alias
242	241	room_alias_localpart = room_alias.localpart
243		yield self.room_creation_handler.create_room(
	242
	243	# getting the RoomCreationHandler during init gives a dependency
	244	# loop
	245	yield self.hs.get_room_creation_handler().create_room(
244	246	fake_requester,
245	247	config={
246	248	"preset": "public_chat",

253	255	except Exception as e:
254	256	logger.error("Failed to join new user to %r: %r", r, e)
255	257
256		# We used to generate default identicons here, but nowadays
257		# we want clients to generate their own as part of their branding
258		# rather than there being consistent matrix-wide ones, so we don't.
259	258	defer.returnValue((user_id, token))
260	259
261	260	@defer.inlineCallbacks

+366

-35

synapse/handlers/room.py less more

20	20	import string
21	21	from collections import OrderedDict
22	22
23		from six import string_types
	23	from six import iteritems, string_types
24	24
25	25	from twisted.internet import defer
26	26

31	31	JoinRules,
32	32	RoomCreationPreset,
33	33	)
34		from synapse.api.errors import AuthError, Codes, StoreError, SynapseError
	34	from synapse.api.errors import AuthError, Codes, NotFoundError, StoreError, SynapseError
35	35	from synapse.storage.state import StateFilter
36	36	from synapse.types import RoomAlias, RoomID, RoomStreamToken, StreamToken, UserID
37	37	from synapse.util import stringutils
	38	from synapse.util.async_helpers import Linearizer
38	39	from synapse.visibility import filter_events_for_client
39	40
40	41	from ._base import BaseHandler

72	73
73	74	self.spam_checker = hs.get_spam_checker()
74	75	self.event_creation_handler = hs.get_event_creation_handler()
	76	self.room_member_handler = hs.get_room_member_handler()
	77
	78	# linearizer to stop two upgrades happening at once
	79	self._upgrade_linearizer = Linearizer("room_upgrade_linearizer")
	80
	81	@defer.inlineCallbacks
	82	def upgrade_room(self, requester, old_room_id, new_version):
	83	"""Replace a room with a new room with a different version
	84
	85	Args:
	86	requester (synapse.types.Requester): the user requesting the upgrade
	87	old_room_id (unicode): the id of the room to be replaced
	88	new_version (unicode): the new room version to use
	89
	90	Returns:
	91	Deferred[unicode]: the new room id
	92	"""
	93	yield self.ratelimit(requester)
	94
	95	user_id = requester.user.to_string()
	96
	97	with (yield self._upgrade_linearizer.queue(old_room_id)):
	98	# start by allocating a new room id
	99	r = yield self.store.get_room(old_room_id)
	100	if r is None:
	101	raise NotFoundError("Unknown room id %s" % (old_room_id,))
	102	new_room_id = yield self._generate_room_id(
	103	creator_id=user_id, is_public=r["is_public"],
	104	)
	105
	106	logger.info("Creating new room %s to replace %s", new_room_id, old_room_id)
	107
	108	# we create and auth the tombstone event before properly creating the new
	109	# room, to check our user has perms in the old room.
	110	tombstone_event, tombstone_context = (
	111	yield self.event_creation_handler.create_event(
	112	requester, {
	113	"type": EventTypes.Tombstone,
	114	"state_key": "",
	115	"room_id": old_room_id,
	116	"sender": user_id,
	117	"content": {
	118	"body": "This room has been replaced",
	119	"replacement_room": new_room_id,
	120	}
	121	},
	122	token_id=requester.access_token_id,
	123	)
	124	)
	125	yield self.auth.check_from_context(tombstone_event, tombstone_context)
	126
	127	yield self.clone_exiting_room(
	128	requester,
	129	old_room_id=old_room_id,
	130	new_room_id=new_room_id,
	131	new_room_version=new_version,
	132	tombstone_event_id=tombstone_event.event_id,
	133	)
	134
	135	# now send the tombstone
	136	yield self.event_creation_handler.send_nonmember_event(
	137	requester, tombstone_event, tombstone_context,
	138	)
	139
	140	old_room_state = yield tombstone_context.get_current_state_ids(self.store)
	141
	142	# update any aliases
	143	yield self._move_aliases_to_new_room(
	144	requester, old_room_id, new_room_id, old_room_state,
	145	)
	146
	147	# and finally, shut down the PLs in the old room, and update them in the new
	148	# room.
	149	yield self._update_upgraded_room_pls(
	150	requester, old_room_id, new_room_id, old_room_state,
	151	)
	152
	153	defer.returnValue(new_room_id)
	154
	155	@defer.inlineCallbacks
	156	def _update_upgraded_room_pls(
	157	self, requester, old_room_id, new_room_id, old_room_state,
	158	):
	159	"""Send updated power levels in both rooms after an upgrade
	160
	161	Args:
	162	requester (synapse.types.Requester): the user requesting the upgrade
	163	old_room_id (unicode): the id of the room to be replaced
	164	new_room_id (unicode): the id of the replacement room
	165	old_room_state (dict[tuple[str, str], str]): the state map for the old room
	166
	167	Returns:
	168	Deferred
	169	"""
	170	old_room_pl_event_id = old_room_state.get((EventTypes.PowerLevels, ""))
	171
	172	if old_room_pl_event_id is None:
	173	logger.warning(
	174	"Not supported: upgrading a room with no PL event. Not setting PLs "
	175	"in old room.",
	176	)
	177	return
	178
	179	old_room_pl_state = yield self.store.get_event(old_room_pl_event_id)
	180
	181	# we try to stop regular users from speaking by setting the PL required
	182	# to send regular events and invites to 'Moderator' level. That's normally
	183	# 50, but if the default PL in a room is 50 or more, then we set the
	184	# required PL above that.
	185
	186	pl_content = dict(old_room_pl_state.content)
	187	users_default = int(pl_content.get("users_default", 0))
	188	restricted_level = max(users_default + 1, 50)
	189
	190	updated = False
	191	for v in ("invite", "events_default"):
	192	current = int(pl_content.get(v, 0))
	193	if current < restricted_level:
	194	logger.info(
	195	"Setting level for %s in %s to %i (was %i)",
	196	v, old_room_id, restricted_level, current,
	197	)
	198	pl_content[v] = restricted_level
	199	updated = True
	200	else:
	201	logger.info(
	202	"Not setting level for %s (already %i)",
	203	v, current,
	204	)
	205
	206	if updated:
	207	try:
	208	yield self.event_creation_handler.create_and_send_nonmember_event(
	209	requester, {
	210	"type": EventTypes.PowerLevels,
	211	"state_key": '',
	212	"room_id": old_room_id,
	213	"sender": requester.user.to_string(),
	214	"content": pl_content,
	215	}, ratelimit=False,
	216	)
	217	except AuthError as e:
	218	logger.warning("Unable to update PLs in old room: %s", e)
	219
	220	logger.info("Setting correct PLs in new room")
	221	yield self.event_creation_handler.create_and_send_nonmember_event(
	222	requester, {
	223	"type": EventTypes.PowerLevels,
	224	"state_key": '',
	225	"room_id": new_room_id,
	226	"sender": requester.user.to_string(),
	227	"content": old_room_pl_state.content,
	228	}, ratelimit=False,
	229	)
	230
	231	@defer.inlineCallbacks
	232	def clone_exiting_room(
	233	self, requester, old_room_id, new_room_id, new_room_version,
	234	tombstone_event_id,
	235	):
	236	"""Populate a new room based on an old room
	237
	238	Args:
	239	requester (synapse.types.Requester): the user requesting the upgrade
	240	old_room_id (unicode): the id of the room to be replaced
	241	new_room_id (unicode): the id to give the new room (should already have been
	242	created with _gemerate_room_id())
	243	new_room_version (unicode): the new room version to use
	244	tombstone_event_id (unicode\|str): the ID of the tombstone event in the old
	245	room.
	246	Returns:
	247	Deferred[None]
	248	"""
	249	user_id = requester.user.to_string()
	250
	251	if not self.spam_checker.user_may_create_room(user_id):
	252	raise SynapseError(403, "You are not permitted to create rooms")
	253
	254	creation_content = {
	255	"room_version": new_room_version,
	256	"predecessor": {
	257	"room_id": old_room_id,
	258	"event_id": tombstone_event_id,
	259	}
	260	}
	261
	262	initial_state = dict()
	263
	264	types_to_copy = (
	265	(EventTypes.JoinRules, ""),
	266	(EventTypes.Name, ""),
	267	(EventTypes.Topic, ""),
	268	(EventTypes.RoomHistoryVisibility, ""),
	269	(EventTypes.GuestAccess, ""),
	270	(EventTypes.RoomAvatar, ""),
	271	)
	272
	273	old_room_state_ids = yield self.store.get_filtered_current_state_ids(
	274	old_room_id, StateFilter.from_types(types_to_copy),
	275	)
	276	# map from event_id to BaseEvent
	277	old_room_state_events = yield self.store.get_events(old_room_state_ids.values())
	278
	279	for k, old_event_id in iteritems(old_room_state_ids):
	280	old_event = old_room_state_events.get(old_event_id)
	281	if old_event:
	282	initial_state[k] = old_event.content
	283
	284	yield self._send_events_for_new_room(
	285	requester,
	286	new_room_id,
	287
	288	# we expect to override all the presets with initial_state, so this is
	289	# somewhat arbitrary.
	290	preset_config=RoomCreationPreset.PRIVATE_CHAT,
	291
	292	invite_list=[],
	293	initial_state=initial_state,
	294	creation_content=creation_content,
	295	)
	296
	297	# XXX invites/joins
	298	# XXX 3pid invites
	299
	300	@defer.inlineCallbacks
	301	def _move_aliases_to_new_room(
	302	self, requester, old_room_id, new_room_id, old_room_state,
	303	):
	304	directory_handler = self.hs.get_handlers().directory_handler
	305
	306	aliases = yield self.store.get_aliases_for_room(old_room_id)
	307
	308	# check to see if we have a canonical alias.
	309	canonical_alias = None
	310	canonical_alias_event_id = old_room_state.get((EventTypes.CanonicalAlias, ""))
	311	if canonical_alias_event_id:
	312	canonical_alias_event = yield self.store.get_event(canonical_alias_event_id)
	313	if canonical_alias_event:
	314	canonical_alias = canonical_alias_event.content.get("alias", "")
	315
	316	# first we try to remove the aliases from the old room (we suppress sending
	317	# the room_aliases event until the end).
	318	#
	319	# Note that we'll only be able to remove aliases that (a) aren't owned by an AS,
	320	# and (b) unless the user is a server admin, which the user created.
	321	#
	322	# This is probably correct - given we don't allow such aliases to be deleted
	323	# normally, it would be odd to allow it in the case of doing a room upgrade -
	324	# but it makes the upgrade less effective, and you have to wonder why a room
	325	# admin can't remove aliases that point to that room anyway.
	326	# (cf https://github.com/matrix-org/synapse/issues/2360)
	327	#
	328	removed_aliases = []
	329	for alias_str in aliases:
	330	alias = RoomAlias.from_string(alias_str)
	331	try:
	332	yield directory_handler.delete_association(
	333	requester, alias, send_event=False,
	334	)
	335	removed_aliases.append(alias_str)
	336	except SynapseError as e:
	337	logger.warning(
	338	"Unable to remove alias %s from old room: %s",
	339	alias, e,
	340	)
	341
	342	# if we didn't find any aliases, or couldn't remove anyway, we can skip the rest
	343	# of this.
	344	if not removed_aliases:
	345	return
	346
	347	try:
	348	# this can fail if, for some reason, our user doesn't have perms to send
	349	# m.room.aliases events in the old room (note that we've already checked that
	350	# they have perms to send a tombstone event, so that's not terribly likely).
	351	#
	352	# If that happens, it's regrettable, but we should carry on: it's the same
	353	# as when you remove an alias from the directory normally - it just means that
	354	# the aliases event gets out of sync with the directory
	355	# (cf https://github.com/vector-im/riot-web/issues/2369)
	356	yield directory_handler.send_room_alias_update_event(
	357	requester, old_room_id,
	358	)
	359	except AuthError as e:
	360	logger.warning(
	361	"Failed to send updated alias event on old room: %s", e,
	362	)
	363
	364	# we can now add any aliases we successfully removed to the new room.
	365	for alias in removed_aliases:
	366	try:
	367	yield directory_handler.create_association(
	368	requester, RoomAlias.from_string(alias),
	369	new_room_id, servers=(self.hs.hostname, ),
	370	send_event=False,
	371	)
	372	logger.info("Moved alias %s to new room", alias)
	373	except SynapseError as e:
	374	# I'm not really expecting this to happen, but it could if the spam
	375	# checking module decides it shouldn't, or similar.
	376	logger.error(
	377	"Error adding alias %s to new room: %s",
	378	alias, e,
	379	)
	380
	381	try:
	382	if canonical_alias and (canonical_alias in removed_aliases):
	383	yield self.event_creation_handler.create_and_send_nonmember_event(
	384	requester,
	385	{
	386	"type": EventTypes.CanonicalAlias,
	387	"state_key": "",
	388	"room_id": new_room_id,
	389	"sender": requester.user.to_string(),
	390	"content": {"alias": canonical_alias, },
	391	},
	392	ratelimit=False
	393	)
	394
	395	yield directory_handler.send_room_alias_update_event(
	396	requester, new_room_id,
	397	)
	398	except SynapseError as e:
	399	# again I'm not really expecting this to fail, but if it does, I'd rather
	400	# we returned the new room to the client at this point.
	401	logger.error(
	402	"Unable to send updated alias events in new room: %s", e,
	403	)
75	404
76	405	@defer.inlineCallbacks
77	406	def create_room(self, requester, config, ratelimit=True,

164	493	visibility = config.get("visibility", None)
165	494	is_public = visibility == "public"
166	495
167		# autogen room IDs and try to create it. We may clash, so just
168		# try a few times till one goes through, giving up eventually.
169		attempts = 0
170		room_id = None
171		while attempts < 5:
172		try:
173		random_string = stringutils.random_string(18)
174		gen_room_id = RoomID(
175		random_string,
176		self.hs.hostname,
177		)
178		yield self.store.store_room(
179		room_id=gen_room_id.to_string(),
180		room_creator_user_id=user_id,
181		is_public=is_public
182		)
183		room_id = gen_room_id.to_string()
184		break
185		except StoreError:
186		attempts += 1
187		if not room_id:
188		raise StoreError(500, "Couldn't generate a room ID.")
	496	room_id = yield self._generate_room_id(creator_id=user_id, is_public=is_public)
189	497
190	498	if room_alias:
191	499	directory_handler = self.hs.get_handlers().directory_handler

215	523	# override any attempt to set room versions via the creation_content
216	524	creation_content["room_version"] = room_version
217	525
218		room_member_handler = self.hs.get_room_member_handler()
219
220	526	yield self._send_events_for_new_room(
221	527	requester,
222	528	room_id,
223		room_member_handler,
224	529	preset_config=preset_config,
225	530	invite_list=invite_list,
226	531	initial_state=initial_state,
227	532	creation_content=creation_content,
228	533	room_alias=room_alias,
229		power_level_content_override=config.get("power_level_content_override", {}),
	534	power_level_content_override=config.get("power_level_content_override"),
230	535	creator_join_profile=creator_join_profile,
231	536	)
232	537

262	567	if is_direct:
263	568	content["is_direct"] = is_direct
264	569
265		yield room_member_handler.update_membership(
	570	yield self.room_member_handler.update_membership(
266	571	requester,
267	572	UserID.from_string(invitee),
268	573	room_id,

300	605	self,
301	606	creator, # A Requester object.
302	607	room_id,
303		room_member_handler,
304	608	preset_config,
305	609	invite_list,
306	610	initial_state,
307	611	creation_content,
308		room_alias,
309		power_level_content_override,
310		creator_join_profile,
	612	room_alias=None,
	613	power_level_content_override=None,
	614	creator_join_profile=None,
311	615	):
312	616	def create(etype, content, **kwargs):
313	617	e = {

323	627	@defer.inlineCallbacks
324	628	def send(etype, content, **kwargs):
325	629	event = create(etype, content, **kwargs)
	630	logger.info("Sending %s in new room", etype)
326	631	yield self.event_creation_handler.create_and_send_nonmember_event(
327	632	creator,
328	633	event,

345	650	content=creation_content,
346	651	)
347	652
348		yield room_member_handler.update_membership(
	653	logger.info("Sending %s in new room", EventTypes.Member)
	654	yield self.room_member_handler.update_membership(
349	655	creator,
350	656	creator.user,
351	657	room_id,

387	693	for invitee in invite_list:
388	694	power_level_content["users"][invitee] = 100
389	695
390		power_level_content.update(power_level_content_override)
	696	if power_level_content_override:
	697	power_level_content.update(power_level_content_override)
391	698
392	699	yield send(
393	700	etype=EventTypes.PowerLevels,

425	732	state_key=state_key,
426	733	content=content,
427	734	)
	735
	736	@defer.inlineCallbacks
	737	def _generate_room_id(self, creator_id, is_public):
	738	# autogen room IDs and try to create it. We may clash, so just
	739	# try a few times till one goes through, giving up eventually.
	740	attempts = 0
	741	while attempts < 5:
	742	try:
	743	random_string = stringutils.random_string(18)
	744	gen_room_id = RoomID(
	745	random_string,
	746	self.hs.hostname,
	747	).to_string()
	748	if isinstance(gen_room_id, bytes):
	749	gen_room_id = gen_room_id.decode('utf-8')
	750	yield self.store.store_room(
	751	room_id=gen_room_id,
	752	room_creator_user_id=creator_id,
	753	is_public=is_public,
	754	)
	755	defer.returnValue(gen_room_id)
	756	except StoreError:
	757	attempts += 1
	758	raise StoreError(500, "Couldn't generate a room ID.")
428	759
429	760
430	761	class RoomContextHandler(object):

+10

-4

synapse/handlers/typing.py less more

62	62	self._member_typing_until = {} # clock time we expect to stop
63	63	self._member_last_federation_poke = {}
64	64
65		# map room IDs to serial numbers
66		self._room_serials = {}
67	65	self._latest_room_serial = 0
68		# map room IDs to sets of users currently typing
69		self._room_typing = {}
	66	self._reset()
70	67
71	68	# caches which room_ids changed at which serials
72	69	self._typing_stream_change_cache = StreamChangeCache(

77	74	self._handle_timeouts,
78	75	5000,
79	76	)
	77
	78	def _reset(self):
	79	"""
	80	Reset the typing handler's data caches.
	81	"""
	82	# map room IDs to serial numbers
	83	self._room_serials = {}
	84	# map room IDs to sets of users currently typing
	85	self._room_typing = {}
80	86
81	87	def _handle_timeouts(self):
82	88	logger.info("Checking for typing timeouts")

+4

-4

synapse/http/server.py less more

467	467	Args:
468	468	request (twisted.web.http.Request): The http request to add CORs to.
469	469	"""
470		request.setHeader("Access-Control-Allow-Origin", "*")
	470	request.setHeader(b"Access-Control-Allow-Origin", b"*")
471	471	request.setHeader(
472		"Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS"
	472	b"Access-Control-Allow-Methods", b"GET, POST, PUT, DELETE, OPTIONS"
473	473	)
474	474	request.setHeader(
475		"Access-Control-Allow-Headers",
476		"Origin, X-Requested-With, Content-Type, Accept, Authorization"
	475	b"Access-Control-Allow-Headers",
	476	b"Origin, X-Requested-With, Content-Type, Accept, Authorization"
477	477	)
478	478
479	479

+4

-5

synapse/http/servlet.py less more

120	120
121	121	Args:
122	122	request: the twisted HTTP request.
123		name (bytes/unicode): the name of the query parameter.
124		default (bytes/unicode\|None): value to use if the parameter is absent,
	123	name (bytes\|unicode): the name of the query parameter.
	124	default (bytes\|unicode\|None): value to use if the parameter is absent,
125	125	defaults to None. Must be bytes if encoding is None.
126	126	required (bool): whether to raise a 400 SynapseError if the
127	127	parameter is absent, defaults to False.
128		allowed_values (list[bytes/unicode]): List of allowed values for the
	128	allowed_values (list[bytes\|unicode]): List of allowed values for the
129	129	string, or None if any value is allowed, defaults to None. Must be
130	130	the same type as name, if given.
131		encoding: The encoding to decode the name to, and decode the string
132		content with.
	131	encoding (str\|None): The encoding to decode the string content with.
133	132
134	133	Returns:
135	134	bytes/unicode\|None: A string value or the default. Unicode if encoding

+4

-1

synapse/push/emailpusher.py less more

84	84	self.timed_call = None
85	85
86	86	def on_new_notifications(self, min_stream_ordering, max_stream_ordering):
87		self.max_stream_ordering = max(max_stream_ordering, self.max_stream_ordering)
	87	if self.max_stream_ordering:
	88	self.max_stream_ordering = max(max_stream_ordering, self.max_stream_ordering)
	89	else:
	90	self.max_stream_ordering = max_stream_ordering
88	91	self._start_processing()
89	92
90	93	def on_new_receipts(self, min_stream_id, max_stream_id):

+2

-2

synapse/push/httppusher.py less more

310	310	]
311	311	}
312	312	}
313		if event.type == 'm.room.member':
	313	if event.type == 'm.room.member' and event.is_state():
314	314	d['notification']['membership'] = event.content['membership']
315	315	d['notification']['user_is_target'] = event.state_key == self.user_id
316		if self.hs.config.push_include_content and 'content' in event:
	316	if self.hs.config.push_include_content and event.content:
317	317	d['notification']['content'] = event.content
318	318
319	319	# We no longer send aliases separately, instead, we send the human

+6

-6

synapse/push/mailer.py less more

25	25	import jinja2
26	26
27	27	from twisted.internet import defer
28		from twisted.mail.smtp import sendmail
29	28
30	29	from synapse.api.constants import EventTypes
31	30	from synapse.api.errors import StoreError

84	83	self.notif_template_html = notif_template_html
85	84	self.notif_template_text = notif_template_text
86	85
	86	self.sendmail = self.hs.get_sendmail()
87	87	self.store = self.hs.get_datastore()
88	88	self.macaroon_gen = self.hs.get_macaroon_generator()
89	89	self.state_handler = self.hs.get_state_handler()

190	190	multipart_msg.attach(html_part)
191	191
192	192	logger.info("Sending email push notification to %s" % email_address)
193		# logger.debug(html_text)
194
195		yield sendmail(
	193
	194	yield self.sendmail(
196	195	self.hs.config.email_smtp_host,
197		raw_from, raw_to, multipart_msg.as_string(),
	196	raw_from, raw_to, multipart_msg.as_string().encode('utf8'),
	197	reactor=self.hs.get_reactor(),
198	198	port=self.hs.config.email_smtp_port,
199	199	requireAuthentication=self.hs.config.email_smtp_user is not None,
200	200	username=self.hs.config.email_smtp_user,

332	332	notif_events, user_id, reason):
333	333	if len(notifs_by_room) == 1:
334	334	# Only one room has new stuff
335		room_id = notifs_by_room.keys()[0]
	335	room_id = list(notifs_by_room.keys())[0]
336	336
337	337	# If the room has some kind of name, use it, but we don't
338	338	# want the generated-from-names one here otherwise we'll

+2

-2

synapse/push/push_rule_evaluator.py less more

123	123
124	124	# XXX: optimisation: cache our pattern regexps
125	125	if condition['key'] == 'content.body':
126		body = self._event["content"].get("body", None)
	126	body = self._event.content.get("body", None)
127	127	if not body:
128	128	return False
129	129

139	139	if not display_name:
140	140	return False
141	141
142		body = self._event["content"].get("body", None)
	142	body = self._event.content.get("body", None)
143	143	if not body:
144	144	return False
145	145

+0

-1

synapse/python_dependencies.py less more

50	50	"daemonize>=2.3.1": ["daemonize"],
51	51	"bcrypt>=3.1.0": ["bcrypt>=3.1.0"],
52	52	"pillow>=3.1.2": ["PIL"],
53		"pydenticon>=0.2": ["pydenticon"],
54	53	"sortedcontainers>=1.4.4": ["sortedcontainers"],
55	54	"psutil>=2.0.0": ["psutil>=2.0.0"],
56	55	"pysaml2>=3.0.0": ["saml2"],

+1

-1

synapse/replication/tcp/client.py less more

105	105
106	106	Can be overriden in subclasses to handle more.
107	107	"""
108		logger.info("Received rdata %s -> %s", stream_name, token)
	108	logger.debug("Received rdata %s -> %s", stream_name, token)
109	109	return self.store.process_replication_rows(stream_name, token, rows)
110	110
111	111	def on_position(self, stream_name, token):

+2

-2

synapse/replication/tcp/protocol.py less more

655	655	"",
656	656	["command", "name"],
657	657	lambda: {
658		(k[0], p.name,): count
	658	(k, p.name,): count
659	659	for p in connected_connections
660	660	for k, count in iteritems(p.inbound_commands_counter)
661	661	},

666	666	"",
667	667	["command", "name"],
668	668	lambda: {
669		(k[0], p.name,): count
	669	(k, p.name,): count
670	670	for p in connected_connections
671	671	for k, count in iteritems(p.outbound_commands_counter)
672	672	},

+2

-0

synapse/rest/__init__.py less more

46	46	register,
47	47	report_event,
48	48	room_keys,
	49	room_upgrade_rest_servlet,
49	50	sendtodevice,
50	51	sync,
51	52	tags,

115	116	sendtodevice.register_servlets(hs, client_resource)
116	117	user_directory.register_servlets(hs, client_resource)
117	118	groups.register_servlets(hs, client_resource)
	119	room_upgrade_rest_servlet.register_servlets(hs, client_resource)

+80

-1

synapse/rest/client/v2_alpha/auth.py less more

67	67	</html>
68	68	"""
69	69
	70	TERMS_TEMPLATE = """
	71	<html>
	72	<head>
	73	<title>Authentication</title>
	74	<meta name='viewport' content='width=device-width, initial-scale=1,
	75	user-scalable=no, minimum-scale=1.0, maximum-scale=1.0'>
	76	<link rel="stylesheet" href="/_matrix/static/client/register/style.css">
	77	</head>
	78	<body>
	79	<form id="registrationForm" method="post" action="%(myurl)s">
	80	<div>
	81	<p>
	82	Please click the button below if you agree to the
	83	<a href="%(terms_url)s">privacy policy of this homeserver.</a>
	84	</p>
	85	<input type="hidden" name="session" value="%(session)s" />
	86	<input type="submit" value="Agree" />
	87	</div>
	88	</form>
	89	</body>
	90	</html>
	91	"""
	92
70	93	SUCCESS_TEMPLATE = """
71	94	<html>
72	95	<head>

132	155	request.write(html_bytes)
133	156	finish_request(request)
134	157	defer.returnValue(None)
	158	elif stagetype == LoginType.TERMS:
	159	session = request.args['session'][0]
	160
	161	html = TERMS_TEMPLATE % {
	162	'session': session,
	163	'terms_url': "%s/_matrix/consent?v=%s" % (
	164	self.hs.config.public_baseurl,
	165	self.hs.config.user_consent_version,
	166	),
	167	'myurl': "%s/auth/%s/fallback/web" % (
	168	CLIENT_V2_ALPHA_PREFIX, LoginType.TERMS
	169	),
	170	}
	171	html_bytes = html.encode("utf8")
	172	request.setResponseCode(200)
	173	request.setHeader(b"Content-Type", b"text/html; charset=utf-8")
	174	request.setHeader(b"Content-Length", b"%d" % (len(html_bytes),))
	175
	176	request.write(html_bytes)
	177	finish_request(request)
	178	defer.returnValue(None)
135	179	else:
136	180	raise SynapseError(404, "Unknown auth stage type")
137	181
138	182	@defer.inlineCallbacks
139	183	def on_POST(self, request, stagetype):
140	184	yield
141		if stagetype == "m.login.recaptcha":
	185	if stagetype == LoginType.RECAPTCHA:
142	186	if ('g-recaptcha-response' not in request.args or
143	187	len(request.args['g-recaptcha-response'])) == 0:
144	188	raise SynapseError(400, "No captcha response supplied")

178	222	finish_request(request)
179	223
180	224	defer.returnValue(None)
	225	elif stagetype == LoginType.TERMS:
	226	if ('session' not in request.args or
	227	len(request.args['session'])) == 0:
	228	raise SynapseError(400, "No session supplied")
	229
	230	session = request.args['session'][0]
	231	authdict = {'session': session}
	232
	233	success = yield self.auth_handler.add_oob_auth(
	234	LoginType.TERMS,
	235	authdict,
	236	self.hs.get_ip_from_request(request)
	237	)
	238
	239	if success:
	240	html = SUCCESS_TEMPLATE
	241	else:
	242	html = TERMS_TEMPLATE % {
	243	'session': session,
	244	'terms_url': "%s/_matrix/consent?v=%s" % (
	245	self.hs.config.public_baseurl,
	246	self.hs.config.user_consent_version,
	247	),
	248	'myurl': "%s/auth/%s/fallback/web" % (
	249	CLIENT_V2_ALPHA_PREFIX, LoginType.TERMS
	250	),
	251	}
	252	html_bytes = html.encode("utf8")
	253	request.setResponseCode(200)
	254	request.setHeader(b"Content-Type", b"text/html; charset=utf-8")
	255	request.setHeader(b"Content-Length", b"%d" % (len(html_bytes),))
	256
	257	request.write(html_bytes)
	258	finish_request(request)
	259	defer.returnValue(None)
181	260	else:
182	261	raise SynapseError(404, "Unknown auth stage type")
183	262

+13

-0

synapse/rest/client/v2_alpha/register.py less more

358	358	[LoginType.MSISDN, LoginType.EMAIL_IDENTITY]
359	359	])
360	360
	361	# Append m.login.terms to all flows if we're requiring consent
	362	if self.hs.config.user_consent_at_registration:
	363	new_flows = []
	364	for flow in flows:
	365	flow.append(LoginType.TERMS)
	366	flows.extend(new_flows)
	367
361	368	auth_result, params, session_id = yield self.auth_handler.check_auth(
362	369	flows, body, self.hs.get_ip_from_request(request)
363	370	)

442	449	yield self._register_msisdn_threepid(
443	450	registered_user_id, threepid, return_dict["access_token"],
444	451	params.get("bind_msisdn")
	452	)
	453
	454	if auth_result and LoginType.TERMS in auth_result:
	455	logger.info("%s has consented to the privacy policy" % registered_user_id)
	456	yield self.store.user_set_consent_version(
	457	registered_user_id, self.hs.config.user_consent_version,
445	458	)
446	459
447	460	defer.returnValue((200, return_dict))

+18

-3

synapse/rest/client/v2_alpha/room_keys.py less more

16	16
17	17	from twisted.internet import defer
18	18
19		from synapse.api.errors import Codes, SynapseError
	19	from synapse.api.errors import Codes, NotFoundError, SynapseError
20	20	from synapse.http.servlet import (
21	21	RestServlet,
22	22	parse_json_object_from_request,

207	207	user_id, version, room_id, session_id
208	208	)
209	209
	210	# Convert room_keys to the right format to return.
210	211	if session_id:
211		room_keys = room_keys['rooms'][room_id]['sessions'][session_id]
	212	# If the client requests a specific session, but that session was
	213	# not backed up, then return an M_NOT_FOUND.
	214	if room_keys['rooms'] == {}:
	215	raise NotFoundError("No room_keys found")
	216	else:
	217	room_keys = room_keys['rooms'][room_id]['sessions'][session_id]
212	218	elif room_id:
213		room_keys = room_keys['rooms'][room_id]
	219	# If the client requests all sessions from a room, but no sessions
	220	# are found, then return an empty result rather than an error, so
	221	# that clients don't have to handle an error condition, and an
	222	# empty result is valid. (Similarly if the client requests all
	223	# sessions from the backup, but in that case, room_keys is already
	224	# in the right format, so we don't need to do anything about it.)
	225	if room_keys['rooms'] == {}:
	226	room_keys = {'sessions': {}}
	227	else:
	228	room_keys = room_keys['rooms'][room_id]
214	229
215	230	defer.returnValue((200, room_keys))
216	231

+89

-0

synapse/rest/client/v2_alpha/room_upgrade_rest_servlet.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2016 OpenMarket Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import logging
	16
	17	from twisted.internet import defer
	18
	19	from synapse.api.constants import KNOWN_ROOM_VERSIONS
	20	from synapse.api.errors import Codes, SynapseError
	21	from synapse.http.servlet import (
	22	RestServlet,
	23	assert_params_in_dict,
	24	parse_json_object_from_request,
	25	)
	26
	27	from ._base import client_v2_patterns
	28
	29	logger = logging.getLogger(__name__)
	30
	31
	32	class RoomUpgradeRestServlet(RestServlet):
	33	"""Handler for room uprade requests.
	34
	35	Handles requests of the form:
	36
	37	POST /_matrix/client/r0/rooms/$roomid/upgrade HTTP/1.1
	38	Content-Type: application/json
	39
	40	{
	41	"new_version": "2",
	42	}
	43
	44	Creates a new room and shuts down the old one. Returns the ID of the new room.
	45
	46	Args:
	47	hs (synapse.server.HomeServer):
	48	"""
	49	PATTERNS = client_v2_patterns(
	50	# /rooms/$roomid/upgrade
	51	"/rooms/(?P<room_id>[^/]*)/upgrade$",
	52	v2_alpha=False,
	53	)
	54
	55	def __init__(self, hs):
	56	super(RoomUpgradeRestServlet, self).__init__()
	57	self._hs = hs
	58	self._room_creation_handler = hs.get_room_creation_handler()
	59	self._auth = hs.get_auth()
	60
	61	@defer.inlineCallbacks
	62	def on_POST(self, request, room_id):
	63	requester = yield self._auth.get_user_by_req(request)
	64
	65	content = parse_json_object_from_request(request)
	66	assert_params_in_dict(content, ("new_version", ))
	67	new_version = content["new_version"]
	68
	69	if new_version not in KNOWN_ROOM_VERSIONS:
	70	raise SynapseError(
	71	400,
	72	"Your homeserver does not support this room version",
	73	Codes.UNSUPPORTED_ROOM_VERSION,
	74	)
	75
	76	new_room_id = yield self._room_creation_handler.upgrade_room(
	77	requester, room_id, new_version
	78	)
	79
	80	ret = {
	81	"replacement_room": new_room_id,
	82	}
	83
	84	defer.returnValue((200, ret))
	85
	86
	87	def register_servlets(hs, http_server):
	88	RoomUpgradeRestServlet(hs).register(http_server)

+27

-18

synapse/rest/consent/consent_resource.py less more

136	136	request (twisted.web.http.Request):
137	137	"""
138	138
139		version = parse_string(request, "v",
140		default=self._default_consent_version)
141		username = parse_string(request, "u", required=True)
142		userhmac = parse_string(request, "h", required=True, encoding=None)
143
144		self._check_hash(username, userhmac)
145
146		if username.startswith('@'):
147		qualified_user_id = username
148		else:
149		qualified_user_id = UserID(username, self.hs.hostname).to_string()
150
151		u = yield self.store.get_user_by_id(qualified_user_id)
152		if u is None:
153		raise NotFoundError("Unknown user")
	139	version = parse_string(request, "v", default=self._default_consent_version)
	140	username = parse_string(request, "u", required=False, default="")
	141	userhmac = None
	142	has_consented = False
	143	public_version = username == ""
	144	if not public_version:
	145	userhmac_bytes = parse_string(request, "h", required=True, encoding=None)
	146
	147	self._check_hash(username, userhmac_bytes)
	148
	149	if username.startswith('@'):
	150	qualified_user_id = username
	151	else:
	152	qualified_user_id = UserID(username, self.hs.hostname).to_string()
	153
	154	u = yield self.store.get_user_by_id(qualified_user_id)
	155	if u is None:
	156	raise NotFoundError("Unknown user")
	157
	158	has_consented = u["consent_version"] == version
	159	userhmac = userhmac_bytes.decode("ascii")
154	160
155	161	try:
156	162	self._render_template(
157	163	request, "%s.html" % (version,),
158		user=username, userhmac=userhmac, version=version,
159		has_consented=(u["consent_version"] == version),
	164	user=username,
	165	userhmac=userhmac,
	166	version=version,
	167	has_consented=has_consented,
	168	public_version=public_version,
160	169	)
161	170	except TemplateNotFound:
162	171	raise NotFoundError("Unknown policy version")

222	231	key=self._hmac_secret,
223	232	msg=userid.encode('utf-8'),
224	233	digestmod=sha256,
225		).hexdigest()
	234	).hexdigest().encode('ascii')
226	235
227	236	if not compare_digest(want_mac, userhmac):
228	237	raise SynapseError(http_client.FORBIDDEN, "HMAC incorrect")

+0

-14

~~synapse/rest/key/v1/__init__.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2015, 2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.

+0

-92

~~synapse/rest/key/v1/server_key_resource.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2014-2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15
16		import logging
17
18		from canonicaljson import encode_canonical_json
19		from signedjson.sign import sign_json
20		from unpaddedbase64 import encode_base64
21
22		from OpenSSL import crypto
23		from twisted.web.resource import Resource
24
25		from synapse.http.server import respond_with_json_bytes
26
27		logger = logging.getLogger(__name__)
28
29
30		class LocalKey(Resource):
31		"""HTTP resource containing encoding the TLS X.509 certificate and NACL
32		signature verification keys for this server::
33
34		GET /key HTTP/1.1
35
36		HTTP/1.1 200 OK
37		Content-Type: application/json
38		{
39		"server_name": "this.server.example.com"
40		"verify_keys": {
41		"algorithm:version": # base64 encoded NACL verification key.
42		},
43		"tls_certificate": # base64 ASN.1 DER encoded X.509 tls cert.
44		"signatures": {
45		"this.server.example.com": {
46		"algorithm:version": # NACL signature for this server.
47		}
48		}
49		}
50		"""
51
52		def __init__(self, hs):
53		self.response_body = encode_canonical_json(
54		self.response_json_object(hs.config)
55		)
56		Resource.__init__(self)
57
58		@staticmethod
59		def response_json_object(server_config):
60		verify_keys = {}
61		for key in server_config.signing_key:
62		verify_key_bytes = key.verify_key.encode()
63		key_id = "%s:%s" % (key.alg, key.version)
64		verify_keys[key_id] = encode_base64(verify_key_bytes)
65
66		x509_certificate_bytes = crypto.dump_certificate(
67		crypto.FILETYPE_ASN1,
68		server_config.tls_certificate
69		)
70		json_object = {
71		u"server_name": server_config.server_name,
72		u"verify_keys": verify_keys,
73		u"tls_certificate": encode_base64(x509_certificate_bytes)
74		}
75		for key in server_config.signing_key:
76		json_object = sign_json(
77		json_object,
78		server_config.server_name,
79		key,
80		)
81
82		return json_object
83
84		def render_GET(self, request):
85		return respond_with_json_bytes(
86		request, 200, self.response_body,
87		)
88
89		def getChild(self, name, request):
90		if name == b'':
91		return self

+0

-68

~~synapse/rest/media/v1/identicon_resource.py~~ less more

0		# Copyright 2015, 2016 OpenMarket Ltd
1		#
2		# Licensed under the Apache License, Version 2.0 (the "License");
3		# you may not use this file except in compliance with the License.
4		# You may obtain a copy of the License at
5		#
6		# http://www.apache.org/licenses/LICENSE-2.0
7		#
8		# Unless required by applicable law or agreed to in writing, software
9		# distributed under the License is distributed on an "AS IS" BASIS,
10		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11		# See the License for the specific language governing permissions and
12		# limitations under the License.
13
14		from pydenticon import Generator
15
16		from twisted.web.resource import Resource
17
18		from synapse.http.servlet import parse_integer
19
20		FOREGROUND = [
21		"rgb(45,79,255)",
22		"rgb(254,180,44)",
23		"rgb(226,121,234)",
24		"rgb(30,179,253)",
25		"rgb(232,77,65)",
26		"rgb(49,203,115)",
27		"rgb(141,69,170)"
28		]
29
30		BACKGROUND = "rgb(224,224,224)"
31		SIZE = 5
32
33
34		class IdenticonResource(Resource):
35		isLeaf = True
36
37		def __init__(self):
38		Resource.__init__(self)
39		self.generator = Generator(
40		SIZE, SIZE, foreground=FOREGROUND, background=BACKGROUND,
41		)
42
43		def generate_identicon(self, name, width, height):
44		v_padding = width % SIZE
45		h_padding = height % SIZE
46		top_padding = v_padding // 2
47		left_padding = h_padding // 2
48		bottom_padding = v_padding - top_padding
49		right_padding = h_padding - left_padding
50		width -= v_padding
51		height -= h_padding
52		padding = (top_padding, bottom_padding, left_padding, right_padding)
53		identicon = self.generator.generate(
54		name, width, height, padding=padding
55		)
56		return identicon
57
58		def render_GET(self, request):
59		name = "/".join(request.postpath)
60		width = parse_integer(request, "width", default=96)
61		height = parse_integer(request, "height", default=96)
62		identicon_bytes = self.generate_identicon(name, width, height)
63		request.setHeader(b"Content-Type", b"image/png")
64		request.setHeader(
65		b"Cache-Control", b"public,max-age=86400,s-maxage=86400"
66		)
67		return identicon_bytes

+0

-2

synapse/rest/media/v1/media_repository.py less more

44	44	from .config_resource import MediaConfigResource
45	45	from .download_resource import DownloadResource
46	46	from .filepath import MediaFilePaths
47		from .identicon_resource import IdenticonResource
48	47	from .media_storage import MediaStorage
49	48	from .preview_url_resource import PreviewUrlResource
50	49	from .storage_provider import StorageProviderWrapper

768	767	self.putChild(b"thumbnail", ThumbnailResource(
769	768	hs, media_repo, media_repo.media_storage,
770	769	))
771		self.putChild(b"identicon", IdenticonResource())
772	770	if hs.config.url_preview_enabled:
773	771	self.putChild(b"preview_url", PreviewUrlResource(
774	772	hs, media_repo, media_repo.media_storage,

+16

-6

synapse/rest/media/v1/preview_url_resource.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14
14	15	import cgi
15	16	import datetime
16	17	import errno

23	24	import sys
24	25	import traceback
25	26
	27	import six
26	28	from six import string_types
27	29	from six.moves import urllib_parse as urlparse
28	30

97	99	# XXX: if get_user_by_req fails, what should we do in an async render?
98	100	requester = yield self.auth.get_user_by_req(request)
99	101	url = parse_string(request, "url")
100		if "ts" in request.args:
	102	if b"ts" in request.args:
101	103	ts = parse_integer(request, "ts")
102	104	else:
103	105	ts = self.clock.time_msec()

179	181	cache_result["expires_ts"] > ts and
180	182	cache_result["response_code"] / 100 == 2
181	183	):
182		defer.returnValue(cache_result["og"])
	184	# It may be stored as text in the database, not as bytes (such as
	185	# PostgreSQL). If so, encode it back before handing it on.
	186	og = cache_result["og"]
	187	if isinstance(og, six.text_type):
	188	og = og.encode('utf8')
	189	defer.returnValue(og)
183	190	return
184	191
185	192	media_info = yield self._download_url(url, user)

212	219	elif _is_html(media_info['media_type']):
213	220	# TODO: somehow stop a big HTML tree from exploding synapse's RAM
214	221
215		file = open(media_info['filename'])
216		body = file.read()
217		file.close()
	222	with open(media_info['filename'], 'rb') as file:
	223	body = file.read()
218	224
219	225	# clobber the encoding from the content-type, or default to utf-8
220	226	# XXX: this overrides any <meta/> or XML charset headers in the body
221	227	# which may pose problems, but so far seems to work okay.
222		match = re.match(r'.; charset=(.*?)(;\|$)', media_info['media_type'], re.I)
	228	match = re.match(
	229	r'.; charset="?(.*?)"?(;\|$)',
	230	media_info['media_type'],
	231	re.I
	232	)
223	233	encoding = match.group(1) if match else "utf-8"
224	234
225	235	og = decode_and_calc_og(body, media_info['uri'], encoding)

+5

-0

synapse/server.py less more

22	22	import logging
23	23
24	24	from twisted.enterprise import adbapi
	25	from twisted.mail.smtp import sendmail
25	26	from twisted.web.client import BrowserLikePolicyForHTTPS
26	27
27	28	from synapse.api.auth import Auth

173	174	'message_handler',
174	175	'pagination_handler',
175	176	'room_context_handler',
	177	'sendmail',
176	178	]
177	179
178	180	# This is overridden in derived application classes

267	269
268	270	def build_room_creation_handler(self):
269	271	return RoomCreationHandler(self)
	272
	273	def build_sendmail(self):
	274	return sendmail
270	275
271	276	def build_state_handler(self):
272	277	return StateHandler(self)

+6

-0

synapse/server.pyi less more

6	6	import synapse.handlers.deactivate_account
7	7	import synapse.handlers.device
8	8	import synapse.handlers.e2e_keys
	9	import synapse.handlers.room
	10	import synapse.handlers.room_member
	11	import synapse.handlers.message
9	12	import synapse.handlers.set_password
10	13	import synapse.rest.media.v1.media_repository
11	14	import synapse.server_notices.server_notices_manager

49	52	def get_room_creation_handler(self) -> synapse.handlers.room.RoomCreationHandler:
50	53	pass
51	54
	55	def get_room_member_handler(self) -> synapse.handlers.room_member.RoomMemberHandler:
	56	pass
	57
52	58	def get_event_creation_handler(self) -> synapse.handlers.message.EventCreationHandler:
53	59	pass
54	60

+2

-2

synapse/state/__init__.py less more

260	260	logger.debug("calling resolve_state_groups from compute_event_context")
261	261
262	262	entry = yield self.resolve_state_groups_for_events(
263		event.room_id, [e for e, _ in event.prev_events],
	263	event.room_id, event.prev_event_ids(),
264	264	)
265	265
266	266	prev_state_ids = entry.state

606	606	return v1.resolve_events_with_store(
607	607	state_sets, event_map, state_res_store.get_events,
608	608	)
609		elif room_version == RoomVersions.VDH_TEST:
	609	elif room_version in (RoomVersions.VDH_TEST, RoomVersions.STATE_V2_TEST):
610	610	return v2.resolve_events_with_store(
611	611	state_sets, event_map, state_res_store,
612	612	)

+12

-8

synapse/state/v2.py less more

52	52
53	53	logger.debug("Computing conflicted state")
54	54
	55	# We use event_map as a cache, so if its None we need to initialize it
	56	if event_map is None:
	57	event_map = {}
	58
55	59	# First split up the un/conflicted state
56	60	unconflicted_state, conflicted_state = _seperate(state_sets)
57	61

154	158	event = yield _get_event(event_id, event_map, state_res_store)
155	159
156	160	pl = None
157		for aid, _ in event.auth_events:
	161	for aid in event.auth_event_ids():
158	162	aev = yield _get_event(aid, event_map, state_res_store)
159	163	if (aev.type, aev.state_key) == (EventTypes.PowerLevels, ""):
160	164	pl = aev

162	166
163	167	if pl is None:
164	168	# Couldn't find power level. Check if they're the creator of the room
165		for aid, _ in event.auth_events:
	169	for aid in event.auth_event_ids():
166	170	aev = yield _get_event(aid, event_map, state_res_store)
167	171	if (aev.type, aev.state_key) == (EventTypes.Create, ""):
168	172	if aev.content.get("creator") == event.sender:

294	298	graph.setdefault(eid, set())
295	299
296	300	event = yield _get_event(eid, event_map, state_res_store)
297		for aid, _ in event.auth_events:
	301	for aid in event.auth_event_ids():
298	302	if aid in auth_diff:
299	303	if aid not in graph:
300	304	state.append(aid)

364	368	event = event_map[event_id]
365	369
366	370	auth_events = {}
367		for aid, _ in event.auth_events:
	371	for aid in event.auth_event_ids():
368	372	ev = yield _get_event(aid, event_map, state_res_store)
369	373
370	374	if ev.rejected_reason is None:

412	416	while pl:
413	417	mainline.append(pl)
414	418	pl_ev = yield _get_event(pl, event_map, state_res_store)
415		auth_events = pl_ev.auth_events
	419	auth_events = pl_ev.auth_event_ids()
416	420	pl = None
417		for aid, _ in auth_events:
	421	for aid in auth_events:
418	422	ev = yield _get_event(aid, event_map, state_res_store)
419	423	if (ev.type, ev.state_key) == (EventTypes.PowerLevels, ""):
420	424	pl = aid

459	463	if depth is not None:
460	464	defer.returnValue(depth)
461	465
462		auth_events = event.auth_events
	466	auth_events = event.auth_event_ids()
463	467	event = None
464	468
465		for aid, _ in auth_events:
	469	for aid in auth_events:
466	470	aev = yield _get_event(aid, event_map, state_res_store)
467	471	if (aev.type, aev.state_key) == (EventTypes.PowerLevels, ""):
468	472	event = aev

+93

-9

synapse/storage/devices.py less more

21	21
22	22	from synapse.api.errors import StoreError
23	23	from synapse.metrics.background_process_metrics import run_as_background_process
	24	from synapse.storage.background_updates import BackgroundUpdateStore
24	25	from synapse.util.caches.descriptors import cached, cachedInlineCallbacks, cachedList
25	26
26		from ._base import Cache, SQLBaseStore, db_to_json
	27	from ._base import Cache, db_to_json
27	28
28	29	logger = logging.getLogger(__name__)
29	30
30
31		class DeviceStore(SQLBaseStore):
	31	DROP_DEVICE_LIST_STREAMS_NON_UNIQUE_INDEXES = (
	32	"drop_device_list_streams_non_unique_indexes"
	33	)
	34
	35
	36	class DeviceStore(BackgroundUpdateStore):
32	37	def __init__(self, db_conn, hs):
33	38	super(DeviceStore, self).__init__(db_conn, hs)
34	39

49	54	index_name="device_lists_stream_user_id",
50	55	table="device_lists_stream",
51	56	columns=["user_id", "device_id"],
	57	)
	58
	59	# create a unique index on device_lists_remote_cache
	60	self.register_background_index_update(
	61	"device_lists_remote_cache_unique_idx",
	62	index_name="device_lists_remote_cache_unique_id",
	63	table="device_lists_remote_cache",
	64	columns=["user_id", "device_id"],
	65	unique=True,
	66	)
	67
	68	# And one on device_lists_remote_extremeties
	69	self.register_background_index_update(
	70	"device_lists_remote_extremeties_unique_idx",
	71	index_name="device_lists_remote_extremeties_unique_idx",
	72	table="device_lists_remote_extremeties",
	73	columns=["user_id"],
	74	unique=True,
	75	)
	76
	77	# once they complete, we can remove the old non-unique indexes.
	78	self.register_background_update_handler(
	79	DROP_DEVICE_LIST_STREAMS_NON_UNIQUE_INDEXES,
	80	self._drop_device_list_streams_non_unique_indexes,
52	81	)
53	82
54	83	@defer.inlineCallbacks

238	267
239	268	def update_remote_device_list_cache_entry(self, user_id, device_id, content,
240	269	stream_id):
241		"""Updates a single user's device in the cache.
	270	"""Updates a single device in the cache of a remote user's devicelist.
	271
	272	Note: assumes that we are the only thread that can be updating this user's
	273	device list.
	274
	275	Args:
	276	user_id (str): User to update device list for
	277	device_id (str): ID of decivice being updated
	278	content (dict): new data on this device
	279	stream_id (int): the version of the device list
	280
	281	Returns:
	282	Deferred[None]
242	283	"""
243	284	return self.runInteraction(
244	285	"update_remote_device_list_cache_entry",

271	312	},
272	313	values={
273	314	"content": json.dumps(content),
274		}
	315	},
	316
	317	# we don't need to lock, because we assume we are the only thread
	318	# updating this user's devices.
	319	lock=False,
275	320	)
276	321
277	322	txn.call_after(self._get_cached_user_device.invalidate, (user_id, device_id,))

288	333	},
289	334	values={
290	335	"stream_id": stream_id,
291		}
	336	},
	337
	338	# again, we can assume we are the only thread updating this user's
	339	# extremity.
	340	lock=False,
292	341	)
293	342
294	343	def update_remote_device_list_cache(self, user_id, devices, stream_id):
295		"""Replace the cache of the remote user's devices.
	344	"""Replace the entire cache of the remote user's devices.
	345
	346	Note: assumes that we are the only thread that can be updating this user's
	347	device list.
	348
	349	Args:
	350	user_id (str): User to update device list for
	351	devices (list[dict]): list of device objects supplied over federation
	352	stream_id (int): the version of the device list
	353
	354	Returns:
	355	Deferred[None]
296	356	"""
297	357	return self.runInteraction(
298	358	"update_remote_device_list_cache",

337	397	},
338	398	values={
339	399	"stream_id": stream_id,
340		}
	400	},
	401
	402	# we don't need to lock, because we can assume we are the only thread
	403	# updating this user's extremity.
	404	lock=False,
341	405	)
342	406
343	407	def get_devices_by_remote(self, destination, from_stream_id):

588	652	combined list of changes to devices, and which destinations need to be
589	653	poked. `destination` may be None if no destinations need to be poked.
590	654	"""
	655	# We do a group by here as there can be a large number of duplicate
	656	# entries, since we throw away device IDs.
591	657	sql = """
592		SELECT stream_id, user_id, destination FROM device_lists_stream
	658	SELECT MAX(stream_id) AS stream_id, user_id, destination
	659	FROM device_lists_stream
593	660	LEFT JOIN device_lists_outbound_pokes USING (stream_id, user_id, device_id)
594	661	WHERE ? < stream_id AND stream_id <= ?
	662	GROUP BY user_id, destination
595	663	"""
596	664	return self._execute(
597	665	"get_all_device_list_changes_for_remotes", None,

717	785	"_prune_old_outbound_device_pokes",
718	786	_prune_txn,
719	787	)
	788
	789	@defer.inlineCallbacks
	790	def _drop_device_list_streams_non_unique_indexes(self, progress, batch_size):
	791	def f(conn):
	792	txn = conn.cursor()
	793	txn.execute(
	794	"DROP INDEX IF EXISTS device_lists_remote_cache_id"
	795	)
	796	txn.execute(
	797	"DROP INDEX IF EXISTS device_lists_remote_extremeties_id"
	798	)
	799	txn.close()
	800
	801	yield self.runWithConnection(f)
	802	yield self._end_background_update(DROP_DEVICE_LIST_STREAMS_NON_UNIQUE_INDEXES)
	803	defer.returnValue(1)

+17

-2

synapse/storage/e2e_room_keys.py less more

117	117	these room keys.
118	118	"""
119	119
	120	try:
	121	version = int(version)
	122	except ValueError:
	123	defer.returnValue({'rooms': {}})
	124
120	125	keyvalues = {
121	126	"user_id": user_id,
122	127	"version": version,

211	216	Raises:
212	217	StoreError: with code 404 if there are no e2e_room_keys_versions present
213	218	Returns:
214		A deferred dict giving the info metadata for this backup version
	219	A deferred dict giving the info metadata for this backup version, with
	220	fields including:
	221	version(str)
	222	algorithm(str)
	223	auth_data(object): opaque dict supplied by the client
215	224	"""
216	225
217	226	def _get_e2e_room_keys_version_info_txn(txn):
218	227	if version is None:
219	228	this_version = self._get_current_version(txn, user_id)
220	229	else:
221		this_version = version
	230	try:
	231	this_version = int(version)
	232	except ValueError:
	233	# Our versions are all ints so if we can't convert it to an integer,
	234	# it isn't there.
	235	raise StoreError(404, "No row found")
222	236
223	237	result = self._simple_select_one_txn(
224	238	txn,

235	249	),
236	250	)
237	251	result["auth_data"] = json.loads(result["auth_data"])
	252	result["version"] = str(result["version"])
238	253	return result
239	254
240	255	return self.runInteraction(

+4

-1

synapse/storage/end_to_end_keys.py less more

39	39	allow_none=True,
40	40	)
41	41
42		new_key_json = encode_canonical_json(device_keys)
	42	# In py3 we need old_key_json to match new_key_json type. The DB
	43	# returns unicode while encode_canonical_json returns bytes.
	44	new_key_json = encode_canonical_json(device_keys).decode("utf-8")
	45
43	46	if old_key_json == new_key_json:
44	47	return False
45	48

+2

-2

synapse/storage/event_federation.py less more

476	476	"is_state": False,
477	477	}
478	478	for ev in events
479		for e_id, _ in ev.prev_events
	479	for e_id in ev.prev_event_ids()
480	480	],
481	481	)
482	482

509	509
510	510	txn.executemany(query, [
511	511	(e_id, ev.room_id, e_id, ev.room_id, e_id, ev.room_id, False)
512		for ev in events for e_id, _ in ev.prev_events
	512	for ev in events for e_id in ev.prev_event_ids()
513	513	if not ev.internal_metadata.is_outlier()
514	514	])
515	515

+116

-53

synapse/storage/events.py less more

37	37	from synapse.storage.background_updates import BackgroundUpdateStore
38	38	from synapse.storage.event_federation import EventFederationStore
39	39	from synapse.storage.events_worker import EventsWorkerStore
	40	from synapse.storage.state import StateGroupWorkerStore
40	41	from synapse.types import RoomStreamToken, get_domain_from_id
41	42	from synapse.util import batch_iter
42	43	from synapse.util.async_helpers import ObservableDeferred

204	205
205	206	# inherits from EventFederationStore so that we can call _update_backward_extremities
206	207	# and _handle_mult_prev_events (though arguably those could both be moved in here)
207		class EventsStore(EventFederationStore, EventsWorkerStore, BackgroundUpdateStore):
	208	class EventsStore(StateGroupWorkerStore, EventFederationStore, EventsWorkerStore,
	209	BackgroundUpdateStore):
208	210	EVENT_ORIGIN_SERVER_TS_NAME = "event_origin_server_ts"
209	211	EVENT_FIELDS_SENDER_URL_UPDATE_NAME = "event_fields_sender_url"
210	212

413	415	)
414	416	if len_1:
415	417	all_single_prev_not_state = all(
416		len(event.prev_events) == 1
	418	len(event.prev_event_ids()) == 1
417	419	and not event.is_state()
418	420	for event, ctx in ev_ctx_rm
419	421	)

437	439	# guess this by looking at the prev_events and checking
438	440	# if they match the current forward extremities.
439	441	for ev, _ in ev_ctx_rm:
440		prev_event_ids = set(e for e, _ in ev.prev_events)
	442	prev_event_ids = set(ev.prev_event_ids())
441	443	if latest_event_ids == prev_event_ids:
442	444	state_delta_reuse_delta_counter.inc()
443	445	break

548	550	result.difference_update(
549	551	e_id
550	552	for event in new_events
551		for e_id, _ in event.prev_events
	553	for e_id in event.prev_event_ids()
552	554	)
553	555
554	556	# Finally, remove any events which are prev_events of any existing events.

866	868	"auth_id": auth_id,
867	869	}
868	870	for event, _ in events_and_contexts
869		for auth_id, _ in event.auth_events
	871	for auth_id in event.auth_event_ids()
870	872	if event.is_state()
871	873	],
872	874	)

2033	2035
2034	2036	logger.info("[purge] finding redundant state groups")
2035	2037
2036		# Get all state groups that are only referenced by events that are
2037		# to be deleted.
2038		# This works by first getting state groups that we may want to delete,
2039		# joining against event_to_state_groups to get events that use that
2040		# state group, then left joining against events_to_purge again. Any
2041		# state group where the left join produce no nulls are referenced
2042		# only by events that are going to be purged.
	2038	# Get all state groups that are referenced by events that are to be
	2039	# deleted. We then go and check if they are referenced by other events
	2040	# or state groups, and if not we delete them.
2043	2041	txn.execute("""
2044		SELECT state_group FROM
2045		(
2046		SELECT DISTINCT state_group FROM events_to_purge
2047		INNER JOIN event_to_state_groups USING (event_id)
2048		) AS sp
2049		INNER JOIN event_to_state_groups USING (state_group)
2050		LEFT JOIN events_to_purge AS ep USING (event_id)
2051		GROUP BY state_group
2052		HAVING SUM(CASE WHEN ep.event_id IS NULL THEN 1 ELSE 0 END) = 0
	2042	SELECT DISTINCT state_group FROM events_to_purge
	2043	INNER JOIN event_to_state_groups USING (event_id)
2053	2044	""")
2054	2045
2055		state_rows = txn.fetchall()
2056		logger.info("[purge] found %i redundant state groups", len(state_rows))
2057
2058		# make a set of the redundant state groups, so that we can look them up
2059		# efficiently
2060		state_groups_to_delete = set([sg for sg, in state_rows])
2061
2062		# Now we get all the state groups that rely on these state groups
2063		logger.info("[purge] finding state groups which depend on redundant"
2064		" state groups")
2065		remaining_state_groups = []
2066		for i in range(0, len(state_rows), 100):
2067		chunk = [sg for sg, in state_rows[i:i + 100]]
2068		# look for state groups whose prev_state_group is one we are about
2069		# to delete
2070		rows = self._simple_select_many_txn(
2071		txn,
2072		table="state_group_edges",
2073		column="prev_state_group",
2074		iterable=chunk,
2075		retcols=["state_group"],
2076		keyvalues={},
2077		)
2078		remaining_state_groups.extend(
2079		row["state_group"] for row in rows
2080
2081		# exclude state groups we are about to delete: no point in
2082		# updating them
2083		if row["state_group"] not in state_groups_to_delete
2084		)
	2046	referenced_state_groups = set(sg for sg, in txn)
	2047	logger.info(
	2048	"[purge] found %i referenced state groups",
	2049	len(referenced_state_groups),
	2050	)
	2051
	2052	logger.info("[purge] finding state groups that can be deleted")
	2053
	2054	state_groups_to_delete, remaining_state_groups = (
	2055	self._find_unreferenced_groups_during_purge(
	2056	txn, referenced_state_groups,
	2057	)
	2058	)
	2059
	2060	logger.info(
	2061	"[purge] found %i state groups to delete",
	2062	len(state_groups_to_delete),
	2063	)
	2064
	2065	logger.info(
	2066	"[purge] de-delta-ing %i remaining state groups",
	2067	len(remaining_state_groups),
	2068	)
2085	2069
2086	2070	# Now we turn the state groups that reference to-be-deleted state
2087	2071	# groups to non delta versions.

2126	2110	logger.info("[purge] removing redundant state groups")
2127	2111	txn.executemany(
2128	2112	"DELETE FROM state_groups_state WHERE state_group = ?",
2129		state_rows
	2113	((sg,) for sg in state_groups_to_delete),
2130	2114	)
2131	2115	txn.executemany(
2132	2116	"DELETE FROM state_groups WHERE id = ?",
2133		state_rows
	2117	((sg,) for sg in state_groups_to_delete),
2134	2118	)
2135	2119
2136	2120	logger.info("[purge] removing events from event_to_state_groups")

2226	2210
2227	2211	logger.info("[purge] done")
2228	2212
	2213	def _find_unreferenced_groups_during_purge(self, txn, state_groups):
	2214	"""Used when purging history to figure out which state groups can be
	2215	deleted and which need to be de-delta'ed (due to one of its prev groups
	2216	being scheduled for deletion).
	2217
	2218	Args:
	2219	txn
	2220	state_groups (set[int]): Set of state groups referenced by events
	2221	that are going to be deleted.
	2222
	2223	Returns:
	2224	tuple[set[int], set[int]]: The set of state groups that can be
	2225	deleted and the set of state groups that need to be de-delta'ed
	2226	"""
	2227	# Graph of state group -> previous group
	2228	graph = {}
	2229
	2230	# Set of events that we have found to be referenced by events
	2231	referenced_groups = set()
	2232
	2233	# Set of state groups we've already seen
	2234	state_groups_seen = set(state_groups)
	2235
	2236	# Set of state groups to handle next.
	2237	next_to_search = set(state_groups)
	2238	while next_to_search:
	2239	# We bound size of groups we're looking up at once, to stop the
	2240	# SQL query getting too big
	2241	if len(next_to_search) < 100:
	2242	current_search = next_to_search
	2243	next_to_search = set()
	2244	else:
	2245	current_search = set(itertools.islice(next_to_search, 100))
	2246	next_to_search -= current_search
	2247
	2248	# Check if state groups are referenced
	2249	sql = """
	2250	SELECT DISTINCT state_group FROM event_to_state_groups
	2251	LEFT JOIN events_to_purge AS ep USING (event_id)
	2252	WHERE state_group IN (%s) AND ep.event_id IS NULL
	2253	""" % (",".join("?" for _ in current_search),)
	2254	txn.execute(sql, list(current_search))
	2255
	2256	referenced = set(sg for sg, in txn)
	2257	referenced_groups \|= referenced
	2258
	2259	# We don't continue iterating up the state group graphs for state
	2260	# groups that are referenced.
	2261	current_search -= referenced
	2262
	2263	rows = self._simple_select_many_txn(
	2264	txn,
	2265	table="state_group_edges",
	2266	column="prev_state_group",
	2267	iterable=current_search,
	2268	keyvalues={},
	2269	retcols=("prev_state_group", "state_group",),
	2270	)
	2271
	2272	prevs = set(row["state_group"] for row in rows)
	2273	# We don't bother re-handling groups we've already seen
	2274	prevs -= state_groups_seen
	2275	next_to_search \|= prevs
	2276	state_groups_seen \|= prevs
	2277
	2278	for row in rows:
	2279	# Note: Each state group can have at most one prev group
	2280	graph[row["state_group"]] = row["prev_state_group"]
	2281
	2282	to_delete = state_groups_seen - referenced_groups
	2283
	2284	to_dedelta = set()
	2285	for sg in referenced_groups:
	2286	prev_sg = graph.get(sg)
	2287	if prev_sg and prev_sg in to_delete:
	2288	to_dedelta.add(sg)
	2289
	2290	return to_delete, to_dedelta
	2291
2229	2292	@defer.inlineCallbacks
2230	2293	def is_event_after(self, event_id1, event_id2):
2231	2294	"""Returns True if event_id1 is after event_id2 in the stream

+1

-1

synapse/storage/prepare_database.py less more

24	24
25	25	# Remember to update this number every time a change is made to database
26	26	# schema files, so the users will be informed on server restarts.
27		SCHEMA_VERSION = 51
	27	SCHEMA_VERSION = 52
28	28
29	29	dir_path = os.path.abspath(os.path.dirname(__file__))
30	30

+1

-1

synapse/storage/room.py less more

46	46	Args:
47	47	room_id (str): The ID of the room to retrieve.
48	48	Returns:
49		A namedtuple containing the room information, or an empty list.
	49	A dict containing the room information, or None if the room is unknown.
50	50	"""
51	51	return self._simple_select_one(
52	52	table="rooms",

+5

-4

synapse/storage/schema/delta/40/device_list_streams.sql less more

19	19	content TEXT NOT NULL
20	20	);
21	21
22		CREATE INDEX device_lists_remote_cache_id ON device_lists_remote_cache(user_id, device_id);
23
24
25	22	-- The last update we got for a user. Empty if we're not receiving updates for
26	23	-- that user.
27	24	CREATE TABLE device_lists_remote_extremeties (

29	26	stream_id TEXT NOT NULL
30	27	);
31	28
32		CREATE INDEX device_lists_remote_extremeties_id ON device_lists_remote_extremeties(user_id, stream_id);
	29	-- we used to create non-unique indexes on these tables, but as of update 52 we create
	30	-- unique indexes concurrently:
	31	--
	32	-- CREATE INDEX device_lists_remote_cache_id ON device_lists_remote_cache(user_id, device_id);
	33	-- CREATE INDEX device_lists_remote_extremeties_id ON device_lists_remote_extremeties(user_id, stream_id);
33	34
34	35
35	36	-- Stream of device lists updates. Includes both local and remotes

+19

-0

synapse/storage/schema/delta/52/add_event_to_state_group_index.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- This is needed to efficiently check for unreferenced state groups during
	16	-- purge. Added events_to_state_group(state_group) index
	17	INSERT into background_updates (update_name, progress_json)
	18	VALUES ('event_to_state_groups_sg_index', '{}');

+36

-0

synapse/storage/schema/delta/52/device_list_streams_unique_idx.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- register a background update which will create a unique index on
	16	-- device_lists_remote_cache
	17	INSERT into background_updates (update_name, progress_json)
	18	VALUES ('device_lists_remote_cache_unique_idx', '{}');
	19
	20	-- and one on device_lists_remote_extremeties
	21	INSERT into background_updates (update_name, progress_json, depends_on)
	22	VALUES (
	23	'device_lists_remote_extremeties_unique_idx', '{}',
	24
	25	-- doesn't really depend on this, but we need to make sure both happen
	26	-- before we drop the old indexes.
	27	'device_lists_remote_cache_unique_idx'
	28	);
	29
	30	-- once they complete, we can drop the old indexes.
	31	INSERT into background_updates (update_name, progress_json, depends_on)
	32	VALUES (
	33	'drop_device_list_streams_non_unique_indexes', '{}',
	34	'device_lists_remote_extremeties_unique_idx'
	35	);

+53

-0

synapse/storage/schema/delta/52/e2e_room_keys.sql less more

	0	/* Copyright 2018 New Vector Ltd
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	/* Change version column to an integer so we can do MAX() sensibly
	16	*/
	17	CREATE TABLE e2e_room_keys_versions_new (
	18	user_id TEXT NOT NULL,
	19	version BIGINT NOT NULL,
	20	algorithm TEXT NOT NULL,
	21	auth_data TEXT NOT NULL,
	22	deleted SMALLINT DEFAULT 0 NOT NULL
	23	);
	24
	25	INSERT INTO e2e_room_keys_versions_new
	26	SELECT user_id, CAST(version as BIGINT), algorithm, auth_data, deleted FROM e2e_room_keys_versions;
	27
	28	DROP TABLE e2e_room_keys_versions;
	29	ALTER TABLE e2e_room_keys_versions_new RENAME TO e2e_room_keys_versions;
	30
	31	CREATE UNIQUE INDEX e2e_room_keys_versions_idx ON e2e_room_keys_versions(user_id, version);
	32
	33	/* Change e2e_rooms_keys to match
	34	*/
	35	CREATE TABLE e2e_room_keys_new (
	36	user_id TEXT NOT NULL,
	37	room_id TEXT NOT NULL,
	38	session_id TEXT NOT NULL,
	39	version BIGINT NOT NULL,
	40	first_message_index INT,
	41	forwarded_count INT,
	42	is_verified BOOLEAN,
	43	session_data TEXT NOT NULL
	44	);
	45
	46	INSERT INTO e2e_room_keys_new
	47	SELECT user_id, room_id, session_id, CAST(version as BIGINT), first_message_index, forwarded_count, is_verified, session_data FROM e2e_room_keys;
	48
	49	DROP TABLE e2e_room_keys;
	50	ALTER TABLE e2e_room_keys_new RENAME TO e2e_room_keys;
	51
	52	CREATE UNIQUE INDEX e2e_room_keys_idx ON e2e_room_keys(user_id, room_id, session_id);

+7

-0

synapse/storage/state.py less more

1256	1256	STATE_GROUP_DEDUPLICATION_UPDATE_NAME = "state_group_state_deduplication"
1257	1257	STATE_GROUP_INDEX_UPDATE_NAME = "state_group_state_type_index"
1258	1258	CURRENT_STATE_INDEX_UPDATE_NAME = "current_state_members_idx"
	1259	EVENT_STATE_GROUP_INDEX_UPDATE_NAME = "event_to_state_groups_sg_index"
1259	1260
1260	1261	def __init__(self, db_conn, hs):
1261	1262	super(StateStore, self).__init__(db_conn, hs)

1273	1274	table="current_state_events",
1274	1275	columns=["state_key"],
1275	1276	where_clause="type='m.room.member'",
	1277	)
	1278	self.register_background_index_update(
	1279	self.EVENT_STATE_GROUP_INDEX_UPDATE_NAME,
	1280	index_name="event_to_state_groups_sg_index",
	1281	table="event_to_state_groups",
	1282	columns=["state_group"],
1276	1283	)
1277	1284
1278	1285	def _store_event_state_mappings_txn(self, txn, events_and_contexts):

+50

-55

tests/handlers/test_e2e_room_keys.py less more

168	168	self.assertEqual(res, 404)
169	169
170	170	@defer.inlineCallbacks
	171	def test_get_missing_backup(self):
	172	"""Check that we get a 404 on querying missing backup
	173	"""
	174	res = None
	175	try:
	176	yield self.handler.get_room_keys(self.local_user, "bogus_version")
	177	except errors.SynapseError as e:
	178	res = e.code
	179	self.assertEqual(res, 404)
	180
	181	@defer.inlineCallbacks
171	182	def test_get_missing_room_keys(self):
172		"""Check that we get a 404 on querying missing room_keys
173		"""
174		res = None
175		try:
176		yield self.handler.get_room_keys(self.local_user, "bogus_version")
177		except errors.SynapseError as e:
178		res = e.code
179		self.assertEqual(res, 404)
180
181		# check we also get a 404 even if the version is valid
182		version = yield self.handler.create_version(self.local_user, {
183		"algorithm": "m.megolm_backup.v1",
184		"auth_data": "first_version_auth_data",
185		})
186		self.assertEqual(version, "1")
187
188		res = None
189		try:
190		yield self.handler.get_room_keys(self.local_user, version)
191		except errors.SynapseError as e:
192		res = e.code
193		self.assertEqual(res, 404)
	183	"""Check we get an empty response from an empty backup
	184	"""
	185	version = yield self.handler.create_version(self.local_user, {
	186	"algorithm": "m.megolm_backup.v1",
	187	"auth_data": "first_version_auth_data",
	188	})
	189	self.assertEqual(version, "1")
	190
	191	res = yield self.handler.get_room_keys(self.local_user, version)
	192	self.assertDictEqual(res, {
	193	"rooms": {}
	194	})
194	195
195	196	# TODO: test the locking semantics when uploading room_keys,
196	197	# although this is probably best done in sytest

344	345	# check for bulk-delete
345	346	yield self.handler.upload_room_keys(self.local_user, version, room_keys)
346	347	yield self.handler.delete_room_keys(self.local_user, version)
347		res = None
348		try:
349		yield self.handler.get_room_keys(
350		self.local_user,
351		version,
352		room_id="!abc:matrix.org",
353		session_id="c0ff33",
354		)
355		except errors.SynapseError as e:
356		res = e.code
357		self.assertEqual(res, 404)
	348	res = yield self.handler.get_room_keys(
	349	self.local_user,
	350	version,
	351	room_id="!abc:matrix.org",
	352	session_id="c0ff33",
	353	)
	354	self.assertDictEqual(res, {
	355	"rooms": {}
	356	})
358	357
359	358	# check for bulk-delete per room
360	359	yield self.handler.upload_room_keys(self.local_user, version, room_keys)

363	362	version,
364	363	room_id="!abc:matrix.org",
365	364	)
366		res = None
367		try:
368		yield self.handler.get_room_keys(
369		self.local_user,
370		version,
371		room_id="!abc:matrix.org",
372		session_id="c0ff33",
373		)
374		except errors.SynapseError as e:
375		res = e.code
376		self.assertEqual(res, 404)
	365	res = yield self.handler.get_room_keys(
	366	self.local_user,
	367	version,
	368	room_id="!abc:matrix.org",
	369	session_id="c0ff33",
	370	)
	371	self.assertDictEqual(res, {
	372	"rooms": {}
	373	})
377	374
378	375	# check for bulk-delete per session
379	376	yield self.handler.upload_room_keys(self.local_user, version, room_keys)

383	380	room_id="!abc:matrix.org",
384	381	session_id="c0ff33",
385	382	)
386		res = None
387		try:
388		yield self.handler.get_room_keys(
389		self.local_user,
390		version,
391		room_id="!abc:matrix.org",
392		session_id="c0ff33",
393		)
394		except errors.SynapseError as e:
395		res = e.code
396		self.assertEqual(res, 404)
	383	res = yield self.handler.get_room_keys(
	384	self.local_user,
	385	version,
	386	room_id="!abc:matrix.org",
	387	session_id="c0ff33",
	388	)
	389	self.assertDictEqual(res, {
	390	"rooms": {}
	391	})

+0

-0

tests/push/__init__.py less more

(New empty file)

+148

-0

tests/push/test_email.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2018 New Vector
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import os
	16
	17	import pkg_resources
	18
	19	from twisted.internet.defer import Deferred
	20
	21	from synapse.rest.client.v1 import admin, login, room
	22
	23	from tests.unittest import HomeserverTestCase
	24
	25	try:
	26	from synapse.push.mailer import load_jinja2_templates
	27	except Exception:
	28	load_jinja2_templates = None
	29
	30
	31	class EmailPusherTests(HomeserverTestCase):
	32
	33	skip = "No Jinja installed" if not load_jinja2_templates else None
	34	servlets = [
	35	admin.register_servlets,
	36	room.register_servlets,
	37	login.register_servlets,
	38	]
	39	user_id = True
	40	hijack_auth = False
	41
	42	def make_homeserver(self, reactor, clock):
	43
	44	# List[Tuple[Deferred, args, kwargs]]
	45	self.email_attempts = []
	46
	47	def sendmail(args, *kwargs):
	48	d = Deferred()
	49	self.email_attempts.append((d, args, kwargs))
	50	return d
	51
	52	config = self.default_config()
	53	config.email_enable_notifs = True
	54	config.start_pushers = True
	55
	56	config.email_template_dir = os.path.abspath(
	57	pkg_resources.resource_filename('synapse', 'res/templates')
	58	)
	59	config.email_notif_template_html = "notif_mail.html"
	60	config.email_notif_template_text = "notif_mail.txt"
	61	config.email_smtp_host = "127.0.0.1"
	62	config.email_smtp_port = 20
	63	config.require_transport_security = False
	64	config.email_smtp_user = None
	65	config.email_app_name = "Matrix"
	66	config.email_notif_from = "test@example.com"
	67
	68	hs = self.setup_test_homeserver(config=config, sendmail=sendmail)
	69
	70	return hs
	71
	72	def test_sends_email(self):
	73
	74	# Register the user who gets notified
	75	user_id = self.register_user("user", "pass")
	76	access_token = self.login("user", "pass")
	77
	78	# Register the user who sends the message
	79	other_user_id = self.register_user("otheruser", "pass")
	80	other_access_token = self.login("otheruser", "pass")
	81
	82	# Register the pusher
	83	user_tuple = self.get_success(
	84	self.hs.get_datastore().get_user_by_access_token(access_token)
	85	)
	86	token_id = user_tuple["token_id"]
	87
	88	self.get_success(
	89	self.hs.get_pusherpool().add_pusher(
	90	user_id=user_id,
	91	access_token=token_id,
	92	kind="email",
	93	app_id="m.email",
	94	app_display_name="Email Notifications",
	95	device_display_name="a@example.com",
	96	pushkey="a@example.com",
	97	lang=None,
	98	data={},
	99	)
	100	)
	101
	102	# Create a room
	103	room = self.helper.create_room_as(user_id, tok=access_token)
	104
	105	# Invite the other person
	106	self.helper.invite(room=room, src=user_id, tok=access_token, targ=other_user_id)
	107
	108	# The other user joins
	109	self.helper.join(room=room, user=other_user_id, tok=other_access_token)
	110
	111	# The other user sends some messages
	112	self.helper.send(room, body="Hi!", tok=other_access_token)
	113	self.helper.send(room, body="There!", tok=other_access_token)
	114
	115	# Get the stream ordering before it gets sent
	116	pushers = self.get_success(
	117	self.hs.get_datastore().get_pushers_by(dict(user_name=user_id))
	118	)
	119	self.assertEqual(len(pushers), 1)
	120	last_stream_ordering = pushers[0]["last_stream_ordering"]
	121
	122	# Advance time a bit, so the pusher will register something has happened
	123	self.pump(100)
	124
	125	# It hasn't succeeded yet, so the stream ordering shouldn't have moved
	126	pushers = self.get_success(
	127	self.hs.get_datastore().get_pushers_by(dict(user_name=user_id))
	128	)
	129	self.assertEqual(len(pushers), 1)
	130	self.assertEqual(last_stream_ordering, pushers[0]["last_stream_ordering"])
	131
	132	# One email was attempted to be sent
	133	self.assertEqual(len(self.email_attempts), 1)
	134
	135	# Make the email succeed
	136	self.email_attempts[0][0].callback(True)
	137	self.pump()
	138
	139	# One email was attempted to be sent
	140	self.assertEqual(len(self.email_attempts), 1)
	141
	142	# The stream ordering has increased
	143	pushers = self.get_success(
	144	self.hs.get_datastore().get_pushers_by(dict(user_name=user_id))
	145	)
	146	self.assertEqual(len(pushers), 1)
	147	self.assertTrue(pushers[0]["last_stream_ordering"] > last_stream_ordering)

+159

-0

tests/push/test_http.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2018 New Vector
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	from mock import Mock
	16
	17	from twisted.internet.defer import Deferred
	18
	19	from synapse.rest.client.v1 import admin, login, room
	20
	21	from tests.unittest import HomeserverTestCase
	22
	23	try:
	24	from synapse.push.mailer import load_jinja2_templates
	25	except Exception:
	26	load_jinja2_templates = None
	27
	28
	29	class HTTPPusherTests(HomeserverTestCase):
	30
	31	skip = "No Jinja installed" if not load_jinja2_templates else None
	32	servlets = [
	33	admin.register_servlets,
	34	room.register_servlets,
	35	login.register_servlets,
	36	]
	37	user_id = True
	38	hijack_auth = False
	39
	40	def make_homeserver(self, reactor, clock):
	41
	42	self.push_attempts = []
	43
	44	m = Mock()
	45
	46	def post_json_get_json(url, body):
	47	d = Deferred()
	48	self.push_attempts.append((d, url, body))
	49	return d
	50
	51	m.post_json_get_json = post_json_get_json
	52
	53	config = self.default_config()
	54	config.start_pushers = True
	55
	56	hs = self.setup_test_homeserver(config=config, simple_http_client=m)
	57
	58	return hs
	59
	60	def test_sends_http(self):
	61	"""
	62	The HTTP pusher will send pushes for each message to a HTTP endpoint
	63	when configured to do so.
	64	"""
	65	# Register the user who gets notified
	66	user_id = self.register_user("user", "pass")
	67	access_token = self.login("user", "pass")
	68
	69	# Register the user who sends the message
	70	other_user_id = self.register_user("otheruser", "pass")
	71	other_access_token = self.login("otheruser", "pass")
	72
	73	# Register the pusher
	74	user_tuple = self.get_success(
	75	self.hs.get_datastore().get_user_by_access_token(access_token)
	76	)
	77	token_id = user_tuple["token_id"]
	78
	79	self.get_success(
	80	self.hs.get_pusherpool().add_pusher(
	81	user_id=user_id,
	82	access_token=token_id,
	83	kind="http",
	84	app_id="m.http",
	85	app_display_name="HTTP Push Notifications",
	86	device_display_name="pushy push",
	87	pushkey="a@example.com",
	88	lang=None,
	89	data={"url": "example.com"},
	90	)
	91	)
	92
	93	# Create a room
	94	room = self.helper.create_room_as(user_id, tok=access_token)
	95
	96	# Invite the other person
	97	self.helper.invite(room=room, src=user_id, tok=access_token, targ=other_user_id)
	98
	99	# The other user joins
	100	self.helper.join(room=room, user=other_user_id, tok=other_access_token)
	101
	102	# The other user sends some messages
	103	self.helper.send(room, body="Hi!", tok=other_access_token)
	104	self.helper.send(room, body="There!", tok=other_access_token)
	105
	106	# Get the stream ordering before it gets sent
	107	pushers = self.get_success(
	108	self.hs.get_datastore().get_pushers_by(dict(user_name=user_id))
	109	)
	110	self.assertEqual(len(pushers), 1)
	111	last_stream_ordering = pushers[0]["last_stream_ordering"]
	112
	113	# Advance time a bit, so the pusher will register something has happened
	114	self.pump()
	115
	116	# It hasn't succeeded yet, so the stream ordering shouldn't have moved
	117	pushers = self.get_success(
	118	self.hs.get_datastore().get_pushers_by(dict(user_name=user_id))
	119	)
	120	self.assertEqual(len(pushers), 1)
	121	self.assertEqual(last_stream_ordering, pushers[0]["last_stream_ordering"])
	122
	123	# One push was attempted to be sent -- it'll be the first message
	124	self.assertEqual(len(self.push_attempts), 1)
	125	self.assertEqual(self.push_attempts[0][1], "example.com")
	126	self.assertEqual(
	127	self.push_attempts[0][2]["notification"]["content"]["body"], "Hi!"
	128	)
	129
	130	# Make the push succeed
	131	self.push_attempts[0][0].callback({})
	132	self.pump()
	133
	134	# The stream ordering has increased
	135	pushers = self.get_success(
	136	self.hs.get_datastore().get_pushers_by(dict(user_name=user_id))
	137	)
	138	self.assertEqual(len(pushers), 1)
	139	self.assertTrue(pushers[0]["last_stream_ordering"] > last_stream_ordering)
	140	last_stream_ordering = pushers[0]["last_stream_ordering"]
	141
	142	# Now it'll try and send the second push message, which will be the second one
	143	self.assertEqual(len(self.push_attempts), 2)
	144	self.assertEqual(self.push_attempts[1][1], "example.com")
	145	self.assertEqual(
	146	self.push_attempts[1][2]["notification"]["content"]["body"], "There!"
	147	)
	148
	149	# Make the second push succeed
	150	self.push_attempts[1][0].callback({})
	151	self.pump()
	152
	153	# The stream ordering has increased, again
	154	pushers = self.get_success(
	155	self.hs.get_datastore().get_pushers_by(dict(user_name=user_id))
	156	)
	157	self.assertEqual(len(pushers), 1)
	158	self.assertTrue(pushers[0]["last_stream_ordering"] > last_stream_ordering)

+2

-2

tests/replication/slave/storage/test_events.py less more

27	27
28	28
29	29	def dict_equals(self, other):
30		me = encode_canonical_json(self._event_dict)
31		them = encode_canonical_json(other._event_dict)
	30	me = encode_canonical_json(self.get_pdu_json())
	31	them = encode_canonical_json(other.get_pdu_json())
32	32	return me == them
33	33
34	34

+118

-0

tests/rest/client/test_consent.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2018 New Vector
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import os
	16
	17	from synapse.api.urls import ConsentURIBuilder
	18	from synapse.rest.client.v1 import admin, login, room
	19	from synapse.rest.consent import consent_resource
	20
	21	from tests import unittest
	22	from tests.server import render
	23
	24	try:
	25	from synapse.push.mailer import load_jinja2_templates
	26	except Exception:
	27	load_jinja2_templates = None
	28
	29
	30	class ConsentResourceTestCase(unittest.HomeserverTestCase):
	31	skip = "No Jinja installed" if not load_jinja2_templates else None
	32	servlets = [
	33	admin.register_servlets,
	34	room.register_servlets,
	35	login.register_servlets,
	36	]
	37	user_id = True
	38	hijack_auth = False
	39
	40	def make_homeserver(self, reactor, clock):
	41
	42	config = self.default_config()
	43	config.user_consent_version = "1"
	44	config.public_baseurl = ""
	45	config.form_secret = "123abc"
	46
	47	# Make some temporary templates...
	48	temp_consent_path = self.mktemp()
	49	os.mkdir(temp_consent_path)
	50	os.mkdir(os.path.join(temp_consent_path, 'en'))
	51	config.user_consent_template_dir = os.path.abspath(temp_consent_path)
	52
	53	with open(os.path.join(temp_consent_path, "en/1.html"), 'w') as f:
	54	f.write("{{version}},{{has_consented}}")
	55
	56	with open(os.path.join(temp_consent_path, "en/success.html"), 'w') as f:
	57	f.write("yay!")
	58
	59	hs = self.setup_test_homeserver(config=config)
	60	return hs
	61
	62	def test_render_public_consent(self):
	63	"""You can observe the terms form without specifying a user"""
	64	resource = consent_resource.ConsentResource(self.hs)
	65	request, channel = self.make_request("GET", "/consent?v=1", shorthand=False)
	66	render(request, resource, self.reactor)
	67	self.assertEqual(channel.code, 200)
	68
	69	def test_accept_consent(self):
	70	"""
	71	A user can use the consent form to accept the terms.
	72	"""
	73	uri_builder = ConsentURIBuilder(self.hs.config)
	74	resource = consent_resource.ConsentResource(self.hs)
	75
	76	# Register a user
	77	user_id = self.register_user("user", "pass")
	78	access_token = self.login("user", "pass")
	79
	80	# Fetch the consent page, to get the consent version
	81	consent_uri = (
	82	uri_builder.build_user_consent_uri(user_id).replace("_matrix/", "")
	83	+ "&u=user"
	84	)
	85	request, channel = self.make_request(
	86	"GET", consent_uri, access_token=access_token, shorthand=False
	87	)
	88	render(request, resource, self.reactor)
	89	self.assertEqual(channel.code, 200)
	90
	91	# Get the version from the body, and whether we've consented
	92	version, consented = channel.result["body"].decode('ascii').split(",")
	93	self.assertEqual(consented, "False")
	94
	95	# POST to the consent page, saying we've agreed
	96	request, channel = self.make_request(
	97	"POST",
	98	consent_uri + "&v=" + version,
	99	access_token=access_token,
	100	shorthand=False,
	101	)
	102	render(request, resource, self.reactor)
	103	self.assertEqual(channel.code, 200)
	104
	105	# Fetch the consent page, to get the consent version -- it should have
	106	# changed
	107	request, channel = self.make_request(
	108	"GET", consent_uri, access_token=access_token, shorthand=False
	109	)
	110	render(request, resource, self.reactor)
	111	self.assertEqual(channel.code, 200)
	112
	113	# Get the version from the body, and check that it's the version we
	114	# agreed to, and that we've consented to it.
	115	version, consented = channel.result["body"].decode('ascii').split(",")
	116	self.assertEqual(consented, "True")
	117	self.assertEqual(version, "1")

+57

-67

tests/rest/client/v1/test_admin.py less more

18	18
19	19	from mock import Mock
20	20
21		from synapse.http.server import JsonResource
22	21	from synapse.rest.client.v1.admin import register_servlets
23		from synapse.util import Clock
24	22
25	23	from tests import unittest
26		from tests.server import (
27		ThreadedMemoryReactorClock,
28		make_request,
29		render,
30		setup_test_homeserver,
31		)
32
33
34		class UserRegisterTestCase(unittest.TestCase):
35		def setUp(self):
36
37		self.clock = ThreadedMemoryReactorClock()
38		self.hs_clock = Clock(self.clock)
	24
	25
	26	class UserRegisterTestCase(unittest.HomeserverTestCase):
	27
	28	servlets = [register_servlets]
	29
	30	def make_homeserver(self, reactor, clock):
	31
39	32	self.url = "/_matrix/client/r0/admin/register"
40	33
41	34	self.registration_handler = Mock()

49	42
50	43	self.secrets = Mock()
51	44
52		self.hs = setup_test_homeserver(
53		self.addCleanup, http_client=None, clock=self.hs_clock, reactor=self.clock
54		)
	45	self.hs = self.setup_test_homeserver()
55	46
56	47	self.hs.config.registration_shared_secret = u"shared"
57	48
58	49	self.hs.get_media_repository = Mock()
59	50	self.hs.get_deactivate_account_handler = Mock()
60	51
61		self.resource = JsonResource(self.hs)
62		register_servlets(self.hs, self.resource)
	52	return self.hs
63	53
64	54	def test_disabled(self):
65	55	"""

68	58	"""
69	59	self.hs.config.registration_shared_secret = None
70	60
71		request, channel = make_request("POST", self.url, b'{}')
72		render(request, self.resource, self.clock)
	61	request, channel = self.make_request("POST", self.url, b'{}')
	62	self.render(request)
73	63
74	64	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
75	65	self.assertEqual(

86	76
87	77	self.hs.get_secrets = Mock(return_value=secrets)
88	78
89		request, channel = make_request("GET", self.url)
90		render(request, self.resource, self.clock)
	79	request, channel = self.make_request("GET", self.url)
	80	self.render(request)
91	81
92	82	self.assertEqual(channel.json_body, {"nonce": "abcd"})
93	83

96	86	Calling GET on the endpoint will return a randomised nonce, which will
97	87	only last for SALT_TIMEOUT (60s).
98	88	"""
99		request, channel = make_request("GET", self.url)
100		render(request, self.resource, self.clock)
	89	request, channel = self.make_request("GET", self.url)
	90	self.render(request)
101	91	nonce = channel.json_body["nonce"]
102	92
103	93	# 59 seconds
104		self.clock.advance(59)
	94	self.reactor.advance(59)
105	95
106	96	body = json.dumps({"nonce": nonce})
107		request, channel = make_request("POST", self.url, body.encode('utf8'))
108		render(request, self.resource, self.clock)
	97	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	98	self.render(request)
109	99
110	100	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
111	101	self.assertEqual('username must be specified', channel.json_body["error"])
112	102
113	103	# 61 seconds
114		self.clock.advance(2)
115
116		request, channel = make_request("POST", self.url, body.encode('utf8'))
117		render(request, self.resource, self.clock)
	104	self.reactor.advance(2)
	105
	106	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	107	self.render(request)
118	108
119	109	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
120	110	self.assertEqual('unrecognised nonce', channel.json_body["error"])

123	113	"""
124	114	Only the provided nonce can be used, as it's checked in the MAC.
125	115	"""
126		request, channel = make_request("GET", self.url)
127		render(request, self.resource, self.clock)
	116	request, channel = self.make_request("GET", self.url)
	117	self.render(request)
128	118	nonce = channel.json_body["nonce"]
129	119
130	120	want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1)

140	130	"mac": want_mac,
141	131	}
142	132	)
143		request, channel = make_request("POST", self.url, body.encode('utf8'))
144		render(request, self.resource, self.clock)
	133	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	134	self.render(request)
145	135
146	136	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
147	137	self.assertEqual("HMAC incorrect", channel.json_body["error"])

151	141	When the correct nonce is provided, and the right key is provided, the
152	142	user is registered.
153	143	"""
154		request, channel = make_request("GET", self.url)
155		render(request, self.resource, self.clock)
	144	request, channel = self.make_request("GET", self.url)
	145	self.render(request)
156	146	nonce = channel.json_body["nonce"]
157	147
158	148	want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1)

168	158	"mac": want_mac,
169	159	}
170	160	)
171		request, channel = make_request("POST", self.url, body.encode('utf8'))
172		render(request, self.resource, self.clock)
	161	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	162	self.render(request)
173	163
174	164	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
175	165	self.assertEqual("@bob:test", channel.json_body["user_id"])

178	168	"""
179	169	A valid unrecognised nonce.
180	170	"""
181		request, channel = make_request("GET", self.url)
182		render(request, self.resource, self.clock)
	171	request, channel = self.make_request("GET", self.url)
	172	self.render(request)
183	173	nonce = channel.json_body["nonce"]
184	174
185	175	want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1)

195	185	"mac": want_mac,
196	186	}
197	187	)
198		request, channel = make_request("POST", self.url, body.encode('utf8'))
199		render(request, self.resource, self.clock)
	188	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	189	self.render(request)
200	190
201	191	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
202	192	self.assertEqual("@bob:test", channel.json_body["user_id"])
203	193
204	194	# Now, try and reuse it
205		request, channel = make_request("POST", self.url, body.encode('utf8'))
206		render(request, self.resource, self.clock)
	195	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	196	self.render(request)
207	197
208	198	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
209	199	self.assertEqual('unrecognised nonce', channel.json_body["error"])

216	206	"""
217	207
218	208	def nonce():
219		request, channel = make_request("GET", self.url)
220		render(request, self.resource, self.clock)
	209	request, channel = self.make_request("GET", self.url)
	210	self.render(request)
221	211	return channel.json_body["nonce"]
222	212
223	213	#

226	216
227	217	# Must be present
228	218	body = json.dumps({})
229		request, channel = make_request("POST", self.url, body.encode('utf8'))
230		render(request, self.resource, self.clock)
	219	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	220	self.render(request)
231	221
232	222	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
233	223	self.assertEqual('nonce must be specified', channel.json_body["error"])

238	228
239	229	# Must be present
240	230	body = json.dumps({"nonce": nonce()})
241		request, channel = make_request("POST", self.url, body.encode('utf8'))
242		render(request, self.resource, self.clock)
	231	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	232	self.render(request)
243	233
244	234	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
245	235	self.assertEqual('username must be specified', channel.json_body["error"])
246	236
247	237	# Must be a string
248	238	body = json.dumps({"nonce": nonce(), "username": 1234})
249		request, channel = make_request("POST", self.url, body.encode('utf8'))
250		render(request, self.resource, self.clock)
	239	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	240	self.render(request)
251	241
252	242	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
253	243	self.assertEqual('Invalid username', channel.json_body["error"])
254	244
255	245	# Must not have null bytes
256	246	body = json.dumps({"nonce": nonce(), "username": u"abcd\u0000"})
257		request, channel = make_request("POST", self.url, body.encode('utf8'))
258		render(request, self.resource, self.clock)
	247	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	248	self.render(request)
259	249
260	250	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
261	251	self.assertEqual('Invalid username', channel.json_body["error"])
262	252
263	253	# Must not have null bytes
264	254	body = json.dumps({"nonce": nonce(), "username": "a" * 1000})
265		request, channel = make_request("POST", self.url, body.encode('utf8'))
266		render(request, self.resource, self.clock)
	255	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	256	self.render(request)
267	257
268	258	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
269	259	self.assertEqual('Invalid username', channel.json_body["error"])

274	264
275	265	# Must be present
276	266	body = json.dumps({"nonce": nonce(), "username": "a"})
277		request, channel = make_request("POST", self.url, body.encode('utf8'))
278		render(request, self.resource, self.clock)
	267	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	268	self.render(request)
279	269
280	270	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
281	271	self.assertEqual('password must be specified', channel.json_body["error"])
282	272
283	273	# Must be a string
284	274	body = json.dumps({"nonce": nonce(), "username": "a", "password": 1234})
285		request, channel = make_request("POST", self.url, body.encode('utf8'))
286		render(request, self.resource, self.clock)
	275	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	276	self.render(request)
287	277
288	278	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
289	279	self.assertEqual('Invalid password', channel.json_body["error"])

292	282	body = json.dumps(
293	283	{"nonce": nonce(), "username": "a", "password": u"abcd\u0000"}
294	284	)
295		request, channel = make_request("POST", self.url, body.encode('utf8'))
296		render(request, self.resource, self.clock)
	285	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	286	self.render(request)
297	287
298	288	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
299	289	self.assertEqual('Invalid password', channel.json_body["error"])
300	290
301	291	# Super long
302	292	body = json.dumps({"nonce": nonce(), "username": "a", "password": "A" * 1000})
303		request, channel = make_request("POST", self.url, body.encode('utf8'))
304		render(request, self.resource, self.clock)
	293	request, channel = self.make_request("POST", self.url, body.encode('utf8'))
	294	self.render(request)
305	295
306	296	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
307	297	self.assertEqual('Invalid password', channel.json_body["error"])

+5

-5

tests/rest/client/v1/test_register.py less more

44	44	)
45	45
46	46	handlers = Mock(registration_handler=self.registration_handler)
47		self.clock = MemoryReactorClock()
48		self.hs_clock = Clock(self.clock)
	47	self.reactor = MemoryReactorClock()
	48	self.hs_clock = Clock(self.reactor)
49	49
50	50	self.hs = self.hs = setup_test_homeserver(
51		self.addCleanup, http_client=None, clock=self.hs_clock, reactor=self.clock
	51	self.addCleanup, http_client=None, clock=self.hs_clock, reactor=self.reactor
52	52	)
53	53	self.hs.get_datastore = Mock(return_value=self.datastore)
54	54	self.hs.get_handlers = Mock(return_value=handlers)

75	75	return_value=(user_id, token)
76	76	)
77	77
78		request, channel = make_request(b"POST", url, request_data)
79		render(request, res, self.clock)
	78	request, channel = make_request(self.reactor, b"POST", url, request_data)
	79	render(request, res, self.reactor)
80	80
81	81	self.assertEquals(channel.result["code"], b"200")
82	82

+7

-15

tests/rest/client/v1/utils.py less more

168	168	path = path + "?access_token=%s" % tok
169	169
170	170	request, channel = make_request(
171		"POST", path, json.dumps(content).encode('utf8')
	171	self.hs.get_reactor(), "POST", path, json.dumps(content).encode('utf8')
172	172	)
173	173	render(request, self.resource, self.hs.get_reactor())
174	174

216	216
217	217	data = {"membership": membership}
218	218
219		request, channel = make_request("PUT", path, json.dumps(data).encode('utf8'))
	219	request, channel = make_request(
	220	self.hs.get_reactor(), "PUT", path, json.dumps(data).encode('utf8')
	221	)
220	222
221	223	render(request, self.resource, self.hs.get_reactor())
222	224

226	228	)
227	229
228	230	self.auth_user_id = temp_id
229
230		@defer.inlineCallbacks
231		def register(self, user_id):
232		(code, response) = yield self.mock_resource.trigger(
233		"POST",
234		"/_matrix/client/r0/register",
235		json.dumps(
236		{"user": user_id, "password": "test", "type": "m.login.password"}
237		),
238		)
239		self.assertEquals(200, code)
240		defer.returnValue(response)
241	231
242	232	def send(self, room_id, body=None, txn_id=None, tok=None, expect_code=200):
243	233	if txn_id is None:

250	240	if tok:
251	241	path = path + "?access_token=%s" % tok
252	242
253		request, channel = make_request("PUT", path, json.dumps(content).encode('utf8'))
	243	request, channel = make_request(
	244	self.hs.get_reactor(), "PUT", path, json.dumps(content).encode('utf8')
	245	)
254	246	render(request, self.resource, self.hs.get_reactor())
255	247
256	248	assert int(channel.result["code"]) == expect_code, (

+29

-66

tests/rest/client/v2_alpha/test_filter.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		import synapse.types
16	15	from synapse.api.errors import Codes
17		from synapse.http.server import JsonResource
18	16	from synapse.rest.client.v2_alpha import filter
19		from synapse.types import UserID
20		from synapse.util import Clock
21	17
22	18	from tests import unittest
23		from tests.server import (
24		ThreadedMemoryReactorClock as MemoryReactorClock,
25		make_request,
26		render,
27		setup_test_homeserver,
28		)
29	19
30	20	PATH_PREFIX = "/_matrix/client/v2_alpha"
31	21
32	22
33		class FilterTestCase(unittest.TestCase):
	23	class FilterTestCase(unittest.HomeserverTestCase):
34	24
35		USER_ID = "@apple:test"
	25	user_id = "@apple:test"
	26	hijack_auth = True
36	27	EXAMPLE_FILTER = {"room": {"timeline": {"types": ["m.room.message"]}}}
37	28	EXAMPLE_FILTER_JSON = b'{"room": {"timeline": {"types": ["m.room.message"]}}}'
38		TO_REGISTER = [filter]
	29	servlets = [filter.register_servlets]
39	30
40		def setUp(self):
41		self.clock = MemoryReactorClock()
42		self.hs_clock = Clock(self.clock)
43
44		self.hs = setup_test_homeserver(
45		self.addCleanup, http_client=None, clock=self.hs_clock, reactor=self.clock
46		)
47
48		self.auth = self.hs.get_auth()
49
50		def get_user_by_access_token(token=None, allow_guest=False):
51		return {
52		"user": UserID.from_string(self.USER_ID),
53		"token_id": 1,
54		"is_guest": False,
55		}
56
57		def get_user_by_req(request, allow_guest=False, rights="access"):
58		return synapse.types.create_requester(
59		UserID.from_string(self.USER_ID), 1, False, None
60		)
61
62		self.auth.get_user_by_access_token = get_user_by_access_token
63		self.auth.get_user_by_req = get_user_by_req
64
65		self.store = self.hs.get_datastore()
66		self.filtering = self.hs.get_filtering()
67		self.resource = JsonResource(self.hs)
68
69		for r in self.TO_REGISTER:
70		r.register_servlets(self.hs, self.resource)
	31	def prepare(self, reactor, clock, hs):
	32	self.filtering = hs.get_filtering()
	33	self.store = hs.get_datastore()
71	34
72	35	def test_add_filter(self):
73		request, channel = make_request(
	36	request, channel = self.make_request(
74	37	"POST",
75		"/_matrix/client/r0/user/%s/filter" % (self.USER_ID),
	38	"/_matrix/client/r0/user/%s/filter" % (self.user_id),
76	39	self.EXAMPLE_FILTER_JSON,
77	40	)
78		render(request, self.resource, self.clock)
	41	self.render(request)
79	42
80	43	self.assertEqual(channel.result["code"], b"200")
81	44	self.assertEqual(channel.json_body, {"filter_id": "0"})
82	45	filter = self.store.get_user_filter(user_localpart="apple", filter_id=0)
83		self.clock.advance(0)
	46	self.pump()
84	47	self.assertEquals(filter.result, self.EXAMPLE_FILTER)
85	48
86	49	def test_add_filter_for_other_user(self):
87		request, channel = make_request(
	50	request, channel = self.make_request(
88	51	"POST",
89	52	"/_matrix/client/r0/user/%s/filter" % ("@watermelon:test"),
90	53	self.EXAMPLE_FILTER_JSON,
91	54	)
92		render(request, self.resource, self.clock)
	55	self.render(request)
93	56
94	57	self.assertEqual(channel.result["code"], b"403")
95	58	self.assertEquals(channel.json_body["errcode"], Codes.FORBIDDEN)

97	60	def test_add_filter_non_local_user(self):
98	61	_is_mine = self.hs.is_mine
99	62	self.hs.is_mine = lambda target_user: False
100		request, channel = make_request(
	63	request, channel = self.make_request(
101	64	"POST",
102		"/_matrix/client/r0/user/%s/filter" % (self.USER_ID),
	65	"/_matrix/client/r0/user/%s/filter" % (self.user_id),
103	66	self.EXAMPLE_FILTER_JSON,
104	67	)
105		render(request, self.resource, self.clock)
	68	self.render(request)
106	69
107	70	self.hs.is_mine = _is_mine
108	71	self.assertEqual(channel.result["code"], b"403")

112	75	filter_id = self.filtering.add_user_filter(
113	76	user_localpart="apple", user_filter=self.EXAMPLE_FILTER
114	77	)
115		self.clock.advance(1)
	78	self.reactor.advance(1)
116	79	filter_id = filter_id.result
117		request, channel = make_request(
118		"GET", "/_matrix/client/r0/user/%s/filter/%s" % (self.USER_ID, filter_id)
	80	request, channel = self.make_request(
	81	"GET", "/_matrix/client/r0/user/%s/filter/%s" % (self.user_id, filter_id)
119	82	)
120		render(request, self.resource, self.clock)
	83	self.render(request)
121	84
122	85	self.assertEqual(channel.result["code"], b"200")
123	86	self.assertEquals(channel.json_body, self.EXAMPLE_FILTER)
124	87
125	88	def test_get_filter_non_existant(self):
126		request, channel = make_request(
127		"GET", "/_matrix/client/r0/user/%s/filter/12382148321" % (self.USER_ID)
	89	request, channel = self.make_request(
	90	"GET", "/_matrix/client/r0/user/%s/filter/12382148321" % (self.user_id)
128	91	)
129		render(request, self.resource, self.clock)
	92	self.render(request)
130	93
131	94	self.assertEqual(channel.result["code"], b"400")
132	95	self.assertEquals(channel.json_body["errcode"], Codes.NOT_FOUND)

134	97	# Currently invalid params do not have an appropriate errcode
135	98	# in errors.py
136	99	def test_get_filter_invalid_id(self):
137		request, channel = make_request(
138		"GET", "/_matrix/client/r0/user/%s/filter/foobar" % (self.USER_ID)
	100	request, channel = self.make_request(
	101	"GET", "/_matrix/client/r0/user/%s/filter/foobar" % (self.user_id)
139	102	)
140		render(request, self.resource, self.clock)
	103	self.render(request)
141	104
142	105	self.assertEqual(channel.result["code"], b"400")
143	106
144	107	# No ID also returns an invalid_id error
145	108	def test_get_filter_no_id(self):
146		request, channel = make_request(
147		"GET", "/_matrix/client/r0/user/%s/filter/" % (self.USER_ID)
	109	request, channel = self.make_request(
	110	"GET", "/_matrix/client/r0/user/%s/filter/" % (self.user_id)
148	111	)
149		render(request, self.resource, self.clock)
	112	self.render(request)
150	113
151	114	self.assertEqual(channel.result["code"], b"400")

+23

-29

tests/rest/client/v2_alpha/test_register.py less more

2	2	from mock import Mock
3	3
4	4	from twisted.python import failure
5		from twisted.test.proto_helpers import MemoryReactorClock
6	5
7	6	from synapse.api.errors import InteractiveAuthIncompleteError
8		from synapse.http.server import JsonResource
9	7	from synapse.rest.client.v2_alpha.register import register_servlets
10		from synapse.util import Clock
11	8
12	9	from tests import unittest
13		from tests.server import make_request, render, setup_test_homeserver
14	10
15	11
16		class RegisterRestServletTestCase(unittest.TestCase):
17		def setUp(self):
	12	class RegisterRestServletTestCase(unittest.HomeserverTestCase):
18	13
19		self.clock = MemoryReactorClock()
20		self.hs_clock = Clock(self.clock)
	14	servlets = [register_servlets]
	15
	16	def make_homeserver(self, reactor, clock):
	17
21	18	self.url = b"/_matrix/client/r0/register"
22	19
23	20	self.appservice = None

45	42	identity_handler=self.identity_handler,
46	43	login_handler=self.login_handler,
47	44	)
48		self.hs = setup_test_homeserver(
49		self.addCleanup, http_client=None, clock=self.hs_clock, reactor=self.clock
50		)
	45	self.hs = self.setup_test_homeserver()
51	46	self.hs.get_auth = Mock(return_value=self.auth)
52	47	self.hs.get_handlers = Mock(return_value=self.handlers)
53	48	self.hs.get_auth_handler = Mock(return_value=self.auth_handler)

57	52	self.hs.config.registrations_require_3pid = []
58	53	self.hs.config.auto_join_rooms = []
59	54
60		self.resource = JsonResource(self.hs)
61		register_servlets(self.hs, self.resource)
	55	return self.hs
62	56
63	57	def test_POST_appservice_registration_valid(self):
64	58	user_id = "@kermit:muppet"

68	62	self.auth_handler.get_access_token_for_user_id = Mock(return_value=token)
69	63	request_data = json.dumps({"username": "kermit"})
70	64
71		request, channel = make_request(
	65	request, channel = self.make_request(
72	66	b"POST", self.url + b"?access_token=i_am_an_app_service", request_data
73	67	)
74		render(request, self.resource, self.clock)
	68	self.render(request)
75	69
76	70	self.assertEquals(channel.result["code"], b"200", channel.result)
77	71	det_data = {

84	78	def test_POST_appservice_registration_invalid(self):
85	79	self.appservice = None # no application service exists
86	80	request_data = json.dumps({"username": "kermit"})
87		request, channel = make_request(
	81	request, channel = self.make_request(
88	82	b"POST", self.url + b"?access_token=i_am_an_app_service", request_data
89	83	)
90		render(request, self.resource, self.clock)
	84	self.render(request)
91	85
92	86	self.assertEquals(channel.result["code"], b"401", channel.result)
93	87
94	88	def test_POST_bad_password(self):
95	89	request_data = json.dumps({"username": "kermit", "password": 666})
96		request, channel = make_request(b"POST", self.url, request_data)
97		render(request, self.resource, self.clock)
	90	request, channel = self.make_request(b"POST", self.url, request_data)
	91	self.render(request)
98	92
99	93	self.assertEquals(channel.result["code"], b"400", channel.result)
100	94	self.assertEquals(channel.json_body["error"], "Invalid password")
101	95
102	96	def test_POST_bad_username(self):
103	97	request_data = json.dumps({"username": 777, "password": "monkey"})
104		request, channel = make_request(b"POST", self.url, request_data)
105		render(request, self.resource, self.clock)
	98	request, channel = self.make_request(b"POST", self.url, request_data)
	99	self.render(request)
106	100
107	101	self.assertEquals(channel.result["code"], b"400", channel.result)
108	102	self.assertEquals(channel.json_body["error"], "Invalid username")

120	114	self.auth_handler.get_access_token_for_user_id = Mock(return_value=token)
121	115	self.device_handler.check_device_registered = Mock(return_value=device_id)
122	116
123		request, channel = make_request(b"POST", self.url, request_data)
124		render(request, self.resource, self.clock)
	117	request, channel = self.make_request(b"POST", self.url, request_data)
	118	self.render(request)
125	119
126	120	det_data = {
127	121	"user_id": user_id,

142	136	self.auth_result = (None, {"username": "kermit", "password": "monkey"}, None)
143	137	self.registration_handler.register = Mock(return_value=("@user:id", "t"))
144	138
145		request, channel = make_request(b"POST", self.url, request_data)
146		render(request, self.resource, self.clock)
	139	request, channel = self.make_request(b"POST", self.url, request_data)
	140	self.render(request)
147	141
148	142	self.assertEquals(channel.result["code"], b"403", channel.result)
149	143	self.assertEquals(channel.json_body["error"], "Registration has been disabled")

154	148	self.hs.config.allow_guest_access = True
155	149	self.registration_handler.register = Mock(return_value=(user_id, None))
156	150
157		request, channel = make_request(b"POST", self.url + b"?kind=guest", b"{}")
158		render(request, self.resource, self.clock)
	151	request, channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
	152	self.render(request)
159	153
160	154	det_data = {
161	155	"user_id": user_id,

168	162	def test_POST_disabled_guest_registration(self):
169	163	self.hs.config.allow_guest_access = False
170	164
171		request, channel = make_request(b"POST", self.url + b"?kind=guest", b"{}")
172		render(request, self.resource, self.clock)
	165	request, channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
	166	self.render(request)
173	167
174	168	self.assertEquals(channel.result["code"], b"403", channel.result)
175	169	self.assertEquals(channel.json_body["error"], "Guest access is disabled")

+123

-0

tests/rest/client/v2_alpha/test_sync.py less more

14	14
15	15	from mock import Mock
16	16
	17	from synapse.rest.client.v1 import admin, login, room
17	18	from synapse.rest.client.v2_alpha import sync
18	19
19	20	from tests import unittest
	21	from tests.server import TimedOutException
20	22
21	23
22	24	class FilterTestCase(unittest.HomeserverTestCase):

64	66	["next_batch", "rooms", "account_data", "to_device", "device_lists"]
65	67	).issubset(set(channel.json_body.keys()))
66	68	)
	69
	70
	71	class SyncTypingTests(unittest.HomeserverTestCase):
	72
	73	servlets = [
	74	admin.register_servlets,
	75	room.register_servlets,
	76	login.register_servlets,
	77	sync.register_servlets,
	78	]
	79	user_id = True
	80	hijack_auth = False
	81
	82	def test_sync_backwards_typing(self):
	83	"""
	84	If the typing serial goes backwards and the typing handler is then reset
	85	(such as when the master restarts and sets the typing serial to 0), we
	86	do not incorrectly return typing information that had a serial greater
	87	than the now-reset serial.
	88	"""
	89	typing_url = "/rooms/%s/typing/%s?access_token=%s"
	90	sync_url = "/sync?timeout=3000000&access_token=%s&since=%s"
	91
	92	# Register the user who gets notified
	93	user_id = self.register_user("user", "pass")
	94	access_token = self.login("user", "pass")
	95
	96	# Register the user who sends the message
	97	other_user_id = self.register_user("otheruser", "pass")
	98	other_access_token = self.login("otheruser", "pass")
	99
	100	# Create a room
	101	room = self.helper.create_room_as(user_id, tok=access_token)
	102
	103	# Invite the other person
	104	self.helper.invite(room=room, src=user_id, tok=access_token, targ=other_user_id)
	105
	106	# The other user joins
	107	self.helper.join(room=room, user=other_user_id, tok=other_access_token)
	108
	109	# The other user sends some messages
	110	self.helper.send(room, body="Hi!", tok=other_access_token)
	111	self.helper.send(room, body="There!", tok=other_access_token)
	112
	113	# Start typing.
	114	request, channel = self.make_request(
	115	"PUT",
	116	typing_url % (room, other_user_id, other_access_token),
	117	b'{"typing": true, "timeout": 30000}',
	118	)
	119	self.render(request)
	120	self.assertEquals(200, channel.code)
	121
	122	request, channel = self.make_request(
	123	"GET", "/sync?access_token=%s" % (access_token,)
	124	)
	125	self.render(request)
	126	self.assertEquals(200, channel.code)
	127	next_batch = channel.json_body["next_batch"]
	128
	129	# Stop typing.
	130	request, channel = self.make_request(
	131	"PUT",
	132	typing_url % (room, other_user_id, other_access_token),
	133	b'{"typing": false}',
	134	)
	135	self.render(request)
	136	self.assertEquals(200, channel.code)
	137
	138	# Start typing.
	139	request, channel = self.make_request(
	140	"PUT",
	141	typing_url % (room, other_user_id, other_access_token),
	142	b'{"typing": true, "timeout": 30000}',
	143	)
	144	self.render(request)
	145	self.assertEquals(200, channel.code)
	146
	147	# Should return immediately
	148	request, channel = self.make_request(
	149	"GET", sync_url % (access_token, next_batch)
	150	)
	151	self.render(request)
	152	self.assertEquals(200, channel.code)
	153	next_batch = channel.json_body["next_batch"]
	154
	155	# Reset typing serial back to 0, as if the master had.
	156	typing = self.hs.get_typing_handler()
	157	typing._latest_room_serial = 0
	158
	159	# Since it checks the state token, we need some state to update to
	160	# invalidate the stream token.
	161	self.helper.send(room, body="There!", tok=other_access_token)
	162
	163	request, channel = self.make_request(
	164	"GET", sync_url % (access_token, next_batch)
	165	)
	166	self.render(request)
	167	self.assertEquals(200, channel.code)
	168	next_batch = channel.json_body["next_batch"]
	169
	170	# This should time out! But it does not, because our stream token is
	171	# ahead, and therefore it's saying the typing (that we've actually
	172	# already seen) is new, since it's got a token above our new, now-reset
	173	# stream token.
	174	request, channel = self.make_request(
	175	"GET", sync_url % (access_token, next_batch)
	176	)
	177	self.render(request)
	178	self.assertEquals(200, channel.code)
	179	next_batch = channel.json_body["next_batch"]
	180
	181	# Clear the typing information, so that it doesn't think everything is
	182	# in the future.
	183	typing._reset()
	184
	185	# Now it SHOULD fail as it never completes!
	186	request, channel = self.make_request(
	187	"GET", sync_url % (access_token, next_batch)
	188	)
	189	self.assertRaises(TimedOutException, self.render, request)

+164

-0

tests/rest/media/v1/test_url_preview.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2018 New Vector Ltd
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15	import os
	16
	17	from mock import Mock
	18
	19	from twisted.internet.defer import Deferred
	20
	21	from synapse.config.repository import MediaStorageProviderConfig
	22	from synapse.util.module_loader import load_module
	23
	24	from tests import unittest
	25
	26
	27	class URLPreviewTests(unittest.HomeserverTestCase):
	28
	29	hijack_auth = True
	30	user_id = "@test:user"
	31
	32	def make_homeserver(self, reactor, clock):
	33
	34	self.storage_path = self.mktemp()
	35	os.mkdir(self.storage_path)
	36
	37	config = self.default_config()
	38	config.url_preview_enabled = True
	39	config.max_spider_size = 9999999
	40	config.url_preview_url_blacklist = []
	41	config.media_store_path = self.storage_path
	42
	43	provider_config = {
	44	"module": "synapse.rest.media.v1.storage_provider.FileStorageProviderBackend",
	45	"store_local": True,
	46	"store_synchronous": False,
	47	"store_remote": True,
	48	"config": {"directory": self.storage_path},
	49	}
	50
	51	loaded = list(load_module(provider_config)) + [
	52	MediaStorageProviderConfig(False, False, False)
	53	]
	54
	55	config.media_storage_providers = [loaded]
	56
	57	hs = self.setup_test_homeserver(config=config)
	58
	59	return hs
	60
	61	def prepare(self, reactor, clock, hs):
	62
	63	self.fetches = []
	64
	65	def get_file(url, output_stream, max_size):
	66	"""
	67	Returns tuple[int,dict,str,int] of file length, response headers,
	68	absolute URI, and response code.
	69	"""
	70
	71	def write_to(r):
	72	data, response = r
	73	output_stream.write(data)
	74	return response
	75
	76	d = Deferred()
	77	d.addCallback(write_to)
	78	self.fetches.append((d, url))
	79	return d
	80
	81	client = Mock()
	82	client.get_file = get_file
	83
	84	self.media_repo = hs.get_media_repository_resource()
	85	preview_url = self.media_repo.children[b'preview_url']
	86	preview_url.client = client
	87	self.preview_url = preview_url
	88
	89	def test_cache_returns_correct_type(self):
	90
	91	request, channel = self.make_request(
	92	"GET", "url_preview?url=matrix.org", shorthand=False
	93	)
	94	request.render(self.preview_url)
	95	self.pump()
	96
	97	# We've made one fetch
	98	self.assertEqual(len(self.fetches), 1)
	99
	100	end_content = (
	101	b'<html><head>'
	102	b'<meta property="og:title" content="~matrix~" />'
	103	b'<meta property="og:description" content="hi" />'
	104	b'</head></html>'
	105	)
	106
	107	self.fetches[0][0].callback(
	108	(
	109	end_content,
	110	(
	111	len(end_content),
	112	{
	113	b"Content-Length": [b"%d" % (len(end_content))],
	114	b"Content-Type": [b'text/html; charset="utf8"'],
	115	},
	116	"https://example.com",
	117	200,
	118	),
	119	)
	120	)
	121
	122	self.pump()
	123	self.assertEqual(channel.code, 200)
	124	self.assertEqual(
	125	channel.json_body, {"og:title": "~matrix~", "og:description": "hi"}
	126	)
	127
	128	# Check the cache returns the correct response
	129	request, channel = self.make_request(
	130	"GET", "url_preview?url=matrix.org", shorthand=False
	131	)
	132	request.render(self.preview_url)
	133	self.pump()
	134
	135	# Only one fetch, still, since we'll lean on the cache
	136	self.assertEqual(len(self.fetches), 1)
	137
	138	# Check the cache response has the same content
	139	self.assertEqual(channel.code, 200)
	140	self.assertEqual(
	141	channel.json_body, {"og:title": "~matrix~", "og:description": "hi"}
	142	)
	143
	144	# Clear the in-memory cache
	145	self.assertIn("matrix.org", self.preview_url._cache)
	146	self.preview_url._cache.pop("matrix.org")
	147	self.assertNotIn("matrix.org", self.preview_url._cache)
	148
	149	# Check the database cache returns the correct response
	150	request, channel = self.make_request(
	151	"GET", "url_preview?url=matrix.org", shorthand=False
	152	)
	153	request.render(self.preview_url)
	154	self.pump()
	155
	156	# Only one fetch, still, since we'll lean on the cache
	157	self.assertEqual(len(self.fetches), 1)
	158
	159	# Check the cache response has the same content
	160	self.assertEqual(channel.code, 200)
	161	self.assertEqual(
	162	channel.json_body, {"og:title": "~matrix~", "og:description": "hi"}
	163	)

+46

-6

tests/server.py less more

20	20	from tests.utils import setup_test_homeserver as _sth
21	21
22	22
	23	class TimedOutException(Exception):
	24	"""
	25	A web query timed out.
	26	"""
	27
	28
23	29	@attr.s
24	30	class FakeChannel(object):
25	31	"""

27	33	wire).
28	34	"""
29	35
	36	_reactor = attr.ib()
30	37	result = attr.ib(default=attr.Factory(dict))
31	38	_producer = None
32	39

49	56	self.result["headers"] = headers
50	57
51	58	def write(self, content):
	59	assert isinstance(content, bytes), "Should be bytes! " + repr(content)
	60
52	61	if "body" not in self.result:
53	62	self.result["body"] = b""
54	63

56	65
57	66	def registerProducer(self, producer, streaming):
58	67	self._producer = producer
	68	self.producerStreaming = streaming
	69
	70	def _produce():
	71	if self._producer:
	72	self._producer.resumeProducing()
	73	self._reactor.callLater(0.1, _produce)
	74
	75	if not streaming:
	76	self._reactor.callLater(0.0, _produce)
59	77
60	78	def unregisterProducer(self):
61	79	if self._producer is None:

97	115	return FakeLogger()
98	116
99	117
100		def make_request(method, path, content=b"", access_token=None, request=SynapseRequest):
	118	def make_request(
	119	reactor,
	120	method,
	121	path,
	122	content=b"",
	123	access_token=None,
	124	request=SynapseRequest,
	125	shorthand=True,
	126	):
101	127	"""
102	128	Make a web request using the given method and path, feed it the
103	129	content, and return the Request and the Channel underneath.
	130
	131	Args:
	132	method (bytes/unicode): The HTTP request method ("verb").
	133	path (bytes/unicode): The HTTP path, suitably URL encoded (e.g.
	134	escaped UTF-8 & spaces and such).
	135	content (bytes or dict): The body of the request. JSON-encoded, if
	136	a dict.
	137	shorthand: Whether to try and be helpful and prefix the given URL
	138	with the usual REST API path, if it doesn't contain it.
	139
	140	Returns:
	141	A synapse.http.site.SynapseRequest.
104	142	"""
105	143	if not isinstance(method, bytes):
106	144	method = method.encode('ascii')

108	146	if not isinstance(path, bytes):
109	147	path = path.encode('ascii')
110	148
111		# Decorate it to be the full path
112		if not path.startswith(b"/_matrix"):
	149	# Decorate it to be the full path, if we're using shorthand
	150	if shorthand and not path.startswith(b"/_matrix"):
113	151	path = b"/_matrix/client/r0/" + path
114	152	path = path.replace(b"//", b"/")
115	153

117	155	content = content.encode('utf8')
118	156
119	157	site = FakeSite()
120		channel = FakeChannel()
	158	channel = FakeChannel(reactor)
121	159
122	160	req = request(site, channel)
123	161	req.process = lambda: b""
124	162	req.content = BytesIO(content)
125	163
126	164	if access_token:
127		req.requestHeaders.addRawHeader(b"Authorization", b"Bearer " + access_token)
	165	req.requestHeaders.addRawHeader(
	166	b"Authorization", b"Bearer " + access_token.encode('ascii')
	167	)
128	168
129	169	if content:
130	170	req.requestHeaders.addRawHeader(b"Content-Type", b"application/json")

150	190	x += 1
151	191
152	192	if x > timeout:
153		raise Exception("Timed out waiting for request to finish.")
	193	raise TimedOutException("Timed out waiting for request to finish.")
154	194
155	195	clock.advance(0.1)
156	196

+1

-9

tests/server_notices/test_resource_limits_server_notices.py less more

3	3
4	4	from synapse.api.constants import EventTypes, ServerNoticeMsgType
5	5	from synapse.api.errors import ResourceLimitError
6		from synapse.handlers.auth import AuthHandler
7	6	from synapse.server_notices.resource_limits_server_notices import (
8	7	ResourceLimitsServerNotices,
9	8	)

12	11	from tests.utils import setup_test_homeserver
13	12
14	13
15		class AuthHandlers(object):
16		def __init__(self, hs):
17		self.auth_handler = AuthHandler(hs)
18
19
20	14	class TestResourceLimitsServerNotices(unittest.TestCase):
21	15	@defer.inlineCallbacks
22	16	def setUp(self):
23		self.hs = yield setup_test_homeserver(self.addCleanup, handlers=None)
24		self.hs.handlers = AuthHandlers(self.hs)
25		self.auth_handler = self.hs.handlers.auth_handler
	17	self.hs = yield setup_test_homeserver(self.addCleanup)
26	18	self.server_notices_sender = self.hs.get_server_notices_sender()
27	19
28	20	# relying on [1] is far from ideal, but the only case where

+99

-3

tests/state/test_v2.py less more

543	543	state_res_store=TestStateResolutionStore(event_map),
544	544	)
545	545
546		self.assertTrue(state_d.called)
547		state_before = state_d.result
	546	state_before = self.successResultOf(state_d)
548	547
549	548	state_after = dict(state_before)
550	549	if fake_event.state_key is not None:

598	597	self.assertEqual(["o", "l", "n", "m", "p"], res)
599	598
600	599
	600	class SimpleParamStateTestCase(unittest.TestCase):
	601	def setUp(self):
	602	# We build up a simple DAG.
	603
	604	event_map = {}
	605
	606	create_event = FakeEvent(
	607	id="CREATE",
	608	sender=ALICE,
	609	type=EventTypes.Create,
	610	state_key="",
	611	content={"creator": ALICE},
	612	).to_event([], [])
	613	event_map[create_event.event_id] = create_event
	614
	615	alice_member = FakeEvent(
	616	id="IMA",
	617	sender=ALICE,
	618	type=EventTypes.Member,
	619	state_key=ALICE,
	620	content=MEMBERSHIP_CONTENT_JOIN,
	621	).to_event([create_event.event_id], [create_event.event_id])
	622	event_map[alice_member.event_id] = alice_member
	623
	624	join_rules = FakeEvent(
	625	id="IJR",
	626	sender=ALICE,
	627	type=EventTypes.JoinRules,
	628	state_key="",
	629	content={"join_rule": JoinRules.PUBLIC},
	630	).to_event(
	631	auth_events=[create_event.event_id, alice_member.event_id],
	632	prev_events=[alice_member.event_id],
	633	)
	634	event_map[join_rules.event_id] = join_rules
	635
	636	# Bob and Charlie join at the same time, so there is a fork
	637	bob_member = FakeEvent(
	638	id="IMB",
	639	sender=BOB,
	640	type=EventTypes.Member,
	641	state_key=BOB,
	642	content=MEMBERSHIP_CONTENT_JOIN,
	643	).to_event(
	644	auth_events=[create_event.event_id, join_rules.event_id],
	645	prev_events=[join_rules.event_id],
	646	)
	647	event_map[bob_member.event_id] = bob_member
	648
	649	charlie_member = FakeEvent(
	650	id="IMC",
	651	sender=CHARLIE,
	652	type=EventTypes.Member,
	653	state_key=CHARLIE,
	654	content=MEMBERSHIP_CONTENT_JOIN,
	655	).to_event(
	656	auth_events=[create_event.event_id, join_rules.event_id],
	657	prev_events=[join_rules.event_id],
	658	)
	659	event_map[charlie_member.event_id] = charlie_member
	660
	661	self.event_map = event_map
	662	self.create_event = create_event
	663	self.alice_member = alice_member
	664	self.join_rules = join_rules
	665	self.bob_member = bob_member
	666	self.charlie_member = charlie_member
	667
	668	self.state_at_bob = {
	669	(e.type, e.state_key): e.event_id
	670	for e in [create_event, alice_member, join_rules, bob_member]
	671	}
	672
	673	self.state_at_charlie = {
	674	(e.type, e.state_key): e.event_id
	675	for e in [create_event, alice_member, join_rules, charlie_member]
	676	}
	677
	678	self.expected_combined_state = {
	679	(e.type, e.state_key): e.event_id
	680	for e in [create_event, alice_member, join_rules, bob_member, charlie_member]
	681	}
	682
	683	def test_event_map_none(self):
	684	# Test that we correctly handle passing `None` as the event_map
	685
	686	state_d = resolve_events_with_store(
	687	[self.state_at_bob, self.state_at_charlie],
	688	event_map=None,
	689	state_res_store=TestStateResolutionStore(self.event_map),
	690	)
	691
	692	state = self.successResultOf(state_d)
	693
	694	self.assert_dict(self.expected_combined_state, state)
	695
	696
601	697	def pairwise(iterable):
602	698	"s -> (s0,s1), (s1,s2), (s2, s3), ..."
603	699	a, b = itertools.tee(iterable)

656	752	result.add(event_id)
657	753
658	754	event = self.event_map[event_id]
659		for aid, _ in event.auth_events:
	755	for aid in event.auth_event_ids():
660	756	stack.append(aid)
661	757
662	758	return list(result)

+15

-0

tests/storage/test_end_to_end_keys.py less more

44	44	self.assertDictContainsSubset({"keys": json, "device_display_name": None}, dev)
45	45
46	46	@defer.inlineCallbacks
	47	def test_reupload_key(self):
	48	now = 1470174257070
	49	json = {"key": "value"}
	50
	51	yield self.store.store_device("user", "device", None)
	52
	53	changed = yield self.store.set_e2e_device_keys("user", "device", now, json)
	54	self.assertTrue(changed)
	55
	56	# If we try to upload the same key then we should be told nothing
	57	# changed
	58	changed = yield self.store.set_e2e_device_keys("user", "device", now, json)
	59	self.assertFalse(changed)
	60
	61	@defer.inlineCallbacks
47	62	def test_get_key_with_device_name(self):
48	63	now = 1470174257070
49	64	json = {"key": "value"}

+1

-1

tests/test_federation.py less more

111	111	"origin_server_ts": 1,
112	112	"type": "m.room.message",
113	113	"origin": "test.serv",
114		"content": "hewwo?",
	114	"content": {"body": "hewwo?"},
115	115	"auth_events": [],
116	116	"prev_events": [("two:test.serv", {}), (most_recent, {})],
117	117	}

+15

-28

tests/test_mau.py less more

20	20
21	21	from synapse.api.constants import LoginType
22	22	from synapse.api.errors import Codes, HttpResponseException, SynapseError
23		from synapse.http.server import JsonResource
24	23	from synapse.rest.client.v2_alpha import register, sync
25		from synapse.util import Clock
26	24
27	25	from tests import unittest
28		from tests.server import (
29		ThreadedMemoryReactorClock,
30		make_request,
31		render,
32		setup_test_homeserver,
33		)
34
35
36		class TestMauLimit(unittest.TestCase):
37		def setUp(self):
38		self.reactor = ThreadedMemoryReactorClock()
39		self.clock = Clock(self.reactor)
40
41		self.hs = setup_test_homeserver(
42		self.addCleanup,
	26
	27
	28	class TestMauLimit(unittest.HomeserverTestCase):
	29
	30	servlets = [register.register_servlets, sync.register_servlets]
	31
	32	def make_homeserver(self, reactor, clock):
	33
	34	self.hs = self.setup_test_homeserver(
43	35	"red",
44	36	http_client=None,
45		clock=self.clock,
46		reactor=self.reactor,
47	37	federation_client=Mock(),
48	38	ratelimiter=NonCallableMock(spec_set=["send_message"]),
49	39	)

62	52	self.hs.config.server_notices_mxid_display_name = None
63	53	self.hs.config.server_notices_mxid_avatar_url = None
64	54	self.hs.config.server_notices_room_name = "Test Server Notice Room"
65
66		self.resource = JsonResource(self.hs)
67		register.register_servlets(self.hs, self.resource)
68		sync.register_servlets(self.hs, self.resource)
	55	return self.hs
69	56
70	57	def test_simple_deny_mau(self):
71	58	# Create and sync so that the MAU counts get updated

192	179	}
193	180	)
194	181
195		request, channel = make_request("POST", "/register", request_data)
196		render(request, self.resource, self.reactor)
	182	request, channel = self.make_request("POST", "/register", request_data)
	183	self.render(request)
197	184
198	185	if channel.code != 200:
199	186	raise HttpResponseException(

205	192	return access_token
206	193
207	194	def do_sync_for_user(self, token):
208		request, channel = make_request(
209		"GET", "/sync", access_token=token.encode('ascii')
	195	request, channel = self.make_request(
	196	"GET", "/sync", access_token=token
210	197	)
211		render(request, self.resource, self.reactor)
	198	self.render(request)
212	199
213	200	if channel.code != 200:
214	201	raise HttpResponseException(

+7

-5

tests/test_server.py less more

56	56	"GET", [re.compile("^/_matrix/foo/(?P<room_id>[^/]*)$")], _callback
57	57	)
58	58
59		request, channel = make_request(b"GET", b"/_matrix/foo/%E2%98%83?a=%E2%98%83")
	59	request, channel = make_request(
	60	self.reactor, b"GET", b"/_matrix/foo/%E2%98%83?a=%E2%98%83"
	61	)
60	62	render(request, res, self.reactor)
61	63
62	64	self.assertEqual(request.args, {b'a': [u"\N{SNOWMAN}".encode('utf8')]})

74	76	res = JsonResource(self.homeserver)
75	77	res.register_paths("GET", [re.compile("^/_matrix/foo$")], _callback)
76	78
77		request, channel = make_request(b"GET", b"/_matrix/foo")
	79	request, channel = make_request(self.reactor, b"GET", b"/_matrix/foo")
78	80	render(request, res, self.reactor)
79	81
80	82	self.assertEqual(channel.result["code"], b'500')

97	99	res = JsonResource(self.homeserver)
98	100	res.register_paths("GET", [re.compile("^/_matrix/foo$")], _callback)
99	101
100		request, channel = make_request(b"GET", b"/_matrix/foo")
	102	request, channel = make_request(self.reactor, b"GET", b"/_matrix/foo")
101	103	render(request, res, self.reactor)
102	104
103	105	self.assertEqual(channel.result["code"], b'500')

114	116	res = JsonResource(self.homeserver)
115	117	res.register_paths("GET", [re.compile("^/_matrix/foo$")], _callback)
116	118
117		request, channel = make_request(b"GET", b"/_matrix/foo")
	119	request, channel = make_request(self.reactor, b"GET", b"/_matrix/foo")
118	120	render(request, res, self.reactor)
119	121
120	122	self.assertEqual(channel.result["code"], b'403')

135	137	res = JsonResource(self.homeserver)
136	138	res.register_paths("GET", [re.compile("^/_matrix/foo$")], _callback)
137	139
138		request, channel = make_request(b"GET", b"/_matrix/foobar")
	140	request, channel = make_request(self.reactor, b"GET", b"/_matrix/foobar")
139	141	render(request, res, self.reactor)
140	142
141	143	self.assertEqual(channel.result["code"], b'400')

+123

-0

tests/test_terms_auth.py less more

	0	# Copyright 2018 New Vector Ltd
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13
	14	import json
	15
	16	import six
	17	from mock import Mock
	18
	19	from twisted.test.proto_helpers import MemoryReactorClock
	20
	21	from synapse.rest.client.v2_alpha.register import register_servlets
	22	from synapse.util import Clock
	23
	24	from tests import unittest
	25
	26
	27	class TermsTestCase(unittest.HomeserverTestCase):
	28	servlets = [register_servlets]
	29
	30	def prepare(self, reactor, clock, hs):
	31	self.clock = MemoryReactorClock()
	32	self.hs_clock = Clock(self.clock)
	33	self.url = "/_matrix/client/r0/register"
	34	self.registration_handler = Mock()
	35	self.auth_handler = Mock()
	36	self.device_handler = Mock()
	37	hs.config.enable_registration = True
	38	hs.config.registrations_require_3pid = []
	39	hs.config.auto_join_rooms = []
	40	hs.config.enable_registration_captcha = False
	41
	42	def test_ui_auth(self):
	43	self.hs.config.user_consent_at_registration = True
	44	self.hs.config.user_consent_policy_name = "My Cool Privacy Policy"
	45	self.hs.config.public_baseurl = "https://example.org"
	46	self.hs.config.user_consent_version = "1.0"
	47
	48	# Do a UI auth request
	49	request, channel = self.make_request(b"POST", self.url, b"{}")
	50	self.render(request)
	51
	52	self.assertEquals(channel.result["code"], b"401", channel.result)
	53
	54	self.assertTrue(channel.json_body is not None)
	55	self.assertIsInstance(channel.json_body["session"], six.text_type)
	56
	57	self.assertIsInstance(channel.json_body["flows"], list)
	58	for flow in channel.json_body["flows"]:
	59	self.assertIsInstance(flow["stages"], list)
	60	self.assertTrue(len(flow["stages"]) > 0)
	61	self.assertEquals(flow["stages"][-1], "m.login.terms")
	62
	63	expected_params = {
	64	"m.login.terms": {
	65	"policies": {
	66	"privacy_policy": {
	67	"en": {
	68	"name": "My Cool Privacy Policy",
	69	"url": "https://example.org/_matrix/consent?v=1.0",
	70	},
	71	"version": "1.0"
	72	},
	73	},
	74	},
	75	}
	76	self.assertIsInstance(channel.json_body["params"], dict)
	77	self.assertDictContainsSubset(channel.json_body["params"], expected_params)
	78
	79	# We have to complete the dummy auth stage before completing the terms stage
	80	request_data = json.dumps(
	81	{
	82	"username": "kermit",
	83	"password": "monkey",
	84	"auth": {
	85	"session": channel.json_body["session"],
	86	"type": "m.login.dummy",
	87	},
	88	}
	89	)
	90
	91	self.registration_handler.check_username = Mock(return_value=True)
	92
	93	request, channel = self.make_request(b"POST", self.url, request_data)
	94	self.render(request)
	95
	96	# We don't bother checking that the response is correct - we'll leave that to
	97	# other tests. We just want to make sure we're on the right path.
	98	self.assertEquals(channel.result["code"], b"401", channel.result)
	99
	100	# Finish the UI auth for terms
	101	request_data = json.dumps(
	102	{
	103	"username": "kermit",
	104	"password": "monkey",
	105	"auth": {
	106	"session": channel.json_body["session"],
	107	"type": "m.login.terms",
	108	},
	109	}
	110	)
	111	request, channel = self.make_request(b"POST", self.url, request_data)
	112	self.render(request)
	113
	114	# We're interested in getting a response that looks like a successful
	115	# registration, not so much that the details are exactly what we want.
	116
	117	self.assertEquals(channel.result["code"], b"200", channel.result)
	118
	119	self.assertTrue(channel.json_body is not None)
	120	self.assertIsInstance(channel.json_body["user_id"], six.text_type)
	121	self.assertIsInstance(channel.json_body["access_token"], six.text_type)
	122	self.assertIsInstance(channel.json_body["device_id"], six.text_type)

+24

-7

tests/unittest.py less more

145	145	return target
146	146
147	147
	148	def INFO(target):
	149	"""A decorator to set the .loglevel attribute to logging.INFO.
	150	Can apply to either a TestCase or an individual test method."""
	151	target.loglevel = logging.INFO
	152	return target
	153
	154
148	155	class HomeserverTestCase(TestCase):
149	156	"""
150	157	A base TestCase that reduces boilerplate for HomeServer-using test cases.

181	188	for servlet in self.servlets:
182	189	servlet(self.hs, self.resource)
183	190
	191	from tests.rest.client.v1.utils import RestHelper
	192
	193	self.helper = RestHelper(self.hs, self.resource, getattr(self, "user_id", None))
	194
184	195	if hasattr(self, "user_id"):
185		from tests.rest.client.v1.utils import RestHelper
186
187		self.helper = RestHelper(self.hs, self.resource, self.user_id)
188
189	196	if self.hijack_auth:
190	197
191	198	def get_user_by_access_token(token=None, allow_guest=False):

250	257	"""
251	258
252	259	def make_request(
253		self, method, path, content=b"", access_token=None, request=SynapseRequest
	260	self,
	261	method,
	262	path,
	263	content=b"",
	264	access_token=None,
	265	request=SynapseRequest,
	266	shorthand=True,
254	267	):
255	268	"""
256	269	Create a SynapseRequest at the path using the method and containing the

262	275	escaped UTF-8 & spaces and such).
263	276	content (bytes or dict): The body of the request. JSON-encoded, if
264	277	a dict.
	278	shorthand: Whether to try and be helpful and prefix the given URL
	279	with the usual REST API path, if it doesn't contain it.
265	280
266	281	Returns:
267	282	A synapse.http.site.SynapseRequest.

269	284	if isinstance(content, dict):
270	285	content = json.dumps(content).encode('utf8')
271	286
272		return make_request(method, path, content, access_token, request)
	287	return make_request(
	288	self.reactor, method, path, content, access_token, request, shorthand
	289	)
273	290
274	291	def render(self, request):
275	292	"""

372	389	self.render(request)
373	390	self.assertEqual(channel.code, 200)
374	391
375		access_token = channel.json_body["access_token"].encode('ascii')
	392	access_token = channel.json_body["access_token"]
376	393	return access_token

+2

-0

tests/utils.py less more

122	122	config.user_directory_search_all_users = False
123	123	config.user_consent_server_notice_content = None
124	124	config.block_events_without_consent_error = None
	125	config.user_consent_at_registration = False
	126	config.user_consent_policy_name = "Privacy Policy"
125	127	config.media_storage_providers = []
126	128	config.autocreate_auto_join_rooms = True
127	129	config.auto_join_rooms = []

+15

-1

tox.ini less more

9	9
10	10	# needed by some of the tests
11	11	lxml
	12
	13	# cyptography 2.2 requires setuptools >= 18.5
	14	#
	15	# older versions of virtualenv (?) give us a virtualenv with the same
	16	# version of setuptools as is installed on the system python (and tox runs
	17	# virtualenv under python3, so we get the version of setuptools that is
	18	# installed on that).
	19	#
	20	# anyway, make sure that we have a recent enough setuptools.
	21	setuptools>=18.5
	22
	23	# we also need a semi-recent version of pip, because old ones fail to
	24	# install the "enum34" dependency of cryptography.
	25	pip>=10
12	26
13	27	setenv =
14	28	PYTHONDONTWRITEBYTECODE = no_byte_code

107	121	basepython = python3.6
108	122	deps =
109	123	flake8
110		commands = /bin/sh -c "flake8 synapse tests scripts scripts-dev scripts/register_new_matrix_user scripts/synapse_port_db synctl {env:PEP8SUFFIX:}"
	124	commands = /bin/sh -c "flake8 synapse tests scripts scripts-dev scripts/hash_password scripts/register_new_matrix_user scripts/synapse_port_db synctl {env:PEP8SUFFIX:}"
111	125
112	126	[testenv:check_isort]
113	127	skip_install = True